Skip to content

ci(soak): gate the HTTP soak on push/PR, dispatch for release candidates - #379

Open
gg582 wants to merge 2 commits into
devfrom
ci/soak-gate
Open

gg582 wants to merge 2 commits into
devfrom
ci/soak-gate

Conversation

@gg582

@gg582 gg582 commented Oct 15, 2026

Copy link
Copy Markdown
Member

Summary

  • What: wraps scripts/ci/soak.sh in a new .github/workflows/soak.yml CI gate so the soak run is no longer a manual release-time step (ROADMAP v4.0 exit criterion: "the soak run is defined and passes on the release commit" — the definition landed in docs/soak-testing.md + the script, but nothing ran it automatically).
  • Triggers (mirrors the c100k gate):
    • push / pull_request to main/master/dev → 10-minute soak (the script's default DURATION=600), gated
    • workflow_dispatch with a duration input → the DURATION=1800 release-candidate run
  • Gate verdict comes straight from soak.sh exit status: fails on server exit mid-run, any socket error / non-2xx, RSS growth > 10%, fd growth > 5, or p99 drift > 20% across slices. Server log is dumped on failure (if: always()), [skip ci] pushes are skipped like the other gates.

Test plan

  • YAML validated
  • Local smoke: DURATION=60 scripts/ci/soak.sh on dev passes (RSS +1.4% limit 10%, fds +0, errors 0, p99 drift 0%, ~173k RPS)
  • The soak job runs on this PR (new workflow file — needs the repo's approval setting for first-time workflows, if any)

gg582 added 2 commits October 15, 2026 22:29
Wrap scripts/ci/soak.sh in .github/workflows/soak.yml so the soak stops
being a manual release-time step. Push and pull_request runs gate on the
10-minute default (alternating keep-alive / Connection: close churn
slices against example/simple-server); workflow_dispatch takes an
explicit duration for the DURATION=1800 release-candidate run recorded
in docs/soak-testing.md as a v4.0 exit criterion. The gate fails on
server exit, any socket error or non-2xx, RSS growth > 10%, fd growth
> 5, or p99 drift > 20% across slices.

Verified locally: DURATION=60 smoke passes on dev (RSS +1.4%, fd +0,
errors 0, p99 drift 0%).
The churn slice passed the header as a single quoted argument
('-H "Connection: close"'). Some wrk builds -- Ubuntu's 4.1.0 among
them -- reject that as an unknown option and exit 1, killing the script
via set -e with the captured wrk output never printed; other builds
accept it but silently ignore the header, so the slice kept doing
keep-alive and the gate passed without ever churning (both observed:
the CI gate failed on the first churn slice, the local smoke did not
churn at all).

- Pass -H and its value as separate argv entries.
- Exclude the read component of 'Socket errors' from the churn slice's
  error budget: with Connection: close the server's close-after-response
  is the mechanism under test and wrk counts it as one read error per
  request (~2.1M over a 60s slice locally). connect/write/timeout errors
  and non-2xx still fail the slice.
- Print the captured wrk output when wrk itself exits non-zero, so the
  next failure mode is visible in the CI log instead of vanishing into
  the command substitution.

Verified locally: DURATION=300 run interleaves both slice types and
passes (RSS +8.5%, fds +0, errors 0 with ~2.1M expected churn read
errors excluded, p99 drift 15.6%).
@gg582

gg582 commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Update: the first CI run failed on the first churn slice — three bugs in soak.sh (the gate script itself, not the workflow), fixed in 17e87018:

  1. Churn header never actually sent: run_slice "$BASE/health" '-H "Connection: close"' bundles the header into one argv. Ubuntu's wrk 4.1.0 rejects it as an unknown option → exit 1 → set -e kills the script (the CI failure). Other wrk builds accept it but silently ignore the header — my local 60s smoke never ran a churn slice at all, which is why it passed locally. Now passed as separate argv entries.
  2. Swallowed diagnostics: wrk's output lived in a command substitution that died with the script on failure; the error message never reached the log. wrk output is now printed on non-zero exit.
  3. Unwinnable error budget: with a real Connection: close, the server's close-after-response is the mechanism under test and wrk reports it as one Socket errors: read per request (~2.1M per 60s slice locally). The churn slice now excludes the read component (connect/write/timeout and non-2xx still fail the slice); keep-alive slices keep the full accounting.

Verified locally end-to-end: DURATION=300 run interleaves both slice types and passes (RSS +8.5%, fds +0, errors 0, p99 drift 15.6%).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant