Repository navigation
Conversation
Wrap scripts/ci/soak.sh in .github/workflows/soak.yml so the soak stops being a manual release-time step. Push and pull_request runs gate on the 10-minute default (alternating keep-alive / Connection: close churn slices against example/simple-server); workflow_dispatch takes an explicit duration for the DURATION=1800 release-candidate run recorded in docs/soak-testing.md as a v4.0 exit criterion. The gate fails on server exit, any socket error or non-2xx, RSS growth > 10%, fd growth > 5, or p99 drift > 20% across slices. Verified locally: DURATION=60 smoke passes on dev (RSS +1.4%, fd +0, errors 0, p99 drift 0%).
The churn slice passed the header as a single quoted argument
('-H "Connection: close"'). Some wrk builds -- Ubuntu's 4.1.0 among
them -- reject that as an unknown option and exit 1, killing the script
via set -e with the captured wrk output never printed; other builds
accept it but silently ignore the header, so the slice kept doing
keep-alive and the gate passed without ever churning (both observed:
the CI gate failed on the first churn slice, the local smoke did not
churn at all).
- Pass -H and its value as separate argv entries.
- Exclude the read component of 'Socket errors' from the churn slice's
error budget: with Connection: close the server's close-after-response
is the mechanism under test and wrk counts it as one read error per
request (~2.1M over a 60s slice locally). connect/write/timeout errors
and non-2xx still fail the slice.
- Print the captured wrk output when wrk itself exits non-zero, so the
next failure mode is visible in the CI log instead of vanishing into
the command substitution.
Verified locally: DURATION=300 run interleaves both slice types and
passes (RSS +8.5%, fds +0, errors 0 with ~2.1M expected churn read
errors excluded, p99 drift 15.6%).
Member
Author
|
Update: the first CI run failed on the first churn slice — three bugs in
Verified locally end-to-end: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/ci/soak.shin a new.github/workflows/soak.ymlCI gate so the soak run is no longer a manual release-time step (ROADMAP v4.0 exit criterion: "the soak run is defined and passes on the release commit" — the definition landed indocs/soak-testing.md+ the script, but nothing ran it automatically).push/pull_requestto main/master/dev → 10-minute soak (the script's defaultDURATION=600), gatedworkflow_dispatchwith adurationinput → theDURATION=1800release-candidate runsoak.shexit status: fails on server exit mid-run, any socket error / non-2xx, RSS growth > 10%, fd growth > 5, or p99 drift > 20% across slices. Server log is dumped on failure (if: always()),[skip ci]pushes are skipped like the other gates.Test plan
DURATION=60 scripts/ci/soak.shon dev passes (RSS +1.4% limit 10%, fds +0, errors 0, p99 drift 0%, ~173k RPS)