Repository navigation
ci: the io_uring and root-set race steps get 40% or more over their longest measured time; a step above 75% of its timeout warns - #1199
Merged
Conversation
…r more over their longest measured time, and a step above 75% of its timeout says so The unit job's iouring step (-timeout=600s) took 520.3 s on main at 64fe9e5 (15% over), the Coverage job's own step 537.6 s at most (12%), the root set's longest binary (internal/engine/epoll) 398.1 s of 540 s (36%), and the base tier's ./test/server step 447.9 s of 600 s (34%). A timeout panic there fails a required check for every unrelated pull request (#1188 hit 594.96 s of 600 s). Timeouts, from the last fourteen CI and Coverage runs on main: unit iouring 600 -> 780 s (50% over the longest 520.3 s) coverage iouring 600 -> 780 s (45% over 537.6 s) unit root set 540 -> 600 s (51% over 398.1 s) coverage root set 540 -> 600 s (52% over 394.0 s), job 40 -> 45 min base tier test/server 600 -> 660 s (47% over 447.9 s) base tier iouring 900 s unchanged (49% over 605.9 s) .github/scripts/timeout-headroom.sh LOG TIMEOUT prints the longest `ok` line of a step as a share of the timeout and emits a warning annotation above 75%; it does not fail the step (a red required check on a slow runner is the harm), and it fails closed on a log with no result. Each step sets its timeout in one variable and passes it to both go test and the script. Its self-test is a Lint step. The Coverage steps tee their output for it. Refs #890
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
Author
|
Observed in this PR's own CI at b76f8a2 (CI run 41268714834, Coverage run 41268714937; both green, no warning annotation), from the
The new Lint step |
…reshold itself is pinned by the self-test * ci.yml (base tier): the second step runs two test binaries one after the other (-p 1), each with its own 660 s, so the steps' limits are no bound of the job's 30 minutes; say so, with the measured figures (root binary 4.3 to 5.3 s, job 17.4 to 18.5 min, a hang in one binary still fits). * test-coverage.yml: the 40 minutes "at their own limits" assume a multi-package invocation ends inside its per-binary -timeout; it does not (the root step took 765 to 866 s). Say so, and replace the stale "about 20 min" with the measured 22.6 to 24.8 min; 45 min still covers one hung binary. The epoll figure is 401.0 s in the run of 86981b3 (50% over, 67% of 600 s). * timeout-headroom-test.sh: fixtures at exactly 75% (no warning) and 1 ms above (one warning), so a change of > to >= fails the self-test; the header of the script says "strictly above". Refs #890
Contributor
Author
|
Review r1 fixes pushed (head 1389383, main 86981b3 merged in); CI run 41271349353 and Coverage run 41271349288 are green, no warning annotation. Observed:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #890
CI only; no test code touched. The race steps that run
./internal/engine/iouring(and the root set, whose longest binary isinternal/engine/epoll) sit close to theirgo test -timeout. A timeout panic on main fails required checks for every unrelated PR (#1188 hit 594.96 s of 600 s).Measured, not guessed
Source: the job logs (
gh api --allow-escape-sequences repos/goceleris/celeris/actions/jobs/<id>/logs) of the last 14 successfulmainpush runs of CI and of Coverage, 1002dcb (2026-10-30) to 64fe9e5 (2026-11-01), theok <package> <seconds>sline of each step.-timeoutapplies to each test binary, so the figure is the longest package of the step. "P90" is nearest-rank of n=14.Notes on the rows:
go testpackage figure (the wall time of the base tier step adds about 20 s of build: 606-625 s).-timeoutis per test binary, and this step runs two of them one after the other (-p 1), so the steps' limits are no bound of the job'stimeout-minutes: 30(900 + 2 x 660 = 2220 s). In practice the root binary takes 4.3 to 5.3 s, the job 17.4 to 18.5 min, and a hang in one binary still ends inside the 30 minutes; only a hang in two binaries at once is cut by the job limit, after the first panic dump is logged.timeout-minutesgoes 40 -> 45. That sum is not a bound either (a multi-package invocation can outlast its per-binary-timeout: the root step took 765 to 866 s for binaries of at most 401 s); the job takes 22.6 to 24.8 min on main, and 45 min covers it plus one hung binary (at most about 260 s more). The unit job has notimeout-minutes(360 default).go teststep the logs show is far from its limit (canary tests 74 s of 300-400 s;./test/server334 s;middleware/sse221 s).Why raise, not split (ci.yml item 66 said "split rather than raise again")
In run 41262675961 the package's 622 top-level tests add up to 517.9 s and no handful dominates: the longest four are 167 s (32%:
TestAsyncChurnNoUseAfterFree60.1 s,TestIouringWriteTimeoutBoundOfAStalledSend81152.8 s,TestAsyncChurnCloseWithStragglerData30.3 s,TestIouringIdleConnTimeoutIsNotGatedByIterations88424.3 s) and the longest fifteen 300 s (58%). A split by-run/-skiplists would have to name hundreds of tests to cut the step in half, would rebuild the race binary per shard, and would need the by-name tallies re-pointed (named-tally.sh, the skip allow-lists) so a rename cannot silently drop a test (theroot-shards.shmodel). That is a larger change than this one; the warning below says when to do it.The tripwire
.github/scripts/timeout-headroom.sh LOG TIMEOUT_SECONDSprints the longestokline of the step'steed log as a share of the timeout and, above 75%, emits a::warning::annotation and a line in the job summary. Choices:okline (a step whose output stopped being captured must not read as "fine").iouring_timeout=780,root_timeout=600, ...) and gives it to bothgo test -timeout=and the script, so the two cannot drift.2>&1 | tee, those steps already run underset -o pipefail, so ago testfailure still fails the step).timeout-headroom-test.shis its self-test, a Lint step likeskip-count-test.sh(exactly 75% does not warn, 1 ms above does, below/above threshold, slowest package picked, a coverage line, non-result lines, empty log, usage errors; a copy without the comparison fails it).Validation
go tool -modfile=.github/tools/go.mod actionlint(the pin the Lint job uses, v1.7.12, with shellcheck): clean on both workflows.timeout-headroom-test.shand shellcheck pass locally; the negative control (comparison removed) fails the self-test.Not touched: test code, the tallies, the
-skiplists, the root shards.Updated after review r1: main merged in (86981b3, #1198), the two job-bound comments corrected as above, Coverage epoll figure updated to 401.0 s, and the 75% boundary pinned in the self-test (a
>=mutant now fails it).