Skip to content

ci: the io_uring and root-set race steps get 40% or more over their longest measured time; a step above 75% of its timeout warns - #1199

Merged
FumingPower3925 merged 3 commits into
mainfrom
ci-iouring-timeout-headroom-890
Nov 1, 2026
Merged

FumingPower3925 merged 3 commits into
mainfrom
ci-iouring-timeout-headroom-890

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Refs #890

CI only; no test code touched. The race steps that run ./internal/engine/iouring (and the root set, whose longest binary is internal/engine/epoll) sit close to their go test -timeout. A timeout panic on main fails required checks for every unrelated PR (#1188 hit 594.96 s of 600 s).

Measured, not guessed

Source: the job logs (gh api --allow-escape-sequences repos/goceleris/celeris/actions/jobs/<id>/logs) of the last 14 successful main push runs of CI and of Coverage, 1002dcb (2026-10-30) to 64fe9e5 (2026-11-01), the ok <package> <seconds>s line of each step. -timeout applies to each test binary, so the figure is the longest package of the step. "P90" is nearest-rank of n=14.

step now min median P90 max headroom over max now new headroom over max after new 75% mark
ci.yml unit: iouring 600 s 497.5 512.2 520.3 520.3 15% 780 s 50% 585 s
test-coverage.yml: iouring own step 600 s 515.9 523.9 531.9 537.6 12% 780 s 45% 585 s
ci.yml unit: root set (epoll is the longest binary) 540 s 361.9 382.3 392.9 398.1 36% 600 s 51% 450 s
test-coverage.yml: root set, other packages (epoll) 540 s 360.9 381.9 391.3 401.0 (86981b3) 35% 600 s 50% 450 s
test-coverage.yml: root set, ./test/server 540 s 302.6 314.4 321.2 321.9 68% 540 s (unchanged) 68% 405 s
ci.yml unit: ./test/server in the root set 540 s 306.3 317.9 333.1 333.9 62% 600 s (same step) 80% 450 s
ci.yml base tier: iouring 900 s 580.0 (6 runs since #1188; 501-518 before it) 596.1 605.9 605.9 49% 900 s (unchanged) 49% 675 s
ci.yml base tier: root + ./test/server 600 s 417.1 (6 runs) 436.5 447.9 447.9 34% 660 s 47% 495 s

Notes on the rows:

  • The unit root set and the base tier were also reported at 513-520 s / 594-620 s as the whole step wall time; the table uses the go test package figure (the wall time of the base tier step adds about 20 s of build: 606-625 s).
  • Base tier iouring: the figure jumped by 80 s at 5aecd53 (engines, conn, h2: a step of the wall clock no longer cuts or holds connections: the stamps are monotonic (Refs #1178) #1188), because that job runs the stepped-clock test the other steps skip; the six runs since are the relevant sample. It is already 49% over its longest, so 900 s stays (a lost cap is meant to end the step in minutes, run 40416230192).
  • Base tier root + ./test/server was not on the list but is the same class (34% over): raised to 660 s. -timeout is per test binary, and this step runs two of them one after the other (-p 1), so the steps' limits are no bound of the job's timeout-minutes: 30 (900 + 2 x 660 = 2220 s). In practice the root binary takes 4.3 to 5.3 s, the job 17.4 to 18.5 min, and a hang in one binary still ends inside the 30 minutes; only a hang in two binaries at once is cut by the job limit, after the first panic dump is logged.
  • Coverage job: at their own limits the steps are 600 + 540 + 780 + 4 x 120 = 2400 s = 40 min (it was 36 min of 40), so timeout-minutes goes 40 -> 45. That sum is not a bound either (a multi-package invocation can outlast its per-binary -timeout: the root step took 765 to 866 s for binaries of at most 401 s); the job takes 22.6 to 24.8 min on main, and 45 min covers it plus one hung binary (at most about 260 s more). The unit job has no timeout-minutes (360 default).
  • Every other go test step the logs show is far from its limit (canary tests 74 s of 300-400 s; ./test/server 334 s; middleware/sse 221 s).

Why raise, not split (ci.yml item 66 said "split rather than raise again")

In run 41262675961 the package's 622 top-level tests add up to 517.9 s and no handful dominates: the longest four are 167 s (32%: TestAsyncChurnNoUseAfterFree 60.1 s, TestIouringWriteTimeoutBoundOfAStalledSend811 52.8 s, TestAsyncChurnCloseWithStragglerData 30.3 s, TestIouringIdleConnTimeoutIsNotGatedByIterations884 24.3 s) and the longest fifteen 300 s (58%). A split by -run/-skip lists would have to name hundreds of tests to cut the step in half, would rebuild the race binary per shard, and would need the by-name tallies re-pointed (named-tally.sh, the skip allow-lists) so a rename cannot silently drop a test (the root-shards.sh model). That is a larger change than this one; the warning below says when to do it.

The tripwire

.github/scripts/timeout-headroom.sh LOG TIMEOUT_SECONDS prints the longest ok line of the step's teed log as a share of the timeout and, above 75%, emits a ::warning:: annotation and a line in the job summary. Choices:

  • It warns, it does not fail. Runs vary about 5% from one to the next (and 15% at engines, conn, h2: a step of the wall clock no longer cuts or holds connections: the stamps are monotonic (Refs #1178) #1188); a hard fail at 75% would put a red required check on unrelated PRs, which is the harm this PR is meant to remove. The timeout panic stays the only hard failure.
  • It fails closed on a log with no ok line (a step whose output stopped being captured must not read as "fine").
  • Each step sets its timeout in one variable (iouring_timeout=780, root_timeout=600, ...) and gives it to both go test -timeout= and the script, so the two cannot drift.
  • The Coverage steps did not tee their output; they do now (2>&1 | tee, those steps already run under set -o pipefail, so a go test failure still fails the step).
  • timeout-headroom-test.sh is its self-test, a Lint step like skip-count-test.sh (exactly 75% does not warn, 1 ms above does, below/above threshold, slowest package picked, a coverage line, non-result lines, empty log, usage errors; a copy without the comparison fails it).

Validation

  • go tool -modfile=.github/tools/go.mod actionlint (the pin the Lint job uses, v1.7.12, with shellcheck): clean on both workflows.
  • timeout-headroom-test.sh and shellcheck pass locally; the negative control (comparison removed) fails the self-test.
  • The script run against the real main logs gives the table's figures (e.g. 520.3 s of 780 s = 67%).
  • CI on this PR is the proof of the step times: see the comment below with the observed times of this PR's own run.

Not touched: test code, the tallies, the -skip lists, the root shards.

Updated after review r1: main merged in (86981b3, #1198), the two job-bound comments corrected as above, Coverage epoll figure updated to 401.0 s, and the 75% boundary pinned in the self-test (a >= mutant now fails it).

…r more over their longest measured time, and a step above 75% of its timeout says so

The unit job's iouring step (-timeout=600s) took 520.3 s on main at 64fe9e5
(15% over), the Coverage job's own step 537.6 s at most (12%), the root set's
longest binary (internal/engine/epoll) 398.1 s of 540 s (36%), and the base
tier's ./test/server step 447.9 s of 600 s (34%). A timeout panic there fails a
required check for every unrelated pull request (#1188 hit 594.96 s of 600 s).

Timeouts, from the last fourteen CI and Coverage runs on main:
  unit iouring        600 -> 780 s (50% over the longest 520.3 s)
  coverage iouring    600 -> 780 s (45% over 537.6 s)
  unit root set       540 -> 600 s (51% over 398.1 s)
  coverage root set   540 -> 600 s (52% over 394.0 s), job 40 -> 45 min
  base tier test/server 600 -> 660 s (47% over 447.9 s)
  base tier iouring   900 s unchanged (49% over 605.9 s)

.github/scripts/timeout-headroom.sh LOG TIMEOUT prints the longest `ok` line of a
step as a share of the timeout and emits a warning annotation above 75%; it
does not fail the step (a red required check on a slow runner is the harm), and
it fails closed on a log with no result. Each step sets its timeout in one
variable and passes it to both go test and the script. Its self-test is a Lint
step. The Coverage steps tee their output for it.

Refs #890
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FumingPower3925

Copy link
Copy Markdown
Contributor Author

Observed in this PR's own CI at b76f8a2 (CI run 41268714834, Coverage run 41268714937; both green, no warning annotation), from the timeout-headroom.sh lines of the job logs:

step longest package took timeout % of timeout
unit: root set internal/engine/epoll 381.4 s 600 s 64%
unit: iouring internal/engine/iouring 512.1 s 780 s 66%
base tier: iouring internal/engine/iouring 584.5 s (step 598 s) 900 s 65%
base tier: root + ./test/server test/server 416.7 s (step 432 s) 660 s 63%
coverage: root set, other packages internal/engine/epoll 373.4 s 600 s 62%
coverage: ./test/server test/server 308.1 s 540 s 57%
coverage: iouring own step internal/engine/iouring 523.7 s 780 s 67%

The new Lint step timeout-headroom.sh self-test passed; every step above ran the headroom check and printed its line.

…reshold itself is pinned by the self-test

* ci.yml (base tier): the second step runs two test binaries one after the other
  (-p 1), each with its own 660 s, so the steps' limits are no bound of the job's
  30 minutes; say so, with the measured figures (root binary 4.3 to 5.3 s, job
  17.4 to 18.5 min, a hang in one binary still fits).
* test-coverage.yml: the 40 minutes "at their own limits" assume a multi-package
  invocation ends inside its per-binary -timeout; it does not (the root step took
  765 to 866 s). Say so, and replace the stale "about 20 min" with the measured
  22.6 to 24.8 min; 45 min still covers one hung binary. The epoll figure is 401.0 s
  in the run of 86981b3 (50% over, 67% of 600 s).
* timeout-headroom-test.sh: fixtures at exactly 75% (no warning) and 1 ms above
  (one warning), so a change of > to >= fails the self-test; the header of the
  script says "strictly above".

Refs #890
@FumingPower3925

Copy link
Copy Markdown
Contributor Author

Review r1 fixes pushed (head 1389383, main 86981b3 merged in); CI run 41271349353 and Coverage run 41271349288 are green, no warning annotation. Observed:

step longest package took timeout %
unit: root set epoll 398.8 s 600 s 66%
unit: iouring iouring 520.5 s 780 s 67%
base tier: iouring iouring 601.7 s 900 s 67%
base tier: root + ./test/server test/server 443.4 s 660 s 67%
coverage: root set epoll 397.4 s 600 s 66%
coverage: ./test/server test/server 320.9 s 540 s 59%
coverage: iouring iouring 529.7 s 780 s 68%

@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 04:33
@FumingPower3925
FumingPower3925 merged commit dbad68e into main Nov 1, 2026
31 of 32 checks passed
@FumingPower3925
FumingPower3925 deleted the ci-iouring-timeout-headroom-890 branch November 1, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant