Skip to content

conn: stepped-clock tests of the HTTP/2 D0, D2, header-block, tombstone and ExpiredWrites deadlines (Refs #1190) - #1191

Merged
FumingPower3925 merged 4 commits into
mainfrom
mono/1190-stepped-h2
Oct 31, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
mono/1190-stepped-h2

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1190

Stepped-clock tests for the HTTP/2 deadlines the flip's tests (#1188) do not pin: the connection's idle clock D0 (and its restart from stream activity, and its ack cap, ExtendIdle on tcp_info bytes_acked), the write window D2 (buffered streams, ExpiredWrites, and pool-handler waiters), the header-block deadline and the tombstone deadline. For each, in both directions of a 75 s step of the wall clock: the step alone must not fire the deadline (cut early), and the time that then really passes must (held past its bound).

Commits

  1. conn: stepped-clock tests ...: internal/conn/stepped_h2_1190_test.go and mono.Manual (internal/mono/stepclock.go).
  2. ci: the canary names the HTTP/2 stepped-clock tests.
  3. Review r1 (Opus), conn, h2 stream: the D0 restart ... and the D2 waiter path ...: F2 two more tests (D0 restart from stream activity; D2 waiter path, in internal/protocol/h2/stream/stepped_wait_1190_test.go), F1 NewManual anchored to the real wall at every skew, F3/F4/F5 assertions, CI names 7 tests. Then a lint nit (real shadowed a built-in).

No production change: every one of these deadlines already reads internal/mono; all seven tests pass on the monotonic clock as they are.

Flake-proof by construction

No sleep that depends on runner speed, no real time in the conn tests. mono.Manual is a clock that moves only when the test moves it: Advance(d) lets d of time pass on both readings, Step(d) steps the wall reading alone. No verdict can come before Advance, whatever the runner's speed. The waiter test (stream package) has a real Go timer of WriteTimeout (40 ms) that the waiter arms itself; the clock being frozen, the timer fires and re-arms against it until the test advances the clock, so the verdict comes one timer period after Advance on any runner (the negative assertions, "no verdict yet", can only pass vacuously on a slow runner, never fail; the positive wait is bounded at 5 s for a 40 ms period).

The clock stands one hour past the REAL wall clock at every CELERIS_MONO_SKEW (review r1 F1): NewManual anchors its readings to the real wall after the canary's skew (the skewed Now() is real + 1 h at +1 h, 0 and -1 h; earlier the hour was put on the monotonic reading alone and at -1 h it cancelled the skew, leaving the clock on the real wall, where two mutants survived the named run). So a site that still reads time.Now is an hour off every stamp of the clock and its test fails, at both signs (the mutant table below, column "new tests -1 h").

On the monotonic clock a step moves nothing, so the "forward" and "backward" subtests test the same thing there (the time that passes fires the deadline); they differ on the wall clock, where the tests are RED. The sweeps that assert "the step alone fired nothing" come after enough time has passed to cross D0's d0Interval gate (100 ms), or the sweep would not look at the idle clock.

Scenarios

test deadline what it asserts, per direction
TestIdleClockAndAckCapIgnoreAStepOfTheWallClock1190 D0 idle clock (IdleTimeout 1 s) and the ack cap a step does not close a conn whose idle clock has run 0.15 s; 1.2 s of time closes it; ExtendIdle gives the first extension, a step and 0.5 s into the extended period do not close, 1.2 s do; no acknowledged progress refuses the next extension; progress extends, a step and 0.5 s into that period do not close, and that period runs out at 1.2 s
TestIdleClockRestartsFromStreamActivityOnAStepOfTheWallClock1190 D0 restart from stream activity (d0At = now, conn/h2.go) a conn with 0.7 s of idle clock that carried a stream is idle 1 s after the stream: not closed 0.65 s after it (step in between), closed 1.25 s after it
TestWriteWindowExpiryIgnoresAStepOfTheWallClock1190 D2 write window of BUFFERED streams, ExpiredWrites (WriteTimeout 1 s) a stalled stream is not reset by a step (sweep and ExpiredWrites both), and is reset with 1.2 s of time
TestWriteWindowRenewalIgnoresAStepOfTheWallClock1190 D2 minimum rate (buffered) a stream that releases its floor each window survives windows separated by steps; one that stops is reset within four windows whatever the wall did
TestWaiterWindowIgnoresAStepOfTheWallClock1190 (h2 stream) D2 for POOL-HANDLER WAITERS (celeris#1170: write_wait.go, judged by the handler's own timer) no verdict before the clock moves, none after the step, the verdict (os.ErrDeadlineExceeded) once the window's time has passed
TestHeaderBlockDeadlineIgnoresAStepOfTheWallClock1190 header-block deadline (ReadHeaderTimeout 1 s) a step does not end an unfinished block; 1.2 s does, with GOAWAY(ENHANCE_YOUR_CALM)
TestTombstoneDeadlineIgnoresAStepOfTheWallClock1190 tombstone (two request timeouts) and the request deadline that makes it the request is not expired at 0.5 s and is at 1.2 s (review F5), the tombstone has 2 s to live, a step does not drop it, it is there at 1.9 s and gone at 2.2 s

Scope, said plainly: ExpiredWrites is D2's SWEEP side for BUFFERED streams only (it skips waiters: !st.waiter); the waiter path is the separate test above.

RED against the wall clock (flipped = false: the clock is the wall reading)

const flipped = false in internal/mono/mono.go (nothing else changed), go test -run '1190$' -v ./internal/conn ./internal/protocol/h2/stream: 20 of the 21 test and subtest results FAIL (7 tests, 14 subtests); the one that passes is the forward renewal subtest (a forward step does not hurt a stream that keeps releasing).

    stepped_h2_1190_test.go:59: a step of 1m15s closed an idle clock that had run for 0.15 s: {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:true}
    stepped_h2_1190_test.go:63: after IdleTimeout (1.2 s of time, step -1m15s) the idle conn was not closed: {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:false}
--- FAIL: TestIdleClockAndAckCapIgnoreAStepOfTheWallClock1190 (0.00s)
    --- FAIL: TestIdleClockAndAckCapIgnoreAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestIdleClockAndAckCapIgnoreAStepOfTheWallClock1190/backward (0.00s)
    stepped_h2_1190_test.go:120: a step of 1m15s ended a write window that had run for nothing: {Wrote:true Expired:1 CloseAfterFlush:false CloseNow:false}
    stepped_h2_1190_test.go:128: ExpiredWrites after the window (1.2 s of time, step -1m15s) returned [], want stream 1
--- FAIL: TestWriteWindowExpiryIgnoresAStepOfTheWallClock1190 (0.00s)
    --- FAIL: TestWriteWindowExpiryIgnoresAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestWriteWindowExpiryIgnoresAStepOfTheWallClock1190/backward (0.00s)
    stepped_h2_1190_test.go:179: a stream that stopped releasing was not reset within four windows (step -1m15s): {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:false}
--- FAIL: TestWriteWindowRenewalIgnoresAStepOfTheWallClock1190 (0.00s)
    --- PASS: TestWriteWindowRenewalIgnoresAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestWriteWindowRenewalIgnoresAStepOfTheWallClock1190/backward (0.00s)
    stepped_h2_1190_test.go:200: a step of 1m15s ended a header block that had run for nothing: {Wrote:true Expired:0 CloseAfterFlush:true CloseNow:false}
    stepped_h2_1190_test.go:206: the header block (1.2 s of time, step -1m15s) was not ended: {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:false}
--- FAIL: TestHeaderBlockDeadlineIgnoresAStepOfTheWallClock1190 (0.00s)
    --- FAIL: TestHeaderBlockDeadlineIgnoresAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestHeaderBlockDeadlineIgnoresAStepOfTheWallClock1190/backward (0.00s)
    stepped_h2_1190_test.go:251: a step of 1m15s dropped a tombstone that had lived for nothing
    stepped_h2_1190_test.go:262: the tombstone outlived its 2 s (2.2 s of time, step -1m15s)
--- FAIL: TestTombstoneDeadlineIgnoresAStepOfTheWallClock1190 (0.00s)
    --- FAIL: TestTombstoneDeadlineIgnoresAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestTombstoneDeadlineIgnoresAStepOfTheWallClock1190/backward (0.00s)
    stepped_h2_1190_test.go:292: 0.65 s after its last stream (step 1m15s) the conn was closed: the idle clock did not restart: {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:true}
    stepped_h2_1190_test.go:296: 1.25 s after its last stream (step -1m15s) the idle conn was not closed: {Wrote:false Expired:0 CloseAfterFlush:false CloseNow:false}
--- FAIL: TestIdleClockRestartsFromStreamActivityOnAStepOfTheWallClock1190 (0.00s)
    --- FAIL: TestIdleClockRestartsFromStreamActivityOnAStepOfTheWallClock1190/forward (0.00s)
    --- FAIL: TestIdleClockRestartsFromStreamActivityOnAStepOfTheWallClock1190/backward (0.00s)
FAIL
FAIL	github.com/goceleris/celeris/internal/conn	0.411s
    stepped_wait_1190_test.go:54: after a step of 1m15s: the wait ended with i/o timeout before its window did
    stepped_wait_1190_test.go:62: the window ended (80ms of time, step -1m15s) and the wait was not judged: held past its bound
--- FAIL: TestWaiterWindowIgnoresAStepOfTheWallClock1190 (5.49s)
    --- FAIL: TestWaiterWindowIgnoresAStepOfTheWallClock1190/forward (0.16s)
    --- FAIL: TestWaiterWindowIgnoresAStepOfTheWallClock1190/backward (5.32s)
FAIL
FAIL	github.com/goceleris/celeris/internal/protocol/h2/stream	5.738s
FAIL

With flipped = true (main): all pass, also with the clock skewed by +1 h and -1 h.

Mutants: one site back to time.Now().UnixNano() (scratch copy; mut1190b.py)

Gates: the scan (TestNoWallStamps); the new tests by name (-run '1190$', conn + h2 stream) with no skew, at +1 h and at -1 h; the canary's whole internal/conn + internal/protocol/h2/stream at +1 h and -1 h. "killed (N)" = N FAIL lines; "pass" = the mutant survived that gate.

mutant scan new tests 0 s new tests +1 h new tests -1 h canary whole +1 h canary whole -1 h
D0 ack cap: ExtendIdle d0At killed (1) killed (3) killed (3) killed (3) killed (4) killed (6)
D0 activity reset (h2.go:1128) killed (1) killed (3) killed (3) killed (3) killed (3) killed (5)
ExpiredWrites: the now Sweep hands it killed (1) killed (6) killed (6) killed (6) killed (7) killed (7)
D2 buffered: noteBuffered window start killed (1) killed (2) killed (2) killed (2) killed (5) killed (78)
D2 waiter: record armedAt (write_wait.go:153) killed (1) killed (3) killed (3) killed (3) killed (16) killed (11)
D2 waiter: judgeWait now (write_wait.go:190) killed (1) killed (3) killed (3) killed (3) killed (11) killed (9)
header block: armHeaderBlock killed (1) killed (2) killed (2) killed (2) killed (4) killed (3)
tombstone: ExpireRequest until killed (1) killed (3) killed (3) killed (3) killed (6) killed (4)
request deadline: ArmRequest (processor.go:1756) killed (1) killed (3) killed (3) killed (3) killed (6) killed (17)
baseline (no mutant) pass pass pass pass pass pass

Every gate kills every mutant at every skew, including the two (header block, D2 noteBuffered) that survived the named run at -1 h before F1, and the two (D0 activity reset, D2 waiter record) that no test reached before F2.

CI wiring and headroom

  • The canary step names the seven tests (-run '1190$' -v, +1 h, -p 1, conn + h2 stream; fails unless seven pass), so a rename cannot pass as "no tests". internal/conn and internal/protocol/h2/stream run whole under the canary at both signs as before.
  • Not added to or skipped from any -race step: six take milliseconds, the waiter's 0.7 s. Test-binary times on this PR's CI run (head e220f3d), against the previous run of this PR (8037241): unit job iouring (race) 496.9 s of 600 s (499.9 s), unit job root set incl. epoll 361.7 s of 540 s (365.6 s), internal/conn there 27.9 s (32.5 s), internal/protocol/h2/stream there 89.3 s, base-tier iouring (race) 590.5 s of 900 s (597.6 s), the canary step 241 s (246 s; the seven named tests are 0.7 s of it). The other near-timeout step, test-coverage.yml's iouring, is not touched by a conn or stream test. 31 checks pass, 1 skipped (CodeRabbit).

Draft. Refs #1190 only.

…the write window (D2, ExpiredWrites), the header-block and the tombstone deadlines (Refs #1190)

mono.Manual is a clock that moves only when a test moves it (Advance: time passes on both readings; Step: the wall reading alone), standing an hour past the real clock so that a site that still reads time.Now is an hour off. The tests need no sleep. For each deadline, in both directions of a 75 s step: the step alone must not fire it, and the time that then passes must. They pass on the monotonic clock and are red on the wall clock (flipped = false: 14 of 15 subtests; the forward step of the renewal case is the one that does not discriminate).
No production change: every one of the deadlines (D0 and its ack cap, D2 and ExpiredWrites, the header block, the tombstone) already reads internal/mono. The wiring: the canary step runs the five tests by name with -v at +1 h and fails when fewer than five pass, so a rename cannot pass as no tests; the whole of internal/conn already runs under the canary at both signs, and the tests take milliseconds, so they stay in the -race steps (they add no measurable time to them).
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 31, 2026
@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/mono/stepclock.go 0.00% 9 Missing ⚠️

📢 Thoughts on this report? Let us know!

…r path are stepped-clock tests; the Manual clock stands an hour past the real wall at every skew (Refs #1190)

Review r1 of #1191. F2a: TestIdleClockRestartsFromStreamActivity...: a conn that carried a
stream is idle for IdleTimeout after the stream, not after the first sweep (red on the wall
clock; the d0At reset at conn/h2.go reads the sweep's now). F2b: TestWaiterWindowIgnores...
(h2 stream): a pool handler's wait is judged at its window's end by its own timer, which
ExpiredWrites skips; the manual clock gives no verdict before Advance, whatever the runner's
speed, and the timer re-arms against the clock. F1: NewManual anchors to the real wall AFTER the
skew (at CELERIS_MONO_SKEW=-1h the hour on the monotonic reading alone cancelled the skew and the
clock stood on the real wall: two mutants passed the named run). F3: the last D0 period runs out.
F4: the 'step alone' sweep crosses the d0Interval gate; a comment says what forward and backward
test on the monotonic clock. F5: the tombstone test asserts the request did not expire at 0.5 s.
CI: the named run covers both packages and wants 7.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 31, 2026 16:18
@FumingPower3925
FumingPower3925 merged commit a4b10df into main Oct 31, 2026
32 checks passed
@FumingPower3925
FumingPower3925 deleted the mono/1190-stepped-h2 branch October 31, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant