Skip to content

engines, conn, h2: a step of the wall clock no longer cuts or holds connections: the stamps are monotonic (Refs #1178) - #1188

Merged
FumingPower3925 merged 4 commits into
mainfrom
mono/1178-step7-flip-push
Oct 31, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
mono/1178-step7-flip-push

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1178

The flip: the native engines' timeouts, the timeout sweep, conn's HTTP/1 header deadline and the HTTP/2 stream deadlines read a clock that a step of the wall clock does not move. Steps 1 to 6 (#1179, #1182, #1184, #1185, #1186, #1187) sent every wall stamp through internal/mono while mono.Now() was still the wall clock (value-neutral, canary at +1 h and -1 h green in every step); this PR changes the clock. Two commits: commit 1 is the tests, RED against the wall clock on purpose; commit 2 is the flip (const flipped = true) and its docs. Needs the Opus review before merge.

Commit 2: the flip

mono.Now() is base + time.Since(start): the wall time taken once at process start plus the monotonic time since, on the wall's scale (so every now - stamp, every int64(time.Second) literal stays valid), one clock read where time.Now().UnixNano() made two. The completeness gates: TestNoWallStamps (static, syntactic, directory walk, stays only, 8 entries with reasons) and TestNoWallFixtures, and the skew canary (-tags celeris_monoskew, clock +1 h / -1 h; a site that still read the wall would be an hour off from every stamp). What CI runs under the canary: internal/mono and the whole of internal/conn and internal/protocol/h2/stream, fixture subsets of both engines, three root tests, and (since review r2) the engines' stepped-clock test at +1 h. The WHOLE-package canary over root and both engines was run locally (below), not in CI. The stepped-clock tests themselves cannot see a missed site after the flip (the stair moves a wall reading that nothing reads any more); the scan and the canary are the gates. Nothing else changes in production code: godoc of mono and drainclock only.

Commit 1: the tests (red against the wall clock)

mono.Stepper / mono.SetStair install a clock whose WALL reading steps (by 75 s every 50 ms, cumulatively, forward or backward) while its monotonic reading runs; internal/mono/monotest holds the scenarios both native engines run, each in both directions and each asserting BOTH bounds (not cut early, not held past its time; "not cut early" alone is passed by a timeout that never fires):

scenario (engine) forward step, commit 1 backward step, commit 1
idle keep-alive conn, IdleTimeout 1 s closed after 50 ms (want >= 600 ms) still open 8.0 s (want <= 7 s)
stalled upload, ReadTimeout 2 s closed after 50 ms (want >= 1.3 s) still open 10.0 s (want <= 9 s)
unfinished header, ReadHeaderTimeout 1 s closed after 47 ms still open 8.0 s
silent conn (protocol undecided), ReadHeaderTimeout 1 s (cut either way) still open 8.0 s
HTTP/2 request not whole, ReadTimeout 1 s answered after 51 ms no answer after 8.0 s
32 MiB response, reader reads nothing for 3 s, WriteTimeout 60 s read 175494 of 33554432 bytes: unexpected EOF (whole)
slow upload still arriving (10 x 100 B over 2 s), ReadTimeout 2 s the server cut a body that was arriving: broken pipe n/a
non-reading peer, WriteTimeout 2 s (positive control) (cut either way) got the whole response: write clock held

conn (internal/conn/stepped_clock_1178_test.go): header deadline after a step of +75 s: -1m5s away, want about 10s; backward 1m25s away; WebSocket idle deadline forward -45s away, want about 30s, backward 1m45s; HTTP/2 request deadline forward a request just armed was expired by a forward step, backward a request past ReadTimeout was not expired. The SSE sentinels (0 and 1 through SetWSIdleDeadline) PASS in commit 1: they pin the conversion's pass-through, they are not red-then-green. Root: the detach snapshot's elapsed time 1m15.02s, want about 20 ms, time.Since(StartTime()) = 1m15s, StartTime() is -1m15s from the real start.

Red output of commit 1 on both engines (iouring shown; epoll is the same, 40 to 51 ms):

monotest.go:158: idle keep-alive conn (IdleTimeout 1 s): closed after 50ms, want no sooner than 600ms (a timeout that fired early)
monotest.go:188: stalled upload (ReadTimeout 2 s): closed after 50ms, want no sooner than 1.3s (a timeout that fired early)
monotest.go:223: unfinished header (ReadHeaderTimeout 1 s): closed after 47ms, want no sooner than 600ms (a timeout that fired early)
monotest.go:272: the request was answered after 51ms, want no sooner than 600ms (a deadline that fired early)
monotest.go:291: a response to a reader that waited 3 s: read 175494 of 33554432 bytes: unexpected EOF
monotest.go:204: the server cut a body that was arriving: write tcp 127.0.0.1:43766->127.0.0.1:36173: write: broken pipe
monotest.go:158: idle keep-alive conn (IdleTimeout 1 s): still open 8.002s after the start, want it closed by 7s (a timeout held past its time)
monotest.go:188: stalled upload (ReadTimeout 2 s): still open 10.001s after the start, want it closed by 9s (a timeout held past its time)
monotest.go:223: unfinished header (ReadHeaderTimeout 1 s): still open 8.008s after the start, want it closed by 7s (a timeout held past its time)
monotest.go:233: silent conn (ReadHeaderTimeout 1 s): still open 8.002s after the start, want it closed by 7s (a timeout held past its time)
monotest.go:264: no answer to a request past ReadTimeout (1 s) after 8.003s: read tcp ...: i/o timeout (a deadline held past its time)
monotest.go:310: a peer that read nothing for 9 s with WriteTimeout 2 s got the whole response: the write clock was held past its time
--- FAIL: TestEngineTimeoutsIgnoreAStepOfTheWallClock1178 (103.07s)   [forward 13.7 s, backward 89.4 s]

Commit 2 makes every one of them pass (the suite takes 75 to 78 s per engine). The suite is deliberately trimmed (per engine: idle, stalled and slow upload, slow and silent header, one HTTP/2 deadline, the sending arm both ways); the other cases (detection deadline, closing bound, D0, D2, header block, tombstones) follow as pins. The scenarios are in internal/mono/monotest so that both engines run the same ones; mono.Stepper and mono.SetStair (internal/mono/stepclock.go) are test helpers that live in the shipped package because other packages' tests import them (a _test.go file cannot be imported); they are internal and dead-code eliminated.

Verification on the final tree

Review r2 fixes (head 05cbb20). CI headroom (MAJOR-1): the engines' stepped-clock test (76 to 80 s per engine under -race) is -skipped by name in the four -race steps that cover the engine packages (ci.yml unit: iouring and the root set that includes epoll; test-coverage.yml: iouring and the root set), stays under -race in the base-tier job (the only -race run of it), and runs without -race with the clock +1 h in the canary step (both engines, -p 1). Per-package test-binary times from CI, before the fix (d3f7ae1) and after (05cbb20), with main (0b4d504):

step timeout main d3f7ae1 05cbb20
ci.yml unit, iouring (-race) 600 s 513.0 s 592.7 s 510.3 s
ci.yml unit, root set incl. epoll (-race) 540 s 383.5 s 461.6 s 389.0 s
test-coverage.yml, iouring (-race, cover) 600 s 523.3 s 594.96 s 507.4 s
test-coverage.yml, root set incl. epoll 540 s 383.6 s 446.6 s 361.0 s
ci.yml base tier, iouring (-race, test kept) 900 s 509.7 s 596.7 s 600.1 s (66.7%)
ci.yml canary step (non-race, whole step) none 110 s 110 s 260 s: the stepped-clock test at +1 h is 76.6 s (iouring) + 75.0 s (epoll) of it

The -skip was checked: go test -race -skip '^TestEngineTimeoutsIgnoreAStepOfTheWallClock1178$' -run AStepOfTheWallClock1178 ./internal/engine/iouring/ reports [no tests to run], and the root sets use one -skip alternation ($skip|^...$). A skipped test prints no --- SKIP line, so the skip tallies are unaffected. The canary line (stepped test, +1 h, no race, -p 1) takes 161 s in docker and 153 s in CI. With the stepped test under +1 h the canary is no longer thin for a missed loop-thread clock read (the reviewer measured 10 to 14 failing lines for a missed per-batch cachedNow, where the earlier canary subset caught it through 1 or 2 side-effect tests at +1 h only).

MINOR-1: TestFallbackLatencyIgnoresAStepOfTheWallClock1178 passed on the wall clock and pinned nothing; it is replaced by TestCollectorLatencyIgnoresAStepOfTheWallClock1178 (root), which asserts the duration the collector records (handler.go: mono.Now() - c.startTimeNs) on the engine-stamped and the fallback path. RED/GREEN, same test, only the constant changes: flipped = false (the wall clock): the request was recorded in latency bucket 9 (bounds [0.001 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 5]): a step of 75 s is not request time for both subtests (FAIL); flipped = true: PASS for both. It also passes at +1 h and -1 h (CI canary). NITs: the scan now also sees time.Since(time.Date(...)) / time.Since(time.Unix(...)) (its godoc says what it cannot see: time.Since(x) of a variable, for which the canary is the backstop); the two comments of handler.go that the flip made false are fixed; stepclock.go stays in the shipped package for the reason above.

Docker battery on d3f7ae1 (linux/arm64, --cpus 2, one container at a time, seccomp unconfined, memlock 8 MiB, go test -count=1, the flipped tree), then the review-r2 fix commit on top (tests and workflows only):

run plain clock +1 h (-tags celeris_monoskew)
engine scenarios (the stepped-clock test), iouring 76.6 s ok (CI canary line, +1 h) 76.4 s ok
engine scenarios, epoll 74.9 s ok 75.0 s ok
internal/mono, conn, h2 stream, drainclock ok (0.7, 9.5, 23.3, 0.0 s) ok (0.8, 9.6, 29.1 s)
internal/engine/iouring, whole ok 544.3 s ok 578.4 s
internal/engine/epoll, whole ok 431.1 s ok 426.8 s
root package, whole ok 46.3 s ok 46.2 s
./test/server, whole ok 1617.4 s ok 1706.9 s

-1 h: my battery was stopped after the +1 h rows (to free the host for the fixes). The review r2 ran the whole packages at -1 h on this head (non-race, -p 1, 8 MiB memlock): mono 0.8 s, conn 9.4 s, h2 stream 18.6 s, root 41.3 s, epoll 433.7 s, iouring 542.6 s, 0 failures (and +1 h with unlimited memlock: mono 1.1, conn 9.4, h2 stream 19.6, root 78.3, epoll 438.5, iouring 544.3 s, 0 failures), plus 7 planted mutants (one routed site pointed back at the wall) killed in CI by the scan, the canary, or both. The earlier discovery run on the tree with steps 2 to 6 (before the flip) had passed the whole iouring, epoll and root packages at both signs.

Commit 1 on the tree before the flip (docker): red as shown above. All CI checks green on 05cbb20 (31 pass, 2 skipped: CodeRabbit and CodSpeed).

Benchmarks (docker arm64, --cpus 2, internal/mono, 3 runs each; the review r2 measured the same)

before: time.Now().UnixNano() after the flip: mono.Now() (seam load + base + time.Since)
one read 33 to 37 ns (review r2: 36.0 to 37.0) 18 to 21 ns (review r2: 20.2 to 20.7)
two procs in parallel 17.3 to 18.4 ns 9.5 to 10.0 ns

mono.Time (once per logged request, StartTime()) 42 to 48 ns; the handler fallback with FromTime is 39 to 41 ns against 41 to 44 for time.Now() + UnixNano(). All 0 allocs. -16 ns on every clock read: one per CQE or epoll batch, two per io_uring keep-alive HTTP/1 request, one per request with a metrics collector.

Release note (draft)

The native engines (io_uring and epoll) now take their connection timeouts from a clock that a step of the system's wall clock does not move: the idle, read, write, header and protocol-detection timeouts, the timeout sweep, conn's HTTP/1 header deadline, the WebSocket idle deadline and the HTTP/2 per-stream deadlines. An NTP step, a VM clock sync or a throttled VM stepping its clock repeatedly no longer closes connections that are making progress (a 32 MiB response was cut at 2.7 MB by repeated 75 s steps) and no longer keeps timed-out ones open. This completes the shutdown change of #1176. Context.StartTime() carries a monotonic reading, so latencies taken from it (time.Since(c.StartTime())) ignore steps, and still returns the true wall time. Context.SetWSIdleDeadline keeps taking a wall-clock Unix time in nanoseconds.

Behaviour to know: Linux's monotonic clock (CLOCK_MONOTONIC, which Go reads) does not advance while the machine is suspended, so connections idle across a suspend are timed out after their full timeout has elapsed awake, not at the first sweep after the resume (IdleTimeout defaults to 600 s). A process restored from a checkpoint (CRIU without a time namespace) onto a host with less uptime sees the monotonic clock go backward and its timeouts wait until it catches up; VM live migration and snapshot restore typically keep the guest's monotonic clock continuous (kvmclock, Firecracker; not verified here).

Residuals (what this does not change)

  • Application time stays wall: middleware rate limit, cache, session, JWT expiry and the stores, and the drivers' pool stamps (lastUsedAt, lastFailAt; self-contained, a step costs a spurious pool recycle, not a cut request).
  • The 8 wall reads listed in the scan's stays (the Date header, a cookie's Expires, two one-second flood windows, the router's promotion TTL, the two public SetWSIdleDeadline values of the websocket middleware, converted by mono.FromWall).
  • The std engine (net/http's own timeouts), drainclock and deferlinger (their own small scales; shutdown deadlines since engines: the shutdown deadlines are on the monotonic clock, not the wall clock (Refs #1176) #1177) are unchanged and are not comparable with mono.Now() (the scan fails a statement that mixes them).
  • base is taken once at start: a host that boots with a wrong wall clock that NTP later corrects keeps a mono.Now() that is off from the true wall by the correction; harmless, since no stamp crosses to a wall boundary unconverted (StartTime() and SetWSIdleDeadline convert with a fresh time.Now()). base is clamped to 2006..2116 so a host without a hardware clock (1970 + uptime) cannot put a stamp next to the sentinels 0 and 1.
  • The stepped-clock tests are blind to a missed site after the flip (the stair moves a wall reading nothing reads any more); the scan and the skew canary are the completeness gates. D0, D2, the header-block deadline, tombstones and ExpiredWrites have no stepped test yet (tracked in Stepped-clock tests for the h2 D0, D2, header-block and tombstone deadlines #1190, milestone v1.6.0); they pass their suites under both canary signs.
  • Not verified: linux/amd64 on real hardware, a real step of a real host's clock (the seam models it), suspend/resume on real hardware, other kernels (6.x linuxkit only).

Draft: not marked ready, not merged. Milestone v1.6.0.

…root package; red against the wall clock (Refs #1178)

Commit 1 of the flip PR, red on purpose: mono.Now() still reads the wall
clock, and the clock the tests install (mono.Stepper, mono.SetStair) steps its
wall reading while its monotonic reading runs. internal/mono/monotest holds the
scenarios both native engines run: an idle keep-alive conn at IdleTimeout, a
stalled and a slow upload at ReadTimeout, a slow header and a silent conn at
ReadHeaderTimeout, an HTTP/2 request at ReadTimeout, a slow reader of a 32 MiB
response and the non-reading peer at WriteTimeout, each with the wall reading
stepping 75 s every 50 ms forward and backward, each asserting both bounds.
conn has the header deadline, the WebSocket idle deadline (the SSE sentinels
pass through) and the HTTP/2 request deadline; the root package has the latency
and StartTime() of a request a step happens in.
…longer moves a timeout (Refs #1178)

Commit 2 of the flip PR: flipped is true. mono.Now() is base + time.Since(start):
the wall time taken once at process start plus the monotonic time since, on the
wall's scale, one clock read where time.Now().UnixNano() made two. Every wall
stamp of the engines, conn, the root package and the HTTP/2 stream code was
routed through it by the steps before (the scan holds it: a wall read it does
not list fails), so the stamps and the comparisons that read them are on one
clock again, and the tests of commit 1 pass.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 31, 2026
@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 7.36196% with 151 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/mono/monotest/monotest.go 0.00% 151 Missing ⚠️

📢 Thoughts on this report? Let us know!

… runs skewed in the canary; the collector's latency under a step is a test (Refs #1178)

Review r2 of #1188. MAJOR-1: the stepped-clock test (80 s per engine under -race) took the coverage
iouring step to 594.96 s of 600 s and the unit job's to 592.7 s: -skip by name in the four race steps
(ci.yml unit iouring and root set, test-coverage.yml iouring and root set); it stays in the base-tier
job, the only race run of it (66 per cent of its timeout), and runs without -race in the canary step
with the clock +1 h, which also makes it the canary's strongest check of the loops' own clock reads.
MINOR-1: TestFallbackLatency... passed on the wall clock; TestCollectorLatencyIgnoresAStepOfTheWallClock1178
asserts the duration the collector records (handler.go) on both the engine-stamped and the fallback path:
red with flipped=false (bucket 9, the 75 s step), green with true. NITs: the scan sees
time.Since(time.Date(...)), its godoc says what it cannot see; two comments of handler.go were false since
the flip.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant