Repository navigation
engines, conn, h2: a step of the wall clock no longer cuts or holds connections: the stamps are monotonic (Refs #1178) - #1188
Merged
Conversation
…root package; red against the wall clock (Refs #1178) Commit 1 of the flip PR, red on purpose: mono.Now() still reads the wall clock, and the clock the tests install (mono.Stepper, mono.SetStair) steps its wall reading while its monotonic reading runs. internal/mono/monotest holds the scenarios both native engines run: an idle keep-alive conn at IdleTimeout, a stalled and a slow upload at ReadTimeout, a slow header and a silent conn at ReadHeaderTimeout, an HTTP/2 request at ReadTimeout, a slow reader of a 32 MiB response and the non-reading peer at WriteTimeout, each with the wall reading stepping 75 s every 50 ms forward and backward, each asserting both bounds. conn has the header deadline, the WebSocket idle deadline (the SSE sentinels pass through) and the HTTP/2 request deadline; the root package has the latency and StartTime() of a request a step happens in.
…longer moves a timeout (Refs #1178) Commit 2 of the flip PR: flipped is true. mono.Now() is base + time.Since(start): the wall time taken once at process start plus the monotonic time since, on the wall's scale, one clock read where time.Now().UnixNano() made two. Every wall stamp of the engines, conn, the root package and the HTTP/2 stream code was routed through it by the steps before (the scan holds it: a wall read it does not list fails), so the stamps and the comparisons that read them are on one clock again, and the tests of commit 1 pass.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… runs skewed in the canary; the collector's latency under a step is a test (Refs #1178) Review r2 of #1188. MAJOR-1: the stepped-clock test (80 s per engine under -race) took the coverage iouring step to 594.96 s of 600 s and the unit job's to 592.7 s: -skip by name in the four race steps (ci.yml unit iouring and root set, test-coverage.yml iouring and root set); it stays in the base-tier job, the only race run of it (66 per cent of its timeout), and runs without -race in the canary step with the clock +1 h, which also makes it the canary's strongest check of the loops' own clock reads. MINOR-1: TestFallbackLatency... passed on the wall clock; TestCollectorLatencyIgnoresAStepOfTheWallClock1178 asserts the duration the collector records (handler.go) on both the engine-stamped and the fallback path: red with flipped=false (bucket 9, the 75 s step), green with true. NITs: the scan sees time.Since(time.Date(...)), its godoc says what it cannot see; two comments of handler.go were false since the flip.
This was referenced Oct 31, 2026
FumingPower3925
marked this pull request as ready for review
October 31, 2026 14:05
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1178
The flip: the native engines' timeouts, the timeout sweep, conn's HTTP/1 header deadline and the HTTP/2 stream deadlines read a clock that a step of the wall clock does not move. Steps 1 to 6 (#1179, #1182, #1184, #1185, #1186, #1187) sent every wall stamp through
internal/monowhilemono.Now()was still the wall clock (value-neutral, canary at +1 h and -1 h green in every step); this PR changes the clock. Two commits: commit 1 is the tests, RED against the wall clock on purpose; commit 2 is the flip (const flipped = true) and its docs. Needs the Opus review before merge.Commit 2: the flip
mono.Now()isbase + time.Since(start): the wall time taken once at process start plus the monotonic time since, on the wall's scale (so everynow - stamp, everyint64(time.Second)literal stays valid), one clock read wheretime.Now().UnixNano()made two. The completeness gates:TestNoWallStamps(static, syntactic, directory walk,staysonly, 8 entries with reasons) andTestNoWallFixtures, and the skew canary (-tags celeris_monoskew, clock +1 h / -1 h; a site that still read the wall would be an hour off from every stamp). What CI runs under the canary:internal/monoand the whole ofinternal/connandinternal/protocol/h2/stream, fixture subsets of both engines, three root tests, and (since review r2) the engines' stepped-clock test at +1 h. The WHOLE-package canary over root and both engines was run locally (below), not in CI. The stepped-clock tests themselves cannot see a missed site after the flip (the stair moves a wall reading that nothing reads any more); the scan and the canary are the gates. Nothing else changes in production code: godoc ofmonoanddrainclockonly.Commit 1: the tests (red against the wall clock)
mono.Stepper/mono.SetStairinstall a clock whose WALL reading steps (by 75 s every 50 ms, cumulatively, forward or backward) while its monotonic reading runs;internal/mono/monotestholds the scenarios both native engines run, each in both directions and each asserting BOTH bounds (not cut early, not held past its time; "not cut early" alone is passed by a timeout that never fires):conn (
internal/conn/stepped_clock_1178_test.go): header deadline after a step of +75 s:-1m5s away, want about 10s; backward1m25s away; WebSocket idle deadline forward-45s away, want about 30s, backward1m45s; HTTP/2 request deadline forwarda request just armed was expired by a forward step, backwarda request past ReadTimeout was not expired. The SSE sentinels (0 and 1 throughSetWSIdleDeadline) PASS in commit 1: they pin the conversion's pass-through, they are not red-then-green. Root: the detach snapshot's elapsed time1m15.02s, want about 20 ms,time.Since(StartTime()) = 1m15s,StartTime() is -1m15s from the real start.Red output of commit 1 on both engines (iouring shown; epoll is the same, 40 to 51 ms):
Commit 2 makes every one of them pass (the suite takes 75 to 78 s per engine). The suite is deliberately trimmed (per engine: idle, stalled and slow upload, slow and silent header, one HTTP/2 deadline, the sending arm both ways); the other cases (detection deadline, closing bound, D0, D2, header block, tombstones) follow as pins. The scenarios are in
internal/mono/monotestso that both engines run the same ones;mono.Stepperandmono.SetStair(internal/mono/stepclock.go) are test helpers that live in the shipped package because other packages' tests import them (a_test.gofile cannot be imported); they are internal and dead-code eliminated.Verification on the final tree
Review r2 fixes (head 05cbb20). CI headroom (MAJOR-1): the engines' stepped-clock test (76 to 80 s per engine under -race) is
-skipped by name in the four -race steps that cover the engine packages (ci.yml unit: iouring and the root set that includes epoll; test-coverage.yml: iouring and the root set), stays under -race in the base-tier job (the only -race run of it), and runs without -race with the clock +1 h in the canary step (both engines,-p 1). Per-package test-binary times from CI, before the fix (d3f7ae1) and after (05cbb20), with main (0b4d504):The
-skipwas checked:go test -race -skip '^TestEngineTimeoutsIgnoreAStepOfTheWallClock1178$' -run AStepOfTheWallClock1178 ./internal/engine/iouring/reports[no tests to run], and the root sets use one-skipalternation ($skip|^...$). A skipped test prints no--- SKIPline, so the skip tallies are unaffected. The canary line (stepped test, +1 h, no race,-p 1) takes 161 s in docker and 153 s in CI. With the stepped test under +1 h the canary is no longer thin for a missed loop-thread clock read (the reviewer measured 10 to 14 failing lines for a missed per-batchcachedNow, where the earlier canary subset caught it through 1 or 2 side-effect tests at +1 h only).MINOR-1:
TestFallbackLatencyIgnoresAStepOfTheWallClock1178passed on the wall clock and pinned nothing; it is replaced byTestCollectorLatencyIgnoresAStepOfTheWallClock1178(root), which asserts the duration the collector records (handler.go:mono.Now() - c.startTimeNs) on the engine-stamped and the fallback path. RED/GREEN, same test, only the constant changes:flipped = false(the wall clock):the request was recorded in latency bucket 9 (bounds [0.001 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 5]): a step of 75 s is not request timefor both subtests (FAIL);flipped = true: PASS for both. It also passes at +1 h and -1 h (CI canary). NITs: the scan now also seestime.Since(time.Date(...))/time.Since(time.Unix(...))(its godoc says what it cannot see:time.Since(x)of a variable, for which the canary is the backstop); the two comments ofhandler.gothat the flip made false are fixed;stepclock.gostays in the shipped package for the reason above.Docker battery on d3f7ae1 (linux/arm64,
--cpus 2, one container at a time, seccomp unconfined, memlock 8 MiB,go test -count=1, the flipped tree), then the review-r2 fix commit on top (tests and workflows only):-tags celeris_monoskew)internal/mono, conn, h2 stream, drainclockinternal/engine/iouring, wholeinternal/engine/epoll, whole./test/server, whole-1 h: my battery was stopped after the +1 h rows (to free the host for the fixes). The review r2 ran the whole packages at -1 h on this head (non-race,
-p 1, 8 MiB memlock): mono 0.8 s, conn 9.4 s, h2 stream 18.6 s, root 41.3 s, epoll 433.7 s, iouring 542.6 s, 0 failures (and +1 h with unlimited memlock: mono 1.1, conn 9.4, h2 stream 19.6, root 78.3, epoll 438.5, iouring 544.3 s, 0 failures), plus 7 planted mutants (one routed site pointed back at the wall) killed in CI by the scan, the canary, or both. The earlier discovery run on the tree with steps 2 to 6 (before the flip) had passed the whole iouring, epoll and root packages at both signs.Commit 1 on the tree before the flip (docker): red as shown above. All CI checks green on 05cbb20 (31 pass, 2 skipped: CodeRabbit and CodSpeed).
Benchmarks (docker arm64,
--cpus 2,internal/mono, 3 runs each; the review r2 measured the same)time.Now().UnixNano()mono.Now()(seam load +base + time.Since)mono.Time(once per logged request,StartTime()) 42 to 48 ns; the handler fallback withFromTimeis 39 to 41 ns against 41 to 44 fortime.Now()+UnixNano(). All 0 allocs. -16 ns on every clock read: one per CQE or epoll batch, two per io_uring keep-alive HTTP/1 request, one per request with a metrics collector.Release note (draft)
The native engines (io_uring and epoll) now take their connection timeouts from a clock that a step of the system's wall clock does not move: the idle, read, write, header and protocol-detection timeouts, the timeout sweep, conn's HTTP/1 header deadline, the WebSocket idle deadline and the HTTP/2 per-stream deadlines. An NTP step, a VM clock sync or a throttled VM stepping its clock repeatedly no longer closes connections that are making progress (a 32 MiB response was cut at 2.7 MB by repeated 75 s steps) and no longer keeps timed-out ones open. This completes the shutdown change of #1176.
Context.StartTime()carries a monotonic reading, so latencies taken from it (time.Since(c.StartTime())) ignore steps, and still returns the true wall time.Context.SetWSIdleDeadlinekeeps taking a wall-clock Unix time in nanoseconds.Behaviour to know: Linux's monotonic clock (
CLOCK_MONOTONIC, which Go reads) does not advance while the machine is suspended, so connections idle across a suspend are timed out after their full timeout has elapsed awake, not at the first sweep after the resume (IdleTimeoutdefaults to 600 s). A process restored from a checkpoint (CRIU without a time namespace) onto a host with less uptime sees the monotonic clock go backward and its timeouts wait until it catches up; VM live migration and snapshot restore typically keep the guest's monotonic clock continuous (kvmclock, Firecracker; not verified here).Residuals (what this does not change)
lastUsedAt,lastFailAt; self-contained, a step costs a spurious pool recycle, not a cut request).stays(the Date header, a cookie'sExpires, two one-second flood windows, the router's promotion TTL, the two publicSetWSIdleDeadlinevalues of the websocket middleware, converted bymono.FromWall).drainclockanddeferlinger(their own small scales; shutdown deadlines since engines: the shutdown deadlines are on the monotonic clock, not the wall clock (Refs #1176) #1177) are unchanged and are not comparable withmono.Now()(the scan fails a statement that mixes them).baseis taken once at start: a host that boots with a wrong wall clock that NTP later corrects keeps amono.Now()that is off from the true wall by the correction; harmless, since no stamp crosses to a wall boundary unconverted (StartTime()andSetWSIdleDeadlineconvert with a freshtime.Now()).baseis clamped to 2006..2116 so a host without a hardware clock (1970 + uptime) cannot put a stamp next to the sentinels 0 and 1.ExpiredWriteshave no stepped test yet (tracked in Stepped-clock tests for the h2 D0, D2, header-block and tombstone deadlines #1190, milestone v1.6.0); they pass their suites under both canary signs.Draft: not marked ready, not merged. Milestone v1.6.0.