Skip to content

iouring: read the clock when a wait ends without a completion, so a handler's write after a quiet gap starts a fresh write clock (Refs #1173) - #1175

Merged
FumingPower3925 merged 4 commits into
mainfrom
flake-1173
Oct 30, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
flake-1173

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1173

Verdict: a PRODUCT bug (a small one) in the io_uring write clock, with a small fix. Release-note worthy. One cause of the CI cut stays open (below), so this is Refs, not a closing PR.

What a user could notice

On the io_uring engine with a short WriteTimeout, a response that a handler writes after the conn has been quiet for most of WriteTimeout (an h2 stream sending a chunk every few hundred ms, as in the issue; any async handler that answers late) can have its conn closed, with an EOF at the client, although nothing is slow. The window is narrow (the sweep must come due in the same loop pass), but real: it is the 1 of 16 streams in the issue's CI run.

Root cause

flushSend starts a conn's write clock (lastActivity) from w.cachedNow, which the loop reads once per batch of completions and at each sweep. In a loop pass whose timed wait ran out with no completion, cachedNow is as old as the last of those (up to the whole idle wait: 1 ms, 5 ms, up to 100 ms, 1 s with no listener).

The window: a handler's write (the pending CAS, then the eventfd Signal) lands after the pass has checked for completions and before drainH2Writes. That pass drains it with the stale clock: flushSend stamps lastActivity = cachedNow and stages the SEND (cs.sending = true; the SQE is submitted only at the top of the next pass). The sweep runs later in the same pass, reads a fresh time.Now(), finds cs.sending && elapsed > WriteTimeout and closes the conn before the SEND reaches the kernel. No descheduling and no slow SEND are needed. What is rare is the sweep coming due in that exact pass (on an idle h2 worker: the 1 s ceiling, a gate hit; a tick timer firing is itself a completion, so that pass is fresh). #1136/#889 fixed the gap-since-last-activity case but assumed the pass had read the clock ("the recv, the wake of a handler goroutine's write, or the sweep refreshed cachedNow before this ran"); the code comments said so, and now do not.

(The 100 us HTTP/2 poll exists only for a worker with no wake eventfd; a normal worker's wake is a completion and was fresh already. On a ring with DEFER_TASKRUN, the default tier, a Signal that arrives while the worker is in user space is posted only at the next enter, which widens the window; on the base tier it is posted at once, the window is still reachable: the statistical test is red 4 of 10 there on main, per the review.)

Fits the CI failure: attempt 1 of run 40848305228 (job io_uring base tier; gh run view --log-failed returns attempt 2, which passed; the log is gh api repos/goceleris/celeris/actions/runs/40848305228/attempts/1/logs, file 19_io_uring base tier ..., lines 5780-5785): 1 of 16 streams cut ... 279 chunks seen, first: EOF after 1.141s. Stream 10 ticks every 250+130 = 380 ms, so its fourth chunk is due at 1140 ms, the EOF came at 1.141 s (at a chunk's write, as the mechanism predicts), and the 12 missed chunks match 291-279 = 12 (4.86 s / 380 ms). Weaker corroboration: 1.141 s is just past the first 1 s sweep ceiling.

Stays open (hence Refs): a worker stall longer than WriteTimeout between staging the SEND and the same pass's sweep (GC, -race, CFS throttling on a shared vCPU) cuts the conn even with a fresh clock, because checkTimeouts counts a staged, unsubmitted SEND as "sending". Not shown; a follow-up could exempt a SEND not yet submitted or restamp at submit.

Evidence

  • Instrumented build, TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring, 8 runs, 12 busy loops: two sweeps found a SEND staged with elapsed 70 ms and age of the stamp when taken 70 ms: the whole elapsed time was the stale stamp.
  • Statistical test (commit 1): the age of cachedNow where a handler's response is staged, after quiet gaps. On main it read 103, 122, 128, 83, 114 ms (WriteTimeout 100 ms): already past the timeout. Red 8 of 12 in my runs; the review measured 6 of 12 (default tier) and 4 of 10 (base tier) on main, 0 of 12 and 0 of 10 with the fix. That is a coin flip on a broken build, so:
  • Deterministic test (commit 3, TestWriteClockStampAfterAnEmptyWaitIsFreshDeterministic1173): right after a ring wait that returned no completion (ringWaitSeam) it ages the worker's clock by 1 s and queues a PING on the conn, then reads the age of the clock at the drain (h2WriteDrainSeam). 20 trials, 19 at emptyIters 1. Plus TestReadClockAfterWaitIsFreshWithNoCompletion1173, a unit test of the refresh (now its own method, readClockAfterWait) with emptyIters in 0..200, independent of the tier. On a ring without DEFER_TASKRUN (the base tier) the poll completion is posted at once, no trial counts there and the deterministic test asserts nothing (no skip: the Portable fallback jobs fail on any skip); the unit test and the statistical test hold that tier.

Fix

The loop reads the clock after every wait: an iteration with no completion calls readClockAfterWait (one vDSO call, about 35 ns, on an iteration that already paid for an io_uring_enter). Iterations with completions are unchanged. The review confirmed every other stamp user is covered (drainDetachQueue, flushDirty, async HTTP/1, SQ-full retries, driver actions, the pendingRelease backstop). The now-redundant closeFDOwed refresh is removed; sweepClock's fresh reads are second reads on empty iterations and stay (the cadence tests and benchmark pin them): follow-up.

Mutants (3 runs each, -race, no load)

mutant of readClockAfterWait deterministic unit statistical
no refresh (main) red 3 of 3 red 3 of 3 red 3 of 3
refresh only if emptyIters >= 100 red 3 of 3 red 3 of 3 red 1 of 3 (the quiet gaps leave emptyIters at 16 to 35: it behaves like main, the earlier "1 of 8" was noise)
>= 5 red 3 of 3 red 3 of 3 green 3 of 3
>= 2 red 3 of 3 red 3 of 3 green 3 of 3

The e2e TestQuietH2StreamIsNotCutByWriteTimeout889 gave no cut on main in 10 runs at 3 busy loops, 15 at 12 and 12 instrumented: it is not a usable reproducer here.

Checks

New tests -race -count=20 -cpu 2 pass (default tier; -count=2 on CELERIS_MAX_IOURING_TIER=base pass); the whole internal/engine/iouring package passes under -race; golangci-lint v2.14 0 issues; GOOS=linux|darwin|windows go vet ./... clean. The seam h2WriteDrainSeam (an atomic.Pointer, nil in production, a nil check per conn with queued writes) is the only non-test code besides the fix. Not verified: the CI matrix, other kernels, the CI cut itself on a build without the fix (the residual cause above is not excluded).

… write after a quiet gap is stamped (Refs #1173)

TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring cut 1 of 16 quiet h2
streams in CI (EOF after 1.141 s, WriteTimeout 100 ms). The cut is the sweep
closing a conn whose SEND is in flight and whose write clock (lastActivity,
stamped by flushSend from w.cachedNow) is older than WriteTimeout. An
instrumented build (flushSend stamping the age of cachedNow, the sweep printing
the conns it finds sending past a third of WriteTimeout) saw, in 8 runs of
that test under 12 busy loops, two sweeps find a SEND in flight 70 ms after a
stamp that was already 70 ms old: the whole age was the stale stamp.

This test holds an HTTP/2 stream's handler through a quiet gap, releases it, and
reads the age of w.cachedNow at drainH2Writes, where the handler's response is
staged (h2WriteDrainSeam, a nil-checked test seam like cancelAcceptSeam). Red on
main: 8 of 12 runs, -race, no load, 60 streams each; the oldest stamp read
103, 122, 128, 83 and 114 ms in the failing runs, WriteTimeout being 100 ms.
…andler's write after a quiet gap starts a fresh write clock (v1.6.0, Refs #1173)

flushSend starts a conn's write clock (lastActivity) from w.cachedNow, which the
loop reads once per batch of completions and at each sweep. A wait that ended
with no completion (the HTTP/2 conns' poll, a timed wait that ran out) left it as
old as the last of those, up to the whole idle wait, and drainH2Writes, which
runs on that iteration, stamped the SEND of a handler's write with it. A stream
quiet for most of WriteTimeout therefore began its SEND with the clock most of
WriteTimeout old (103 to 128 ms against a WriteTimeout of 100 in the test), and
a sweep that found the SEND still in flight closed the conn, mid-stream.

The iteration that woke with no completion now reads the clock: one vDSO call
where the worker has nothing to be quick about.

The new test: 0 of 12 runs red (oldest stamp 0.34 ms), 0 of 5 under 12 busy
loops, -race -count=20 -cpu 2 passes; the package passes under -race.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 30, 2026
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

… the comments that stated the old invariant (v1.6.0, Refs #1173)

The statistical test catches a build without the fix about half the time, and a
refresh gated on the empty-iteration count (>= 2, >= 5, >= 100) not at all or as
rarely. Added:

- TestWriteClockStampAfterAnEmptyWaitIsFreshDeterministic1173: right after a ring
  wait that returned no completion (ringWaitSeam) it ages the worker's clock by a
  second and queues a PING on the conn, as a handler's write landing after the
  loop looked for completions would; drainH2Writes of that iteration stages the
  SEND, and h2WriteDrainSeam reads the age of the clock. 20 trials, 19 at
  emptyIters 1. Without the fix and with a refresh gated on emptyIters >= 2, 5 or
  100 it fails 3 of 3; with the fix it passes (worst age 4 to 24 us). On a ring
  without DEFER_TASKRUN (the base tier) the poll completion is posted at once, so
  no trial counts there and the test asserts nothing: the unit test holds the rule.
- TestReadClockAfterWaitIsFreshWithNoCompletion1173: the refresh is its own method
  (readClockAfterWait), tested with emptyIters in 0..200; fails 3 of 3 for the
  same four mutants.

The mechanism, as the review read it from the code: the sweep that closes the conn
runs later in the same loop pass as the drain, and the SEND is only staged, so no
descheduling is needed; the window is a handler's write landing after the pass has
checked for completions and before the drain, on a pass whose wait timed out. The
100 us HTTP/2 poll exists only for a worker without a wake eventfd.

Comments: the cachedNow field, flushSend and the adoption's fresh read in
transplant.go said the iteration had already read the clock; the closeFDOwed
refresh, now redundant, is removed, and sweepClock's note says its fresh reads are
second ones since this change (the cadence tests and benchmark pin it: follow-up).
h2WriteDrainSeam is an atomic.Pointer, as testHookCheckTimeouts is.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 30, 2026 14:41
@FumingPower3925
FumingPower3925 merged commit 40ed250 into main Oct 30, 2026
30 checks passed
@FumingPower3925
FumingPower3925 deleted the flake-1173 branch October 30, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant