Repository navigation
iouring: read the clock when a wait ends without a completion, so a handler's write after a quiet gap starts a fresh write clock (Refs #1173) - #1175
Merged
Conversation
… write after a quiet gap is stamped (Refs #1173) TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring cut 1 of 16 quiet h2 streams in CI (EOF after 1.141 s, WriteTimeout 100 ms). The cut is the sweep closing a conn whose SEND is in flight and whose write clock (lastActivity, stamped by flushSend from w.cachedNow) is older than WriteTimeout. An instrumented build (flushSend stamping the age of cachedNow, the sweep printing the conns it finds sending past a third of WriteTimeout) saw, in 8 runs of that test under 12 busy loops, two sweeps find a SEND in flight 70 ms after a stamp that was already 70 ms old: the whole age was the stale stamp. This test holds an HTTP/2 stream's handler through a quiet gap, releases it, and reads the age of w.cachedNow at drainH2Writes, where the handler's response is staged (h2WriteDrainSeam, a nil-checked test seam like cancelAcceptSeam). Red on main: 8 of 12 runs, -race, no load, 60 streams each; the oldest stamp read 103, 122, 128, 83 and 114 ms in the failing runs, WriteTimeout being 100 ms.
…andler's write after a quiet gap starts a fresh write clock (v1.6.0, Refs #1173) flushSend starts a conn's write clock (lastActivity) from w.cachedNow, which the loop reads once per batch of completions and at each sweep. A wait that ended with no completion (the HTTP/2 conns' poll, a timed wait that ran out) left it as old as the last of those, up to the whole idle wait, and drainH2Writes, which runs on that iteration, stamped the SEND of a handler's write with it. A stream quiet for most of WriteTimeout therefore began its SEND with the clock most of WriteTimeout old (103 to 128 ms against a WriteTimeout of 100 in the test), and a sweep that found the SEND still in flight closed the conn, mid-stream. The iteration that woke with no completion now reads the clock: one vDSO call where the worker has nothing to be quick about. The new test: 0 of 12 runs red (oldest stamp 0.34 ms), 0 of 5 under 12 busy loops, -race -count=20 -cpu 2 passes; the package passes under -race.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
… the comments that stated the old invariant (v1.6.0, Refs #1173) The statistical test catches a build without the fix about half the time, and a refresh gated on the empty-iteration count (>= 2, >= 5, >= 100) not at all or as rarely. Added: - TestWriteClockStampAfterAnEmptyWaitIsFreshDeterministic1173: right after a ring wait that returned no completion (ringWaitSeam) it ages the worker's clock by a second and queues a PING on the conn, as a handler's write landing after the loop looked for completions would; drainH2Writes of that iteration stages the SEND, and h2WriteDrainSeam reads the age of the clock. 20 trials, 19 at emptyIters 1. Without the fix and with a refresh gated on emptyIters >= 2, 5 or 100 it fails 3 of 3; with the fix it passes (worst age 4 to 24 us). On a ring without DEFER_TASKRUN (the base tier) the poll completion is posted at once, so no trial counts there and the test asserts nothing: the unit test holds the rule. - TestReadClockAfterWaitIsFreshWithNoCompletion1173: the refresh is its own method (readClockAfterWait), tested with emptyIters in 0..200; fails 3 of 3 for the same four mutants. The mechanism, as the review read it from the code: the sweep that closes the conn runs later in the same loop pass as the drain, and the SEND is only staged, so no descheduling is needed; the window is a handler's write landing after the pass has checked for completions and before the drain, on a pass whose wait timed out. The 100 us HTTP/2 poll exists only for a worker without a wake eventfd. Comments: the cachedNow field, flushSend and the adoption's fresh read in transplant.go said the iteration had already read the clock; the closeFDOwed refresh, now redundant, is removed, and sweepClock's note says its fresh reads are second ones since this change (the cadence tests and benchmark pin it: follow-up). h2WriteDrainSeam is an atomic.Pointer, as testHookCheckTimeouts is.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1173
Verdict: a PRODUCT bug (a small one) in the io_uring write clock, with a small fix. Release-note worthy. One cause of the CI cut stays open (below), so this is
Refs, not a closing PR.What a user could notice
On the io_uring engine with a short
WriteTimeout, a response that a handler writes after the conn has been quiet for most ofWriteTimeout(an h2 stream sending a chunk every few hundred ms, as in the issue; any async handler that answers late) can have its conn closed, with an EOF at the client, although nothing is slow. The window is narrow (the sweep must come due in the same loop pass), but real: it is the 1 of 16 streams in the issue's CI run.Root cause
flushSendstarts a conn's write clock (lastActivity) fromw.cachedNow, which the loop reads once per batch of completions and at each sweep. In a loop pass whose timed wait ran out with no completion,cachedNowis as old as the last of those (up to the whole idle wait: 1 ms, 5 ms, up to 100 ms, 1 s with no listener).The window: a handler's write (the
pendingCAS, then the eventfdSignal) lands after the pass has checked for completions and beforedrainH2Writes. That pass drains it with the stale clock:flushSendstampslastActivity = cachedNowand stages the SEND (cs.sending = true; the SQE is submitted only at the top of the next pass). The sweep runs later in the same pass, reads a freshtime.Now(), findscs.sending && elapsed > WriteTimeoutand closes the conn before the SEND reaches the kernel. No descheduling and no slow SEND are needed. What is rare is the sweep coming due in that exact pass (on an idle h2 worker: the 1 s ceiling, a gate hit; a tick timer firing is itself a completion, so that pass is fresh). #1136/#889 fixed the gap-since-last-activity case but assumed the pass had read the clock ("the recv, the wake of a handler goroutine's write, or the sweep refreshed cachedNow before this ran"); the code comments said so, and now do not.(The 100 us HTTP/2 poll exists only for a worker with no wake eventfd; a normal worker's wake is a completion and was fresh already. On a ring with DEFER_TASKRUN, the default tier, a
Signalthat arrives while the worker is in user space is posted only at the next enter, which widens the window; on the base tier it is posted at once, the window is still reachable: the statistical test is red 4 of 10 there on main, per the review.)Fits the CI failure: attempt 1 of run 40848305228 (job
io_uring base tier;gh run view --log-failedreturns attempt 2, which passed; the log isgh api repos/goceleris/celeris/actions/runs/40848305228/attempts/1/logs, file19_io_uring base tier ..., lines 5780-5785):1 of 16 streams cut ... 279 chunks seen, first: EOF after 1.141s. Stream 10 ticks every 250+130 = 380 ms, so its fourth chunk is due at 1140 ms, the EOF came at 1.141 s (at a chunk's write, as the mechanism predicts), and the 12 missed chunks match 291-279 = 12 (4.86 s / 380 ms). Weaker corroboration: 1.141 s is just past the first 1 s sweep ceiling.Stays open (hence
Refs): a worker stall longer thanWriteTimeoutbetween staging the SEND and the same pass's sweep (GC,-race, CFS throttling on a shared vCPU) cuts the conn even with a fresh clock, becausecheckTimeoutscounts a staged, unsubmitted SEND as "sending". Not shown; a follow-up could exempt a SEND not yet submitted or restamp at submit.Evidence
TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring, 8 runs, 12 busy loops: two sweeps found a SEND staged withelapsed 70 msandage of the stamp when taken 70 ms: the whole elapsed time was the stale stamp.cachedNowwhere a handler's response is staged, after quiet gaps. On main it read 103, 122, 128, 83, 114 ms (WriteTimeout 100 ms): already past the timeout. Red 8 of 12 in my runs; the review measured 6 of 12 (default tier) and 4 of 10 (base tier) on main, 0 of 12 and 0 of 10 with the fix. That is a coin flip on a broken build, so:TestWriteClockStampAfterAnEmptyWaitIsFreshDeterministic1173): right after a ring wait that returned no completion (ringWaitSeam) it ages the worker's clock by 1 s and queues a PING on the conn, then reads the age of the clock at the drain (h2WriteDrainSeam). 20 trials, 19 atemptyIters1. PlusTestReadClockAfterWaitIsFreshWithNoCompletion1173, a unit test of the refresh (now its own method,readClockAfterWait) withemptyItersin 0..200, independent of the tier. On a ring without DEFER_TASKRUN (the base tier) the poll completion is posted at once, no trial counts there and the deterministic test asserts nothing (no skip: the Portable fallback jobs fail on any skip); the unit test and the statistical test hold that tier.Fix
The loop reads the clock after every wait: an iteration with no completion calls
readClockAfterWait(one vDSO call, about 35 ns, on an iteration that already paid for anio_uring_enter). Iterations with completions are unchanged. The review confirmed every other stamp user is covered (drainDetachQueue, flushDirty, async HTTP/1, SQ-full retries, driver actions, the pendingRelease backstop). The now-redundant closeFDOwed refresh is removed;sweepClock's fresh reads are second reads on empty iterations and stay (the cadence tests and benchmark pin them): follow-up.Mutants (3 runs each,
-race, no load)readClockAfterWaitemptyIters >= 100emptyItersat 16 to 35: it behaves like main, the earlier "1 of 8" was noise)>= 5>= 2The e2e
TestQuietH2StreamIsNotCutByWriteTimeout889gave no cut on main in 10 runs at 3 busy loops, 15 at 12 and 12 instrumented: it is not a usable reproducer here.Checks
New tests
-race -count=20 -cpu 2pass (default tier;-count=2onCELERIS_MAX_IOURING_TIER=basepass); the wholeinternal/engine/iouringpackage passes under-race;golangci-lint v2.140 issues;GOOS=linux|darwin|windows go vet ./...clean. The seamh2WriteDrainSeam(anatomic.Pointer, nil in production, a nil check per conn with queued writes) is the only non-test code besides the fix. Not verified: the CI matrix, other kernels, the CI cut itself on a build without the fix (the residual cause above is not excluded).