Repository navigation
iouring: a SEND still in our own SQ ring when the sweep comes gets a fresh write clock (Refs #1173) - #1196
Merged
Conversation
…t the peer's to be cut for (RED, Refs #1173) A staged SEND is submitted at the top of the next loop iteration; the sweep runs later in the same one and counts it as in flight, measuring WriteTimeout from the stamp taken at staging. A worker stall longer than WriteTimeout in between closes a conn whose peer was never handed a byte. Unit tests (mono.Manual clock, real ring) over the five SEND arming sites, with the positive control (a SEND the kernel holds, to a peer that takes nothing, is still cut at WriteTimeout), and an end-to-end HTTP/2 test with a stall injected between the drain and the sweep. Red on main.
…fresh write clock (Refs #1173) checkTimeouts counted a staged, unsubmitted SEND as in flight, so a worker stall longer than WriteTimeout between staging and the same iteration's sweep cut a conn whose peer had been handed nothing. The sweep now reads the SQEs the kernel has not consumed ([head, tail) of the SQ ring) and restamps the write clock of the conn of each SEND among them. Nothing on the send path; one scan of the unconsumed SQEs per sweep.
…ep benchmark with SENDs staged (Refs #1173)
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… remainder only the stall, a dirty conn, a detached conn; the generation and wrap guards (RED, Refs #1173)
… its bytes: a remainder, a dirty conn and a detached conn included (Refs #1173) Review r1 of #1196. The restamp moved the write clock of a SEND found in the SQ ring to the sweep's time, so a partial send's remainder got a whole new window whenever a sweep and a partial completion fell in the same pass. The clocks now move by now - passClock (the clock the pass read when it started, which is what flushSend stamps): a fresh SEND still ends at now, a remainder is excused the pass's stall only. The same excuse covers a conn on the dirty list with bytes and no SEND (SQ ring full) and the stallSince clock of a detached conn. config.go says so; the test comments say the stall is in the iteration that stages the response, not after the staging.
…een drained with the stall injected (Refs #1173)
…entical operands (staticcheck SA4000, Refs #1173)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1173
Follow-up of #1175, which fixed one cause of the CI cut and said one stays open. This is that one: a worker stall in the loop iteration that stages a response, between the read of the iteration's clock and that iteration's timeout sweep.
What a user could notice
On the io_uring engine with a short
WriteTimeout, a conn can be closed (EOF at the client) although the peer is reading normally, when the worker stalls for longer thanWriteTimeoutin the iteration that stages the response, before its timeout sweep. A stall like that is a handler of another conn running inline in the same batch, a GC assist,-race, or a CFS quota throttling a CPU-limited container. epoll is not affected (it writes inline; it has no staged-but-not-submitted state).Root cause (read from the code, confirmed by deterministic tests)
flushSendstages the SEND SQE and stamps the conn's write clock (lastActivity) withw.cachedNow, the clock the iteration read when it started; the SQE is submitted to the kernel at the top of the NEXT iteration. The sweep (checkTimeouts) runs later in the SAME iteration, reads a fresh clock, and countscs.sendingas "a SEND in flight":elapsed > WriteTimeoutcloses the conn. While the SEND sits in our own SQ ring the peer has been handed nothing, so a stall of ours (anywhere between the iteration's clock read and the sweep, handlers of other conns earlier in the batch included) is charged to the peer. Sites that arm a SEND:flushSend(fresh and remainder arms),flushSendLink(linked and the one-free-slot fallback); the partial-send re-send,flushDirtyand the h2 deadline sweep arm throughflushSend.Fix
restampUnsubmittedSends(passClock, now)at the start of the sweep gives back to a conn the time the ENGINE held its bytes in this pass,now - passClock(passClock isw.cachedNowas the sweep found it, before it sets it). The conn's clocks move by that, never tonow:now, as if the stall had not happened;WriteTimeoutruns from the first SEND of the response; a partial completion does not restart it) and is excused this pass's stall only. (The first version of this PR restamped tonow, which handed a remainder a whole new window whenever a sweep and a partial completion fell in the same pass, soWriteTimeoutof a large response to a slow reader would have depended on that coincidence, against the measured statements inconfig.go. Review r1 found it; fixed here with a failing-first test.)Whose bytes the kernel has not got:
[sqHead, sqTail)of the SQ ring), whose conn is still the one that placed it (generation) and still has it in flight:lastActivity, andstallSincefor a detached conn (WebSocket/SSE);A stalled peer is still cut: its SEND is in the kernel, not in our ring, and a conn that stays in the engine's hands pass after pass gets one pass's stall back per sweep, which a sweep interval outgrows. A stall in the iteration is excused once.
Why not "record the SEND's queue position per conn, like
recvArmSeq": it needs a field onconnStateand a store at four arming sites on the send path, and a missed site is a silent hole. Reading the ring at the sweep covers every arming site, including future ones, and puts nothing on the send path.What remains uncovered, stated exactly:
WriteTimeout, whichever is larger; a stall longer than that is a different problem.Detached conns (SSE/WebSocket, review MINOR-4): their write bound is
stallSince, notlastActivity. On a TCP socket the sweep already restarts that clock when the peer's acknowledged byte count moved (tcp_info), which a stall does not stop, so a healthy peer was in practice saved by that evidence; for a fixed-file or non-TCP descriptor there is no evidence and the clock is all there is, and there a stall cut the conn the same way.stallSincenow gets the same give-back (test below), so it no longer depends on the evidence. No remaining case, no follow-up issue needed.User-visible text:
config.go(the io_uringWriteTimeoutclock paragraph) now says a sweep that finds the connection's bytes still in the engine's own hands gives back the stall, and only that.Tests (the fix is commit 46eb88a; the tests that are red without it are in 37c85da, run RED on the previous head before the fix existed)
TestSendHeldInTheSQRingIsNotCutByWriteTimeout1173(5 subtests, one per SEND-arming site: flushSend, SEND_ZC, linked, one-free-slot, remainder re-send): real ring,mono.Manualclock, no sleeps. Stage a SEND, stall 2x WriteTimeout, sweep: not cut. A realSubmit(the kernel has it, the peer takes nothing): not cut at WriteTimeout/2 (kills "exempt without restamp"), cut after another WriteTimeout.TestStalledPeerIsStillCutByWriteTimeout1173: the positive control alone.TestStalledWorkerDoesNotCutAConnWithAStagedSend1173: end to end on a real HTTP/2 engine. A seam stalls the clock by 1.2 s in the iteration that stages the response (the seam runs before the drain stages it, soflushSendstamps the clock read before the stall and the sweep of the same iteration reads the clock after it); the client reads normally and a PING after the response must be answered. On main the close is deferred behind the SEND: the response arrives and the conn then ends (EOF). It goes on over streams until one has been drained with the stall injected.TestRemainderHeldInTheSQRingGetsBackOnlyTheStall1173(review MAJOR-1): a partial remainder in the ring across a stall of 0.5 WT gets back 0.5 WT, not a window. Red on 275399e.TestDirtyConnWithBytesTheKernelNeverHadIsNotCutByAStall1173(MINOR-2): SQ ring full, flushSend -> dirty, stall: not cut; a conn that stays dirty is still cut. Red on 275399e.TestDetachedConnWithASendInTheSQRingIsNotCutByAStall1173(MINOR-4). Red on 275399e.TestRestampIgnoresASendOfAnEarlierOwner1173(generation/ABA guard) andTestRestampFindsASendAcrossAWrappedRing1173(the 32-bit counters wrap; MINOR-3, adopted from the review's probes). They pass on 275399e and fail or pass on main as shown; they pin the guards.-race, in the Docker container below: on main (e449c2c) + the new test files: the 5 SQ-ring subtests, remainder, dirty, wrap, detached and the end-to-end all FAIL; the positive control and the ABA test pass. On 275399e + the new test files: remainder, dirty and detached FAIL (the rest pass). On the head: all 8 test functions pass,-count=20ok in 1.2 s (all tests together), each one under 0.05 s.-tags celeris_monoskew,CELERIS_MONO_SKEW=3600sand-3600s), the whole io_uring list ofci.yml(66 names, the new ones added): ok at both.-count=20failure of my own (the end-to-end test asserted that the seam fired once, which it does not when a response goes out inline) is fixed in 3ae3b81 by going on over streams until it has fired.Measured (docker arm64 golang:1.27.3, 2-CPU quota:
--cpus 2 --security-opt seccomp=unconfined --ulimit memlock=-1, linuxkit 7.0.12,-racefor tests)internal/engine/iouringpackage,-race, with the fix (3ae3b81): ok, 596 s (it was 594 s before this round). epoll is untouched by the diff (its jobs run in CI).TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring,-race, 3 busy loops, on the first version of the fix (the e2e path is unchanged by this round): main 0 cut of 102, head 0 of 102, and a throwaway counter (printed whenever the sweep restamped a SEND whose stamp was older than WriteTimeout) 0 events in 102 runs. The end-to-end test does not reproduce this cause at a measurable rate here; the evidence for the cause is the deterministic RED/GREEN above and the code. 0 of 102 bounds the rate under about 3 percent at 95 percent confidence.checkTimeoutsand the new function).BenchmarkCheckTimeoutsStaged1173(4096 conns, real ring, one sweep; main, head, and an A/A twin of main, interleaved, 6 rounds, this round's code): staged=0 medians 23.6 / 24.2 / 24.8 us (main / head / main2), staged=32 24.1 / 23.8 / 23.4, staged=256 24.2 / 23.6 / 23.2 us; head is inside the A/A spread, 0 allocs.BenchmarkRequestLoop884(4 rounds, first version; the send path and the loop are unchanged since) showed the head inside the main/main2 spread.GOOS=linux go vet, gofmt clean); CI runs lint on the head.Release note
io_uring: a worker stall longer than
WriteTimeoutin the loop iteration that stages a response no longer closes the connection of a peer that is reading normally (celeris#1173).