Repository navigation
iouring: a worker with nothing to send answers a request already in the kernel before it exits (Refs #1166 #1208) - #1226
Merged
Conversation
…n the kernel (RED) (#1208) Refs #1166 #1208. io_uring closes these conns without an answer (a FIN) when its worker has no send pending and no parsed request in progress: it exits on its first look at the shutdown, and h1RequestInProgress does not look at the kernel. epoll and adaptive answer. The next commit fixes it.
…he kernel before it exits (#1208) Refs #1166 #1208. On its first look at the shutdown a worker with no send pending and no parsed request in progress exited at once, and h1RequestInProgress never looks at the kernel, so a request head in a conn's receive queue (or as a recv CQE not reaped) was reaped without being served: the conn was closed without an answer (epoll's h1InProgress counts those bytes and answers). The worker now asks the kernel once per iteration inside the 250 ms floor (h1UnreadInKernel: TIOCINQ on the sockets, then the unreaped CQEs, the look of the gate's first pass without its marks) and stays to serve what it finds; past the floor the gate's first pass marks such a request (gateRead) and h1RequestInProgress now counts a marked conn. Shutdown path only: the request loop is untouched.
…lf, the gateRead count, the floor (#1208) Refs #1166 #1208. Three mutants of the previous fix survived the end-to-end cells: h1RequestInProgress without the gateRead count, h1UnreadInKernel without the completion-queue look (postedRecvs: on a DEFER_TASKRUN kernel every rescue there comes from TIOCINQ, so only a base-tier ring or a ring that posts at once reaches it), and the floor bound. The exit decision becomes sendDrainDone, and unit tests (the gate rig, which arms its own recv, so independent of the ring's tier; the base-tier job runs this package) kill each. Notes in the comments: a head fragment or bare CRLF found in the kernel is a request in progress and holds the drain to its budget, as on epoll; a fixed-file conn marked by a CQE of a bare CRLF is never closed by the gate and does the same (dev-only setting).
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
FumingPower3925
marked this pull request as ready for review
November 2, 2026 20:24
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1166 #1208
Replaces #1224 (closed), which was stacked on #1220: #1220 is merged, and this is its 5 commits rebased onto main (
git rebase --onto origin/main 4429b9d; clean, no conflicts). The B-hold-other cell and its helpers are on main since #1220 as "pinning dropped for io_uring"; commit 1 flips them, so it is RED on main for io_uring (re-checked below on this base).The bug
On its first look at the cancelled context, an io_uring worker with no send pending and no parsed request in progress exits at once (
worker.go, the send-drain branch:if !w.hasPendingSends() && !w.h1RequestInProgress() { w.shutdown(); return }).h1RequestInProgressexplicitly does not look at the kernel (its comment: those requests are "held at the floor", which such a worker never reaches), so a request head sitting in a conn's receive queue, or as a recv CQE not reaped, is reaped byshutdown()without being served and the conn is closed without an answer. epoll'sh1InProgresscounts "bytes in the kernel's receive queue of a conn that holds nothing" as in progress, and answers them: this is #1104's own stated contract ("a request that reached the kernel before the gate is answered") and the gap that #1166 closed for a request already parsed. OnConnect is only one way to make a worker's iteration long enough for a head to land unread while it has nothing to send; the axis is "does the worker have anything to drain when it first sees the shutdown" (review r2 of #1220, MAJOR-3).RED / GREEN
Commit 1 (
B-hold-other, and theB-accept-holdwant for io_uring flipped to answered), docker arm64--cpus 2,-count=3:B-hold-other: 24 keep-alive conns the loops are already reading (each served once); one loop is held in theOnConnectof another, silent conn; all 24 send a GET or a 64 KiB POST; Shutdown begins (observed: the FIN of an idle conn of the free loop, never slept for); the loop is released. epoll answers all, withConnection: close; io_uring dropped the held loop's conns. adaptive starts on epoll, so its cell is epoll's (adaptive running on io_uring after a switch is the io_uring worker and is not exercised by a cell: static).The fix (commit 2, shutdown path only)
h1UnreadInKernel(new,shutdown_gate_884.go): the look of the gate's first pass (sweepUnreadAtGate: TIOCINQ on the sockets first, then the unreaped CQEs) without its marks. Skips what the gate skips: closing conns, ended detached streams, detached streams, HTTP/2, a fixed-file conn's socket (its CQE is still seen), and a dispatch goroutine's busy conn (already counted byh1RequestInProgress).holdPostedRecvsis split intoeachPostedRecv(walk) plus its marking callback, so the CQ walk is shared.!hasPendingSends && !h1RequestInProgress && (pastFloor || !h1UnreadInKernel()). The kernel look runs once per iteration only inside the 250 ms floor, while the worker serves as an ordinary iteration, so the look itself can never keep a worker past the floor (see the release note for what a request it finds can do): the floor,stopAccepting,refuseRecv,closeIdlePacedanddrainEndare untouched (the 250 ms semantics are unchanged; the sweep still marks at the floor).h1RequestInProgressnow counts agateReadconn (found by the floor's sweep) as in progress, so a worker that reaches the floor with only unread, marked heads waits for their answers instead of exiting between the mark and the read. Bounded as the drain is (drainEnd).sweepUnreadAtGateat drain start) was not taken as is, for its own two caveats: it setsCloseAfteron every HTTP/1 conn ~250 ms earlier than today, andgateReadstays set until the conn is released, so a marked conn whose partial head never completes would hold the worker todrainEnd. The non-marking look has neither.What was checked
test/server(plain and-race) andinternal/engine/iouring(plain;-racefor the broad filter) with-run 'Shutdown|Gate|Drain|Unread|Close|Sweep|Start|PauseAccept|_884|_885|_896|_1104|_1138|_1163|_1165|_1168|_1208': all pass (plain 171 s and 133 s class, race likewise). Thengo test -run 'Shutdown|Gate|Drain|Unread|Sweep' ./internal/engine/iouringonce more on the final head: ok, 85 s.BenchmarkRequestLoop884, base 4429b9d vs head, 6 alternating rounds, 1 s each, docker--cpus 2): the diff is in the shutdown branch only, and the numbers are noise (the four sub-benchmarks move -29%, -44%, -6%, +16% with no direction; the same binary varies that much between rounds). No hot-path code changed. Raw:scratchpad/b1208/out/bench.txt.internal/engine/iouringpackage under-race(as the base-tier job runs it): ok;test/servershutdown suites under-race: ok; the new unit tests-racex20: ok;_1208_cells plain x10 (360 PASS) and-racex20 with two busy loops (720 PASS), 0 FAIL, 0 SKIP.holdPostedRecvs' CQ walk (reached througheachPostedRecv) is the path that finds a request: the unit tests above exercise it deterministically, but CI's base-tier result is the condition of the review's approval. This PR targets main, so CI runs the base-tier job: the reviewer's condition.Behaviour change to know about (delta review)
A request head fragment (or a bare CRLF) that is unread in a conn's receive queue when the worker first looks at the shutdown used to be dropped at about 104 ms (the worker exited and closed the conn). It is now read on the look's say-so and becomes #1166's "request in progress", so it holds
Shutdownto its budget (drainEnd, capped byWriteTimeout), exactly as on epoll and as a fragment read before the shutdown already did. Probe, loop held inOnConnect, 1.5 s budget:"GET /ping HTTP/1.1\r\nHo": #1220 epoll 1.501 s, io_uring 104 ms; #1224 epoll 1.501 s, io_uring 1.500 s. Garbage (a TLS hello) is still closed at ~100 ms on both. A bare CRLF (RFC 9112 2.2 says a server SHOULD ignore it) holdsShutdownto the budget on both native engines, also before this PR: a separate, older matter, for its own issue. Godoc: bullet 3 of theShutdownlist says so. A fixed-file conn (development setting) marked by a CQE of a bare CRLF is never closed by the gate, whose TIOCINQ cannot ask its slot, and does the same.Tests added after the delta review (the three surviving mutants)
Mutants of commit 2 that the end-to-end cells did not kill, and the unit tests (
shutdown_unread_look_1208_linux_test.go, on the gate rig, which arms its own recv, so they do not depend on the ring's tier; the base-tier job runs this whole package, so they run there too) that kill each, checked by applying the mutant in a copy:h1RequestInProgresswithout thegateReadcountTestGateH1RequestInProgressCountsAMarkedConn1208h1UnreadInKernelwithout the completion-queue half (postedRecvs); on this kernel every end-to-end rescue comes from TIOCINQ, only a base-tier ring or one that posts at once reaches the CQ halfTestGateH1UnreadInKernel1208/a_recv_CQE_with_bytes,_unreaped,_the_socket_queue_empty_(the_CQ_half)(recv posted by the test itself, socket queue asserted empty)pastFloor := false)TestGateSendDrainDoneAsksTheKernelInsideTheFloorOnly1208(the exit decision is nowsendDrainDone(now), a method, so the floor is testable)Also pinned there: EOF CQE, another generation's CQE, HTTP/2 conn, closing conn, ended and running detached streams are not asked, and the look does not mark (
gateRead).Release note
io_uring: a request head that reached a connection's receive queue before
Shutdownwas first looked at is answered (withConnection: close) even when the worker has no response going out, as epoll does; before, it was closed without an answer. A head fragment or bare CRLF found there is now a request in progress and holdsShutdownuntil it completes or the budget ends, as on epoll (before, io_uring dropped it at once). Only the shutdown path changed.