Skip to content

iouring: a SEND still in our own SQ ring when the sweep comes gets a fresh write clock (Refs #1173) - #1196

Merged
FumingPower3925 merged 8 commits into
mainfrom
iouring-unsubmitted-send-1173
Nov 1, 2026
Merged

FumingPower3925 merged 8 commits into
mainfrom
iouring-unsubmitted-send-1173

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1173

Follow-up of #1175, which fixed one cause of the CI cut and said one stays open. This is that one: a worker stall in the loop iteration that stages a response, between the read of the iteration's clock and that iteration's timeout sweep.

What a user could notice

On the io_uring engine with a short WriteTimeout, a conn can be closed (EOF at the client) although the peer is reading normally, when the worker stalls for longer than WriteTimeout in the iteration that stages the response, before its timeout sweep. A stall like that is a handler of another conn running inline in the same batch, a GC assist, -race, or a CFS quota throttling a CPU-limited container. epoll is not affected (it writes inline; it has no staged-but-not-submitted state).

Root cause (read from the code, confirmed by deterministic tests)

flushSend stages the SEND SQE and stamps the conn's write clock (lastActivity) with w.cachedNow, the clock the iteration read when it started; the SQE is submitted to the kernel at the top of the NEXT iteration. The sweep (checkTimeouts) runs later in the SAME iteration, reads a fresh clock, and counts cs.sending as "a SEND in flight": elapsed > WriteTimeout closes the conn. While the SEND sits in our own SQ ring the peer has been handed nothing, so a stall of ours (anywhere between the iteration's clock read and the sweep, handlers of other conns earlier in the batch included) is charged to the peer. Sites that arm a SEND: flushSend (fresh and remainder arms), flushSendLink (linked and the one-free-slot fallback); the partial-send re-send, flushDirty and the h2 deadline sweep arm through flushSend.

Fix

restampUnsubmittedSends(passClock, now) at the start of the sweep gives back to a conn the time the ENGINE held its bytes in this pass, now - passClock (passClock is w.cachedNow as the sweep found it, before it sets it). The conn's clocks move by that, never to now:

  • a fresh SEND was stamped with passClock, so it ends at now, as if the stall had not happened;
  • a partial send's remainder keeps the existing io_uring rule (WriteTimeout runs from the first SEND of the response; a partial completion does not restart it) and is excused this pass's stall only. (The first version of this PR restamped to now, which handed a remainder a whole new window whenever a sweep and a partial completion fell in the same pass, so WriteTimeout of a large response to a slow reader would have depended on that coincidence, against the measured statements in config.go. Review r1 found it; fixed here with a failing-first test.)

Whose bytes the kernel has not got:

  • a SEND among the SQEs the kernel has not consumed ([sqHead, sqTail) of the SQ ring), whose conn is still the one that placed it (generation) and still has it in flight: lastActivity, and stallSince for a detached conn (WebSocket/SSE);
  • a conn on the dirty list with bytes and no SEND (the SQ ring was full when they were staged).

A stalled peer is still cut: its SEND is in the kernel, not in our ring, and a conn that stays in the engine's hands pass after pass gets one pass's stall back per sweep, which a sweep interval outgrows. A stall in the iteration is excused once.

Why not "record the SEND's queue position per conn, like recvArmSeq": it needs a field on connState and a store at four arming sites on the send path, and a missed site is a silent hole. Reading the ring at the sweep covers every arming site, including future ones, and puts nothing on the send path.

What remains uncovered, stated exactly:

  • a stall between this iteration's sweep and the next iteration's submit: no handler runs in that window (park/shutdown/listen bookkeeping only), and a peer that is reading takes the SEND at that submit and its completion is reaped before the next sweep; it can only matter for a SEND that then blocks in the kernel, which is the peer's;
  • a conn that is already closing: its drain bound is 5 s or WriteTimeout, whichever is larger; a stall longer than that is a different problem.

Detached conns (SSE/WebSocket, review MINOR-4): their write bound is stallSince, not lastActivity. On a TCP socket the sweep already restarts that clock when the peer's acknowledged byte count moved (tcp_info), which a stall does not stop, so a healthy peer was in practice saved by that evidence; for a fixed-file or non-TCP descriptor there is no evidence and the clock is all there is, and there a stall cut the conn the same way. stallSince now gets the same give-back (test below), so it no longer depends on the evidence. No remaining case, no follow-up issue needed.

User-visible text: config.go (the io_uring WriteTimeout clock paragraph) now says a sweep that finds the connection's bytes still in the engine's own hands gives back the stall, and only that.

Tests (the fix is commit 46eb88a; the tests that are red without it are in 37c85da, run RED on the previous head before the fix existed)

  • TestSendHeldInTheSQRingIsNotCutByWriteTimeout1173 (5 subtests, one per SEND-arming site: flushSend, SEND_ZC, linked, one-free-slot, remainder re-send): real ring, mono.Manual clock, no sleeps. Stage a SEND, stall 2x WriteTimeout, sweep: not cut. A real Submit (the kernel has it, the peer takes nothing): not cut at WriteTimeout/2 (kills "exempt without restamp"), cut after another WriteTimeout.
  • TestStalledPeerIsStillCutByWriteTimeout1173: the positive control alone.
  • TestStalledWorkerDoesNotCutAConnWithAStagedSend1173: end to end on a real HTTP/2 engine. A seam stalls the clock by 1.2 s in the iteration that stages the response (the seam runs before the drain stages it, so flushSend stamps the clock read before the stall and the sweep of the same iteration reads the clock after it); the client reads normally and a PING after the response must be answered. On main the close is deferred behind the SEND: the response arrives and the conn then ends (EOF). It goes on over streams until one has been drained with the stall injected.
  • TestRemainderHeldInTheSQRingGetsBackOnlyTheStall1173 (review MAJOR-1): a partial remainder in the ring across a stall of 0.5 WT gets back 0.5 WT, not a window. Red on 275399e.
  • TestDirtyConnWithBytesTheKernelNeverHadIsNotCutByAStall1173 (MINOR-2): SQ ring full, flushSend -> dirty, stall: not cut; a conn that stays dirty is still cut. Red on 275399e.
  • TestDetachedConnWithASendInTheSQRingIsNotCutByAStall1173 (MINOR-4). Red on 275399e.
  • TestRestampIgnoresASendOfAnEarlierOwner1173 (generation/ABA guard) and TestRestampFindsASendAcrossAWrappedRing1173 (the 32-bit counters wrap; MINOR-3, adopted from the review's probes). They pass on 275399e and fail or pass on main as shown; they pin the guards.
  • Results, -race, in the Docker container below: on main (e449c2c) + the new test files: the 5 SQ-ring subtests, remainder, dirty, wrap, detached and the end-to-end all FAIL; the positive control and the ABA test pass. On 275399e + the new test files: remainder, dirty and detached FAIL (the rest pass). On the head: all 8 test functions pass, -count=20 ok in 1.2 s (all tests together), each one under 0.05 s.
  • Skew canary (-tags celeris_monoskew, CELERIS_MONO_SKEW=3600s and -3600s), the whole io_uring list of ci.yml (66 names, the new ones added): ok at both.
  • The earlier -count=20 failure of my own (the end-to-end test asserted that the seam fired once, which it does not when a response goes out inline) is fixed in 3ae3b81 by going on over streams until it has fired.

Measured (docker arm64 golang:1.27.3, 2-CPU quota: --cpus 2 --security-opt seccomp=unconfined --ulimit memlock=-1, linuxkit 7.0.12, -race for tests)

  • Whole internal/engine/iouring package, -race, with the fix (3ae3b81): ok, 596 s (it was 594 s before this round). epoll is untouched by the diff (its jobs run in CI).
  • Stress, TestQuietH2StreamIsNotCutByWriteTimeout889/io_uring, -race, 3 busy loops, on the first version of the fix (the e2e path is unchanged by this round): main 0 cut of 102, head 0 of 102, and a throwaway counter (printed whenever the sweep restamped a SEND whose stamp was older than WriteTimeout) 0 events in 102 runs. The end-to-end test does not reproduce this cause at a measurable rate here; the evidence for the cause is the deterministic RED/GREEN above and the code. 0 of 102 bounds the rate under about 3 percent at 95 percent confidence.
  • Hot path: the send path is untouched (diff: one call in checkTimeouts and the new function). BenchmarkCheckTimeoutsStaged1173 (4096 conns, real ring, one sweep; main, head, and an A/A twin of main, interleaved, 6 rounds, this round's code): staged=0 medians 23.6 / 24.2 / 24.8 us (main / head / main2), staged=32 24.1 / 23.8 / 23.4, staged=256 24.2 / 23.6 / 23.2 us; head is inside the A/A spread, 0 allocs. BenchmarkRequestLoop884 (4 rounds, first version; the send path and the loop are unchanged since) showed the head inside the main/main2 spread.
  • golangci-lint v2.14: 0 issues (first version; this round's code was vetted GOOS=linux go vet, gofmt clean); CI runs lint on the head.

Release note

io_uring: a worker stall longer than WriteTimeout in the loop iteration that stages a response no longer closes the connection of a peer that is reading normally (celeris#1173).

…t the peer's to be cut for (RED, Refs #1173)

A staged SEND is submitted at the top of the next loop iteration; the sweep
runs later in the same one and counts it as in flight, measuring WriteTimeout
from the stamp taken at staging. A worker stall longer than WriteTimeout in
between closes a conn whose peer was never handed a byte.

Unit tests (mono.Manual clock, real ring) over the five SEND arming sites,
with the positive control (a SEND the kernel holds, to a peer that takes
nothing, is still cut at WriteTimeout), and an end-to-end HTTP/2 test with a
stall injected between the drain and the sweep. Red on main.
…fresh write clock (Refs #1173)

checkTimeouts counted a staged, unsubmitted SEND as in flight, so a worker
stall longer than WriteTimeout between staging and the same iteration's sweep
cut a conn whose peer had been handed nothing. The sweep now reads the SQEs the
kernel has not consumed ([head, tail) of the SQ ring) and restamps the write
clock of the conn of each SEND among them. Nothing on the send path; one scan
of the unconsumed SQEs per sweep.
@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/engine/iouring/worker.go 96.15% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

… remainder only the stall, a dirty conn, a detached conn; the generation and wrap guards (RED, Refs #1173)
… its bytes: a remainder, a dirty conn and a detached conn included (Refs #1173)

Review r1 of #1196. The restamp moved the write clock of a SEND found in the
SQ ring to the sweep's time, so a partial send's remainder got a whole new
window whenever a sweep and a partial completion fell in the same pass. The
clocks now move by now - passClock (the clock the pass read when it started,
which is what flushSend stamps): a fresh SEND still ends at now, a remainder is
excused the pass's stall only. The same excuse covers a conn on the dirty list
with bytes and no SEND (SQ ring full) and the stallSince clock of a detached
conn. config.go says so; the test comments say the stall is in the iteration
that stages the response, not after the staging.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 01:10
@FumingPower3925
FumingPower3925 merged commit fce3cc5 into main Nov 1, 2026
33 of 34 checks passed
@FumingPower3925
FumingPower3925 deleted the iouring-unsubmitted-send-1173 branch November 1, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant