Skip to content

engines: an HTTP/2 pool handler's wait for the client's window is bounded as a minimum rate, not by an absolute deadline: stage 3 of the per-stream deadlines (v1.6.0, Refs #1170 #889) - #1180

Merged
FumingPower3925 merged 28 commits into
mainfrom
l19s3-pool-wait-rate
Oct 31, 2026

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Draft. Needs the user's sign-off on option B (see "The trade"). Lane L19 H2-TIMEOUTS, stage 3: an HTTP/2 pool handler blocked for the client's window is bounded by WriteTimeout as a minimum rate (D2's rule), not by an absolute deadline. Refs #1170 #889. Built on main d5c58c0 (stage 2, #1167, merged) per the design note design-1170-pool-wait-rate.md v4 and its reviews (Opus r2 on the design, r10 on this PR). Decisions of the reviews are implemented as given; where building them showed something else it is listed under "Findings".

What it does

On epoll, io_uring and adaptive, a pool handler whose response the connection's 4 MiB outbound budget could not buffer (sendRest) or whose StreamWriter.Write finds both windows shut waited under deadline = now + WriteTimeout from its first block (#906), whatever the client read. Go's HTTP/2 client (net/http's Transport with unencrypted HTTP/2, the bundled copy of x/net/http2's), 4 x 8 MiB on a 1 MB/s link, lost 3 of 4 at WriteTimeout 1 s and 5 s (the RST reaches the client at ~17 s behind the queued bytes): each stream has 4 MiB in flight ahead of the next, so the 2nd, 3rd and 4th stream's first grant comes 5.2, 9.4 and 13.6 s after their handler began to wait.

Now the wait is a record in the stall books (write_wait.go), judged by holdsLocked (floor min(64 KiB, held) per window, bank and seed, connection credit, share and idle grace, tail grace) by the handler's own timer at the end of each window. Credit = the bytes the handler sends. The sweep skips waiter records (verdicts and hint), so a record has one judge and a stream gets one RST_STREAM(INTERNAL_ERROR) (only the call that cancels the stream sends it: CancelFirst); the handler gets the same os.ErrDeadlineExceeded error as before, with a text that says why.

rule
record made by the first blocking call (the fast path reads no clock and takes no lock), held = pending = bytes still to send, seed = bytes in flight (4 floors at most) and no more than the bytes sent since the stream's previous record ended (review F2: a fresh seed for every Write let a client that granted each Write once in five windows hold a StreamWriter at a fifth of the floor for ever; D2's detached streams had the same hole, same fix)
who judges the waiter, on its own timer: the bound holds when the event loop cannot run (#906's property). A grant that arrives as the window ends is sent, not judged. A record that has gone while the stream lives is remade without a seed and never resets a cancelled stream
option B: the idle allowance a stream below its share on a connection that moves was held 3 windows; now ceil(in flight / what the connection released) windows, at least 3, at most 12 (writeIdleCapWindows, one named constant), the largest the run has seen (review r10 F3: a run that began in the tail branch had an allowance of 0 and was reset at once, where main held it; unit test red first). In flight = connHigh - connectionWindow (the highest the connection window has been, a CAS-max in the two places the processor raises it): no scan, no lock, nothing on the send path. All stalled streams get it (buffered tails too)
the horizon (review r10 F1) a connection counts as moving for four windows after its other streams last released 64 KiB (the bank's length); what carries is what the OTHER streams released, never the stream's own (a client that took its floor every fourth window would live for ever on its own). Go's client returns connection window in ~1 MiB steps, so some 1 s windows have neither a grant nor a send, and a queue of siblings' bytes can leave the connection without a release between the first grants and a stream's own. Such windows count as idle ones (the allowance bounds them: the bound is 17 windows in all, the horizon is inside it; pinned by TestHorizonWindowsAreIdleWindows1170, review r11 N1: without the count a decoy releasing a floor in one window out of four held an abandoned stream for 49 windows)
the wedge (a waiter holds a lock the loops' handlers wait for) the rule has no ceiling, so a client at the floor would hold such a lock for the whole download. While any event loop of the server has been in one batch of events (all the handlers it runs) for max(WriteTimeout/2, 250 ms), a waiter that has waited a WriteTimeout is reset at its next judgement: about 1.5 WriteTimeouts, as on main. The 250 ms floor matters only for WriteTimeout under 500 ms (at the 60 s default the threshold is 30 s). stream.LoopWatch: a slot per loop, Enter/Leave around each batch in the epoll Loop.run and the io_uring Worker.run and before each park (review r10 F2: a parked loop kept the stamp of its last batch and read as stuck for ever, until ResumeAccept; epoll 2 of 5 runs, traced), per engine, never a global. The server says so: one Warn line per episode ("the minimum rate of a pool handler's wait for window is suspended"), and the reason of each reset of a pool handler's wait at Debug level (rate or loop-blocked)
ServerOwes the sendWindowWaiters term is dropped for WriteTimeout > 0 (as A2 was for buffered tails); kept for WriteTimeout <= 0
WriteTimeout <= 0 no record, no timer, no fallback, as before

The bounds (unit tests; D2's table with one more column): a stream the client never grants is reset in window 1 (default window) / 5 (>= 256 KiB) alone, 4 / 8 beside a moving sibling with little in flight, and up to 13 / 17 beside a moving connection with a long queue in flight (B), 17 in all (the horizon windows idle inside the allowance). A pool handler's wait, a buffered tail and a one-byte dribbler are the same rule. The documented limit: parallel downloads are safe while N x W / L stays under about 16 x WriteTimeout (N streams, W the client's stream window, L the link); measured with Go's client (all shapes complete, epoll and adaptive; io_uring from 5 s): 4 x 8 MiB on 1 MB/s at 1 s, 2 s, 5 s and with the link idle 1.5 s before the downloads; 2 x 8 MiB on 500 KB/s at 1 s; 3 x 6 MiB on 1 MB/s at 1 s, 2 s, 5 s; 8 x 6 MiB and 8 x 8 MiB on 2 MB/s at 1 s; 4 x 8 MiB on 500 KB/s at 2 s. (Without the horizon the review measured, at WriteTimeout 1 s, 2 x 8 MiB @ 500 KB/s cut 1 of 2, 8 x 6 MiB @ 2 MB/s 2 of 8 and 4 x 8 MiB after an idle link 3 of 4; with a horizon of one window I measured 3 x 6 MiB @ 1 MB/s still cutting 2 of 3, which the four-window horizon fixed.) For slow links use 2 s or more, 5 s under 1 MB/s. A wedge is bounded by about 1.5 WriteTimeouts (measured 1.30 s at 1 s).

The trade (needs the user)

B stretches the standing decision "WINDOW_UPDATE(1) dribblers must not hold a stream": a stream that is paused or dribbling beside a connection that moves is held up to 17 windows (17 s at WriteTimeout 1 s, 17 minutes at the 60 s default; cap 12, per the review) where D2 held it 8. The horizon (which option A keeps too) cuts the connection-level price of holding streams by four: before, every held window needed the connection to release its floor; now other streams releasing one floor in one window out of four is enough, so the crumb clause (D2's accepted unbounded hold) also holds on a connection that moves below the floor as a whole (crumbs plus a quarter-floor decoy). The decoy has to be a non-stalled stream, for example a fresh GET of ~70 KB every four WriteTimeouts (creditConn counts every DATA reservation of the connection while any stream is stalled); a stalled decoy is itself reset. It pins no more memory, and the hold time is the same 17 windows. It does not widen what an attacker can pin (a connection whose other streams release a floor once every four windows already keeps all its stalled streams for ever under D2's crumb clause, memory per connection is 4 MiB + MCS x body either way, and in-flight is cheap to inflate: advertise a huge connection window and never return it, so the cap is what an attacker gets). It lengthens the clean-up of a truly abandoned stream. D2's own cell for this (TestH2PausedAndDribbledStreamsOnAHealthyConnAreBounded889) moves from [3.5 s, 7 s] to [3.5 s, 18 s]: measured 12.2 to 13.8 s. Without B (option A) the PR is the same minus the scaling (idleAllowance returns 3): at WriteTimeout 1 s the 2nd and 3rd waiters in the Go cell are cut again, at 5 s all four complete; the documented limit becomes (N-1) x W / L under 8 x WriteTimeout, and that cell goes back to [3.5 s, 7 s].

Findings (what building the decisions showed)

  1. The ServerOwes cell the review asked for is green on main. A client that grants 1 MiB every 2 s (or 4 s) with ReadTimeout 1 s, WriteTimeout 5 s completes on main: a connection with a pool handler in flight is shielded from the connection clocks by the engines' pool gate whatever ServerOwes says. So the term only affects the h2OwedAt bookkeeping: after this PR a silent client's connection closes up to ReadTimeout after the reset. Pinned by the unit test; the cell is a guard.
  2. io_uring at WriteTimeout 1 s is not in the Go-Transport cells: the engine's own send clock closes the connection at 17 to 24 s (every stream of the client ends in an unexpected EOF); on main three streams were cut by the absolute bound first and that clock closed the fourth; after this PR the send clock is the only thing that cuts at WriteTimeout 1 s. A client that stops reading for a WriteTimeout (a 1 s pause at 1 s) loses its connection to the engines' write clock on epoll as well, with or without this rule (pre-existing, review M5; documented with the 2 s guidance).
  3. The issue's second shape (16 streams of 512 KiB on 300 KB/s at 1 s) is not a waiting case with Go's Transport (every body fits its 4 MiB window; 16 of 16 complete on main). Its waiting equivalent is 8 x 8 MiB on 2 MB/s (cell).
  4. Leave before the poll cannot be told from correct end to end, but Leave before a park can (review r10 F2): the parked-loop cell (TestH2PoolWaiterAfterTheLoopsParked1170: PauseAccept, the h1 connections close, a Go client downloads 4 x 8 MiB) is red 2 of 3 runs on each engine without the fix (statistical: it depends on whether the loop's last batch had events), and TestParkedLoopsAreNotStuck1170 kills the mutant deterministically. Adaptive has no PauseAccept of its own; its sub-engines are these two.
  5. CI honesty. On three heads the coverage shard failed the threshold cell's "under" case (a blocking route holds a loop 0.2 s, threshold 0.5 s). I could not reproduce it locally with the CI coverage step's exact flags (-race -covermode=atomic -coverpkg=..., -count=3: 3 of 3 green, 2 loops and 4), and measured the premise: with those flags a handler that holds a loop 200 ms shows a batch age of about 220 ms (20 ms resolution), not 500 ms. The earlier explanation (cells starving each other) was unproven. The best remaining cause is review F1 (the rate rule, empty windows made likelier by a blocked loop), which the horizon addresses. The CI logs did not record which rule cut a stream; they do now (the cells print "resets by reason" and the episodes from the server's log, and assert the reason: "under" must see no loop-blocked reset and no episode, "over" must see both). The "under" hold is 0.1 s now. If it fails again its message names the rule.
  6. Stream stays 1024 bytes and Manager 416 bytes (the same malloc class as main): a first build had grown the Manager to 440 and the epoll no-stream sweep was +6 to 8% with it.
  7. A pool handler that ignores its error and sleeps keeps its connection open (the pool gate), as on main.
  8. Review M3 (the stamps and windows use the wall clock, as D2 does; an NTP step of 250 ms or more during a batch can trip the fallback once): left, to align with timeouts: the idle/read/write/header deadlines and the sweep stamp the wall clock, so repeated wall-clock steps cut connections (follow-up of #1176) #1178 when it lands.

Commits (tests first)

Tests red on main (b9b1247..dae6d04), the change (4ca25ca), units, docs and fixes (677bc1f..07246e9), the review-r10 round: red tests for F3 (tests: an idle run that begins in a tail ...), then the fixes (horizon, running max, cap 12, park Leave, logging, the reason-asserting cells), the docs, and TestParkedLoopsAreNotStuck1170.

Evidence (docker arm64 --cpus 2, one container at a time; amd64 unmeasured)

  • Red on main (the final cells, net/http client): the Go 4 x 8 MiB cell 5 of 5 (epoll and adaptive at 1 s and 5 s, io_uring at 5 s; 3 of 4 streams reset in each), the 8 x 8 MiB cell 2 of 2, the under-threshold loop cell 2 of 2, the parked-loop cell 2 of 2 (epoll, io_uring; the absolute bound cuts there); the "over" cells fail on main only because the log lines do not exist there. Correction of the earlier PR text: a waiter with a 1 MiB window that grants nothing and the one-byte dribbler are GREEN on main (reset at 5.05 to 5.38 s there too; the earlier "red 3 of 3" came from a cell whose connection window was shut); they are guards, with the default-window waiter, the cap cell, the pool StreamWriter dribbler, the bursty reader, WriteTimeout off, the silent client, and the wedge (8 trials per engine). Green on the head: every cell on epoll, io_uring and adaptive (io_uring from 5 s).
  • Shape table (Go's client, -cpu 2, after the horizon; epoll and adaptive, 0 cut in each): 2 x 8 MiB @ 500 KB/s WT 1 s; 3 x 6 MiB @ 1 MB/s WT 1 s and 2 s; 8 x 6 MiB @ 2 MB/s WT 1 s; 4 x 8 MiB @ 1 MB/s WT 1 s and 2 s, also with the link idle 1.5 s first; 8 x 8 MiB @ 2 MB/s WT 1 s; 4 x 8 MiB @ 500 KB/s WT 2 s; and at 5 s the same shapes on epoll, io_uring and adaptive. With a one-window horizon (cap 12): 3 x 6 MiB @ 1 MB/s at 1 s cut 2 of 3 on both engines; the rest were complete.
  • Unit: internal/protocol/h2/stream (231 s), internal/conn (89 s), internal/tcpack, internal/resource, internal/protocol/... -race -count=5 ok; internal/engine/epoll -race ok (401 s), internal/engine/iouring -race ok (507 s); the new and D2 cells -race -count=2 (serialised by WriteTimeout) and the whole test/server H2|889|Shutdown|SSE|Stream|1165|884|1170 set non-race: ok except D2's paused/dribbled cell, whose bound B moved (above; ok after the change). The threshold cell passes -race -covermode=atomic -coverpkg=github.com/goceleris/celeris -count=3 locally.
  • golangci-lint v2.14.0: 0 issues (also -tags netem889); GOOS=windows and darwin go vet ok; apidump ok (53 packages); size tests ok; root-shards.sh check ok (76 tests in shards: a 33, b 32, c 11: a shard of its own for the HTTP/2 pool handlers blocked in AwaitSendWindow are cut by an absolute WriteTimeout bound even when the client reads at a healthy rate #1170 cells — shards a and b had run at 79% of their -timeout=660s (523 s, 519 s, 497 s, review r11 N4); the workflows' shard: matrices, their margin comments and codecov.yml's after_n_builds (7 to 8) follow).
  • CI at 9831d4f: 31 pass, 2 skipping, none failing. Step times with the new shard c: Coverage shards a/b/c 6m14s / 6m50s / 6m44s, Unit shards a/b/c 6m47s / 7m06s / 6m31s (job totals; was 8m58s / 9m20s at 00ad43f for a and b), Lint 2m02s. The threshold cell passed on the coverage shard again (it had failed on three earlier heads; see Findings 5: one green run is not proof of a cause).

Mutants (each applied, the unit tests (or the named test) run, file restored)

mutant killed by
the 34 of this PR: the sweep judges waiters; the waiter in the hint; no credit; the waiters' bytes not in what the connection holds; the timer not re-armed after a hold; the record not ended; fallback removed; threshold 0; seed refreshed per record; cap removed; allowance re-derived each window / fixed by the first window (F3); the judge expires a missing record / a cancelled stream; ServerOwes term kept; reset without CancelFirst; a grant at the window end judged; no high mark from the conn WINDOW_UPDATE frame; a remade record seeded; a buffer joining a waiter not converted; pending not synced; no first-record seed cap; horizon removed / own release shelters / one window / unbounded (F1); no Leave at the park (F2); the episode told every time; horizon windows not counted as idle (r11 N1: 49 windows instead of 13); idleMax not reset for a new run unit, each by name (write_wait_1170_test.go, internal/conn/h2_wait_rate_1170_test.go, park_loopwatch_1170_linux_test.go); the timer one by timeout; Enter removed on epoll / io_uring by the wedge cell
rate := connRel (drop prevRel); the max(armedAt+wt, now+wt) clamp reverted survive, near-equivalent: the horizon never holds more than three empty windows and the minimum allowance is 3, so prevRel cannot change a verdict; the clamp has no path (review r11)
epoll Leave before the poll removed survives: not observable end to end (the loops wake every few ms even when idle); TestLoopWatch1170 pins the semantics
fallback conn-local not expressible: a Manager has no notion of its own loop's slot (the watch is the engine's)
D2's 52 mutants re-run against the new holdsLocked all killed (unit); n11 is now killed by TestWriteBankIsNotADebt889; two patterns adapted to the new code (p1, l7)

Hot path and cost (benchmarks interleaved with main, medians; minimums agree)

benchmark main head
sweep, epoll, no streams, 4096 conns 31.36 us 31.78 us +1.3%
sweep, epoll, 256 1428 ns 1479 ns +3.6%
sweep, io_uring, 4096 / 256 44.42 us / 2196 ns 44.78 us / 2236 ns +0.8% / +1.8%
conn-window reader, 1 waiter, 892's absolute deadline (reserve) vs the rate 496 ns 517 ns +4.3% (about 21 ns per wait: a record lookup, a credit per send)
the same, 8 waiters 1706 ns 1752 ns +2.7%
the same without WriteTimeout (no record, no timer) 256 / 1248 ns 261 / 1278 ns +2.1% / +2.4%
the loop stamp (two atomic stores per batch): epoll keep-alive dispatch, io_uring busy request loop; A/B/A/B x 4 rounds x 3 runs 21.8 us / 6.28 us 19.5 us / 5.96 us -10.4% / -5.0%: no cost resolvable; the noise of the machine is ±9% (main against itself, first half vs second half: 20.3 vs 22.1 us); minimums -4.5% / +1.1%

The send path is unchanged; nothing runs per send on a connection with no stalled stream.

Unverified

amd64 (every number is arm64); adaptive's io_uring side (its cells ran on epoll); the 1 s cells on a loaded CI runner (a stuck loop switches the rule off, to the old bound, and says so in the log); the user's acceptance of option B.

Release note (draft)

HTTP/2 on epoll, io_uring and adaptive: a handler on the worker pool that blocks for the client's flow-control window (a response over the connection's 4 MiB output budget, a StreamWriter Write) is bounded by WriteTimeout as a minimum rate, as a response buffered for the window already is, not by an absolute deadline from the start of its wait. A stream that receives at least 64 KiB per WriteTimeout window on average is never cut, however long it is; one that receives nothing is reset with RST_STREAM(INTERNAL_ERROR) after 1 to 5 windows (8 beside siblings that keep the connection moving), and later when the client has a long queue of bytes in flight on the connection: a stream beside a connection that moves is held for as many windows as the client needs to read what is in flight, at most 12 more (so up to 17 windows), and the connection counts as moving for four windows after its other streams last released 64 KiB. Parallel downloads on one connection are safe while (streams x client stream window / link rate) is under about 16 x WriteTimeout; for slow links use 2 s or more, 5 s under 1 MB/s. A handler that holds a lock across a response write still makes the requests that wait for the lock wait for the write; while any event loop of the server has been in one batch of events for half a WriteTimeout (at least 250 ms) every such wait that has lasted a WriteTimeout is bounded by WriteTimeout in all, as before, and the server logs it once per episode. With WriteTimeout: -1 nothing changes.

… not an absolute deadline (red on main, Refs #1170 #889)

Go's x/net/http2.Transport 4 x 8 MiB on a 1 MB/s link, 8 x 8 MiB on 2 MB/s, the
bound of a stalled waiter and of a one-byte dribbler, a pool StreamWriter whose
client grants each Write once in five windows, a client whose updates are further
apart than ReadTimeout, a silent client's connection, WriteTimeout off, the wedge
of a lock held across the write by a compliant reader, and the loop-blocked
threshold.
…elds a pool handler's connection), the silent-client cell allows the handler's sleep
… rule of every stalled stream (Refs #1170 #889)

sendRest and a pool StreamWriter's Write no longer wait under an absolute
WriteTimeout from the first block (#906). The wait is a record in the stall
books, credited with the bytes the handler sends and judged by holdsLocked at the
end of each window by the handler's own timer (the sweep skips it), so the bound
holds when the loop cannot run; exactly one RST_STREAM, INTERNAL_ERROR, and the
handler gets the same os.ErrDeadlineExceeded error.

Also: the idle allowance of a stream below its share on a moving connection is
ceil(in flight / the connection's release) windows, 3 to 16 (option B); the seed
counts only what was sent since the stream's last record (review r2 F2, D2's
detached streams too); a record that went missing is remade; the waiter term of
ServerOwes is dropped for WriteTimeout > 0; while an event loop is stuck in one
batch for half a WriteTimeout (at least 250 ms) a waiter that has waited a
WriteTimeout is bounded as before (LoopWatch, stamped by the epoll loop and the
io_uring worker); a reset is sent only by the call that cancelled the stream.
…s held by a waiter), a test for the books following the handler's count
…hy loop for the waiter in half of them), the Go cell has an idle loop beside it
…owance, the wedge fallback; the slow cells in the shards; one clock read less per wait (Refs #1170)
…encrypted HTTP/2 for the Go-client cells (the x/net one is deprecated), the under-threshold block is 200 ms (Refs #1170)
… the waiters' bytes summed where a window begins, the loop watch carried by the wait), a test deadlock (Refs #1170)
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 30, 2026
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…hold; the threshold pair and the 1 s Go cells run fewer at a time (Refs #1170)
…andby parks on a channel, not in epoll_wait, and its last stamp aged into a stuck loop that switched the rule off; CI) (Refs #1170)
…allowance is the largest the run saw (red; review r10 F3)
… other streams' release), the allowance of an idle run is the largest it saw, cap 12 (review r10 F1, F3; Refs #1170)
…) and tells an episode of a stuck loop once (Warn); the cells assert the reason; the parked-loop cell (review r10 F2, M1; Refs #1170)
… moved lately is moving through four empty windows), the F2 cell without adaptive (review r10 F1)
… the log lines, the pause); the threshold cell's under hold is 0.1 s (Refs #1170)
@codecov

codecov Bot commented Oct 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.85017% with 32 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/protocol/h2/stream/write_wait.go 86.66% 24 Missing ⚠️
internal/conn/response.go 81.25% 6 Missing ⚠️
internal/protocol/h2/stream/write_deadline.go 95.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

…iew r11 N1), each run has its own allowance; a shard of its own for the #1170 cells (N4); main merged
…room of the examples, the 17 in all (review r11 N2, N3, N5; Refs #1170)
@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 31, 2026 06:54
@FumingPower3925
FumingPower3925 merged commit 4ba79cb into main Oct 31, 2026
33 checks passed
@FumingPower3925
FumingPower3925 deleted the l19s3-pool-wait-rate branch October 31, 2026 06:54
FumingPower3925 added a commit that referenced this pull request Oct 31, 2026
…added (Refs #1178)

#1180 (stage 3 of the per-stream deadlines) added write_wait.go: the waiter
record's armedAt and start (beginWait), the judge's clock (judgeWait) and two
time.Unix(0, end) conversions for its timers, and two tests that write a wall
reading into armedAt. All step 5. The engines' LoopSlot.Enter(now) stamps take
the engines' own now, so they move with steps 2 and 3, and judgeWait compares
them with its own read: that edge is listed in the design note (R14).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant