Repository navigation
engines: an HTTP/2 pool handler's wait for the client's window is bounded as a minimum rate, not by an absolute deadline: stage 3 of the per-stream deadlines (v1.6.0, Refs #1170 #889) - #1180
Merged
Conversation
… not an absolute deadline (red on main, Refs #1170 #889) Go's x/net/http2.Transport 4 x 8 MiB on a 1 MB/s link, 8 x 8 MiB on 2 MB/s, the bound of a stalled waiter and of a one-byte dribbler, a pool StreamWriter whose client grants each Write once in five windows, a client whose updates are further apart than ReadTimeout, a silent client's connection, WriteTimeout off, the wedge of a lock held across the write by a compliant reader, and the loop-blocked threshold.
…t (a decision of the r2 review)
…elds a pool handler's connection), the silent-client cell allows the handler's sleep
… the stream's window is what limits them
… rule of every stalled stream (Refs #1170 #889) sendRest and a pool StreamWriter's Write no longer wait under an absolute WriteTimeout from the first block (#906). The wait is a record in the stall books, credited with the bytes the handler sends and judged by holdsLocked at the end of each window by the handler's own timer (the sweep skips it), so the bound holds when the loop cannot run; exactly one RST_STREAM, INTERNAL_ERROR, and the handler gets the same os.ErrDeadlineExceeded error. Also: the idle allowance of a stream below its share on a moving connection is ceil(in flight / the connection's release) windows, 3 to 16 (option B); the seed counts only what was sent since the stream's last record (review r2 F2, D2's detached streams too); a record that went missing is remade; the waiter term of ServerOwes is dropped for WriteTimeout > 0; while an event loop is stuck in one batch for half a WriteTimeout (at least 250 ms) a waiter that has waited a WriteTimeout is bounded as before (LoopWatch, stamped by the epoll loop and the io_uring worker); a reset is sent only by the call that cancelled the stream.
…ack and the single reset (Refs #1170)
…s held by a waiter), a test for the books following the handler's count
…hy loop for the waiter in half of them), the Go cell has an idle loop beside it
…owance, the wedge fallback; the slow cells in the shards; one clock read less per wait (Refs #1170)
…encrypted HTTP/2 for the Go-client cells (the x/net one is deprecated), the under-threshold block is 200 ms (Refs #1170)
… the waiters' bytes summed where a window begins, the loop watch carried by the wait), a test deadlock (Refs #1170)
…ce starve each other's loops under -race)
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…hold; the threshold pair and the 1 s Go cells run fewer at a time (Refs #1170)
…andby parks on a channel, not in epoll_wait, and its last stamp aged into a stuck loop that switched the rule off; CI) (Refs #1170)
…allowance is the largest the run saw (red; review r10 F3)
… other streams' release), the allowance of an idle run is the largest it saw, cap 12 (review r10 F1, F3; Refs #1170)
…) and tells an episode of a stuck loop once (Warn); the cells assert the reason; the parked-loop cell (review r10 F2, M1; Refs #1170)
… moved lately is moving through four empty windows), the F2 cell without adaptive (review r10 F1)
… the log lines, the pause); the threshold cell's under hold is 0.1 s (Refs #1170)
…asured 12 to 14 s, was 4 s); lint (Refs #1170)
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
FumingPower3925
marked this pull request as ready for review
October 31, 2026 06:54
FumingPower3925
added a commit
that referenced
this pull request
Oct 31, 2026
…added (Refs #1178) #1180 (stage 3 of the per-stream deadlines) added write_wait.go: the waiter record's armedAt and start (beginWait), the judge's clock (judgeWait) and two time.Unix(0, end) conversions for its timers, and two tests that write a wall reading into armedAt. All step 5. The engines' LoopSlot.Enter(now) stamps take the engines' own now, so they move with steps 2 and 3, and judgeWait compares them with its own read: that edge is listed in the design note (R14).
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. Needs the user's sign-off on option B (see "The trade"). Lane L19 H2-TIMEOUTS, stage 3: an HTTP/2 pool handler blocked for the client's window is bounded by
WriteTimeoutas a minimum rate (D2's rule), not by an absolute deadline. Refs #1170 #889. Built on main d5c58c0 (stage 2, #1167, merged) per the design notedesign-1170-pool-wait-rate.mdv4 and its reviews (Opus r2 on the design, r10 on this PR). Decisions of the reviews are implemented as given; where building them showed something else it is listed under "Findings".What it does
On epoll, io_uring and adaptive, a pool handler whose response the connection's 4 MiB outbound budget could not buffer (
sendRest) or whoseStreamWriter.Writefinds both windows shut waited underdeadline = now + WriteTimeoutfrom its first block (#906), whatever the client read. Go's HTTP/2 client (net/http's Transport with unencrypted HTTP/2, the bundled copy of x/net/http2's), 4 x 8 MiB on a 1 MB/s link, lost 3 of 4 at WriteTimeout 1 s and 5 s (the RST reaches the client at ~17 s behind the queued bytes): each stream has 4 MiB in flight ahead of the next, so the 2nd, 3rd and 4th stream's first grant comes 5.2, 9.4 and 13.6 s after their handler began to wait.Now the wait is a record in the stall books (
write_wait.go), judged byholdsLocked(floor min(64 KiB, held) per window, bank and seed, connection credit, share and idle grace, tail grace) by the handler's own timer at the end of each window. Credit = the bytes the handler sends. The sweep skips waiter records (verdicts and hint), so a record has one judge and a stream gets one RST_STREAM(INTERNAL_ERROR) (only the call that cancels the stream sends it:CancelFirst); the handler gets the sameos.ErrDeadlineExceedederror as before, with a text that says why.held = pending = bytes still to send, seed = bytes in flight (4 floors at most) and no more than the bytes sent since the stream's previous record ended (review F2: a fresh seed for every Write let a client that granted each Write once in five windows hold a StreamWriter at a fifth of the floor for ever; D2's detached streams had the same hole, same fix)ceil(in flight / what the connection released)windows, at least 3, at most 12 (writeIdleCapWindows, one named constant), the largest the run has seen (review r10 F3: a run that began in the tail branch had an allowance of 0 and was reset at once, where main held it; unit test red first). In flight =connHigh - connectionWindow(the highest the connection window has been, a CAS-max in the two places the processor raises it): no scan, no lock, nothing on the send path. All stalled streams get it (buffered tails too)TestHorizonWindowsAreIdleWindows1170, review r11 N1: without the count a decoy releasing a floor in one window out of four held an abandoned stream for 49 windows)stream.LoopWatch: a slot per loop,Enter/Leavearound each batch in the epollLoop.runand the io_uringWorker.runand before each park (review r10 F2: a parked loop kept the stamp of its last batch and read as stuck for ever, until ResumeAccept; epoll 2 of 5 runs, traced), per engine, never a global. The server says so: one Warn line per episode ("the minimum rate of a pool handler's wait for window is suspended"), and the reason of each reset of a pool handler's wait at Debug level (rateorloop-blocked)sendWindowWaitersterm is dropped for WriteTimeout > 0 (as A2 was for buffered tails); kept for WriteTimeout <= 0The bounds (unit tests; D2's table with one more column): a stream the client never grants is reset in window 1 (default window) / 5 (>= 256 KiB) alone, 4 / 8 beside a moving sibling with little in flight, and up to 13 / 17 beside a moving connection with a long queue in flight (B), 17 in all (the horizon windows idle inside the allowance). A pool handler's wait, a buffered tail and a one-byte dribbler are the same rule. The documented limit: parallel downloads are safe while N x W / L stays under about 16 x WriteTimeout (N streams, W the client's stream window, L the link); measured with Go's client (all shapes complete, epoll and adaptive; io_uring from 5 s): 4 x 8 MiB on 1 MB/s at 1 s, 2 s, 5 s and with the link idle 1.5 s before the downloads; 2 x 8 MiB on 500 KB/s at 1 s; 3 x 6 MiB on 1 MB/s at 1 s, 2 s, 5 s; 8 x 6 MiB and 8 x 8 MiB on 2 MB/s at 1 s; 4 x 8 MiB on 500 KB/s at 2 s. (Without the horizon the review measured, at WriteTimeout 1 s, 2 x 8 MiB @ 500 KB/s cut 1 of 2, 8 x 6 MiB @ 2 MB/s 2 of 8 and 4 x 8 MiB after an idle link 3 of 4; with a horizon of one window I measured 3 x 6 MiB @ 1 MB/s still cutting 2 of 3, which the four-window horizon fixed.) For slow links use 2 s or more, 5 s under 1 MB/s. A wedge is bounded by about 1.5 WriteTimeouts (measured 1.30 s at 1 s).
The trade (needs the user)
B stretches the standing decision "WINDOW_UPDATE(1) dribblers must not hold a stream": a stream that is paused or dribbling beside a connection that moves is held up to 17 windows (17 s at WriteTimeout 1 s, 17 minutes at the 60 s default; cap 12, per the review) where D2 held it 8. The horizon (which option A keeps too) cuts the connection-level price of holding streams by four: before, every held window needed the connection to release its floor; now other streams releasing one floor in one window out of four is enough, so the crumb clause (D2's accepted unbounded hold) also holds on a connection that moves below the floor as a whole (crumbs plus a quarter-floor decoy). The decoy has to be a non-stalled stream, for example a fresh GET of ~70 KB every four WriteTimeouts (
creditConncounts every DATA reservation of the connection while any stream is stalled); a stalled decoy is itself reset. It pins no more memory, and the hold time is the same 17 windows. It does not widen what an attacker can pin (a connection whose other streams release a floor once every four windows already keeps all its stalled streams for ever under D2's crumb clause, memory per connection is 4 MiB + MCS x body either way, and in-flight is cheap to inflate: advertise a huge connection window and never return it, so the cap is what an attacker gets). It lengthens the clean-up of a truly abandoned stream. D2's own cell for this (TestH2PausedAndDribbledStreamsOnAHealthyConnAreBounded889) moves from [3.5 s, 7 s] to [3.5 s, 18 s]: measured 12.2 to 13.8 s. Without B (option A) the PR is the same minus the scaling (idleAllowancereturns 3): at WriteTimeout 1 s the 2nd and 3rd waiters in the Go cell are cut again, at 5 s all four complete; the documented limit becomes (N-1) x W / L under 8 x WriteTimeout, and that cell goes back to [3.5 s, 7 s].Findings (what building the decisions showed)
ServerOwessays. So the term only affects theh2OwedAtbookkeeping: after this PR a silent client's connection closes up to ReadTimeout after the reset. Pinned by the unit test; the cell is a guard.Leavebefore the poll cannot be told from correct end to end, butLeavebefore a park can (review r10 F2): the parked-loop cell (TestH2PoolWaiterAfterTheLoopsParked1170: PauseAccept, the h1 connections close, a Go client downloads 4 x 8 MiB) is red 2 of 3 runs on each engine without the fix (statistical: it depends on whether the loop's last batch had events), andTestParkedLoopsAreNotStuck1170kills the mutant deterministically. Adaptive has no PauseAccept of its own; its sub-engines are these two.-race -covermode=atomic -coverpkg=...,-count=3: 3 of 3 green, 2 loops and 4), and measured the premise: with those flags a handler that holds a loop 200 ms shows a batch age of about 220 ms (20 ms resolution), not 500 ms. The earlier explanation (cells starving each other) was unproven. The best remaining cause is review F1 (the rate rule, empty windows made likelier by a blocked loop), which the horizon addresses. The CI logs did not record which rule cut a stream; they do now (the cells print "resets by reason" and the episodes from the server's log, and assert the reason: "under" must see noloop-blockedreset and no episode, "over" must see both). The "under" hold is 0.1 s now. If it fails again its message names the rule.Streamstays 1024 bytes andManager416 bytes (the same malloc class as main): a first build had grown the Manager to 440 and the epoll no-stream sweep was +6 to 8% with it.Commits (tests first)
Tests red on main (
b9b1247..dae6d04), the change (4ca25ca), units, docs and fixes (677bc1f..07246e9), the review-r10 round: red tests for F3 (tests: an idle run that begins in a tail ...), then the fixes (horizon, running max, cap 12, parkLeave, logging, the reason-asserting cells), the docs, andTestParkedLoopsAreNotStuck1170.Evidence (docker arm64
--cpus 2, one container at a time; amd64 unmeasured)-cpu 2, after the horizon; epoll and adaptive, 0 cut in each): 2 x 8 MiB @ 500 KB/s WT 1 s; 3 x 6 MiB @ 1 MB/s WT 1 s and 2 s; 8 x 6 MiB @ 2 MB/s WT 1 s; 4 x 8 MiB @ 1 MB/s WT 1 s and 2 s, also with the link idle 1.5 s first; 8 x 8 MiB @ 2 MB/s WT 1 s; 4 x 8 MiB @ 500 KB/s WT 2 s; and at 5 s the same shapes on epoll, io_uring and adaptive. With a one-window horizon (cap 12): 3 x 6 MiB @ 1 MB/s at 1 s cut 2 of 3 on both engines; the rest were complete.internal/protocol/h2/stream(231 s),internal/conn(89 s),internal/tcpack,internal/resource,internal/protocol/...-race -count=5ok;internal/engine/epoll-raceok (401 s),internal/engine/iouring-raceok (507 s); the new and D2 cells-race -count=2(serialised by WriteTimeout) and the wholetest/serverH2|889|Shutdown|SSE|Stream|1165|884|1170set non-race: ok except D2's paused/dribbled cell, whose bound B moved (above; ok after the change). The threshold cell passes-race -covermode=atomic -coverpkg=github.com/goceleris/celeris-count=3locally.-tags netem889);GOOS=windowsanddarwingo vetok;apidumpok (53 packages); size tests ok;root-shards.sh checkok (76 tests in shards: a 33, b 32, c 11: a shard of its own for the HTTP/2 pool handlers blocked in AwaitSendWindow are cut by an absolute WriteTimeout bound even when the client reads at a healthy rate #1170 cells — shards a and b had run at 79% of their-timeout=660s(523 s, 519 s, 497 s, review r11 N4); the workflows'shard:matrices, their margin comments andcodecov.yml'safter_n_builds(7 to 8) follow).9831d4f: 31 pass, 2 skipping, none failing. Step times with the new shard c: Coverage shards a/b/c 6m14s / 6m50s / 6m44s, Unit shards a/b/c 6m47s / 7m06s / 6m31s (job totals; was 8m58s / 9m20s at00ad43ffor a and b), Lint 2m02s. The threshold cell passed on the coverage shard again (it had failed on three earlier heads; see Findings 5: one green run is not proof of a cause).Mutants (each applied, the unit tests (or the named test) run, file restored)
CancelFirst; a grant at the window end judged; no high mark from the conn WINDOW_UPDATE frame; a remade record seeded; a buffer joining a waiter not converted; pending not synced; no first-record seed cap; horizon removed / own release shelters / one window / unbounded (F1); noLeaveat the park (F2); the episode told every time; horizon windows not counted as idle (r11 N1: 49 windows instead of 13);idleMaxnot reset for a new runwrite_wait_1170_test.go,internal/conn/h2_wait_rate_1170_test.go,park_loopwatch_1170_linux_test.go); the timer one by timeout;Enterremoved on epoll / io_uring by the wedge cellrate := connRel(dropprevRel); themax(armedAt+wt, now+wt)clamp revertedprevRelcannot change a verdict; the clamp has no path (review r11)Leavebefore the poll removedTestLoopWatch1170pins the semanticsholdsLockedn11is now killed byTestWriteBankIsNotADebt889; two patterns adapted to the new code (p1,l7)Hot path and cost (benchmarks interleaved with main, medians; minimums agree)
The send path is unchanged; nothing runs per send on a connection with no stalled stream.
Unverified
amd64 (every number is arm64); adaptive's io_uring side (its cells ran on epoll); the 1 s cells on a loaded CI runner (a stuck loop switches the rule off, to the old bound, and says so in the log); the user's acceptance of option B.
Release note (draft)