Skip to content

engines: HTTP/2 WriteTimeout bounds a response that waits for the client's window, as a minimum rate: stage 2 of the per-stream deadlines (v1.6.0, Refs #889) - #1167

Merged
FumingPower3925 merged 34 commits into
mainfrom
l19d-d2-write-deadline
Oct 30, 2026
Merged

FumingPower3925 merged 34 commits into
mainfrom
l19d-d2-write-deadline

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Lane L19 H2-TIMEOUTS, stage 2 of the per-stream deadlines: D2, the write deadline of an HTTP/2 stream on epoll, io_uring and adaptive (std unchanged). Refs #889. Built on the merged stage 1 (#1150) per design v3.1 (lane evidence: design-per-stream-deadlines.md). Draft.

What it does

A response whose bytes wait for the client's flow-control window (the tail of an inline handler's body, or a pool handler's, buffered in the stream; Manager.outboundHeld) had no bound at all (#1134 Finding 1): a client that granted no window and kept pinging held the stream, and its buffer, for ever (measured: 6 s, 60 s, no reset). Now WriteTimeout bounds it as a minimum rate, #1119's rule at the h2 layer, refined by review r7 so that it does not cut a reader that is slow but moving (final rule and its bounds: design v3.5):

rule
window a stream is stalled from the moment its buffer goes from empty to non-empty; its window is WriteTimeout long
floor at the end of a window the stream must have released (DATA the windows let out, FlushOutbound) at least F = min(64 KiB, what it held when the window began); if so the next window begins from that instant; if not it is reset with RST_STREAM(INTERNAL_ERROR) (net/http's code at a write deadline, and #906's). 64 KiB per window is #1119's parkedMinAcked: about 64 KiB/s at a 1 s WriteTimeout, about 1 KiB/s at the 60 s default
carry-over (r7 F1) what a stream released beyond F carries to the next windows, up to 4 floors (256 KiB), and a stream's first window is covered by the bytes it had in flight when it stalled (the peer's initial window less what is left of it), capped the same way. A client that sends few WINDOW_UPDATEs (nghttp2/curl: at half the window) releases in big steps with empty windows between: it survives while it releases on average F per window and its updates are no further than about five windows apart
no ceiling a stream that keeps that average is never cut, however long the response (std's deadline is absolute)
bytes, not frames a PING, a SETTINGS, a WINDOW_UPDATE that lets no byte out, a WINDOW_UPDATE of one byte are not progress (a dribbler on a 64 KiB window is reset within two windows: the seed covers the first)
the connection's rate (r7 F6) when the connection as a whole released at least its floor in the window, a stream below F is held if it took half its fair share (F / 2N, N the streams stalled), and for 3 consecutive windows if it took less (paused, dribbled, or queued behind its siblings' bytes in the same socket); the fourth is the verdict. When the connection itself released less than its floor, the first window is the verdict, as for a single stream. The tail (the end of a response) gets a grace inside the same bound (r8 F-A, r9 F-A2): a stream that holds under one floor and had half of it released, and any stream while the whole connection holds under one floor (the last bytes are on the wire; the backlog behind them can take more than a window to drain), is held for 3 windows without proof and reset in the 4th. A window held on the bank alone pays a whole floor, however small the rest (r9 F-H). The true bounds of a stream the client never reads: reset in window 1+k alone, 4+k with a sibling that keeps the connection above the floor, k = min(4, whole 64 KiB in the bytes in flight when it stalled): 1 and 4 WriteTimeouts for the default 64 KiB window, 5 and 8 for a window of 256 KiB or more; with a rest under 64 KiB (a response a little longer than the window): 5 (default window), 6 (128 KiB), 8 (>= 256 KiB) (pinned in unit tests). A 1-byte WINDOW_UPDATE dribbler: window max(2, 1+k): 2 (default window), 3 (128 KiB), 4 (192 KiB), 5 (>= 256 KiB). Parallel downloads on one connection are fine while the bytes ahead of a stream in the client's socket (N x W / link) take under 3 x WriteTimeout; at WriteTimeout 1 s on a slow link they are reset where main completes them: set WriteTimeout to 5 s or more for slow-link servers
the connection window a stream whose own send window has room but whose bytes wait for the connection window is credited with what the whole conn released in the window (held while the conn moves, with no idle bound: waiting for the conn is not its fault). A stream whose own window is shut is judged by its own releases and the rules above, even when the conn window is shut too
A2 retired for WriteTimeout > 0 a stalled stream no longer takes the conn out of the conn scan's exemption (A2, #1134: the conn of a silent client was closed at ReadTimeout without a word): the stream is bounded by WriteTimeout, reset, and then the conn is stream-less and its clocks apply (as M2 did for A1). A silent client's unread response, and the buffer it pins, therefore lives for up to 8 x WriteTimeout (5 with the default 64 KiB window and a rest under 64 KiB, 1 with a larger rest; 8 for a window of 256 KiB or more whose in-flight bytes fit the kernel buffers), not ReadTimeout: the conn used to be closed at ReadTimeout and take the buffer with it (r8: a silent client with a 1 MiB window kept its buffer past 5 s at WriteTimeout 1 s where main closed it within 2 s; 5 minutes at the 60 s default). A pool handler blocked in AwaitSendWindow stays counted (its bound is #906's)
#906 unchanged a pool handler blocked in AwaitSendWindow keeps its absolute WriteTimeout from the start of the wait (it cuts a client that reads at a healthy rate: #1170, filed as the next stage, not changed here); a stream both would reset gets one RST_STREAM (the second finds it cancelled or gone)

State: a small per-conn map on the Manager (stalls, guarded by stallMu, made on the first stall), a record {window start, held, pending, released, conn counters} per stream that holds buffered bytes, and the fourth sweep hint DeadlineHints.WrDL. Nothing in Stream (it has 120 free bytes: that is not the reason; the reason is that only a stalled stream, a rare case, needs the state, and the conn credit needs conn-wide state anyway; the per-stream-field alternative is in the design note). Every path that ends a stalled stream ends its record (7 sites, and the sweep's net for a stream gone from the map); the leak test pins each. The common send costs one atomic load of stallN in ReserveSendWindow (to count the conn's releases while any stream is stalled).

The RST_STREAM is queued behind the frames the stream already queued (through the conn writer's Resetter, as resetHandlerStream does): written straight to the connection it could overtake DATA still in the write queue. The bytes the stream held are dropped and refunded to the outbound budget; the stream leaves the conn as a reset from the peer does (a pool handler still running on it retires it); its identifier goes in the reset window, so the client's late WINDOW_UPDATE, RST_STREAM and PRIORITY are ignored (cell).

Review r7 (Opus, REQUEST-CHANGES: two MAJOR, three MINOR) and what changed

Tests first (h2_write_rate_889_linux_test.go), the rule after.

  • F6, parallel downloads on a slow link (new against main at any ReadTimeout): unless the conn window was shut each stream was judged on its own bytes, so N streams sharing a slow socket (link/N each, below the floor, the conn far above it) were reset: 2 streams on 100 KB/s both at 2.3 s, 4 on 200 KB/s all four; main completes all. Red on 46947d4: 6 of 6 (3 engines x 2 shapes); green now: all complete at 5.25 s. Complete rule: the connection's rate and the share/idle bound above. Whether the scheduler can starve a stream of a healthy client: a stalled stream is flushed only by its own WINDOW_UPDATE (or the conn window opening, for the starved), so who gets bytes is the client's choice and the order the socket delivers them; the server has nothing to make fairer. A stream queued behind its siblings' bytes sees no release for N x W / link (4 streams of 64 KiB on 200 KB/s: 1.3 s), which the 3 idle windows absorb. I chose the bounded grace (K = 3) over credit for bytes in flight (cannot tell a paused stream from a queued one: unbounded shelter) and over a per-connection judgement for all (same shelter). A paused or dribbled stream among healthy siblings: red on the previous head 6 of 6 (reset at 1.2 s), now reset at 4.3 s (paused) and 5.3 s (dribbled; the seed covers its first window).
  • F1, few WINDOW_UPDATEs: progress counted only when an update arrived and nothing carried over. Real curl 8.14.1 (nghttp2 1.64.0) at 300 KB/s, 16 MiB, WriteTimeout 1 s: reset after 7.9 s at 2.5 MB before (curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR), complete in 54.5 s now (the review measured main at 54.5 s too); 8 MiB at 1m: complete both. Raw-framer cells (CI-able): a client that updates at half a 1 MiB window reading at 300 KB/s (4.6x the floor): reset at 1.0 s before, complete now; half of 6 MiB at 1 MB/s (gap 3 s): reset at 1.0 s, complete; Go-Transport-like (4 KiB updates): green both. Red on 46947d4: 6 of 9 (the 3 guards green). The fix is the bounded carry-over (4 floors) and the seed; the exact guarantee is in the table. When ReadTimeout <= WriteTimeout (including the 60 s defaults) main closed the whole conn at the same gap, so the new cut was where WriteTimeout < ReadTimeout or ReadTimeout is off.
  • F2: the verdict is for the record phase 1 found due (a record ended and re-made while the windows were read was expired ~2 us into its window); nothing is reset whose buffer went out whole between the verdict and the reset. Unit tests (the reviewer's interleaving).
  • F3: WriteTimeout near math.MaxInt64 overflowed armedAt + wt: every stalled stream had to release 64 KiB per sweep (Go's Transport at 500 KB/s got INTERNAL_ERROR after 8.4 s). Timeouts are clamped to 100 years in SetWriteTimeout, SetRequestTimeout and SetHeaderBlockTimeout (stage 1's request deadline had the same overflow); tests for all three.
  • F4: "the renewal keeps the first window's held" turned out to be an equivalent mutant at the verdict level (the connection's floor, connHeld, carries the same bytes; a unit test pins the intended behaviour), and the own-window-shut shelter, killed only by a unit test before, is now also a cell: a paused stream on a healthy conn is reset in the fourth window.
  • F5: the window clock is the wall clock (UnixNano, the engines' sweep now, as D1's): a forward step judges every stalled stream at once. Not changed: the sweep's now is a wall-clock value; documented.
  • Docs: the WriteTimeout godoc says the guarantee exactly and drops "never cut" without its conditions; the detached-stream paragraph no longer says "no minimum rate" for an h2 stream; the release note says a silent client's buffer lives for WriteTimeout.

Review r8 (Opus, APPROVE-WITH-FIXES: one MAJOR, six MINOR) and what changed

  • F-B MAJOR, CI timeout margin: the root package's rest step (test/server, -race, -timeout=540s) took 539.2 s at the merge of epoll, io_uring: every HTTP/1 response after Shutdown carries Connection: close and ends its conn (v1.6.0, Refs #885 #884) #1169 (0.8 s from the limit; main alone 501 s). The 14 slowest tests still in it (8 to 34 s each, 216 s together, none of this PR's) moved to shards a and b (.github/root-test-shards.txt, root-shards.sh check ok). CI before and after (runs 40816689641 and 40827919267): rest 539.2 s -> 312.7 s; shard a 237.7 s -> 337.9 s, shard b 250.1 s -> 351.0 s (their timeout is 660 s).
  • F-A, a shared download reset a few bytes before its end (WriteTimeout 1 s): when less than a floor is held, "release all of it in one window" is the verdict, and the grant for the last bytes waits behind the siblings' bytes in the socket. A tail (stream's or connection's) that released half of itself is a grace window inside the idle bound; nothing is the verdict at once. Cells: 2 streams x 384 KiB on 100 KB/s and 8 x 128 KiB on 200 KB/s (red before: 4 of 12 cell runs, the 2 x 384 KiB on epoll only, 8 x 128 KiB on all three; green now), unit tests for each clause, small-tail-short (which pinned the old behaviour) changed to the new one: a tail that released half is held, under half is reset. The two r8 shapes of 21 s are not cells.
  • F-C, the bounds (documented 4 x WriteTimeout, really 5 alone and 8 with a moving sibling for a window >= 256 KiB): the true bounds are derived from the code and pinned (TestWriteUnreadStreamBounds889, TestWriteDribblerBounds889); godoc, release note, this body and the design say 1 and 4 WriteTimeouts for the default window, 5 and 8 for >= 256 KiB, a 1-byte dribbler in window 2 or 5.
  • F-D, queueing behind siblings: stated as a condition in the godoc and release note (N x W / link under 3 WriteTimeouts; WriteTimeout >= 5 s for slow links). I measured scaling the grace (clamp(ceil(N/2), 3, 8)): no change on 16 streams of 512 KiB on 300 KB/s (16 of 16 reset either way). The reset there is the cascade of the pool handlers' absolute bound (below): six streams reset at 4.5 s on main too, after which the client keeps consuming their queued bytes, grants nothing for them, and the connection itself releases less than its floor for a window. So the grace is not scaled.
  • F-E, mutants that survived: eight were killed by new unit tests: the idle count not restarted on a share hold, a floor hold or a starved hold; the share ignoring the stream count, without its halving, or quartered; the seed taken as the initial window; the connection's floor fixed at 64 KiB; the bank kept on the share path; the tail grace unbounded and each of its two clauses. One is equivalent (the renewal keeps the first window's held). Table below.
  • F-F, the clamp: moved to resource.Config.WithDefaults (resolveTimeout), so every engine's now + timeout (including detectDeadline = now + ReadHeaderTimeout) is safe, and kept in the h2 setters; a test per timeout.
  • F-G, one unreproduced race failure (SparseWindowUpdates/io_uring/halfwindow/W6MiB): 240 runs (60 x io_uring and epoll, -race, --cpus 2, plain and under 6 busy loops) did not fail. By reading, the seed (initWin - windowSize at the first buffering) is read after the window is debited in every buffering path; I found no hole. The cell's failure message names how and when the stream ended.
  • Item 8, checked: the Go-Transport parallel loss is fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906's absolute bound, not this rule. Four parallel 8 MiB downloads by Go's Transport on a 1 MB/s link, WriteTimeout 1 s: three of four reset with INTERNAL_ERROR at 16.8 s, on main and on this head. Trace (a print in sendRest, expireWrite printing nothing): sendRest deadline: stream 5, 7, 9 left 4194304: the 4 MiB output budget is taken by the first stream's tail, the other inline handlers are moved to the pool, their 4 MiB remainders wait for the window in sendRest under the absolute deadline = now + WriteTimeout, the client reads the first stream first and grants nothing for the others within that second, and sendRest resets them: a client reading at a healthy rate (4 x the floor) is cut with no D2 verdict involved. It needs the minimum-rate treatment of this PR (not done here; for an issue).

Review r9 (Opus, REQUEST-CHANGES: one BLOCKER, one MAJOR) and what changed

Tests first (754fae1, red on 4feb280), the rule after (bed1f51; one block of holdsLocked each).

  • F-H BLOCKER, an unread stream with a small rest was reset after tens of thousands of windows, and its connection never closed. The bank paid total - floor and the floor of a stream whose rest is 1000 bytes is 1000: a 64 KiB to 256 KiB seed lasted 65,536 windows (default window) or 262,145 (>= 256 KiB): 45 and 182 days at the 60 s default, with ReadTimeout and IdleTimeout not applying meanwhile (A2 retired): 8 of 8 connections open at 30 s on epoll and io_uring with all timeouts at 1 s, where main closes at 1.0 to 1.1 s. Fix (fixD): a window held on the bank alone pays a whole 64 KiB (charge = writeMinRelease when released < floor; bank = min(max(total - charge, 0), 256 KiB)). Not fixB (cap the bank by the floor): it truncates the carry-over of a streamed response (TestWriteBankSurvivesASmallRenewal889, killed by mutant r9b). Red on 4feb280: TestH2SilentClientWithASmallRemainderIsLetGo889, 6 of 6 (3 engines x default and 1 MiB window; the connection is still open at limit + 10 s); green now: epoll closes at 5.1 s and 8.1 s, io_uring at 9.1 s and 13.7 s, adaptive 5.2 s and 8.2 s (docker --cpus 2; limits 16 s and 24 s). Unit: TestWriteUnreadSmallRemainderBounds889 (12 shapes: rests of 1, 100, 10000, 60000 bytes x windows of 65535, 128 KiB, 1 MiB: windows 5, 6, 8), red before.
  • F-A2 MAJOR, parallel downloads lost their last 2 to 8 bytes when the end-of-transfer backlog exceeded one window (20 x 128 KiB on 100 KB/s at WriteTimeout 5 s: 9 of 20 reset; 12 x 256 KiB on 300 KB/s at 1 s; main completes all). The connection's tail clause required half of its floor released in the window; a slow drain does not. Fix (fixC, one line): while the whole connection holds under one floor the stream is held without that proof, inside the idle bound (3 windows, reset in the 4th). Cell: .../12streams/300KBps/256KiB (backlog 2.6 s, longer than a window), red on 4feb280 on epoll and adaptive (6 of 12 streams reset; io_uring happened to complete), green now on all three. The reviewer's 20-stream shape takes 26 s; this one is its scaled-down class and runs in 10.5 s.
  • The four pinned cases changed on purpose (they pinned the r8 behaviour that F-A2 shows is wrong): TestExpiredWritesFloor889/small-tail-under-half (a tail that released under half is now held, bounded by the idle windows; was reset at once), TestExpiredWritesBatch889 (40 streams of 100 bytes are no longer expired at the first window; the batch limit is shown after the idle windows), TestWriteTailGraceIsBounded889/nothing-released-is-the-verdict (renamed nothing-released-is-held-for-the-idle-windows: held three windows, reset in the fourth), TestWriteConnTailGrace889/under-half (held whatever went out; reset in the fourth window). TestExpiredWritesJudgesTheRecordItFoundDue889 now buffers 100000 bytes after the renewal (a 1000-byte tail would be held by the grace and hide the re-check mutant).
  • The Coverage job failed at 7e7f5ce (TestExpiredWritesBatch889: second batch 6, want 8): a test, not the rule. The batch test (and TestWriteConnTailGrace889) took start before it made its records; a record armed more than the 1 ms slack after start is not due at the first verdict, so its idle count lags a window and its reset lands one window late. The coverage build is slow enough for that (2 of 40 records), a normal build is not (not reproduced in 20 runs there). Reproduced deterministically by pausing 2 ms between the records (red, -count=3, both tests: first batch 1, reset in window 5), fixed by taking start after the records (aec8e2e); the whole package then passes -covermode=atomic -count=20 (173 s) and the stall/write tests -race -count=5. Not a code defect: the rule does not depend on the order of the records.
  • NITs: the dribbler bound in docs is window max(2, 1+k) (measured 3 at 128 KiB, 4 at 192 KiB); resolveTimeout's comment no longer says "N stays N" without the clamp; the rule comments (write_deadline.go head and holdsLocked, Config.WriteTimeout) describe the tail grace; HTTP/2 pool handlers blocked in AwaitSendWindow are cut by an absolute WriteTimeout bound even when the client reads at a healthy rate #1170 (the pool handlers' absolute bound) is linked above.
  • Mutants (below): for fixD, r9a (the charge is the rest again) and r9d (charged only when nothing was released, killed by a dribbler with a small rest); for the bank, r9b (fixB), r9e (a negative bank, killed by TestWriteBankIsNotADebt889), r9f (half a floor charged); for fixC, r9c; and the existing mutants that the new rule had let survive (n8 the re-check, p8 the connection's floor fixed at 64 KiB) are killed again by TestExpiredWritesJudgesTheRecordItFoundDue889 (changed) and TestWriteSmallConnShareHoldClearsIdle889 (new).

Commits (tests first)

3883c56 tests, red on main 33 cells on 3 engines: 24 red, 9 green guards
a4d068b the change write_deadline.go, Manager, FlushOutbound, ReserveSendWindow, H2State.Sweep, 7 removal sites; unit tests
the pinned cell #1134's "a pinging client holds an unread 1 MiB response" is flipped: red on main 3 of 3
r7 tests (red on 46947d4) 18 cells: sparse updates 6 red (3 guards green), shared link 6 red, paused/dribbled stream 6 red
r7 fix carry-over and seed, connection rate, share and idle bound, re-check, clamp, docs
r9 tests (red on 4feb280) 754fae1: the small-remainder cell 6 of 6 red, the 12-stream shared download red on epoll and adaptive, unit tests red
r9 fix bed1f51 (fixD + fixC in holdsLocked), then docs (ad7210a), tests for the surviving mutants, the merge of main (7e7f5ce), and the test-timing fix for the Coverage job (aec8e2e)

Evidence

  • Unit (internal/protocol/h2/stream, internal/conn): the floor and windows, the carry-over and its cap, the average-rate guarantee (floor on average held for ever, half of it reset once the bank is spent), the seed, the dribbler within two windows, the idle stream on a healthy conn (3 held, 4th reset; its share holds it; a slow conn is the verdict at once), the conn credit, a record ends with every removal path (9 paths), one RST, the re-check, nothing held means no reset, the three clamps, the common path free, a direct send makes no record, frames-to-progress, the sweep end to end.
  • r9 round, on the head merged with main after tests: the port 0 hold tests ask whether the sockets the engine opened are closed, not how many sockets of the process have the port number (v1.6.0, Refs #1104) #1171 (7e7f5ce; docker --cpus 2, arm64): the 10 write-deadline cells -race -count=5 -cpu 2 ok (932 s); internal/conn, internal/tcpack, internal/resource, internal/protocol/... -race -count=5 ok (stream package 269 s); internal/engine/epoll -race ok (380 s) and internal/engine/iouring -race ok (491 s); the whole test/server H2|889|Shutdown|SSE|Stream|1165|884 set non-race ok (924 s). (A first non-race run of that set had 8 failing cells while the Mac slept and two other lanes' containers ran: timing cells with 1 s windows cut by the stalled VM clock; the clean re-run is the one above.)
  • CI: the first push of the r7 cells made the root package's rest step (-race, -timeout=540s) time out; the 9 slow cells are listed in .github/root-test-shards.txt (shards a and b, root-shards.sh check ok) so that the rest stays inside its timeout. CI at the merged head 56ca76a: 29 pass, 2 skipping.
  • golangci-lint v2.14.0: 0 issues (also -tags netem889) at 7e7f5ce; GOOS=windows and darwin go vet ok; apidump ok (53 packages) and its own tests; root-shards.sh check ok (65 tests in shards, a=33 b=32); size tests pass (connState 448, H2State 648 bytes = the 704 malloc class, Stream unchanged).
  • CI evidence: every cell runs in the ordinary test/server package, no privileges, so CI runs them all. The curl numbers above are a one-off measurement (docker, curl 8.14.1/nghttp2 1.64.0), not a cell: a CI cell that skips without curl would trip the skip rule.

Mutants (each applied, the named tests run, file restored; logs/mutant-d2-*.log)

mutant killed by
no expiry unit, 12 cells
any byte is the floor unit, 9 cells
floor off by one unit x3
the window never restarts unit x3, 6 cells
a shut own window gets the conn's credit unit
no conn credit for a starved stream unit, 2 cells
any release counts as a full floor unit, 3 cells
a direct send arms a record unit (PoolDirect)
A2 not retired unit, 3 cells
CANCEL for INTERNAL_ERROR unit x2, 6 cells
no RST queued unit x3
a second reset; the identifier not remembered unit each
the hint never lowered unit x3, 6 cells
no conn count in ReserveSendWindow unit, 3 cells
no carry-over unit x3, 5 cells (sparse)
the bank uncapped unit
no seed unit x3, 6 cells
idle windows unbounded (a paused stream sheltered for ever) unit, 6 cells
no idle grace (a queued stream reset at once) unit, 3 cells (shared link)
any release is a share (a dribbler on a healthy conn) unit, 3 cells
a slow conn is held (the verdict not at once) unit
the record not re-checked unit
reset with nothing held unit
timeouts not clamped unit x3
r8: the idle count not restarted on a share hold / a floor hold / a starved hold unit x1 each
r8: the share ignoring the stream count; without its halving; quartered unit x1-3 each
r8: the seed taken as the initial window (not what is in flight) unit
r8: the connection's floor fixed at 64 KiB unit x3
r8: the bank kept on the share path unit
r8: no tail grace; the stream clause alone off; the conn clause alone off; the tail grace unbounded unit (floor, tail, conn tail, beside-a-big-sibling), 4 cells (no tail grace)
r9: the charge is the rest again (r9a, fixD removed) unit, 6 cells (small remainder)
r9: the bank capped by the floor (r9b, fixB) unit (carry-over, average rate, bank survives a small renewal)
r9: the half-proof back on the connection's tail clause (r9c, fixC removed) unit (floor, conn tail, batch, tail grace), 3 cells (12 streams)
r9: charged only when nothing was released (r9d) unit (dribbler with a small rest; new)
r9: a negative bank (r9e) unit (TestWriteBankIsNotADebt889; new)
r9: half a floor charged (r9f) unit
r9: the connection's floor fixed at 64 KiB (p8, survived after fixC until TestWriteSmallConnShareHoldClearsIdle889), the re-check (n8, survived until its test used a 100000-byte tail) unit each
a record outlives its stream, 7 ways the leak test, one path each
equivalent: the renewal keeps the first window's held survives: the connection's floor (connHeld) carries the same bytes, so the verdict is the same; the unit test pins the intended behaviour

Hot path and cost

Docker arm64 --cpus 2 (a shared machine; amd64 unmeasured), main (3f8d83f) vs head, interleaved, 4 rounds x 2 runs per engine (the r8 round; the previous round: epoll 4096 +6.1% / +10.5% min), medians (minimums): the sweep is where D2 adds a per-conn cost (one more atomic, hints 8 bytes longer):

benchmark (nothing due) main head
sweep, epoll, 4096 h2 conns 29.47 us (28.72) 31.21 us (30.79) +5.9% (+7.2%): about +0.4 ns per conn
sweep, epoll, 256 1336 ns (1268) 1384 (1314) +3.6% (+3.6%)
sweep, io_uring, 4096 40.69 us (39.28) 43.30 us (41.82) +6.4% (+6.5%)
sweep, io_uring, 256 2139 ns (2094) 2197 (2151) +2.7% (+2.7%)

r9 round (same method, 4 rounds x 2 runs; the host was busy with other lanes, load about 5, so every absolute number is 25 to 40% slower than the r8 round and the percentages are noisier): epoll 4096 37.60 us (35.06) -> 40.49 us (38.99): +7.7% (+11.2%), epoll 256 1673 -> 1822 ns (+8.9%); io_uring 4096 53.09 -> 52.88 us (-0.4%), io_uring 256 2746 -> 2718 ns (-1.0%). The r9 change is inside the verdict, not on the sweep's scan, so these are the same cost as r8 seen through a noisier machine: on epoll about +0.7 ns per conn here (+0.4 in the quiet round), on io_uring within noise. (The review measured +6.6 to +7.9% at 4096 on epoll: the same order.) At 10 000 h2 conns and 40 sweeps a second the addition is about 0.2 ms of a second. The conn and stream benchmarks (ProcessH2Post889, StreamedResponse889, OneByteGrant911) are within the noise of the shared machine, allocations unchanged (previous round: ProcessH2Post889/single 673 vs 668 ns, StreamedResponse889/closed 559 vs 548 ns; the review: +2 to 3% medians, minimums equal or lower). The common send pays one atomic load of stallN in ReserveSendWindow. A stalled stream's flush takes the books' lock once: BenchmarkOneByteGrantWriteDeadline889 on 61 to 73 ns, off 53 to 122 ns (noise-bound). The window verdict allocates when more than 32 streams of a conn are due at once.

Release notes (draft)

  • HTTP/2 on epoll, io_uring and adaptive: WriteTimeout now bounds a response that waits for the client's flow-control window, as a minimum rate, the rule it already applies to a response parked on the socket: in each WriteTimeout window the client must let at least 64 KiB of the stream's waiting bytes out (all of them, if fewer are waiting), or the stream is reset with RST_STREAM(INTERNAL_ERROR), as net/http resets a stream at its write deadline. The guarantee: a stream that releases on average at least that per window, lagging that rate by no more than four windows' worth (256 KiB; the bytes in flight when it stalls count for its first window), is never cut, however long its response takes (unlike std, whose deadline is absolute); a client that updates in big steps (curl/nghttp2: half the window) survives while the gap between its updates (half the window divided by its read rate) stays under about five WriteTimeouts. A client that PINGs, or sends WINDOW_UPDATEs of a few bytes, no longer holds a stream (and the memory of its response) for ever: a stream it never reads is reset after 1 WriteTimeout (4 with a sibling that keeps the connection moving) for the default 64 KiB window and 5 (8) for a window of 256 KiB or more; if what is left to send is under 64 KiB, after 5 (default window), 6 (128 KiB) or 8 (256 KiB or more) WriteTimeouts, and its connection is closed then.
  • Parallel downloads on one connection are held while the connection moves at least that much and each takes half its fair share; a stream that takes less is held for three more windows and reset in the fourth. Streams queued behind their siblings' bytes in the client's socket are cut when those bytes take more than three WriteTimeouts to read (N streams of W bytes in flight on a link of L bytes/s: N x W / L): with WriteTimeout 1 s on a slow link they can be reset where main completed them; set WriteTimeout to 5 s or more for servers whose clients are on slow links.
  • A connection with such a stream is no longer closed by ReadTimeout or IdleTimeout while the stream lives: the stream's WriteTimeout bounds it, and the connection's clocks apply once it is gone. A silent client's unread response, and the buffer it pins (up to the 4 MiB outbound budget plus one body per connection), used to be dropped with the whole connection at ReadTimeout and now lives for up to 8 WriteTimeouts (5 with the default 64 KiB window and a response a little longer than it, 1 when the rest is 64 KiB or more; 8 minutes at the 60 s default for a 256 KiB or larger window). With WriteTimeout: -1 nothing changes.
  • Timeouts longer than a hundred years (a math.MaxInt64 write, read, read-header or idle timeout) are kept as a hundred years in every engine, so that now + timeout cannot overflow and make every deadline due at once.
  • A handler that is blocked waiting for window keeps its own absolute bound of WriteTimeout from the start of the wait (unchanged); a pool handler cut by it while its client reads at a healthy rate (parallel downloads over the 4 MiB output budget) is a known limit, see below.
  • std is unchanged.

Findings, not fixed here

  1. A pool handler blocked in AwaitSendWindow (sendRest) is bounded by fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906's absolute WriteTimeout, a buffered tail by this rate: the two policies differ, and the absolute one cuts a client that reads at a healthy rate (Go's Transport, 4 parallel 8 MiB downloads on a 1 MB/s link, 3 of 4 reset: traced above, identical on main). It needs this PR's treatment as a follow-up (an issue) or here.
  2. The windows run on the wall clock (the engines' sweep now): a forward clock step judges every stalled stream at once, as D1's does (r7 F5).
  3. ExpiredWrites allocates when more than 32 streams of one conn are due at once. The per-expiry budget (2 ms per conn) is the stage 1 one.
  4. The share/idle rule can hold a dribbler for 4 windows on a healthy conn, and a stream that releases its share every window for as long as the conn moves; an attacker pays the floor in bytes the socket carries for each conn it uses, and the buffer stays under the outbound budget.

Unverified

  • amd64 (every number is arm64); adaptive on its io_uring side (its cells ran on epoll); the review ran its F1/F6 probes on epoll only (the rule is in internal/protocol/h2/stream, engine-independent), and the new cells ran here on all three engines.
  • Browsers' update policies (Chromium flushes small updates on a timer, Firefox acks in large steps: from memory, not measured); the cells' timing on a loaded CI runner (1 s windows; the paused-stream cell expects a reset in [3.5 s, 7 s]).

…iteTimeout as a minimum rate (red on main)

Refs #889
…(D2)

A stream that holds buffered bytes for the peer's window releases at least
min(64 KiB, what it held at the window's start) per WriteTimeout or is reset
with RST_STREAM(INTERNAL_ERROR). Streams starved by the connection window are
credited with the conn's releases only if their own window has room. A small
per-conn map on the Manager holds the records of stalled streams, ended by every
removal path; the fourth hint, WrDL, is read by the sweep. A conn whose stream
is stalled is exempt from the conn clocks for WriteTimeout > 0 (A2 retired).

Refs #889
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…nd off; the conn benchmarks run with WriteTimeout

Refs #889
…e is flipped: its stream is reset at WriteTimeout

Red on main (3 of 3 engines), as the cell's comment said it would be.

Refs #889
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.54455% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/protocol/h2/stream/write_deadline.go 95.32% 8 Missing ⚠️
internal/protocol/h2/stream/manager.go 87.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

… append to a stalled stream; frames-to-progress units

Refs #889
…arse WINDOW_UPDATEs, a shared slow link, and the bound on a paused stream (red on the previous head)

Refs #889
…er and seed, the connection's rate and a bound on a stream below its share, the record judged is the one found due, deadlines that cannot overflow

F1: surplus released beyond the floor carries to later windows, up to 4 windows' worth, and a stream's first window is covered by its bytes in flight (the same cap).
F6: on a connection that releases at least the floor a stream below its floor is held while it takes half its fair share of the floor, and for 3 consecutive windows if it takes less.
F2: the verdict is for the record found due; nothing is reset that has nothing held. F3: timeouts are clamped to a hundred years.

Refs #889
…oor-1); the sweep unit with a client of window 0

Refs #889
…end makes no record; the credited cell lasts beyond the idle grace

Refs #889
… an unread stream; a silent client's buffer lives for WriteTimeout; the detached-stream paragraph

Refs #889
… shared are not reset a few bytes short (red: 4 of 12)

Refs #889
…he true bounds of an unread stream, the idle count, the share, the seed and the conn floor

F-A (r8): the stream's tail, or the connection's, under one floor was judged on all of it in one window; reset 8 to 16 bytes before the end of a shared download. Half of it released is a grace window inside the idle bound.
F-C/F-E: the bounds are pinned (an unread stream is reset in window 1+k alone and 1+k+3 with a moving sibling, k = min(4, in flight / 64 KiB)) and eight mutants that survived are killed.

Refs #889
…te for parallel downloads and sparse updates, the pool handler's absolute bound

Refs #889
…losed), the end of a shared download whose drain exceeds one window is not cut (red on the previous head)

Refs #889
…tail is held without proof (r9 F-H, F-A2)

An unread stream whose rest is under 64 KiB spent a few bytes a window of its bank and lived for tens of thousands of windows (the connection never closed: ReadTimeout and IdleTimeout do not apply while a response is held); a window that passes on the carried bank now pays a whole floor.

When the whole connection holds under one floor the end of the transfer is on the wire and the drain of the backlog behind it can take more than a window (9 of 20 parallel downloads reset 2 to 8 bytes before the end): that tail is held for the idle windows without the half-released proof.

Four pinned cases change on purpose: TestExpiredWritesFloor889/small-tail-under-half (held), TestExpiredWritesBatch889 (the batch limit is shown after the idle windows), TestWriteTailGraceIsBounded889 (nothing released is held for 3 windows, reset in the 4th), TestWriteConnTailGrace889 (held for the idle windows whatever went out).

Refs #889
…l remainder and the dribbler, resolveTimeout's clamp

Refs #889
…check, the connection's floor on a small connection, the bank as a debt, a dribbler's charge)

Refs #889
…ws begin before the records are made fail (red; the CI Coverage job at 7e7f5ce)

Refs #889
…e (green; the CI Coverage job at 7e7f5ce)

A record armed more than the 1 ms slack after the window's start is not due at the first verdict, so its idle count lags a window and the batch/the reset lands one window late. The slow runner (coverage instrumentation) made 2 of 40 records late.

Refs #889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant