Repository navigation
engines: HTTP/2 WriteTimeout bounds a response that waits for the client's window, as a minimum rate: stage 2 of the per-stream deadlines (v1.6.0, Refs #889) - #1167
Merged
Conversation
…iteTimeout as a minimum rate (red on main) Refs #889
…(D2) A stream that holds buffered bytes for the peer's window releases at least min(64 KiB, what it held at the window's start) per WriteTimeout or is reset with RST_STREAM(INTERNAL_ERROR). Streams starved by the connection window are credited with the conn's releases only if their own window has room. A small per-conn map on the Manager holds the records of stalled streams, ended by every removal path; the fourth hint, WrDL, is read by the sweep. A conn whose stream is stalled is exempt from the conn clocks for WriteTimeout > 0 (A2 retired). Refs #889
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… append to a stalled stream; frames-to-progress units Refs #889
… into l19d-d2-write-deadline
…arse WINDOW_UPDATEs, a shared slow link, and the bound on a paused stream (red on the previous head) Refs #889
…er and seed, the connection's rate and a bound on a stream below its share, the record judged is the one found due, deadlines that cannot overflow F1: surplus released beyond the floor carries to later windows, up to 4 windows' worth, and a stream's first window is covered by its bytes in flight (the same cap). F6: on a connection that releases at least the floor a stream below its floor is held while it takes half its fair share of the floor, and for 3 consecutive windows if it takes less. F2: the verdict is for the record found due; nothing is reset that has nothing held. F3: timeouts are clamped to a hundred years. Refs #889
…oor-1); the sweep unit with a client of window 0 Refs #889
…end makes no record; the credited cell lasts beyond the idle grace Refs #889
… an unread stream; a silent client's buffer lives for WriteTimeout; the detached-stream paragraph Refs #889
…tep timed out at 540 s with them) Refs #889
…s (it was at 539.2 s of 540 s) Refs #889
… shared are not reset a few bytes short (red: 4 of 12) Refs #889
…he true bounds of an unread stream, the idle count, the share, the seed and the conn floor F-A (r8): the stream's tail, or the connection's, under one floor was judged on all of it in one window; reset 8 to 16 bytes before the end of a shared download. Half of it released is a grace window inside the idle bound. F-C/F-E: the bounds are pinned (an unread stream is reset in window 1+k alone and 1+k+3 with a moving sibling, k = min(4, in flight / 64 KiB)) and eight mutants that survived are killed. Refs #889
…ath.MaxInt64 overflowed now + timeout) Refs #889
…te for parallel downloads and sparse updates, the pool handler's absolute bound Refs #889
Refs #889
…losed), the end of a shared download whose drain exceeds one window is not cut (red on the previous head) Refs #889
…tail is held without proof (r9 F-H, F-A2) An unread stream whose rest is under 64 KiB spent a few bytes a window of its bank and lived for tens of thousands of windows (the connection never closed: ReadTimeout and IdleTimeout do not apply while a response is held); a window that passes on the carried bank now pays a whole floor. When the whole connection holds under one floor the end of the transfer is on the wire and the drain of the backlog behind it can take more than a window (9 of 20 parallel downloads reset 2 to 8 bytes before the end): that tail is held for the idle windows without the half-released proof. Four pinned cases change on purpose: TestExpiredWritesFloor889/small-tail-under-half (held), TestExpiredWritesBatch889 (the batch limit is shown after the idle windows), TestWriteTailGraceIsBounded889 (nothing released is held for 3 windows, reset in the 4th), TestWriteConnTailGrace889 (held for the idle windows whatever went out). Refs #889
…l remainder and the dribbler, resolveTimeout's clamp Refs #889
…check, the connection's floor on a small connection, the bank as a debt, a dribbler's charge) Refs #889
… into l19d-d2-write-deadline
FumingPower3925
marked this pull request as ready for review
October 30, 2026 13:01
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lane L19 H2-TIMEOUTS, stage 2 of the per-stream deadlines: D2, the write deadline of an HTTP/2 stream on epoll, io_uring and adaptive (std unchanged). Refs #889. Built on the merged stage 1 (#1150) per design v3.1 (lane evidence:
design-per-stream-deadlines.md). Draft.What it does
A response whose bytes wait for the client's flow-control window (the tail of an inline handler's body, or a pool handler's, buffered in the stream;
Manager.outboundHeld) had no bound at all (#1134 Finding 1): a client that granted no window and kept pinging held the stream, and its buffer, for ever (measured: 6 s, 60 s, no reset). NowWriteTimeoutbounds it as a minimum rate, #1119's rule at the h2 layer, refined by review r7 so that it does not cut a reader that is slow but moving (final rule and its bounds: design v3.5):WriteTimeoutlongFlushOutbound) at least F =min(64 KiB, what it held when the window began); if so the next window begins from that instant; if not it is reset with RST_STREAM(INTERNAL_ERROR) (net/http's code at a write deadline, and #906's). 64 KiB per window is #1119'sparkedMinAcked: about 64 KiB/s at a 1 s WriteTimeout, about 1 KiB/s at the 60 s defaultF / 2N, N the streams stalled), and for 3 consecutive windows if it took less (paused, dribbled, or queued behind its siblings' bytes in the same socket); the fourth is the verdict. When the connection itself released less than its floor, the first window is the verdict, as for a single stream. The tail (the end of a response) gets a grace inside the same bound (r8 F-A, r9 F-A2): a stream that holds under one floor and had half of it released, and any stream while the whole connection holds under one floor (the last bytes are on the wire; the backlog behind them can take more than a window to drain), is held for 3 windows without proof and reset in the 4th. A window held on the bank alone pays a whole floor, however small the rest (r9 F-H). The true bounds of a stream the client never reads: reset in window 1+k alone, 4+k with a sibling that keeps the connection above the floor, k = min(4, whole 64 KiB in the bytes in flight when it stalled): 1 and 4 WriteTimeouts for the default 64 KiB window, 5 and 8 for a window of 256 KiB or more; with a rest under 64 KiB (a response a little longer than the window): 5 (default window), 6 (128 KiB), 8 (>= 256 KiB) (pinned in unit tests). A 1-byte WINDOW_UPDATE dribbler: window max(2, 1+k): 2 (default window), 3 (128 KiB), 4 (192 KiB), 5 (>= 256 KiB). Parallel downloads on one connection are fine while the bytes ahead of a stream in the client's socket (N x W / link) take under 3 x WriteTimeout; at WriteTimeout 1 s on a slow link they are reset where main completes them: set WriteTimeout to 5 s or more for slow-link serversWriteTimeout > 0a stalled stream no longer takes the conn out of the conn scan's exemption (A2, #1134: the conn of a silent client was closed at ReadTimeout without a word): the stream is bounded by WriteTimeout, reset, and then the conn is stream-less and its clocks apply (as M2 did for A1). A silent client's unread response, and the buffer it pins, therefore lives for up to 8 x WriteTimeout (5 with the default 64 KiB window and a rest under 64 KiB, 1 with a larger rest; 8 for a window of 256 KiB or more whose in-flight bytes fit the kernel buffers), not ReadTimeout: the conn used to be closed at ReadTimeout and take the buffer with it (r8: a silent client with a 1 MiB window kept its buffer past 5 s at WriteTimeout 1 s where main closed it within 2 s; 5 minutes at the 60 s default). A pool handler blocked inAwaitSendWindowstays counted (its bound is #906's)AwaitSendWindowkeeps its absolute WriteTimeout from the start of the wait (it cuts a client that reads at a healthy rate: #1170, filed as the next stage, not changed here); a stream both would reset gets one RST_STREAM (the second finds it cancelled or gone)State: a small per-conn map on the Manager (
stalls, guarded bystallMu, made on the first stall), a record{window start, held, pending, released, conn counters}per stream that holds buffered bytes, and the fourth sweep hintDeadlineHints.WrDL. Nothing inStream(it has 120 free bytes: that is not the reason; the reason is that only a stalled stream, a rare case, needs the state, and the conn credit needs conn-wide state anyway; the per-stream-field alternative is in the design note). Every path that ends a stalled stream ends its record (7 sites, and the sweep's net for a stream gone from the map); the leak test pins each. The common send costs one atomic load ofstallNinReserveSendWindow(to count the conn's releases while any stream is stalled).The RST_STREAM is queued behind the frames the stream already queued (through the conn writer's
Resetter, asresetHandlerStreamdoes): written straight to the connection it could overtake DATA still in the write queue. The bytes the stream held are dropped and refunded to the outbound budget; the stream leaves the conn as a reset from the peer does (a pool handler still running on it retires it); its identifier goes in the reset window, so the client's late WINDOW_UPDATE, RST_STREAM and PRIORITY are ignored (cell).Review r7 (Opus, REQUEST-CHANGES: two MAJOR, three MINOR) and what changed
Tests first (
h2_write_rate_889_linux_test.go), the rule after.curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR), complete in 54.5 s now (the review measured main at 54.5 s too); 8 MiB at 1m: complete both. Raw-framer cells (CI-able): a client that updates at half a 1 MiB window reading at 300 KB/s (4.6x the floor): reset at 1.0 s before, complete now; half of 6 MiB at 1 MB/s (gap 3 s): reset at 1.0 s, complete; Go-Transport-like (4 KiB updates): green both. Red on 46947d4: 6 of 9 (the 3 guards green). The fix is the bounded carry-over (4 floors) and the seed; the exact guarantee is in the table. When ReadTimeout <= WriteTimeout (including the 60 s defaults) main closed the whole conn at the same gap, so the new cut was where WriteTimeout < ReadTimeout or ReadTimeout is off.WriteTimeoutnearmath.MaxInt64overflowedarmedAt + wt: every stalled stream had to release 64 KiB per sweep (Go's Transport at 500 KB/s got INTERNAL_ERROR after 8.4 s). Timeouts are clamped to 100 years inSetWriteTimeout,SetRequestTimeoutandSetHeaderBlockTimeout(stage 1's request deadline had the same overflow); tests for all three.connHeld, carries the same bytes; a unit test pins the intended behaviour), and the own-window-shut shelter, killed only by a unit test before, is now also a cell: a paused stream on a healthy conn is reset in the fourth window.UnixNano, the engines' sweepnow, as D1's): a forward step judges every stalled stream at once. Not changed: the sweep'snowis a wall-clock value; documented.Review r8 (Opus, APPROVE-WITH-FIXES: one MAJOR, six MINOR) and what changed
reststep (test/server,-race,-timeout=540s) took 539.2 s at the merge of epoll, io_uring: every HTTP/1 response after Shutdown carries Connection: close and ends its conn (v1.6.0, Refs #885 #884) #1169 (0.8 s from the limit; main alone 501 s). The 14 slowest tests still in it (8 to 34 s each, 216 s together, none of this PR's) moved to shards a and b (.github/root-test-shards.txt,root-shards.sh checkok). CI before and after (runs 40816689641 and 40827919267): rest 539.2 s -> 312.7 s; shard a 237.7 s -> 337.9 s, shard b 250.1 s -> 351.0 s (their timeout is 660 s).small-tail-short(which pinned the old behaviour) changed to the new one: a tail that released half is held, under half is reset. The two r8 shapes of 21 s are not cells.TestWriteUnreadStreamBounds889,TestWriteDribblerBounds889); godoc, release note, this body and the design say 1 and 4 WriteTimeouts for the default window, 5 and 8 for >= 256 KiB, a 1-byte dribbler in window 2 or 5.clamp(ceil(N/2), 3, 8)): no change on 16 streams of 512 KiB on 300 KB/s (16 of 16 reset either way). The reset there is the cascade of the pool handlers' absolute bound (below): six streams reset at 4.5 s on main too, after which the client keeps consuming their queued bytes, grants nothing for them, and the connection itself releases less than its floor for a window. So the grace is not scaled.held). Table below.resource.Config.WithDefaults(resolveTimeout), so every engine'snow + timeout(includingdetectDeadline = now + ReadHeaderTimeout) is safe, and kept in the h2 setters; a test per timeout.SparseWindowUpdates/io_uring/halfwindow/W6MiB): 240 runs (60 x io_uring and epoll,-race,--cpus 2, plain and under 6 busy loops) did not fail. By reading, the seed (initWin - windowSizeat the first buffering) is read after the window is debited in every buffering path; I found no hole. The cell's failure message names how and when the stream ended.sendRest,expireWriteprinting nothing):sendRest deadline: stream 5, 7, 9 left 4194304: the 4 MiB output budget is taken by the first stream's tail, the other inline handlers are moved to the pool, their 4 MiB remainders wait for the window insendRestunder the absolutedeadline = now + WriteTimeout, the client reads the first stream first and grants nothing for the others within that second, andsendRestresets them: a client reading at a healthy rate (4 x the floor) is cut with no D2 verdict involved. It needs the minimum-rate treatment of this PR (not done here; for an issue).Review r9 (Opus, REQUEST-CHANGES: one BLOCKER, one MAJOR) and what changed
Tests first (
754fae1, red on4feb280), the rule after (bed1f51; one block ofholdsLockedeach).total - floorand the floor of a stream whose rest is 1000 bytes is 1000: a 64 KiB to 256 KiB seed lasted 65,536 windows (default window) or 262,145 (>= 256 KiB): 45 and 182 days at the 60 s default, with ReadTimeout and IdleTimeout not applying meanwhile (A2 retired): 8 of 8 connections open at 30 s on epoll and io_uring with all timeouts at 1 s, where main closes at 1.0 to 1.1 s. Fix (fixD): a window held on the bank alone pays a whole 64 KiB (charge = writeMinReleasewhenreleased < floor;bank = min(max(total - charge, 0), 256 KiB)). Not fixB (cap the bank by the floor): it truncates the carry-over of a streamed response (TestWriteBankSurvivesASmallRenewal889, killed by mutantr9b). Red on4feb280:TestH2SilentClientWithASmallRemainderIsLetGo889, 6 of 6 (3 engines x default and 1 MiB window; the connection is still open at limit + 10 s); green now: epoll closes at 5.1 s and 8.1 s, io_uring at 9.1 s and 13.7 s, adaptive 5.2 s and 8.2 s (docker--cpus 2; limits 16 s and 24 s). Unit:TestWriteUnreadSmallRemainderBounds889(12 shapes: rests of 1, 100, 10000, 60000 bytes x windows of 65535, 128 KiB, 1 MiB: windows 5, 6, 8), red before..../12streams/300KBps/256KiB(backlog 2.6 s, longer than a window), red on4feb280on epoll and adaptive (6 of 12 streams reset; io_uring happened to complete), green now on all three. The reviewer's 20-stream shape takes 26 s; this one is its scaled-down class and runs in 10.5 s.TestExpiredWritesFloor889/small-tail-under-half(a tail that released under half is now held, bounded by the idle windows; was reset at once),TestExpiredWritesBatch889(40 streams of 100 bytes are no longer expired at the first window; the batch limit is shown after the idle windows),TestWriteTailGraceIsBounded889/nothing-released-is-the-verdict(renamednothing-released-is-held-for-the-idle-windows: held three windows, reset in the fourth),TestWriteConnTailGrace889/under-half(held whatever went out; reset in the fourth window).TestExpiredWritesJudgesTheRecordItFoundDue889now buffers 100000 bytes after the renewal (a 1000-byte tail would be held by the grace and hide the re-check mutant).7e7f5ce(TestExpiredWritesBatch889: second batch 6, want 8): a test, not the rule. The batch test (andTestWriteConnTailGrace889) tookstartbefore it made its records; a record armed more than the 1 ms slack afterstartis not due at the first verdict, so its idle count lags a window and its reset lands one window late. The coverage build is slow enough for that (2 of 40 records), a normal build is not (not reproduced in 20 runs there). Reproduced deterministically by pausing 2 ms between the records (red,-count=3, both tests: first batch 1, reset in window 5), fixed by takingstartafter the records (aec8e2e); the whole package then passes-covermode=atomic -count=20(173 s) and the stall/write tests-race -count=5. Not a code defect: the rule does not depend on the order of the records.resolveTimeout's comment no longer says "N stays N" without the clamp; the rule comments (write_deadline.gohead andholdsLocked,Config.WriteTimeout) describe the tail grace; HTTP/2 pool handlers blocked in AwaitSendWindow are cut by an absolute WriteTimeout bound even when the client reads at a healthy rate #1170 (the pool handlers' absolute bound) is linked above.r9a(the charge is the rest again) andr9d(charged only when nothing was released, killed by a dribbler with a small rest); for the bank,r9b(fixB),r9e(a negative bank, killed byTestWriteBankIsNotADebt889),r9f(half a floor charged); for fixC,r9c; and the existing mutants that the new rule had let survive (n8the re-check,p8the connection's floor fixed at 64 KiB) are killed again byTestExpiredWritesJudgesTheRecordItFoundDue889(changed) andTestWriteSmallConnShareHoldClearsIdle889(new).Commits (tests first)
3883c56tests, red on maina4d068bthe changewrite_deadline.go,Manager,FlushOutbound,ReserveSendWindow,H2State.Sweep, 7 removal sites; unit tests46947d4)4feb280)754fae1: the small-remainder cell 6 of 6 red, the 12-stream shared download red on epoll and adaptive, unit tests redbed1f51(fixD + fixC inholdsLocked), then docs (ad7210a), tests for the surviving mutants, the merge of main (7e7f5ce), and the test-timing fix for the Coverage job (aec8e2e)Evidence
internal/protocol/h2/stream,internal/conn): the floor and windows, the carry-over and its cap, the average-rate guarantee (floor on average held for ever, half of it reset once the bank is spent), the seed, the dribbler within two windows, the idle stream on a healthy conn (3 held, 4th reset; its share holds it; a slow conn is the verdict at once), the conn credit, a record ends with every removal path (9 paths), one RST, the re-check, nothing held means no reset, the three clamps, the common path free, a direct send makes no record, frames-to-progress, the sweep end to end.7e7f5ce; docker--cpus 2, arm64): the 10 write-deadline cells-race -count=5 -cpu 2ok (932 s);internal/conn,internal/tcpack,internal/resource,internal/protocol/...-race -count=5ok (stream package 269 s);internal/engine/epoll-raceok (380 s) andinternal/engine/iouring-raceok (491 s); the wholetest/serverH2|889|Shutdown|SSE|Stream|1165|884set non-race ok (924 s). (A first non-race run of that set had 8 failing cells while the Mac slept and two other lanes' containers ran: timing cells with 1 s windows cut by the stalled VM clock; the clean re-run is the one above.)reststep (-race,-timeout=540s) time out; the 9 slow cells are listed in.github/root-test-shards.txt(shards a and b,root-shards.sh checkok) so that the rest stays inside its timeout. CI at the merged head56ca76a: 29 pass, 2 skipping.-tags netem889) at7e7f5ce;GOOS=windowsanddarwingo vetok;apidumpok (53 packages) and its own tests;root-shards.sh checkok (65 tests in shards, a=33 b=32); size tests pass (connState 448,H2State648 bytes = the 704 malloc class,Streamunchanged).test/serverpackage, no privileges, so CI runs them all. The curl numbers above are a one-off measurement (docker, curl 8.14.1/nghttp2 1.64.0), not a cell: a CI cell that skips without curl would trip the skip rule.Mutants (each applied, the named tests run, file restored;
logs/mutant-d2-*.log)PoolDirect)ReserveSendWindowr9a, fixD removed)r9b, fixB)r9c, fixC removed)r9d)r9e)TestWriteBankIsNotADebt889; new)r9f)p8, survived after fixC untilTestWriteSmallConnShareHoldClearsIdle889), the re-check (n8, survived until its test used a 100000-byte tail)heldconnHeld) carries the same bytes, so the verdict is the same; the unit test pins the intended behaviourHot path and cost
Docker arm64
--cpus 2(a shared machine; amd64 unmeasured), main (3f8d83f) vs head, interleaved, 4 rounds x 2 runs per engine (the r8 round; the previous round: epoll 4096 +6.1% / +10.5% min), medians (minimums): the sweep is where D2 adds a per-conn cost (one more atomic, hints 8 bytes longer):r9 round (same method, 4 rounds x 2 runs; the host was busy with other lanes, load about 5, so every absolute number is 25 to 40% slower than the r8 round and the percentages are noisier): epoll 4096 37.60 us (35.06) -> 40.49 us (38.99): +7.7% (+11.2%), epoll 256 1673 -> 1822 ns (+8.9%); io_uring 4096 53.09 -> 52.88 us (-0.4%), io_uring 256 2746 -> 2718 ns (-1.0%). The r9 change is inside the verdict, not on the sweep's scan, so these are the same cost as r8 seen through a noisier machine: on epoll about +0.7 ns per conn here (+0.4 in the quiet round), on io_uring within noise. (The review measured +6.6 to +7.9% at 4096 on epoll: the same order.) At 10 000 h2 conns and 40 sweeps a second the addition is about 0.2 ms of a second. The conn and stream benchmarks (
ProcessH2Post889,StreamedResponse889,OneByteGrant911) are within the noise of the shared machine, allocations unchanged (previous round:ProcessH2Post889/single673 vs 668 ns,StreamedResponse889/closed559 vs 548 ns; the review: +2 to 3% medians, minimums equal or lower). The common send pays one atomic load ofstallNinReserveSendWindow. A stalled stream's flush takes the books' lock once:BenchmarkOneByteGrantWriteDeadline889on 61 to 73 ns, off 53 to 122 ns (noise-bound). The window verdict allocates when more than 32 streams of a conn are due at once.Release notes (draft)
WriteTimeout: -1nothing changes.math.MaxInt64write, read, read-header or idle timeout) are kept as a hundred years in every engine, so thatnow + timeoutcannot overflow and make every deadline due at once.Findings, not fixed here
AwaitSendWindow(sendRest) is bounded by fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906's absolute WriteTimeout, a buffered tail by this rate: the two policies differ, and the absolute one cuts a client that reads at a healthy rate (Go's Transport, 4 parallel 8 MiB downloads on a 1 MB/s link, 3 of 4 reset: traced above, identical on main). It needs this PR's treatment as a follow-up (an issue) or here.now): a forward clock step judges every stalled stream at once, as D1's does (r7 F5).ExpiredWritesallocates when more than 32 streams of one conn are due at once. The per-expiry budget (2 ms per conn) is the stage 1 one.Unverified
internal/protocol/h2/stream, engine-independent), and the new cells ran here on all three engines.