Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
b9b1247
tests: a pool handler's wait for window is bounded by a minimum rate,…
FumingPower3925 Oct 30, 2026
3e6ba49
tests: a bursty reader whose frames are further apart than ReadTimeou…
FumingPower3925 Oct 30, 2026
f921e60
tests: the bursty reader is a guard (green on main: the pool gate shi…
FumingPower3925 Oct 30, 2026
dae6d04
tests: the connection window is open in the raw-client cells, so that…
FumingPower3925 Oct 30, 2026
4ca25ca
h2: a pool handler's wait for window is a minimum rate, judged by the…
FumingPower3925 Oct 30, 2026
677bc1f
tests: units for the wait's books, the seed, the allowance, the fallb…
FumingPower3925 Oct 30, 2026
148eb7a
tests: the wedge cell puts the connection over the budget (the lock i…
FumingPower3925 Oct 30, 2026
5cf7d59
tests: the wedge cell repeats the trial on fresh connections (a healt…
FumingPower3925 Oct 30, 2026
500b817
bench: the connection-window reader with the wait as a rate (Refs #1170)
FumingPower3925 Oct 30, 2026
db65b88
docs: Config.WriteTimeout on a pool handler's wait as a rate, the all…
FumingPower3925 Oct 30, 2026
58a7fe0
tests: lint (gofmt, nolintlint, revive), net/http's Transport with un…
FumingPower3925 Oct 30, 2026
07246e9
stream: the Manager stays in its size class (connHigh in the padding,…
FumingPower3925 Oct 30, 2026
934fd54
tests: the Go-client cells run one WriteTimeout at a time (five at on…
FumingPower3925 Oct 30, 2026
c1fa8e4
Merge remote-tracking branch 'origin/main' into l19s3-pool-wait-rate
FumingPower3925 Oct 30, 2026
128ca1f
shards: the weight of the stage 3 cells
FumingPower3925 Oct 30, 2026
781cc40
tests: CI starved the cells that race the loops against a 0.5 s thres…
FumingPower3925 Oct 30, 2026
6f9f4b1
engines: a parked loop clears its stamp (an adaptive sub-engine in st…
FumingPower3925 Oct 30, 2026
9a59192
tests: an idle run that begins in a tail keeps an allowance, and the …
FumingPower3925 Oct 30, 2026
149704a
stream: a connection that moved carries through one empty window (the…
FumingPower3925 Oct 30, 2026
26c911a
tests, h2: the server says why a pool handler's wait was reset (Debug…
FumingPower3925 Oct 30, 2026
303726e
stream: the horizon is the windows the bank covers (a connection that…
FumingPower3925 Oct 30, 2026
bd1f87d
docs: Config.WriteTimeout (cap 12, the horizon, the batch, the floor,…
FumingPower3925 Oct 30, 2026
6dc8597
tests: the measurement probe of the shapes is not part of the PR
FumingPower3925 Oct 30, 2026
00ad43f
tests: the paused/dribbled stream of D2 has the bound of option B (me…
FumingPower3925 Oct 30, 2026
ad534c9
Merge branch 'main' of github.com:goceleris/celeris into l19s3-pool-w…
FumingPower3925 Oct 31, 2026
3e4fe36
tests: the horizon windows are idle windows (the 17-window bound, rev…
FumingPower3925 Oct 31, 2026
2b287c5
docs: the four-window price of the horizon, io_uring at 1 s, the head…
FumingPower3925 Oct 31, 2026
9831d4f
tests: lint (QF1002)
FumingPower3925 Oct 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/root-test-shards.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
# Add a test when it takes 3 s or more; keep the shards within ~10% of each other (the CI seconds below).
#
# shard a: ~221 s of CI test time, shard b: ~218 s; the rest: ~114 s (incl. TestAdaptiveSettledRouteRetime592, 37 s).
# After the write-deadline cells of celeris#889 and #1170: shards a and b ~360 s, shard c ~370 s (unit run 41114943307 had a and b at 519 and 497 s of 660 s before shard c).

a TestUnmatchedChainGoexitIsLoud848
a TestShutdownHooksRunAfterTheDrain
Expand Down Expand Up @@ -73,6 +74,20 @@ b TestH2ConnWindowStarvedStreamsAreJudgedPerConnection889
b TestH2FlowingDownloadAndQuietStreamAreNotCut889
b TestH2SilentClientWithAStalledResponseKeepsTheConnUntilTheStreamIsReset889

# The pool handler's wait for window as a minimum rate (celeris#1170, stage 3), a shard of its own: 5 to 136 s each, 430 s together.
# (Shards a and b ran at 79% of their timeout with these in them: review r11 N4.)
c TestH2GoTransportParallelDownloadsOnASlowLinkAreNotCut1170
c TestH2PoolWaiterHoldingALockDoesNotWedgeTheServer1170
c TestH2PoolWaiterThatGrantsNothingIsReset1170
c TestH2PoolWaiterLoopBlockedThreshold1170
c TestH2GoTransportManyParallelDownloadsAreNotCut1170
c TestH2GoTransportQueueBeyondTheCapIsCut1170
c TestH2PoolWaiterBurstyReaderIsNotClosedByReadTimeout1170
c TestH2PoolWaiterOfASilentClientEndsTheConnection1170
c TestH2PoolWaiterDribblerIsReset1170
c TestH2PoolStreamWriterDribblerIsReset1170
c TestH2PoolWaiterAfterTheLoopsParked1170

# The rest step of the Unit job (test/server, -race, -timeout=540s) was at 539.2 s on the merge of #1169 (main alone: 501 s), 0.8 s from its timeout:
# the 14 slowest tests of the root packages still in it (8 to 34 s each, 216 s of CI test time) move to the shards, which have 300 s of room.
a TestShutdownConnCloseContract884
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1463,8 +1463,10 @@ jobs:
# the runner today), because a skip is absent, not a pass.
#
# -timeout=660s: the shards took 214.7 s (a) and 225.5 s (b) on the runner (run
# 38348908017; 215.3 s and 220.5 s on a linux/arm64 container with 4 CPUs), so
# 660 s keeps a margin of 2.9x or more.
# 38348908017; 215.3 s and 220.5 s on a linux/arm64 container with 4 CPUs). With the
# write-deadline cells (celeris#889, #1170) in a and b they ran at 519 s and 497 s
# (79% of the timeout); the cells of #1170 have a shard of their own (c) now, and
# a, b and c are about 360 s each, a margin of 1.8x.
unit-root-shard:
name: Unit root shard ${{ matrix.shard }}
runs-on: ubuntu-latest
Expand All @@ -1474,7 +1476,7 @@ jobs:
matrix:
# Every shard named in .github/root-test-shards.txt (the guard in the
# `unit` job and in test-coverage.yml fails when the two differ).
shard: [a, b]
shard: [a, b, c]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/test-coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -286,11 +286,13 @@ jobs:
# own 8 MiB memlock). Each shard writes a profile of the root package for its
# tests and uploads it under the `root` flag; the `coverage` job uploads the
# rest under the same flag, and Codecov adds the uploads of a commit together
# (codecov.yml waits for all seven). The shard's tests must each run and PASS
# (codecov.yml waits for all eight). The shard's tests must each run and PASS
# by name, as in ci.yml. Not a required check, like the job above.
#
# -timeout=660s: the shards took 215.5 s (a) and 221.2 s (b) under coverage on
# the runner (run 38348908034), so 660 s keeps a margin of 3x.
# the runner (run 38348908034). With the write-deadline cells (celeris#889, #1170) in
# a and b they ran at 523 s and 497 s (79% of the timeout); the cells of #1170 have
# a shard of their own (c) now, and a, b and c are about 360 s each, a margin of 1.8x.
coverage-root-shard:
name: Coverage root shard ${{ matrix.shard }}
runs-on: ubuntu-latest
Expand All @@ -300,7 +302,7 @@ jobs:
matrix:
# Every shard named in .github/root-test-shards.txt (the guard in the
# `coverage` job's root step fails when the two differ).
shard: [a, b]
shard: [a, b, c]
permissions:
contents: read # checkout
id-token: write # Codecov upload over OIDC, as the job above
Expand Down
10 changes: 5 additions & 5 deletions codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ codecov:
require_ci_to_pass: false
notify:
wait_for_ci: false
# Seven uploads per commit: root (the coverage job's), the two root-package
# Eight uploads per commit: root (the coverage job's), the three root-package
# shards (coverage-root-shard jobs, also flagged root; celeris#890),
# compress, metrics, otel, protobuf. Waiting for all seven keeps Codecov
# compress, metrics, otel, protobuf. Waiting for all eight keeps Codecov
# from posting a partial, misleading number. The root upload carries two
# profile files (the root step's and ./test/server's, which run with
# -coverpkg on the root package); that is still one upload.
after_n_builds: 7
after_n_builds: 8

coverage:
precision: 2
Expand Down Expand Up @@ -142,9 +142,9 @@ comment:
# One comment per pull request, edited in place on every push.
behavior: default
require_changes: false
# The same seven uploads as codecov.notify.after_n_builds above: a comment
# The same eight uploads as codecov.notify.after_n_builds above: a comment
# posted after five would show a partial number.
after_n_builds: 7
after_n_builds: 8

github_checks:
# No per-line "not covered" annotations in the Files tab. Many io_uring and
Expand Down
69 changes: 53 additions & 16 deletions config.go
Original file line number Diff line number Diff line change
Expand Up @@ -376,8 +376,10 @@ type Config struct {
// client's HTTP/2 flow-control window is not parked (nothing is queued on
// the socket); WriteTimeout bounds its stream in its own right, as a
// minimum rate, on the native engines: in each WriteTimeout window a stream
// that holds buffered bytes must have had at least 64 KiB of them (or all
// it held, if less) sent, released by the client's WINDOW_UPDATEs, or it is
// that holds bytes waiting for the window, buffered or still held by a handler
// on the worker pool that is blocked in its write, must have had at least
// 64 KiB of them (or all it held, if less) sent, released by the client's
// WINDOW_UPDATEs, or it is
// reset with RST_STREAM(INTERNAL_ERROR), as net/http resets a stream at its
// write deadline. There is no ceiling, where std's deadline is absolute: a
// stream that releases at least that on average, lagging the rate by no
Expand All @@ -389,22 +391,39 @@ type Config struct {
// updates stays under about five WriteTimeouts: half the window divided by the
// rate it reads at. When the connection as a whole releases at least 64 KiB
// in a window, a stream that takes less (parallel downloads share the
// client's socket) is held while it takes half its fair share of it, and for
// three windows in a row when it takes less: a download queued behind its
// siblings' bytes in the client's socket is cut where the bytes ahead of it
// take longer than three WriteTimeouts to read (N streams of W bytes in flight
// on a link of L bytes/s: N x W / L), so a server whose clients are on slow
// links should set WriteTimeout to 5s or more. When the connection itself
// releases less than 64 KiB while it holds more, the first window is the
// verdict. The end of a response is given grace: a stream that holds less than
// client's socket) is held while it takes half its fair share of it, and,
// when it takes less, for as many windows in a row as the client needs to read
// what is in flight on the connection at the rate the connection moves (the
// bytes sent and not yet granted back at the connection level over what the
// connection released in the window), at least three and at most twelve. The
// connection counts as moving for four windows after its other streams last
// released 64 KiB, so a connection whose other streams release a floor only once
// in four windows keeps its stalled streams for as long as the allowance lasts: a
// client that returns window in big steps leaves some windows with neither a grant
// nor a send. Those windows count as idle ones, so the bound below holds. A
// download queued behind its siblings' bytes in the client's socket is cut where
// the bytes ahead of it take longer than that to read (N streams of W bytes in
// flight on a link of L bytes/s: N x W / L over about 16 WriteTimeouts, the cut
// beginning between 16.8 and 21), so a server whose clients are on slow links
// should set WriteTimeout to 2s or more, and 5s where the links are slower than a
// megabyte a second. Measured with Go's HTTP/2 client on epoll and adaptive: 4 x
// 8 MiB on 1 MB/s and 8 x 8 MiB on 2 MB/s complete at 1s, 2s and 5s, with about one
// window of headroom; on io_uring the engine's own send clock closes such a
// connection at 1s, so io_uring is measured from 5s. A stream that is paused, or
// dribbles one byte at a time, beside siblings that keep the connection moving is
// held for as long: up to 17 windows in all (13 for the default window), against 8
// when little is in flight. When the connection itself
// releases less than 64 KiB while it holds more, for more windows than the
// bank covers, the verdict is the first window that does not move. The end of a response is given grace: a stream that holds less than
// 64 KiB, and has had half of it sent in the window, or any stream while the
// connection as a whole holds less than 64 KiB (the last bytes of the transfer
// are on the wire, and the grant for them can wait behind its siblings' bytes in
// the client's socket for more than a window), is held for three windows more
// and reset in the fourth if nothing moves it. A stream the client never
// grants is reset after 1+k windows alone, or 4+k with siblings that keep the
// connection moving, k being the number of whole 64 KiB in the bytes in flight
// when it stalled (about the peer's initial window), at most four: 1 and 4
// connection moving and little in flight on it, k being the number of whole
// 64 KiB in the bytes in flight when it stalled (about the peer's initial
// window), at most four: 1 and 4
// WriteTimeouts for the default 64 KiB window, 5 and 8 for a window of 256 KiB or
// more. When what it has left to send is under 64 KiB (the response is a
// little longer than the window), the bytes in flight are spent a whole floor a
Expand All @@ -418,10 +437,28 @@ type Config struct {
// (WriteTimeout is its bound), so a silent client's unread response, and the
// buffer it pins, lives for up to 8 WriteTimeouts (5, with the default window)
// where it used to be dropped with the whole connection at ReadTimeout. A
// handler blocked waiting for window keeps its own, absolute, bound of
// WriteTimeout from the start of the wait, whatever the client reads: parallel
// downloads on one connection whose responses exceed the connection's 4 MiB
// output budget can lose the later ones to it. A request error read while a
// handler on the worker pool that blocks in its write for the client's window
// (a response over the connection's 4 MiB output budget, a StreamWriter's
// Write) is bounded by the same rule, judged by its own timer from the moment it
// first blocks, so that parallel downloads on one connection are not lost to a
// deadline that is absolute (celeris#906 had WriteTimeout in all from the first
// block): its write returns an error that wraps os.ErrDeadlineExceeded when the
// stream is reset. While an event loop of the server has been in one batch of
// events (all the handlers it runs, not one) for half a WriteTimeout (at least
// 250 ms, which matters only for a WriteTimeout under 500 ms: at the 60s default
// the threshold is 30s), as a synchronous handler that waits for a lock a waiting
// handler holds across its response write makes it, or any slow blocking call,
// every such wait that has lasted a WriteTimeout is bounded by WriteTimeout in
// all, as before: about 1.5 WriteTimeouts from the start of the freeze, whatever
// the client reads. The server says so once per episode, in a Warn log line
// ("the minimum rate of a pool handler's wait for window is suspended"), and
// logs the reason of each such reset at Debug level (rate, or loop-blocked). A
// client that stops reading altogether for a WriteTimeout (a pause of 1s at
// WriteTimeout 1s) loses its connection to the engines' write clock on epoll and
// io_uring alike, with or without this rule, and on io_uring so does a client that
// reads at 1 MB/s at WriteTimeout 1s (a send that does not complete for a
// WriteTimeout); use 2s or more for clients that pause, 5s for io_uring on slow
// links. A request error read while a
// response is queued (a malformed request) puts the connection on the
// closing drain below, which counts each byte the client acknowledges and
// has no floor: a client that sends one and then takes a few bytes per
Expand Down
6 changes: 6 additions & 0 deletions internal/conn/h2.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ type H2Config struct {
// wrapping os.ErrDeadlineExceeded. The engines pass the server's
// WriteTimeout. 0 = no bound.
WriteTimeout time.Duration
// LoopWatch is the engine's watch on its event loops: while one of them is
// stuck in a batch for half a WriteTimeout, a pool handler's wait for window is
// bounded by WriteTimeout in all (celeris#1170, stream.LoopWatch). Nil: never.
LoopWatch *stream.LoopWatch
// ReadTimeout is the request deadline of a stream (the server's
// ReadTimeout): a request that is not whole this long after its HEADERS is
// answered 408 and reset with RST_STREAM(NO_ERROR) (celeris#889). Absolute,
Expand Down Expand Up @@ -626,6 +630,8 @@ func NewH2State(handler stream.Handler, cfg H2Config, write func([]byte), wake *
proc.GetManager().SetHints(&s.hints)
proc.GetManager().SetRequestTimeout(cfg.ReadTimeout)
proc.GetManager().SetWriteTimeout(cfg.WriteTimeout)
s.adapter.loops = cfg.LoopWatch
s.adapter.log = cfg.Logger
proc.SetHeaderBlockTimeout(cfg.ReadHeaderTimeout)
if lg := cfg.Logger; lg != nil {
proc.OnHandlerPanic = func(id uint32, r any, stack []byte) {
Expand Down
1 change: 1 addition & 0 deletions internal/conn/h2_sweep_889_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,7 @@ func TestSweepResetsAStalledStreamWithInternalError889(t *testing.T) {
for w := 1; w <= 4; w++ {
b.SetWindowSize(65536)
m.FlushOutbound(b, func(uint32, bool, []byte) {})
m.UpdateConnectionWindow(65536) // the client reads, and returns the connection window: nothing queued ahead
res = r.st.Sweep(now+int64(w)*int64(1100*time.Millisecond), r.write)
if (w < 4) != (res.Expired == 0) {
t.Fatalf("window %d: %+v", w, res)
Expand Down
39 changes: 39 additions & 0 deletions internal/conn/h2_wait_rate_1170_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package conn

import (
"testing"
"time"

"golang.org/x/net/http2"
)

// celeris#1170 (review r2 F4): the reset of a wait that was judged goes only if the
// call cancelled the stream. A stream the peer reset between the verdict and the
// enqueue (its RST_STREAM was processed in between) gets no second RST_STREAM
// (RFC 9113 §5.1), which #906's expiry sent.
func TestWaitExpiryAfterThePeersResetSendsNoSecondRST1170(t *testing.T) {
for _, tc := range []struct {
name string
peerFirst bool
wantResets int
}{{"the server alone", false, 1}, {"the peer reset it first", true, 0}} {
t.Run(tc.name, func(t *testing.T) {
w := newWaitBound893(t, "async", time.Hour) // the handler waits; its own timer will not fire
time.Sleep(150 * time.Millisecond) // it is blocked in sendRest
s, ok := w.st.processor.GetManager().GetStream(1)
if !ok {
t.Fatal("no stream 1")
}
if tc.peerFirst {
s.Cancel() // the peer's RST_STREAM, as the processor applies it
}
w.st.adapter.resetStreamOnce(s, http2.ErrCodeInternal)
<-w.done // the cancel ends the handler's wait
w.st.DrainWriteQueue(w.write)
_, _, _, rst := w.wire1()
if len(rst) != tc.wantResets {
t.Fatalf("%d RST_STREAM frames on the wire (%v), want %d", len(rst), rst, tc.wantResets)
}
})
}
}
Loading
Loading