Skip to content

Shutdown: document what a silent conn and a head in the kernel get, per engine, and pin it (Refs #1208) - #1220

Merged
FumingPower3925 merged 7 commits into
mainfrom
shutdown-contract-1208
Nov 2, 2026
Merged

FumingPower3925 merged 7 commits into
mainfrom
shutdown-contract-1208

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1208

Per the maintainer's decision (2026-11-01): document both shapes, fix the Shutdown godoc, do not change std, add tests that pin the per-engine behaviour. No engine code changes here. The one real engine gap this found (io_uring, below) is fixed in #1224, stacked on this PR.

Revised after review r2 (Opus): three claims of the first version were false on io_uring and are corrected; the removed sentence ("a conn that has sent nothing is answered close when its first request is") is not restored (it is false on epoll, std, and on an io_uring worker with nothing to send).

Verified table (docker arm64, --cpus 2, golang:1.27.3, kernel 7.0.12-linuxkit, io_uring tier high; one tier and one arch only)

"Sends pending" = every worker has a response going out (16 clients that do not read an 8 MiB response), i.e. a server under load when Shutdown comes.

conn state when Shutdown begins std epoll io_uring, nothing to send io_uring, sends pending adaptive
idle keep-alive closed at once FIN in 1 ms FIN in ~1 ms FIN at ~252 ms (serves on for the first 250 ms of its send drain; a request before then is answered, close) the engine it runs on (epoll at start)
request in progress answered, close answered, close answered, close answered, close same
silent conn, accepted no FIN until its first request (dropped: FIN; reset only if body bytes stay unread) or net/http's 5 s: Shutdown returned at 5.3 to 6.3 s FIN in 1 ms; a later request is dropped same FIN at ~252 ms; a first request 50 ms in is answered, close (8/8) epoll
silent conn held back by TCP_DEFER_ACCEPT (default) n/a no FIN; first request reset (once the loops have closed their listeners) no FIN; first request reset first request answered, close (3/3) epoll
head in the kernel, loop reading the conn (busy loop) n/a answered, close answered, close (a pending send keeps the worker in its drain) answered, close answered, close
head in the kernel, loop held in another conn's OnConnect (24 conns it was reading) n/a answered, close dropped (FIN) answered, close answered, close
head in the kernel, accept not finished (OnConnect held) GET: FIN; POST with 64 KiB: RST; 5 B or 2 KiB body: FIN answered, close dropped (FIN, GET and POST), 40/40 answered, close answered, close

Mechanism of the io_uring "dropped" cells (review MAJOR-3, confirmed): a worker with no send pending and no parsed request exits on its first look at the shutdown, and h1RequestInProgress does not look at the kernel (epoll's twin does). OnConnect was never the axis. Fixed in #1224; after it the io_uring column reads answered in both dropped cells, and this PR's cells and godoc line flip there.

Changes

  1. Commit a768c9a (tests): test/server/shutdown_conn_shapes_1208_linux_test.go and shutdown_doc_1208_test.go (a godoc test in the style of doc_limits_891_test.go; RED on main, because the godoc says the false thing).
  2. Commit 7d75a50 (docs): Server.Shutdown godoc, README, the two native gate comments, the test: the shutdown gate e2e waits until the engine has read the upload's head (Refs #1166) #1207 test comment.
  3. Commit a12256e (tests, r2): cells A-accepted-draining, A-deferred-draining (io_uring), B-hold-other, B-accept-hold-draining, and requireDoc takes the issue tag (the new doc test prints Shutdown: a conn that sent nothing is not answered, and a request head already in the kernel is reset on std but answered on epoll; the godoc says otherwise #1208, not v1.6.0 polish: documentation #891). The doc test wants the corrected godoc and is RED on this commit alone.
  4. Commit 8a22618 (docs, r2): godoc bullets 1, 3 and 4, README, the io_uring gate comment, the test: the shutdown gate e2e waits until the engine has read the upload's head (Refs #1166) #1207 test comment, to the table above: the io_uring send-drain window, the deferred conn, "reset only for unread body bytes", and "measured on arm64, 2 CPUs, kernel 7.0, io_uring high tier" instead of "on every engine".
  5. Commit 4429b9d (flake fix): A-deferred-draining is io_uring only. On epoll, a loop busy sending closes its listener a few ms late and a first request that arrives before is answered, not reset (1 failure in 20 runs, found by my own loaded run); the godoc says so.

"The shutdown has begun" is observed, never slept for, except where noted in the cells: an idle conn's FIN (natives), the listener's refusal (std). A cell in which every conn lands on one loop (p = 2^-23) fails rather than skips, since an unlisted skip fails CI's tally. The send-pending state is read from /proc/net/tcp (the server-side send queue of each staller is non-empty), not slept for.

Cost: the _1208_ set takes about 2.9 s plain and 4.6 to 5.0 s under -race in the docker box (2 CPUs, 40 cells in parallel; the draining cells hold 16 x 8 MiB responses each), about 2 s more than the first version. Flake check on the final head: plain x20 and -race x20 with two busy loops: 720 PASS, 0 FAIL, 0 SKIP each. golangci-lint v2.14 (CI's pin, GOOS=linux, whole root module): 0 issues.

For the issue thread (not written by me)

Sentences for the docs repo (not in this PR; match the godoc as of #1224 for the head-in-the-kernel line)

  • "What a connection gets depends on what it has sent when Shutdown begins. A request in progress is read on and answered, with Connection: close, on every engine. An idle keep-alive connection is closed at once by std and epoll, and by an io_uring worker with no response going out; an io_uring worker that has one serves on for the first 250 ms of its send drain, closes its idle connections at 250 ms, and answers, with Connection: close, a request one of them sends before then."
  • "A request head that has reached the kernel without the server having read it is answered, with Connection: close, on epoll and adaptive, and on io_uring (from iouring: a worker with nothing to send answers a request already in the kernel before it exits (Refs #1166 #1208) #1224; before, only while its worker had a response going out or a request in progress); std (net/http) drops it: the connection is closed without an answer, and reset if body bytes are still unread (a body of 2 KiB or less is read with the head and the close is a FIN)."
  • "A connection that has sent nothing is not answered by std, or by epoll and an io_uring worker with no response going out, and a request it sends later is dropped there; those engines close it at once when they have accepted it. By default the kernel holds such a connection back for about a second (TCP_DEFER_ACCEPT, see DisableDeferAccept): there Shutdown neither closes it nor lets a request through (the first request gets a reset). An io_uring worker that has a response going out keeps accepting for its first 250 ms and answers a first request in that window, held back or not. std keeps the connection until its first request is read (and dropped) or net/http closes it at five seconds, so a client that connects and says nothing can hold Shutdown on std for about five seconds, within the Shutdown context."
  • "Measured on arm64, in docker with 2 CPUs, on a 7.0 kernel, io_uring's high tier."
  • Keep-alive rule (epoll, io_uring: every HTTP/1 response after Shutdown carries Connection: close and ends its conn (v1.6.0, Refs #885 #884) #1169) and the budget are unchanged: every HTTP/1 response written after Shutdown began ends its connection; the budget is the Shutdown context (and ShutdownTimeout for a cancel of StartWithContext).

…rnel, per engine (#1208)

Refs #1208. Characterization tests: shape A (a conn that sent nothing) and
shape B (a request head in the kernel, unread) on std, epoll, io_uring and
adaptive, with the cells tagged as in the issue's table. The doc test is RED:
the Shutdown godoc still says a silent conn is answered close when its first
request is, and says nothing of the head in the kernel.
…nel get, per engine (#1208)

Refs #1208. The Shutdown godoc said a conn that has sent nothing is answered
close when its first request is; no engine does. It now lists what each state of
a conn gets when Shutdown begins (idle, in progress, head in the kernel, silent),
per engine, as measured by the tests of the previous commit. README and the two
native gate comments say the same; the #1207 test comment no longer calls the
native side unverified.
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…; parametrize requireDoc's tag (#1208)

Refs #1208. New cells: A-accepted-draining and A-deferred-draining (a response
going out on every worker: epoll closes at once, io_uring serves on for 250 ms
and answers a first request), B-hold-other (the loop is held in another conn's
OnConnect: epoll answers, io_uring drops), B-accept-hold-draining (io_uring
answers when its worker has a response going out). requireDoc takes the issue tag
for its failures. The doc test now wants the corrected godoc (RED here).
…xis of a head in the kernel (#1208)

Refs #1208. Review r2 of #1220: an io_uring worker that has a response going out
serves on for the first 250 ms of its send drain, so its idle and silent conns
are closed at 250 ms and a first request before then is answered (also for a
conn TCP_DEFER_ACCEPT holds back); a head in the kernel is answered on io_uring
only while the worker is in its drain, a worker with nothing to send exits at
once (an OnConnect callback is one way to the window, not the axis); std's reset
needs unread body bytes (2 KiB gives a FIN). 'Measured on every engine' becomes
what was measured. README and the two native gate comments say the same.
…t conn's FIN (#1208)

Refs #1208. A flake of the previous commit (1 in 20 runs): an epoll loop busy
sending closes its listener a few ms late, and a held-back conn's first request
that arrives before is answered, not reset. The godoc says so.
…serve in; A-accepted-draining checks Connection: close on io_uring's answer (#1208)

Refs #1208. Two NITs of the delta review of #1220.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 2, 2026 19:29
@FumingPower3925
FumingPower3925 merged commit 834d9de into main Nov 2, 2026
34 checks passed
@FumingPower3925
FumingPower3925 deleted the shutdown-contract-1208 branch November 2, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant