Repository navigation
test: the shutdown gate e2e waits until the engine has read the upload's head (Refs #1166) - #1207
Merged
FumingPower3925 merged 3 commits intoNov 1, 2026
Conversation
…d's head, not for a sleep (Refs #1166) TestShutdownIdleAndInProgressH1ConnsWithoutHTTP2_1165/std/slow-downloads failed once on main (CI run 41283250149, Unit root shard b): "the response to the upload: read: connection reset by peer: a request in progress was not answered". The test began the Shutdown 300 ms after it had written the upload's head and first 64 KiB, which it took for "the engine has read them". On std the conn count it waited for is bumped when the conn is accepted (the StateNew hook), before its goroutine has run; a goroutine that parses the head after Shutdown began is not serving a request in progress but a new one: net/http returns from the conn right after readRequest when the server is shutting down (server.go:2080, go1.27.3) and closes it with the body unread, an RST. With the accept path held for 700 ms (an OnConnect that sleeps for the upload's conn) the unfixed test fails 5 of 5 on std; the failure needs no stall in CI, only a loop late to a conn under -race with six 16 MiB responses being formatted. The test now waits, from /proc/net/tcp, until the client's socket has no unacknowledged byte and the server's socket no unread one for the upload and the downloads, which holds only once the engine has the head and the 64 KiB (on std the bufio buffer is 4 KiB, so the queue empties only when the handler reads the body, past the shutting-down check). The sleep stays for the downloads' responses to be in flight. What the test asserts is unchanged.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
FumingPower3925
marked this pull request as ready for review
November 1, 2026 11:14
FumingPower3925
deleted the
fix/1166-shutdown-gate-test-read-precondition
branch
November 1, 2026 11:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1166
What failed
CI run 41283250149 (push to main after #1201), job "Unit root shard b" (passed on rerun, attempt 2):
TestShutdownIdleAndInProgressH1ConnsWithoutHTTP2_1165/std/slow-downloads,shutdown_h1_gate_1165_linux_test.go:196: "the response to the upload: read tcp ...: connection reset by peer: a request in progress was not answered". The epoll, io_uring and adaptive cells and std/no-downloads passed in the same run.Cause: a test race, not an engine defect
The test began the Shutdown 300 ms after it wrote the upload's head and first 64 KiB, taking the sleep for "the engine has read them". The conn count it waits for is no proof of that on std:
activeConnsis bumped in the StateNew hook, when net/http accepts the conn and beforego c.serve. A conn goroutine that parses the head afterShutdownbegan serves a new request, not one in progress: net/http returns from the conn right afterreadRequestwhen the server is shutting down (net/http/server.go:2080, go1.27.3,if c.server.shuttingDown() { return }) and closes it with the body unread, which sends an RST. Plain net/http probe: head sent afterShutdownbegan gets "connection reset by peer"; sent before, it is answered.CI saw
read: reset, the injection belowwrite: reset: same mechanism, the difference is whether the close came after the client's ~2 MiB write fitted into the socket buffers or while it was still going.Fix (test only)
The test now waits, from
/proc/net/tcp, until for the upload and the downloads the client socket has no unacknowledged byte (tx_queue 0) and the server socket no unread one (rx_queue 0). For the upload this is exact on std: the bufio buffer is 4 KiB, so about 61 KiB stay in the kernel until the handler reads the body, which is past the shutting-down check. For the downloads (a bare GET) std's queue empties before that check, and on io_uring an empty receive queue means the kernel handed the bytes over, not that the engine parsed them; the kept 300 ms sleep covers both, and no assertion reads the downloads. What the test asserts is unchanged; the 300 ms sleep stays for the downloads' responses to be in flight. Where/proc/net/tcpcannot be read the wait is skipped with a log line.Evidence
Arm64 docker,
--cpus 2,seccomp=unconfined,--ulimit memlock=-1,-race, one container at a time.Reproduction, injection (not committed):
OnConnect: func(string){ if stallNext.CompareAndSwap(true,false){ time.Sleep(700*time.Millisecond) } }in the test Config, withstallNext.Store(true)set (after the other conns are counted) just before the upload's dial, std/slow-downloads only (-count=5; non-race build):Natural rate on main (unfixed binary, no injection): 0 failures in 40 runs of each of the 8 cells with 0 busy loops, 40 with 1, 40 with 2, plus 30 (0 loops) and 60 (2 loops) of the std cells: 0 of 170 for std/slow-downloads, 0 of 120 for each other cell. So the natural rate is under about 2% here, and CI hit it once. CI history: 6 failed ci.yml runs in the last 60, none in this test (other tests: H2 pool waiter 1170, mono wall stamps, io_uring probes 1200, lint).
Fixed binary, all 8 cells, 2 busy loops,
-count=100: 800 of 800 PASS, no "has not read" Fatalf (no native engine leaves bytes unread),golangci-lint(repo config) on./test/server/...: 0 issues.Contract boundary, for the maintainers
"In progress" in this test is "the engine has read the head". A request whose head is still in the kernel when Shutdown begins is reset on std (
server.go:2080); epoll reads and answers it at its first gate pass (poolwait_gate_885.go); io_uring not verified here. The r1 review of #1163 wrote "Std answers it" about that shape; the injection andserver.go:2080say otherwise for a head not yet parsed. Not widened here; filed separately as a contract question.