Skip to content

test: the shutdown gate e2e waits until the engine has read the upload's head (Refs #1166) - #1207

Merged
FumingPower3925 merged 3 commits into
mainfrom
fix/1166-shutdown-gate-test-read-precondition
Nov 1, 2026
Merged

FumingPower3925 merged 3 commits into
mainfrom
fix/1166-shutdown-gate-test-read-precondition

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1166

What failed

CI run 41283250149 (push to main after #1201), job "Unit root shard b" (passed on rerun, attempt 2): TestShutdownIdleAndInProgressH1ConnsWithoutHTTP2_1165/std/slow-downloads, shutdown_h1_gate_1165_linux_test.go:196: "the response to the upload: read tcp ...: connection reset by peer: a request in progress was not answered". The epoll, io_uring and adaptive cells and std/no-downloads passed in the same run.

Cause: a test race, not an engine defect

The test began the Shutdown 300 ms after it wrote the upload's head and first 64 KiB, taking the sleep for "the engine has read them". The conn count it waits for is no proof of that on std: activeConns is bumped in the StateNew hook, when net/http accepts the conn and before go c.serve. A conn goroutine that parses the head after Shutdown began serves a new request, not one in progress: net/http returns from the conn right after readRequest when the server is shutting down (net/http/server.go:2080, go1.27.3, if c.server.shuttingDown() { return }) and closes it with the body unread, which sends an RST. Plain net/http probe: head sent after Shutdown began gets "connection reset by peer"; sent before, it is answered.

CI saw read: reset, the injection below write: reset: same mechanism, the difference is whether the close came after the client's ~2 MiB write fitted into the socket buffers or while it was still going.

Fix (test only)

The test now waits, from /proc/net/tcp, until for the upload and the downloads the client socket has no unacknowledged byte (tx_queue 0) and the server socket no unread one (rx_queue 0). For the upload this is exact on std: the bufio buffer is 4 KiB, so about 61 KiB stay in the kernel until the handler reads the body, which is past the shutting-down check. For the downloads (a bare GET) std's queue empties before that check, and on io_uring an empty receive queue means the kernel handed the bytes over, not that the engine parsed them; the kept 300 ms sleep covers both, and no assertion reads the downloads. What the test asserts is unchanged; the 300 ms sleep stays for the downloads' responses to be in flight. Where /proc/net/tcp cannot be read the wait is skipped with a log line.

Evidence

Arm64 docker, --cpus 2, seccomp=unconfined, --ulimit memlock=-1, -race, one container at a time.

Reproduction, injection (not committed): OnConnect: func(string){ if stallNext.CompareAndSwap(true,false){ time.Sleep(700*time.Millisecond) } } in the test Config, with stallNext.Store(true) set (after the other conns are counted) just before the upload's dial, std/slow-downloads only (-count=5; non-race build):

  • unfixed test: 5 of 5 FAIL ("the rest of the upload, sent ~390 ms into the shutdown: ... connection reset by peer ... (celeris#1166)")
  • fixed test: 5 of 5 PASS (the wait rides out the stall)

Natural rate on main (unfixed binary, no injection): 0 failures in 40 runs of each of the 8 cells with 0 busy loops, 40 with 1, 40 with 2, plus 30 (0 loops) and 60 (2 loops) of the std cells: 0 of 170 for std/slow-downloads, 0 of 120 for each other cell. So the natural rate is under about 2% here, and CI hit it once. CI history: 6 failed ci.yml runs in the last 60, none in this test (other tests: H2 pool waiter 1170, mono wall stamps, io_uring probes 1200, lint).

Fixed binary, all 8 cells, 2 busy loops, -count=100: 800 of 800 PASS, no "has not read" Fatalf (no native engine leaves bytes unread), golangci-lint (repo config) on ./test/server/...: 0 issues.

Contract boundary, for the maintainers

"In progress" in this test is "the engine has read the head". A request whose head is still in the kernel when Shutdown begins is reset on std (server.go:2080); epoll reads and answers it at its first gate pass (poolwait_gate_885.go); io_uring not verified here. The r1 review of #1163 wrote "Std answers it" about that shape; the injection and server.go:2080 say otherwise for a head not yet parsed. Not widened here; filed separately as a contract question.

…d's head, not for a sleep (Refs #1166)

TestShutdownIdleAndInProgressH1ConnsWithoutHTTP2_1165/std/slow-downloads failed
once on main (CI run 41283250149, Unit root shard b): "the response to the
upload: read: connection reset by peer: a request in progress was not answered".

The test began the Shutdown 300 ms after it had written the upload's head and
first 64 KiB, which it took for "the engine has read them". On std the conn
count it waited for is bumped when the conn is accepted (the StateNew hook),
before its goroutine has run; a goroutine that parses the head after Shutdown
began is not serving a request in progress but a new one: net/http returns from
the conn right after readRequest when the server is shutting down
(server.go:2080, go1.27.3) and closes it with the body unread, an RST. With the
accept path held for 700 ms (an OnConnect that sleeps for the upload's conn) the
unfixed test fails 5 of 5 on std; the failure needs no stall in CI, only a loop
late to a conn under -race with six 16 MiB responses being formatted.

The test now waits, from /proc/net/tcp, until the client's socket has no
unacknowledged byte and the server's socket no unread one for the upload and the
downloads, which holds only once the engine has the head and the 64 KiB (on std
the bufio buffer is 4 KiB, so the queue empties only when the handler reads the
body, past the shutting-down check). The sleep stays for the downloads'
responses to be in flight. What the test asserts is unchanged.
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 11:14
@FumingPower3925
FumingPower3925 merged commit 02e02bf into main Nov 1, 2026
31 of 32 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/1166-shutdown-gate-test-read-precondition branch November 1, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant