Skip to content

iouring: a close seam and an ownership trap; the stale closes the canary found are tests' own (Refs #1242) - #1245

Merged
FumingPower3925 merged 6 commits into
mainfrom
fix/iouring-goexit-double-close-1242
Nov 3, 2026
Merged

FumingPower3925 merged 6 commits into
mainfrom
fix/iouring-goexit-double-close-1242

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Refs #1242

What the issue guessed, and what is true

The stale-close canary (FLAKES13) printed its hit in the -v log during TestGoexitedWorkerLeavesTheGroupAndThePauseDoesNotWaitForIt848. That test runs its worker in a child process (child848); the parent only waits for it, and prints the child's output only when the child fails. So the hit was made by the parent process, in whatever ran just before. Not the worker's Goexit path.

This PR finds the closes with a deterministic trap instead of a timing canary, and the result is:

  • The engine closes each descriptor once on every worker exit path (normal stop, pause and resume, Goexit, Goexit in the pause's close, ring setup failure, first-submit failure, bind failure, panic). There is no second close in leaveGroupAfterExit, shutdown, stopAccepting, closeListenerAfterDrain or releaseFailedInit, so there is no engine change.
  • Seven tests closed a number the engine had already closed, which after the first close names whatever opens next in the process (the canary's sockets, a sibling test's engine, the Go runtime's own). Those are the stale closes, and they are fixed here.

The trap

fdCloseSeam (fd_close_seam_1242.go) is an atomic test seam under every close of an engine-owned descriptor (closeFD, closeFDErr; the sites that called unix.Close now call them; one load of a nil pointer in production, like ringWaitSeam). The test trap (fdTrap1242) does not close(2): it dup3s a placeholder pipe end over the number. That closes the file and keeps the number taken, so nothing can re-open it, and a later close of a number that still names the placeholder is the engine closing something it no longer owns, whatever else runs. No timing, no reuse race. The failure names the number and both sites:

fd 9 closed a second time with nothing in between that could have re-opened it:
    first close:  closeFD (driver.go:793) <- closeUnlessOwed (fd_lifetime.go:561) <- (*Worker).finishClose (worker.go:5830) <- (*Worker).closeConn (worker.go:5355)
    second close: closeFD (driver.go:793) <- gateRig884.func3 (shutdown_gate_884_linux_test.go:258)

TestFDTrapFlagsOnlyAStaleClose1242 and TestARemovedFDTrapClosesPlainly1242 are the trap's own controls (a removed trap closes plainly, for a worker an earlier test leaked) (a second close is reported with both sites; a number released by someone else and opened again is a first close; a number not open is reported).

Tests

  • TestEveryExitPathClosesEachDescriptorOnce1242: one child process per path (cancel with open connections, pause-resume, goexit with two pauses and resumes, goexit in the pause's close, ring setup failure, first submit failure, bind failure, a ring wait that fails for good, a lost listener) and TestPanickedWorkerClosesEachDescriptorOnce1242 (the process dies; the child writes each close and fault to stderr, the parent reads them). Green on main by design: they pin the pairing. Mutants confirm they bite: a duplicated closeFD(w.listenFD) in leaveGroupAfterExit fails goexit and the panic test; dropping w.listenFD = -1 in releaseFailedInit fails ring-setup-failure and initial-submit-failure (the ring-wait and lost-listener paths were added after the mutant runs and have none of their own), each naming both sites.
  • The stale closes are RED in the first commit (run on it: TestLeaveGroupAfterExitClosesTheRingAndTheListener884 and, through gateRig884, TestGatePassClosesAtMostTheCapPerPass884, TestGatePassSkipsHTTP2AndBusyConns884, TestGatePassWaitsForTheKernelSendQueue884, TestGatePassHoldsAConnWithARecvCQEPosted1104, TestGatePassHoldsAConnWithBytesInTheSocketQueue1104) and GREEN in the second, which makes a test close a number only while it still names the file it was taken for (fdID1242, fstat dev+inode; fdOpen704 asked only "is anything open here"). The other five sites have no trap of their own; the package-wide run below is their evidence.

Evidence (golang:1.27.3, 2 CPUs, memlock 8 MiB, -race)

  • Package-wide trap run on main's tests (every unix.Close in the test files routed through the trap, window of 256 placeholders): the only second closes with a known first close are the seven tests' (about 80 in the gate rig alone). No engine site. A first version of the trap also reported about 1,240 second closes with no recorded first close (RegisterConn, closeOpFD, closeFDsInParallel); they are the trap's own: a test registered a number the trap held as a placeholder, so F_DUPFD returned a second number naming the placeholder, where a real process gets EBADF. The trap now reports a second close only for a number it closed itself (and opFD has one close per driverConn, guarded by opFDOpen). After the second commit: none left (7 reports remain, all of closes of made-up numbers, 65537 and MaxInt32, which the tests hand to refuseAdopt/onAcceptedFD; EBADF, nothing double).
  • Canary (OPUS_CANARY=1, whole package -race -covermode=atomic, skipping the three tests the notes name): main: 2 stale closes (one after TestGatePassWaitsForTheKernelSendQueue884, one after TestLeaveGroupAfterExitClosesTheRingAndTheListener884, which is also the run's only failure); this branch: 0 and the package passes. Targeted, -count=30 of the seven tests: main 29 stale closes and 26 failures; this branch 0 and 0, with TestReportWorkerExitLeavesTheGroupBeforeTheFlag884 no longer skipped.
  • The package (go test -race -count=1 -v ./internal/engine/iouring/, as ci.yml runs it) passes on this branch with 8 MiB memlock; the two-worker variants of the setup-failure paths run with one worker there and with two in the CI iouring job.
  • TestDriverRegisterRacingShutdownReleasesEverySocket stays out of a canary run: it compares the whole fd table before and after, which the canary's own sockets change; it has no close of its own to fix.

Checked by reading, no test

The base-tier fallback in Listen (NewRing then Close per probe, once each; a failed mmap closes its ring fd once, ring.go) has no injection point. Two closes are outside the seam by design: the wake eventfd is closed by internal/wakefd, which retires the number under its lock first (#655), and a hijacked connection's descriptor is closed through the *os.File that net.FileConn was given (hijackConn). Routing that one through closeFD would leave the file's finalizer to close the number again at a later GC, after it has been reused, which is the bug this PR hunts.

Note for #1243

The seam touches worker.go in the lines that called unix.Close (no function of the shutdown send-drain look), and shutdown_gate_884_linux_test.go in gateRig884 and TestGatePassWaitsForTheKernelSendQueue884.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

…mber closed twice, and the exit paths pinned with it (Refs #1242)

Every close of an engine-owned descriptor goes through closeFD/closeFDErr
and fdCloseSeam. The trap dup3s a placeholder over each closed number, so a
later close of a number still naming it is a stale close whatever else runs.

The worker exit paths (cancel, pause and resume, Goexit, Goexit in the
pause's close, ring setup failure, first submit failure, bind failure, a
panic) close each number once. The stale closes the canary saw are two
tests' own: leaveGroupAfterExit's unit test cleans up with an open-check,
and the shutdown gate tests' rig closes what a pass closed. Both fail here.
…ot (Refs #1242)

Seven tests closed a number the engine had already closed: the cleanup of
TestLeaveGroupAfterExitClosesTheRingAndTheListener884 (an open-check, then
Ring.Close and close(2) again), the shutdown gate rig and
TestGatePassWaitsForTheKernelSendQueue884, and the close-accounting,
unregister-close, transplant egress, refused-adopt and wake-eventfd tests.
Once a number is free the next open in the process takes it, so the second
close destroys whatever that is: the canary's sockets, a sibling test's
engine still shutting down, the Go runtime's own. A test now closes a number
only while it still names the file it was taken for (fdID1242).
@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

…il-state close properly, and list the hijack close beside wakefd (Refs #1242)

A trap that has been removed closes plainly, so a worker an earlier test
leaked cannot dup3 a placeholder that is gone onto a reused number. The ring
of the leave-group test is judged by the worker's own state and by what the
number names (an io_uring file), not by an inode that rings may share. The
nil-state close test fails when finishClose leaves the descriptor open. The
unregister-close test no longer dup3s over a number that something else
took. The seam's comment names the hijack close as outside it.
… instead of dialing once (Refs #1242)

ResumeAccept returns before the workers have re-created their listeners (they
do it on their next iteration), so the dial that followed the pause-resume
sub-test's resume was refused whenever the worker was slow to run: always on
one CPU shared with other work under -race -covermode=atomic (Coverage, run
41638098277). Every dial of these tests now retries to a 15 s bound, and the
polls for an exit, a lost listener or a free port are bounded waits, not
fixed windows; a premise that is still false then fails with what was seen.
@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 3, 2026 15:53
@FumingPower3925
FumingPower3925 merged commit c81d84e into main Nov 3, 2026
31 of 32 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/iouring-goexit-double-close-1242 branch November 3, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant