Skip to content

tests: the port 0 hold tests ask whether the sockets the engine opened are closed, not how many sockets of the process have the port number (v1.6.0, Refs #1104) - #1171

Merged
FumingPower3925 merged 1 commit into
mainfrom
flake-1104
Oct 29, 2026

Conversation

@FumingPower3925

Copy link
Copy Markdown
Contributor

Summary

Refs #1104. Test-only. The cause of the one CI failure is not established; this makes the next occurrence say what it is. No product bug was found.

TestPort0HoldIsReleasedWhenListenFails895 (and the ...IsHeldBetweenBeingChosenAndBeingBound895 final check), in both internal/engine/iouring and internal/engine/epoll, counted every socket of the test process whose local port equals the port the hold chose (census895). After Listen returns the hold is gone and the port is free, so that count also takes in a socket the process has on that port number that is not the engine's, and the failure ("1 sockets still bound to port 37483") could not tell the two apart. What the engine promises is narrower: every socket it opened for that Listen (the hold, the workers' / loops' listen sockets) is closed when Listen returns, and the port is bindable at once.

The tests now name the engine's sockets by inode while Listen runs (the hold in portHoldHook, each listen socket in listenAddrOf), and after Listen returns fail if any of those inodes is still open in /proc/self/fd, with the process's sockets on the port listed. A socket of the process that is not the engine's is logged with its address and does not fail. bindWithin656 / the plain bind afterwards are kept.

What I found

  • Product, by reading: on the failed-start path every worker's shutdown() / stopAccepting / releaseFailedInit closes its listen fd on the worker's thread before wg.Done, and hold.Close() (an *os.File, blocking, not in the poller) is deferred in Listen: all synchronous before Listen returns. The census reads /proc/self/fd, so a socket the kernel keeps alive behind a closed fd (the io_uring: Listen returns while its listen sockets still hold the port, for ~2 ms, until the kernel tears the ring down #896 accept-reference case) is not visible to it at all.
  • Not reproduced: 0 of 3000 runs of the failing test alone (race, --cpus 2, 8 MiB memlock), 0 of 5000 runs of it with its predecessor under 3 busy loops, one full-package -race -v pass in the CI shape (dual stack required) green. epoll: an async route's closing response is sent whole; the EBUSY timed-wait arm reaches its wait; the port 0 hold tests name the extra socket (Refs #1068, #890, #1107, #1108, #1104) #1106 reports 0 of 35 000 and about 1 CI failure in 1 100 executions.
  • At the start of the failing test the process holds 5 leftover sockets from earlier tests (3 connected ones, 2 bound-only, all on 127.0.0.1, SO_REUSEPORT 0). A kernel probe (linux-arm64, 7.0.12 linuxkit) shows the hold's bind(0) never lands on a port such a socket has (0 of 20 000, accepted children of closed listeners, client sockets, same-address listeners), but does land on a port a socket at another address has (127.0.0.2, ::1: 2 800 of 20 000 binds with 2 000 such listeners). Also P = 37483 is odd, and bind(0) prefers odd ports while connect() prefers even ones, so a stray dial taking the port after the release is unlikely. So neither explanation in the issue is shown. The new check turns a recurrence into one of two outcomes: it fails naming an engine socket (a product bug, with its fd and kind), or it passes and logs the foreign owner with its address.
  • What the new oracle no longer catches: an engine socket that never passes through portHoldHook or listenAddrOf. Each test asserts it saw at least one hold and one listen socket (the IsHeld test: one per worker/loop), so a broken recording fails instead of passing empty.

Evidence (golang:1.27.3 linux/arm64 container, -race, --ulimit memlock=8388608, --cpus 2, -test.cpu 2)

  • Old oracle, foreign socket on the port number (TestPort0FailedListenIgnoresASocketOfAnotherOwner1104, listener arm) with the old assertion put back: 20 of 20 FAIL on both engines, with the issue's message ("1 sockets still bound to port N after Listen failed, want 0"). New oracle: 30 of 30 PASS on both engines (whole TestPort0* set).
  • Mutants, 5 runs each, all caught on both engines: hold not closed (M1: 5/5 red, names the port 0 hold); a worker's / loop's listen socket not closed (iouring shutdown; epoll stopAccepting and shutdown, 5/5 red, names the listen socket); address filter of the hold recording removed (N1: the bound-with-reuseport arm, which looks exactly like the hold but sits on 127.0.0.2, 5/5 red); listenAddrOf recording disabled (N2: 5/5 red at the "named at least one of each" check). Unmutated: green.
  • Load: 2000 runs of the three tests under 3 busy loops per engine: 0 failures (the product closes everything).
  • A retried start (ring setup ENOMEM at 8 MiB opens extra holds) is why the IsHeld check is "at least", found by running it 20 times.

Checks

go vet linux/darwin/windows and -tags validation; mage apiCheck (no API change); golangci-lint 2.15 (not 2.14), also with --build-tags validation: 0 issues in the changed files (3 errcheck hits in adopt_*_test.go are on main and unrelated). Not run: a full CI-shape package pass on this head (CI will).

Out of scope, seen under load (--cpus 2, 3 busy loops, 30 runs of -run '895|896'): TestListenReleasesThePortAfterALongSendDrain{Natural,SQFull}896 failed 4 of 30 on its own guard ("the shutdown never reached cancelAccept").

…d are closed, not how many sockets of the process have the port number (v1.6.0, Refs #1104)
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 29, 2026
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between f6561b1 and 7dbe4c0.

Files selected for processing: 4
  • internal/engine/epoll/listen_port0_hold_895_linux_test.go
  • internal/engine/epoll/listen_port0_owned_1104_linux_test.go
  • internal/engine/iouring/listen_port0_hold_895_linux_test.go
  • internal/engine/iouring/listen_port0_owned_1104_linux_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

Linux tests for epoll and io_uring now track engine-owned sockets by inode. They verify those sockets close after successful and failed Listen calls, including when another process socket occupies the selected port.

Changes: Port-0 socket ownership tests

Layer / File(s) Summary
Socket ownership tracking
internal/engine/epoll/listen_port0_owned_1104_linux_test.go, internal/engine/iouring/listen_port0_owned_1104_linux_test.go
Both engines add inode-based trackers. The trackers identify listener sockets and discover matching port-hold sockets through /proc/self/fd.
Listen lifecycle checks
internal/engine/epoll/listen_port0_hold_895_linux_test.go, internal/engine/epoll/listen_port0_owned_1104_linux_test.go, internal/engine/iouring/listen_port0_hold_895_linux_test.go, internal/engine/iouring/listen_port0_owned_1104_linux_test.go
The tests record engine sockets during successful and failed Listen calls and verify they close. The failed-listen tests retain the port-rebind check.
Foreign socket cases
internal/engine/epoll/listen_port0_hold_895_linux_test.go, internal/engine/iouring/listen_port0_hold_895_linux_test.go
New subtests place a listener or an SO_REUSEPORT socket on the selected port. They check that engine-owned sockets close even when the port census finds a socket.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other · Severity of issue fixed: Low

Possibly related issues

  • Issue 1104: The changes address the test failure caused by counting a socket on the selected port without distinguishing engine-owned sockets from other process sockets.
Merge Risk: Minimal · up to 7dbe4

The updated port-0 tests have no established issue that should delay merging, subject to normal checks.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Title check Warning The title describes the test change, but it uses the invalid type "tests" instead of the required "test". It also does not use the required issue-reference format for a fix. Change the title to use an allowed type and the required issue reference, for example: "test: verify engine-owned port-0 sockets close (celeris#1104)".
Passed Passed checks: 3 passed
Check name Status Explanation
Description check Passed The description directly explains the port-0 test changes, the socket-ownership issue, validation results, and out-of-scope failures.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant