Skip to content

iouring tests: stop the engines a failed test started, and make the ring-wait and cancel-accept seams atomic - #1230

Merged
FumingPower3925 merged 11 commits into
mainfrom
fix/iouring-test-enomem-race-1229
Nov 3, 2026
Merged

FumingPower3925 merged 11 commits into
mainfrom
fix/iouring-test-enomem-race-1229

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

iouring tests: stop the engines a failed test started, and make the ring-wait and cancel-accept seams atomic. Refs #1229

What happened on the Coverage job

On the Coverage job (8 MiB RLIMIT_MEMLOCK, ring ENOMEM fallbacks in the log) four 896 tests (TestListenReleasesThePortWithH2HandlersRunning{Natural,SQFull}896, ...AfterALongSendDrain{Natural,SQFull}896) failed their first iteration on a 2 s dial timeout. The cause of that timeout is not established: it did not reproduce locally at 8 MiB or 4.5 MiB, nor with a forced high-to-base tier fallback. What the log does show is what the failure then did to the rest of the package. startHold896 returned the engine without registering its stop, and t.Fatalf ends the test with runtime.Goexit, so the engine's context was never cancelled and its worker kept running (the racing reader is created at listen_release_stop_accepting_896_linux_test.go:121). Every later test that installs a package seam then had those workers read it: they were answered the injected EBUSY/EIO (counts off, "the retry has no ceiling" after 6 s) and read ringWaitSeam while setRingWaitSeam884's cleanup wrote it (DATA RACE, then "race detected during execution of test" on whatever test was running).

The change

  • engineRun896.reapAtCleanup, registered by start896, startHold896 and startGate1173: cancels Listen and waits for it when the test ends, first releasing the handler the test holds (holdHandler896.unblock, the gate), which Listen waits for.
  • ringWaitSeam and cancelAcceptSeam are atomic.Pointer, as h2WriteDrainSeam and the other seams are; one load per ring wait, as before. Setters go through setRingWaitSeam884 / setCancelAcceptSeam1229.
  • Failing-first: the first commit adds the tests and they fail on it (the engine helpers' engines are still running after their test ended, every time; the two seam late-reader tests report a DATA RACE under -race in most runs, not in every one); the second commit fixes them.
  • Loops and gate (c0ba6fa): the 896 iteration helpers defer run.reap(t), so a finished iteration's engine and its 32 MiB /big body are not kept until the test ends (the cleanup holds a reaper896, not the engine); the /gate handler's started signal is non-blocking, so a second handler reaches the gate the cleanup closes; the race-test helper returns when -run filters out its subtest instead of waiting for the test timeout.
  • Diagnostics (dial_watch_1229_linux_test.go, test-only, reads /proc, cannot fail or hang the test): see below.

On this PR's own Coverage runs (two runs, heads 1b731a3 and 0539eb7) the dial timeout recurred (four 896 tests in the first run, ...H2HandlersRunning{Natural,SQFull}896 in the second, on the 2nd and 4th engine of the loop, not the first), and nothing else failed: no 884/1173 failure and no DATA RACE, which is what the change is for. The second run's message, from the first diagnostics: tier=high workers=1, and one LISTEN row for the port with an empty accept queue. That row proves little: for a LISTEN socket /proc/net/tcp shows tx_queue as always 0 and rx_queue as the accept queue only, and a connection that TCP_DEFER_ACCEPT holds back sits in the request queue and is not counted.

What the dial-timeout message now holds, and how to read it

Go closes the socket of a failed dial before DialTimeout returns, so the client's row is gone when the report is made; the watch therefore keeps its own snapshot of the port's rows from half the timeout on (every 250 ms, the last one with a row), and the report prints both.

  • Near the deadline: a client in SYN_SENT, with its retrnsmt count, is a SYN nobody answered. The snapshot also gives the number of SYN_RECV request sockets of the port and the summed softnet backlog length (a SYN that sits unprocessed).
  • After the timeout: LISTEN alone, nothing in SYN_RECV, means the listener never saw the SYN. A client in FIN_WAIT or TIME_WAIT, a server-side ESTABLISHED, CLOSE_WAIT or SYN_RECV row, or a LISTEN row with rx_queue above 0, means the handshake got through and the test did not see it.
  • Deltas over the dial: TcpExt ListenDrops, ListenOverflows, TCPReqQFullDrop, TCPReqQFullDoCookies, TCPDeferAcceptDrop, TCPBacklogDrop, TCPMemoryPressures, PFMemallocDrop, IpExt InNoRoutes, /proc/net/snmp Tcp AttemptFails, EstabResets, RetransSegs, InSegs, InErrs, and the softnet dropped and time_squeeze columns (the summed backlog length is softnet_stat column 11, input and process queues together, on every kernel since 5.10).
  • The largest gap between the ticks of a 10 ms ticker running during the dial, measured with the clock (a Go ticker delivers the time the tick was scheduled for, so its value would show a 10 ms gap after any stall) and closed with the gap since the last tick when the watch stops. A gap of a second or more means the test process or the VM was not running.

The parsers are tested on fixed /proc text (a real 15-column and a real 13-column softnet row among it), and eleven mutants of them (states swapped, name/value pairing shifted, wrong softnet column, backlog column 11 changed to 14, retransmit count parsed in base 10, delta always 0, tick not using the clock, no final gap, no snapshot, snapshot threshold of half the timeout removed) each fail a test.

The largest gap between 10 ms ticks: 10ms line in the Coverage run of head 6481ffc came from a ticker that could not see a stall (a Go ticker delivers the time the tick was scheduled for); it is not evidence of anything.

What this change does not do is remove the trigger: the Coverage job is not green with this change alone. A controlled experiment on CI points at #1226 (dispatch runs of main: 4 of 12 failed the 896 family; main with #1226 reverted: 0 of 12), so #1229 stays open for that.

Not changed: .github/workflows/test-coverage.yml, portHoldHook (read once by Listen at start), and the dial timeout itself.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 2, 2026
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@codecov

codecov Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/engine/iouring/worker.go 87.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 3, 2026 05:21
@FumingPower3925
FumingPower3925 merged commit dc925bd into main Nov 3, 2026
32 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/iouring-test-enomem-race-1229 branch November 3, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant