Repository navigation
iouring tests: stop the engines a failed test started, and make the ring-wait and cancel-accept seams atomic - #1230
Conversation
… cancel-accept seams are atomic (Refs #1229)
|
Caution Review failedThe pull request is closed.
Tip 🎯 Customize your risk assessments
✅ Automatic approvals, with your rules
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. CodeRabbit Verify
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
iouring tests: stop the engines a failed test started, and make the ring-wait and cancel-accept seams atomic. Refs #1229
What happened on the Coverage job
On the Coverage job (8 MiB RLIMIT_MEMLOCK, ring ENOMEM fallbacks in the log) four 896 tests (
TestListenReleasesThePortWithH2HandlersRunning{Natural,SQFull}896,...AfterALongSendDrain{Natural,SQFull}896) failed their first iteration on a 2 s dial timeout. The cause of that timeout is not established: it did not reproduce locally at 8 MiB or 4.5 MiB, nor with a forced high-to-base tier fallback. What the log does show is what the failure then did to the rest of the package.startHold896returned the engine without registering its stop, andt.Fatalfends the test withruntime.Goexit, so the engine's context was never cancelled and its worker kept running (the racing reader is created atlisten_release_stop_accepting_896_linux_test.go:121). Every later test that installs a package seam then had those workers read it: they were answered the injected EBUSY/EIO (counts off, "the retry has no ceiling" after 6 s) and readringWaitSeamwhilesetRingWaitSeam884's cleanup wrote it (DATA RACE, then "race detected during execution of test" on whatever test was running).The change
engineRun896.reapAtCleanup, registered bystart896,startHold896andstartGate1173: cancels Listen and waits for it when the test ends, first releasing the handler the test holds (holdHandler896.unblock, the gate), which Listen waits for.ringWaitSeamandcancelAcceptSeamareatomic.Pointer, ash2WriteDrainSeamand the other seams are; one load per ring wait, as before. Setters go throughsetRingWaitSeam884/setCancelAcceptSeam1229.run.reap(t), so a finished iteration's engine and its 32 MiB/bigbody are not kept until the test ends (the cleanup holds areaper896, not the engine); the/gatehandler'sstartedsignal is non-blocking, so a second handler reaches the gate the cleanup closes; the race-test helper returns when-runfilters out its subtest instead of waiting for the test timeout.dial_watch_1229_linux_test.go, test-only, reads /proc, cannot fail or hang the test): see below.On this PR's own Coverage runs (two runs, heads 1b731a3 and 0539eb7) the dial timeout recurred (four 896 tests in the first run,
...H2HandlersRunning{Natural,SQFull}896in the second, on the 2nd and 4th engine of the loop, not the first), and nothing else failed: no 884/1173 failure and no DATA RACE, which is what the change is for. The second run's message, from the first diagnostics:tier=high workers=1, and one LISTEN row for the port with an empty accept queue. That row proves little: for a LISTEN socket/proc/net/tcpshows tx_queue as always 0 and rx_queue as the accept queue only, and a connection that TCP_DEFER_ACCEPT holds back sits in the request queue and is not counted.What the dial-timeout message now holds, and how to read it
Go closes the socket of a failed dial before
DialTimeoutreturns, so the client's row is gone when the report is made; the watch therefore keeps its own snapshot of the port's rows from half the timeout on (every 250 ms, the last one with a row), and the report prints both.retrnsmtcount, is a SYN nobody answered. The snapshot also gives the number of SYN_RECV request sockets of the port and the summed softnet backlog length (a SYN that sits unprocessed)./proc/net/snmpTcp AttemptFails, EstabResets, RetransSegs, InSegs, InErrs, and the softnet dropped and time_squeeze columns (the summed backlog length is softnet_stat column 11, input and process queues together, on every kernel since 5.10).The parsers are tested on fixed /proc text (a real 15-column and a real 13-column softnet row among it), and eleven mutants of them (states swapped, name/value pairing shifted, wrong softnet column, backlog column 11 changed to 14, retransmit count parsed in base 10, delta always 0, tick not using the clock, no final gap, no snapshot, snapshot threshold of half the timeout removed) each fail a test.
The
largest gap between 10 ms ticks: 10msline in the Coverage run of head 6481ffc came from a ticker that could not see a stall (a Go ticker delivers the time the tick was scheduled for); it is not evidence of anything.What this change does not do is remove the trigger: the Coverage job is not green with this change alone. A controlled experiment on CI points at #1226 (dispatch runs of main: 4 of 12 failed the 896 family; main with #1226 reverted: 0 of 12), so #1229 stays open for that.
Not changed:
.github/workflows/test-coverage.yml,portHoldHook(read once by Listen at start), and the dial timeout itself.