[hydrangeas20]

Fixes the server-shutdown failure on macOS (test_stop_server_started_by_another_kernel), hardens stop_server() against stale and reused PIDs (including pid files that cannot prove ownership), and adds GitHub Actions CI. No changes to inference, scheduling, the streaming protocol, benchmark results or dependencies.

Root cause

_alive(pid) read /proc/<pid>/stat and treated a missing file as "dead". macOS has no /proc. For a server started by an earlier notebook kernel, stop_server() therefore stopped waiting immediately after SIGTERM and checked /health while uvicorn was still shutting down. Reproduced on Linux by making /proc reads fail: stop_server() raised "something is still serving" 0.00 s after SIGTERM, although the server then exited normally.

Replacing it with plain os.kill(pid, 0) would not work either: that call succeeds on a zombie, so an exited but unreaped server would look alive until the 20 s deadline (verified).

Three further defects found while fixing it:

• Stale PIDs could hit unrelated processes. The pid file held only a PID, and stop_server() called killpg() on it without verification. A new test reproduces it: with a legacy pid file whose PID now belongs to an unrelated session leader, the old code sent it SIGTERM.
• "Port free" was inferred from /health failing, even if something still listened on the port.
• A failed or timed-out start left the pid file, _PROC and the log handle behind; the log handle was never closed.

Changes

benchmarks/colab_helpers.py (public signatures unchanged):

Area	Behaviour now
Liveness	kill(pid, 0) first (ESRCH = gone, EPERM = exists), then the process state from /proc on Linux or ps -o stat= elsewhere. Zombies count as exited. An unreadable state counts as alive, so shutdown waits rather than declaring a live process dead.
Ownership	The pid file is JSON {pid, port, start}. start is the process start time (/proc starttime or ps -o lstart=), fixed at fork. A PID is signalled only if its start time and command line (-m echorelay --port N) still match; otherwise the file is stale and is removed without signalling. A matching command line alone never authorizes a signal. A pid file with no start time (a legacy PID-only file, "start": null, or no start key) cannot confirm ownership: if its process is alive and its command line matches, stop_server() sends nothing, keeps the file unchanged and raises with manual-stop instructions; if the process has exited or its command line does not match, the file is removed without signalling. If ownership cannot be verified at all (no /proc, no ps), stop_server() raises and keeps the file.
PID pinning (Linux)	For a server recorded in the pid file, the process is pinned with a pidfd (os.pidfd_open) before its identity is checked. An identity read after the pinned process exited is not trusted (it may describe a successor with the same PID); nothing is signalled once the pinned process has exited; and exit is awaited on the pidfd with poll(), which a reused PID cannot fool. Where pidfds are unavailable (macOS, old kernels, seccomp sandboxes that refuse pidfd_open), the helpers fall back to the unpinned path. A server started by the current kernel is still handled through its Popen object (an unreaped child's PID cannot be reused).
Shutdown	SIGTERM to the process group → bounded wait (monotonic deadline, backoff polling) → identity re-checked → SIGKILL → bounded wait → raise if still alive. A child of this kernel is reaped; an already-exited child is never signalled, since its PID may be reused.
Port	After stopping, waits until nothing accepts TCP connections on 127.0.0.1:PORT, and raises otherwise, saying whether the listener answers /health.
Startup	On a crash or health timeout, the process, pid file and log handle are cleaned up before the error is raised.

Timeouts are module constants (STOP_GRACE_S=20, STOP_KILL_S=10, PORT_RELEASE_S=5, START_TIMEOUT_S=300), because start_server(**settings) forwards every keyword to the server. New imports are standard library only (select, socket, typing.IO).

tests/test_colab_helpers.py: 33 tests in total (28 new). They cover:

• Another kernel: a server started by an earlier kernel, stopped on both the /proc path and the ps path (the macOS path, simulated).
• Normal shutdown: graceful SIGTERM only, repeated stop_server(), and an already-exited server.
• Pid files: stale legacy, mismatched-start and garbage files; PID reuse detected by start time; unverifiable ownership.
• Pid files without a start time: a live process with a matching command line is never signalled, the file is kept unchanged and the error repeats on retry, for a bare-PID file, "start": null and a missing start key, on both the /proc and ps paths; such a file for an exited process is removed. Run against the previous commit (84025d6), all six of these cases fail because it signalled the lookalike process.
• pidfd (Linux): a pinned process that exits before signalling gets no signal; an identity read after the pinned exit is rejected (and shown to be accepted without the pin); the pidfd wait is bounded and wakes on exit; a refused pidfd_open falls back cleanly.
• Escalation: SIGKILL after SIGTERM is ignored (with and without a pidfd), and an unkillable process.
• Liveness: the ps-based path, the unknown-state path, and Linux zombies.
• Port and startup: a non-HTTP listener holding the port, and cleanup after a crashed or never-healthy start.

Spawned processes are killed in teardown even when a test fails. Waiting uses bounded polling and pipe handshakes, not fixed sleeps.

.github/workflows/ci.yml:

• Linux: offline suite on Python 3.11 and 3.13 × all four combinations of ECHORELAY_FINAL_PRIORITY / ECHORELAY_CANCEL_QUEUED_PARTIAL_ON_END (8 jobs, fail-fast: false).
• macOS: Python 3.13, default flags.
• Evidence: a benchmark evidence-verifier job.
• Exclusions and limits: real-model tests are excluded (-m "not whisper"); permissions: contents: read, persist-credentials: false, job timeouts, pip cache keyed on pyproject.toml; no secrets or GPU.

README.md: CI badge.

Validation

Local runs, Linux, Python 3.11.17 and 3.13.16, at 770e29c:

Command	3.11	3.13
python -m pytest -q -m "not whisper"	237 passed, 1 deselected	237 passed, 1 deselected
same, all 4 flag combinations	4/4 configs pass (237 passed each)	4/4 configs pass (237 passed each)
python -m pytest -q tests/test_colab_helpers.py	33 passed; 10/10 repeats	33 passed; 10/10 repeats
python -m benchmarks.verify_evidence	OK: 3 manifests, 170 files, 0 problems	same
git diff --check	clean	—
actionlint .github/workflows/ci.yml	no problems	—
ruff check on changed files	only the 2 pre-existing findings (SIM115: the log handle must stay open; PLW1510 in save_provenance)	—

The deselected test is the real-model Whisper test. The flag-matrix runs were made just before the final change in 770e29c (pidfd wait switched from select() to poll(), helper code only); the full suite and the helper repeats were re-run on both versions after it.

GitHub Actions (run 40150692081) on head 770e29c: 10/10 jobs completed with success: 8 Linux matrix jobs (py3.11 and py3.13 × both flags), macOS py3.13, and Benchmark evidence.

Not verified locally: macOS itself. The macOS path was exercised on Linux by disabling /proc and using ps; the real macOS run is the CI job above (passed). The pidfd tests are skipped on macOS; I could not retrieve the macOS job log through the API, so its exact skip count is unconfirmed.

Remaining risks

• macOS check-then-signal window: without pidfds, the identity check and killpg() are separate steps. Signalling the wrong process would need the server to exit and its PID to be reused within milliseconds between them.
• Linux process-group signal: the pidfd pins the server process, but killpg() still addresses the group by number. A group ID cannot be reused while any member is alive, so the window is microseconds wide and needs a full PID wraparound.
• Legacy pid files need manual cleanup: a pid file written before this change has no start time. If its server is still running, stop_server() (and therefore start_server()) refuses until the user stops it (kill -TERM -- -<pid>) and deletes the file. This is the intended fail-safe behaviour, but it is a visible change for anyone upgrading with a server running.
• No /proc and no ps: stop_server() refuses to signal and raises (fail-safe), and liveness falls back to kill(pid, 0), which cannot detect zombies.
• Start-time granularity: ps lstart has one-second resolution. A PID reused within the same second and running -m echorelay --port N would be indistinguishable on macOS. Linux uses /proc clock ticks.
• Port check scope: it covers 127.0.0.1 only, which is where the helpers bind the server.

https://claude.ai/code/session_017DfLG9q5h7eiawGNwaBswq

---

Generated by Claude Code