fix(gates): safe_node can cap each check's memory (MEM_MB), so one heavy file cannot take down a node

[ScottMoore0]

Running safe_node.ts over BendHub ran out of memory for me. One heavy check (bend2 plus the kernel it starts) can pass 3 GB, and with several checks at once the OOM killer fires. Twice the run died and lost its results; once it took down the machine.

Change (gates/safe_node.ts only):

• Each check runs in its own process group (detached: true).
• With MEM_MB set, the runner sums the group's RSS every 0.5 s with ps -A -o pgid=,rss=, which works on macOS and Linux. Past MEM_MB it kills the group, and the file reports memory cap: the check passed N MB as an infrastructure error, like a timeout.
• With MEM_MB unset (the default), nothing is capped and verdicts are unchanged. Checks still run in their own process groups, and the runner kills every live group on exit, SIGINT, SIGTERM and SIGHUP, so no check outlives it (SIGKILL to the runner's process group can't be trapped; see the review thread).
• The 30 s deadline now kills the whole group, so a kernel that bend2 started does not outlive its check.

What I ran (Linux x86_64 under WSL2):

• MEM_MB=300 on a heavy BendHub file (hairpin.bend) and a light demo: the heavy one ends as e (memory cap: the check passed 300 MB) in about 1 s, and the light one still agrees.

• MEM_MB unset, all 1,666 tests/ and demos/*/PROOF.bend files: every verdict class matches the unpatched safe_node.ts on the same main.

• All of BendHub (4,832 files) at a 3 GB cap, 4 at a time: the run completed, where uncapped runs at 8 and 16 at a time were killed by the OOM killer. 15 files passed 3 GB and were reported as e.

• After review: with 4 heavy hub checks running, SIGTERM, SIGINT or SIGHUP to the runner leaves 0 checks behind after 1.5 s (8d6888e left all 4); a 2 s deadline leaves nothing running; with MEM_MB unset all 1,666 files keep their verdict and output text.

Not run: on the minis (macOS). The repo-gate workflow checks out main's gates/, so CI will not exercise this change.

Found and tested with Claude Code, running Claude Opus 5.5, during an audit of --verdict.

[Lorenzobattistela]

Thanks! The cap works, and having the timeout kill the whole group is a real improvement: on main, a grandchild that holds the pipe can stall a timed-out check for its full runtime. One regression needs fixing before merge.

1. Checks are orphaned when the runner dies. With detached: true (safe_node.ts:66), signals sent to the runner's group no longer reach the checks. We tested this by sending SIGTERM, SIGHUP or SIGKILL to the runner's process group. On main, bend2 and its children die at once. On this branch, bend2 and the kernel keep running after the runner exits, and they keep going past the timeout because nothing is left to kill them. safe.ts runs safe_node over ssh on the nodes, so an aborted gate or a dropped ssh session (SIGHUP) can leave heavy BendHub checks using memory on the node. That is the OOM this PR is meant to prevent. This also happens with MEM_MB unset, so the "unset, nothing changes" line in the body isn't quite accurate.

Suggested fix: keep a Set of live child pids, and on exit, SIGINT, SIGTERM and SIGHUP run process.kill(-pid, "SIGKILL") for each one before exiting.

2. Nit: process.kill(-(kid.pid ?? 0)) (:68). Because -0 === 0, this would kill the runner's own group if pid were ever undefined. It can't happen today (a failed spawn emits close, which clears the timer first), but one refactor could change that. Something like if (kid.pid) process.kill(-kid.pid, ...) else kid.kill(...) would guard it.

3. Minor: synchronous ps polling. spawnSync ps takes ~9 ms on a Mac with ~750 processes. With 8 parallel checks it blocks the event loop about 15% of the time. That's acceptable, but an async or single shared poller would be cleaner. With MEM_MB unset, ps never runs.

What we checked: with MEM_MB unset, verdicts on 219 files match main. MEM_MB=40 on a demo PROOF gives e "memory cap" in ~0.5 s, and a fake 600 MB check under MEM_MB=300 is killed in ~1 s. Repo gate 54/54 and ping 62/62 on current main. Not run: safe hub, Linux, test/perf gates.

[ScottMoore0]

Thanks for the review. I've addressed all three in 397a5b9 - re: the orphans, the runner keeps its live check groups and kills each one on exit, SIGINT, SIGTERM and SIGHUP before exiting. Re: -0, process.kill(-pid) runs only when kid.pid is set, otherwise kid.kill("SIGKILL"). Re: polling, there is one shared asynchronous ps every 500 ms which covers all live groups, at most one in flight, and it doesn't start with MEM_MB unset.

When I reran on Linux, with MEM_MB unset, all 1,666 of the tests and demos maintain their verdict and output text against main's runner, while MEM_MB=300 kills a heavy hub check in around 1 s.

This doesn't cover SIGKILL to the runner's process group, as the runner can't trap SIGKILL, and the checks sit in their own groups, thus they survive it, whereas on main they die with the group. SIGKILL to the runner's pid alone orphans them on main also. If that case matters, the other approach is to drop 'detached' and kill by process tree via ps -o pid=,ppid= which keeps the checks in the runner's group at the cost of a small race with a process spawned mid-kill. I am happy to switch, if you'd prefer that.