[hhff]

What

A new worker or planner doesn't start while the shared machine's memory is at or over the supervisor's machine_memory_ceiling.

• machineMemory() returns { inUse, max } in bytes. It reads the cgroup v2 cap first (memory.max, memory.current, and memory.stat's file less shmem, through DEVTEAM_CGROUP). Without a cap it uses /proc/meminfo (MemTotal less MemAvailable, through DEVTEAM_MEMINFO). With neither (macOS), or when a file can't be read, it returns null and blocks nothing. machineLine now uses it, so there's one copy of the code.
• worker / planner: a new start is refused with, for example, "the machine's memory is at 27.2 of 32.0 GB (85%), at or over its 80% ceiling: start this worker when some sessions finish, or pass --force (it is logged)". --force starts it anyway, and the worker-slots act records what was overridden. A resume of a session that already ran is never refused. That includes a planner whose plan branch is only on origin, and the accounts-tick relaunches. --resume on a session that never ran counts as a new start. A resume that has to start fresh because its worktree is gone notes the memory in its act.
• New supervisor setting machine_memory_ceiling: default 0.8, valid from 0.5 to 0.95. It is not in OWNER_SETTINGS. validate reports an invalid value as an error, and the check uses 0.8 in its place, so no value can make the ceiling looser than the allowed range.
• Next step: while memory is over the ceiling, idle-capacity and idle-planning are suppressed and a memory-full finding takes their place. It says how full the memory is and suggests lowering max_active_tickets.
• process/supervisor.md §4/§5: the machine is shared; a start refused for memory means waiting, not --force; a supervisor that keeps hitting the ceiling lowers its own max_active_tickets, without asking the owner.
• Tests never read the real machine's memory: test/lib/cli.mjs, test/lib/kit.mjs and the CI job env point both paths nowhere unless a test sets its own.

Why

Several supervisors share one Render box (32 GB cgroup, 4 CPUs). Each one starts workers up to its own max_active_tickets, and nothing checks the machine as a whole. Today the box hit its memory limit and the container restarted three times. The last time, Website had about 8 Claude sessions plus an npm ci running, alongside the other fleets.

Tested

• New test/memory-ceiling.test.mjs uses a fake cgroup directory. It covers: refused over the ceiling (worker, planner, and --resume on a session that never ran); --force overrides and is logged; a real resume over the ceiling is allowed; a resume that starts fresh is allowed and noted in its act; allowed under the ceiling; the supervisor's own ceiling (0.9) is used; invalid values (0.99, 1, 2, 0.4, "0.9", null) fail validate and fall back to 0.8; shmem counts as in use; the /proc/meminfo fallback; no cgroup and no meminfo means allowed and no memory-full finding; idle-capacity is suppressed and memory-full is shown, and idle-capacity comes back under the ceiling.
• test/machine.test.mjs gains a check that machine status reads meminfo.
• Ran locally on Node 24: memory-ceiling, machine, kit, fleet-park, next-step, accounts, recovery, ooda-knowledge and dashboard. 308 passed, 1 skipped.

What Fable found (adversarial review)

Fixed:

• memory.stat's file includes shmem (tmpfs, /dev/shm), which the kernel can't drop, so in-use memory was undercounted. Now shmem is subtracted from file.
• A planner moved from another machine, whose plan branch is only on origin, was refused as a new start on --resume. It now counts as having run.
• A resume that starts fresh because its worktree is gone started over the ceiling without any record. Its act now says so.
• readText can throw on EACCES/EIO. machineMemory now returns null in that case and doesn't crash.
• Docs: the §4 table gains a memory-full row; "a resume always starts" now says a session that never ran isn't a resume; "at or over" is the accurate wording at the boundary.
• Some tests spawn the kit directly: test/lib/kit.mjs now sets the two env vars at module scope.

Not fixed, left for follow-up:

• The ceiling is per supervisor, but the memory is shared, so in practice the box's limit is the highest ceiling any supervisor on it sets (up to 0.95). A machine-level cap that the operator owns, which each supervisor's setting could only lower, would close this.
• DEVTEAM_CGROUP and DEVTEAM_MEMINFO pointed at nothing skip the check, and no act records it. This gives no more power than --force already does, but it leaves no trace.
• On cgroup v1, or when the cap is set on a parent cgroup, the kit reads the host's /proc/meminfo. This is noted in the code comment.

🤖 Generated with Claude Code

[hhff]

Reviewed at d8246c6:

• machine_memory_ceiling only gates when a new session starts. It's clamped to 0.5–0.95, and an invalid value counts as 0.8, so it grants no authority.
• --resume of a session that already ran is never refused, so box-recover relaunches after a restart are safe. --resume on a session that never ran counts as a new start.
• --force past the ceiling is written into the worker-slots act.
• CI sets DEVTEAM_CGROUP and DEVTEAM_MEMINFO to nowhere, so a busy runner can't refuse test workers. That's test-only.
• The follow-ups (a machine-wide cap the operator owns, and cgroup v1) are noted.

CI passes 9 of 9. Merging. This is the fix for today's three memory restarts on the shared box.