ci: cap runner container memory

CLAUDE.md

@@ -71,7 +71,7 @@ Since every runner is Ubuntu 24.04, GNU shell is fine — but **arch must never
 
 To pin a job to one host, add the runner's automatic arch label: `runs-on: [self-hosted, wavekat-ci, X64]` or `..., ARM64]`.
 
-**No GitHub cloud caches on `wavekat-ci`** — no `cache: npm`/`cache: pnpm` on `setup-node`, no `Swatinem/rust-cache`, no `actions/cache` for dependency stores. The runner volume already keeps the pnpm store and cargo dirs warm, and the cloud round-trip goes over the runners' own uplink: `wavekat-platform` once spent 10 of a 15-minute timeout restoring a 255 MB pnpm tarball into a store that already had every package. Details in `docs/06-self-hosted-runners.md`.
+**No GitHub cloud caches on `wavekat-ci`** — no `cache: npm`/`cache: pnpm` on `setup-node`, no `Swatinem/rust-cache`, no `actions/cache` for dependency stores. The runner volume already keeps the pnpm store and cargo dirs warm, so the cloud round-trip is pure cost — slow downloads of packages already on disk. Runner containers are also memory-capped (`RUNNER_MEMORY`); `Killed`/exit 137 means the host or cap is too small, not a code bug. Details in `docs/06-self-hosted-runners.md`.
 
 ## SEO & GEO — every new page must be both
 

docs/06-self-hosted-runners.md

@@ -197,13 +197,9 @@ no `actions/cache` for dependency stores. The runners already keep that state
 on disk, and the cloud round-trip goes through the runners' own uplink instead of
 GitHub's own network.
 
-The cost is not theoretical. `wavekat-platform` run 34731393671 spent
-**10 minutes** of a 15-minute job timeout in `Setup Node`, downloading a
-255 MB pnpm-store tarball at ~0.4 MB/s — and the `pnpm install` that followed
-reused 325 of 325 packages from `/home/runner/runner/.pnpm-store`, the store
-that was already on the volume. `Swatinem/rust-cache` in `wavekat-asr` costs
-up to ~2.5 minutes per job the same way, saving a target dir that
-`CARGO_TARGET_DIR` already keeps on the volume.
+The pnpm store and cargo dirs are already on the volume, so the cache step only
+adds a slow download: on a large workspace it can take most of a job's timeout
+restoring a tarball of packages that `pnpm install` then finds already present.
 
 Where the warmth actually lives:
 
@@ -217,6 +213,37 @@ A workflow that also runs on GitHub-hosted runners can keep its cache there
 and skip it here — `wavekat-voice/release.yml` does
 `cache: ${{ runner.environment == 'github-hosted' && 'pnpm' || '' }}`.
 
+### Memory: cap every container, size the count to the host
+
+`Killed` / `exit code 137` in a step (usually `pnpm install` or a vitest
+shard) is the kernel's OOM killer, not the job. Confirm on the host with
+`journalctl -k | grep "Out of memory"`.
+
+Uncapped containers can each use all of the host's RAM, so a busy pool
+overcommits it and the kernel picks a victim host-wide — any container, or
+anything else running on the machine. Removing the cloud caches (above) makes
+this easier to hit, because jobs no longer spend their first minutes on
+network I/O and reach their memory peak together.
+
+`setup-gha-runners-docker.sh` passes `--memory` with swap disabled.
+`RUNNER_MEMORY` defaults to `(host RAM − RUNNER_HOST_RESERVE_GB) / COUNT`,
+reserve 4 GB, and warns below 3 GB, which is roughly what a heavy Node test
+job needs. A job that outgrows its cap dies inside its own cgroup instead of
+taking a neighbour down. If the default comes out under 3 GB, lower
+`RUNNER_COUNT` rather than the cap.
+
+To shrink a host, remove the top instances first (this de-registers them,
+so it needs a remove token — `gh auth refresh -s admin:org`), then re-create
+the rest in place:
+
+```sh
+RUNNER_INSTANCES="3 4" ./scripts/uninstall-gha-runners-docker.sh
+RUNNER_COUNT=2 RUNNER_KEEP_VOLUME=1 RUNNER_SKIP_BUILD=1 ./scripts/setup-gha-runners-docker.sh
+```
+
+The setup script warns if instances above `RUNNER_COUNT` are still installed,
+because lowering the count alone leaves them running.
+
 ### Re-creating the runners without re-registering them
 
 Changing a `docker run` flag means re-creating the containers, and the setup

scripts/setup-gha-runners-docker.sh

@@ -21,6 +21,10 @@
 #   RUNNER_COUNT=6 RUNNER_PREFIX=aoc-m3l RUNNER_LABELS=aoc-m3l,gpu \
 #     ./setup-gha-runners-docker.sh
 #
+#   # Shrink a host and cap each container's memory (see RUNNER_MEMORY):
+#   RUNNER_COUNT=2 RUNNER_KEEP_VOLUME=1 RUNNER_SKIP_BUILD=1 \
+#     ./setup-gha-runners-docker.sh
+#
 # Re-running is safe: existing containers/services are torn down and
 # re-registered. The registration token is only consumed on the first
 # start of a runner — subsequent restarts use the cached credentials.
@@ -66,6 +70,25 @@ SKIP_BUILD="${RUNNER_SKIP_BUILD:-0}"
 # Set it to a space-separated list to override. Set it to the empty string
 # to opt out entirely and let Docker do whatever it would have done.
 RUNNER_DNS_FALLBACK="${RUNNER_DNS_FALLBACK:-1.1.1.1 8.8.8.8}"
+# Hard memory cap per container (`docker run --memory`, swap disabled).
+#
+# Without one, every container may use all of the host's RAM, so N busy
+# runners overcommit it and the *kernel's* OOM killer picks the victim —
+# host-wide, from any container or anything else on the machine — and a job
+# dies with `Killed` / exit 137 through no fault of its own. With a cap, a job
+# that outgrows its share fails inside its own cgroup and nobody else is
+# touched.
+#
+# Left unset, it is derived: (host RAM − RUNNER_HOST_RESERVE_GB) / COUNT,
+# rounded down to whole GB. Set it explicitly (e.g. `3g`) to override, or to
+# the empty string to run uncapped.
+RUNNER_HOST_RESERVE_GB="${RUNNER_HOST_RESERVE_GB:-4}"
+if [[ -z "${RUNNER_MEMORY+set}" ]]; then
+  host_gb="$(awk '/^MemTotal/ {printf "%d", $2 / 1048576}' /proc/meminfo)"
+  per_gb=$(( (host_gb - RUNNER_HOST_RESERVE_GB) / COUNT ))
+  (( per_gb >= 1 )) || per_gb=1
+  RUNNER_MEMORY="${per_gb}g"
+fi
 if [[ -z "${RUNNER_DNS+set}" ]]; then
   # Loopback is the host's own stub (unreachable from a container) and
   # fe80:: is link-local; both are exactly what Docker already discards.
@@ -154,6 +177,18 @@ if [[ -n "${RUNNER_DNS}" ]]; then
   DNS_FLAGS+=$'\n'"  --dns-option timeout:2 --dns-option attempts:3 \\"
   log "container resolvers: ${RUNNER_DNS}"
 fi
+MEM_FLAGS=""
+if [[ -n "${RUNNER_MEMORY}" ]]; then
+  MEM_FLAGS=$'\n'"  --memory ${RUNNER_MEMORY} --memory-swap ${RUNNER_MEMORY} \\"
+  log "per-runner memory cap: ${RUNNER_MEMORY} × ${COUNT}"
+  # A heavy Node test job alone can reach ~2 GB RSS; under ~3 GB per
+  # runner those jobs will OOM routinely.
+  if [[ "${RUNNER_MEMORY}" =~ ^([0-9]+)g$ ]] && (( BASH_REMATCH[1] < 3 )); then
+    warn "${RUNNER_MEMORY} per runner is below the ~3g heavy jobs need — lower RUNNER_COUNT"
+  fi
+else
+  warn "RUNNER_MEMORY is empty — containers run uncapped"
+fi
 
 sudo tee "${UNIT_PATH}" >/dev/null <<EOF
 [Unit]
@@ -167,7 +202,7 @@ Type=simple
 EnvironmentFile=/etc/gha-runner/%i.env
 ExecStartPre=-/usr/bin/docker rm -f gha-runner-%i
 ExecStart=/usr/bin/docker run --rm \\
-  --name gha-runner-%i \\${DNS_FLAGS}
+  --name gha-runner-%i \\${DNS_FLAGS}${MEM_FLAGS}
   --hostname gha-runner-%i \\
   -v gha-runner-%i:/home/runner/runner \\
   -e RUNNER_ORG=\${RUNNER_ORG} \\
@@ -237,6 +272,17 @@ done
 # systemd's auto-restart unable to recover if config.sh failed on the
 # first attempt (no token in the env file → no way to register).
 
+# Lowering RUNNER_COUNT does not remove the runners above it — they keep
+# taking jobs, uncapped by this run's arithmetic. Say so rather than leave
+# the host silently over budget.
+extra="$(systemctl list-units --all 'gha-runner@*.service' --no-legend 2>/dev/null \
+  | grep -oE 'gha-runner@[0-9]+' | cut -d@ -f2 \
+  | awk -v n="${COUNT}" '$1 > n' | sort -n | tr '\n' ' ')"
+if [[ -n "${extra}" ]]; then
+  warn "runner instance(s) ${extra}are above RUNNER_COUNT=${COUNT} and still installed — remove them with:"
+  warn "  RUNNER_INSTANCES='${extra% }' ./scripts/uninstall-gha-runners-docker.sh"
+fi
+
 log "done — ${COUNT} runner(s) registered to ${ORG}"
 log "check status:  systemctl list-units 'gha-runner@*.service'"
 log "live logs:     journalctl -u 'gha-runner@1.service' -f"

scripts/uninstall-gha-runners-docker.sh

@@ -8,6 +8,7 @@
 # Usage:
 #   ./uninstall-gha-runners-docker.sh
 #   RUNNER_TOKEN=AAAA... ./uninstall-gha-runners-docker.sh   # remove-token
+#   RUNNER_INSTANCES="3 4" ./uninstall-gha-runners-docker.sh  # only these
 #
 # A *remove* token can be fetched via:
 #   gh api -X POST /orgs/wavekat/actions/runners/remove-token --jq .token
@@ -17,6 +18,9 @@ set -euo pipefail
 ORG="${RUNNER_ORG:-wavekat}"
 COUNT="${RUNNER_COUNT:-4}"
 PREFIX="${RUNNER_PREFIX:-$(hostname -s)}"
+# Which instance numbers to remove. Defaults to 1..RUNNER_COUNT; name a
+# subset to shrink a host (e.g. 4 → 2 runners) without touching the rest.
+INSTANCES="${RUNNER_INSTANCES:-$(seq 1 "${COUNT}")}"
 
 log()  { printf '\033[1;36m==>\033[0m %s\n' "$*"; }
 warn() { printf '\033[1;33m!!\033[0m %s\n' "$*" >&2; }
@@ -49,7 +53,7 @@ EOF
 TOKEN="$(get_token)"
 [[ -n "${TOKEN}" ]] || die "got empty remove token"
 
-for i in $(seq 1 "${COUNT}"); do
+for i in ${INSTANCES}; do
   NAME="${PREFIX}-${i}"
   log "removing runner ${NAME}"