diff --git a/CLAUDE.md b/CLAUDE.md index 4d1db4e..ae83034 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,7 @@ Since every runner is Ubuntu 24.04, GNU shell is fine — but **arch must never To pin a job to one host, add the runner's automatic arch label: `runs-on: [self-hosted, wavekat-ci, X64]` or `..., ARM64]`. -**No GitHub cloud caches on `wavekat-ci`** — no `cache: npm`/`cache: pnpm` on `setup-node`, no `Swatinem/rust-cache`, no `actions/cache` for dependency stores. The runner volume already keeps the pnpm store and cargo dirs warm, and the cloud round-trip goes over the runners' own uplink: `wavekat-platform` once spent 10 of a 15-minute timeout restoring a 255 MB pnpm tarball into a store that already had every package. Details in `docs/06-self-hosted-runners.md`. +**No GitHub cloud caches on `wavekat-ci`** — no `cache: npm`/`cache: pnpm` on `setup-node`, no `Swatinem/rust-cache`, no `actions/cache` for dependency stores. The runner volume already keeps the pnpm store and cargo dirs warm, so the cloud round-trip is pure cost — slow downloads of packages already on disk. Runner containers are also memory-capped (`RUNNER_MEMORY`); `Killed`/exit 137 means the host or cap is too small, not a code bug. Details in `docs/06-self-hosted-runners.md`. ## SEO & GEO — every new page must be both diff --git a/docs/06-self-hosted-runners.md b/docs/06-self-hosted-runners.md index dc7a720..6a6fd88 100644 --- a/docs/06-self-hosted-runners.md +++ b/docs/06-self-hosted-runners.md @@ -197,13 +197,9 @@ no `actions/cache` for dependency stores. The runners already keep that state on disk, and the cloud round-trip goes through the runners' own uplink instead of GitHub's own network. -The cost is not theoretical. `wavekat-platform` run 34731393671 spent -**10 minutes** of a 15-minute job timeout in `Setup Node`, downloading a -255 MB pnpm-store tarball at ~0.4 MB/s — and the `pnpm install` that followed -reused 325 of 325 packages from `/home/runner/runner/.pnpm-store`, the store -that was already on the volume. `Swatinem/rust-cache` in `wavekat-asr` costs -up to ~2.5 minutes per job the same way, saving a target dir that -`CARGO_TARGET_DIR` already keeps on the volume. +The pnpm store and cargo dirs are already on the volume, so the cache step only +adds a slow download: on a large workspace it can take most of a job's timeout +restoring a tarball of packages that `pnpm install` then finds already present. Where the warmth actually lives: @@ -217,6 +213,37 @@ A workflow that also runs on GitHub-hosted runners can keep its cache there and skip it here — `wavekat-voice/release.yml` does `cache: ${{ runner.environment == 'github-hosted' && 'pnpm' || '' }}`. +### Memory: cap every container, size the count to the host + +`Killed` / `exit code 137` in a step (usually `pnpm install` or a vitest +shard) is the kernel's OOM killer, not the job. Confirm on the host with +`journalctl -k | grep "Out of memory"`. + +Uncapped containers can each use all of the host's RAM, so a busy pool +overcommits it and the kernel picks a victim host-wide — any container, or +anything else running on the machine. Removing the cloud caches (above) makes +this easier to hit, because jobs no longer spend their first minutes on +network I/O and reach their memory peak together. + +`setup-gha-runners-docker.sh` passes `--memory` with swap disabled. +`RUNNER_MEMORY` defaults to `(host RAM − RUNNER_HOST_RESERVE_GB) / COUNT`, +reserve 4 GB, and warns below 3 GB, which is roughly what a heavy Node test +job needs. A job that outgrows its cap dies inside its own cgroup instead of +taking a neighbour down. If the default comes out under 3 GB, lower +`RUNNER_COUNT` rather than the cap. + +To shrink a host, remove the top instances first (this de-registers them, +so it needs a remove token — `gh auth refresh -s admin:org`), then re-create +the rest in place: + +```sh +RUNNER_INSTANCES="3 4" ./scripts/uninstall-gha-runners-docker.sh +RUNNER_COUNT=2 RUNNER_KEEP_VOLUME=1 RUNNER_SKIP_BUILD=1 ./scripts/setup-gha-runners-docker.sh +``` + +The setup script warns if instances above `RUNNER_COUNT` are still installed, +because lowering the count alone leaves them running. + ### Re-creating the runners without re-registering them Changing a `docker run` flag means re-creating the containers, and the setup diff --git a/scripts/setup-gha-runners-docker.sh b/scripts/setup-gha-runners-docker.sh index e326409..ee9b079 100755 --- a/scripts/setup-gha-runners-docker.sh +++ b/scripts/setup-gha-runners-docker.sh @@ -21,6 +21,10 @@ # RUNNER_COUNT=6 RUNNER_PREFIX=aoc-m3l RUNNER_LABELS=aoc-m3l,gpu \ # ./setup-gha-runners-docker.sh # +# # Shrink a host and cap each container's memory (see RUNNER_MEMORY): +# RUNNER_COUNT=2 RUNNER_KEEP_VOLUME=1 RUNNER_SKIP_BUILD=1 \ +# ./setup-gha-runners-docker.sh +# # Re-running is safe: existing containers/services are torn down and # re-registered. The registration token is only consumed on the first # start of a runner — subsequent restarts use the cached credentials. @@ -66,6 +70,25 @@ SKIP_BUILD="${RUNNER_SKIP_BUILD:-0}" # Set it to a space-separated list to override. Set it to the empty string # to opt out entirely and let Docker do whatever it would have done. RUNNER_DNS_FALLBACK="${RUNNER_DNS_FALLBACK:-1.1.1.1 8.8.8.8}" +# Hard memory cap per container (`docker run --memory`, swap disabled). +# +# Without one, every container may use all of the host's RAM, so N busy +# runners overcommit it and the *kernel's* OOM killer picks the victim — +# host-wide, from any container or anything else on the machine — and a job +# dies with `Killed` / exit 137 through no fault of its own. With a cap, a job +# that outgrows its share fails inside its own cgroup and nobody else is +# touched. +# +# Left unset, it is derived: (host RAM − RUNNER_HOST_RESERVE_GB) / COUNT, +# rounded down to whole GB. Set it explicitly (e.g. `3g`) to override, or to +# the empty string to run uncapped. +RUNNER_HOST_RESERVE_GB="${RUNNER_HOST_RESERVE_GB:-4}" +if [[ -z "${RUNNER_MEMORY+set}" ]]; then + host_gb="$(awk '/^MemTotal/ {printf "%d", $2 / 1048576}' /proc/meminfo)" + per_gb=$(( (host_gb - RUNNER_HOST_RESERVE_GB) / COUNT )) + (( per_gb >= 1 )) || per_gb=1 + RUNNER_MEMORY="${per_gb}g" +fi if [[ -z "${RUNNER_DNS+set}" ]]; then # Loopback is the host's own stub (unreachable from a container) and # fe80:: is link-local; both are exactly what Docker already discards. @@ -154,6 +177,18 @@ if [[ -n "${RUNNER_DNS}" ]]; then DNS_FLAGS+=$'\n'" --dns-option timeout:2 --dns-option attempts:3 \\" log "container resolvers: ${RUNNER_DNS}" fi +MEM_FLAGS="" +if [[ -n "${RUNNER_MEMORY}" ]]; then + MEM_FLAGS=$'\n'" --memory ${RUNNER_MEMORY} --memory-swap ${RUNNER_MEMORY} \\" + log "per-runner memory cap: ${RUNNER_MEMORY} × ${COUNT}" + # A heavy Node test job alone can reach ~2 GB RSS; under ~3 GB per + # runner those jobs will OOM routinely. + if [[ "${RUNNER_MEMORY}" =~ ^([0-9]+)g$ ]] && (( BASH_REMATCH[1] < 3 )); then + warn "${RUNNER_MEMORY} per runner is below the ~3g heavy jobs need — lower RUNNER_COUNT" + fi +else + warn "RUNNER_MEMORY is empty — containers run uncapped" +fi sudo tee "${UNIT_PATH}" >/dev/null < n' | sort -n | tr '\n' ' ')" +if [[ -n "${extra}" ]]; then + warn "runner instance(s) ${extra}are above RUNNER_COUNT=${COUNT} and still installed — remove them with:" + warn " RUNNER_INSTANCES='${extra% }' ./scripts/uninstall-gha-runners-docker.sh" +fi + log "done — ${COUNT} runner(s) registered to ${ORG}" log "check status: systemctl list-units 'gha-runner@*.service'" log "live logs: journalctl -u 'gha-runner@1.service' -f" diff --git a/scripts/uninstall-gha-runners-docker.sh b/scripts/uninstall-gha-runners-docker.sh index e830112..39d5dc9 100755 --- a/scripts/uninstall-gha-runners-docker.sh +++ b/scripts/uninstall-gha-runners-docker.sh @@ -8,6 +8,7 @@ # Usage: # ./uninstall-gha-runners-docker.sh # RUNNER_TOKEN=AAAA... ./uninstall-gha-runners-docker.sh # remove-token +# RUNNER_INSTANCES="3 4" ./uninstall-gha-runners-docker.sh # only these # # A *remove* token can be fetched via: # gh api -X POST /orgs/wavekat/actions/runners/remove-token --jq .token @@ -17,6 +18,9 @@ set -euo pipefail ORG="${RUNNER_ORG:-wavekat}" COUNT="${RUNNER_COUNT:-4}" PREFIX="${RUNNER_PREFIX:-$(hostname -s)}" +# Which instance numbers to remove. Defaults to 1..RUNNER_COUNT; name a +# subset to shrink a host (e.g. 4 → 2 runners) without touching the rest. +INSTANCES="${RUNNER_INSTANCES:-$(seq 1 "${COUNT}")}" log() { printf '\033[1;36m==>\033[0m %s\n' "$*"; } warn() { printf '\033[1;33m!!\033[0m %s\n' "$*" >&2; } @@ -49,7 +53,7 @@ EOF TOKEN="$(get_token)" [[ -n "${TOKEN}" ]] || die "got empty remove token" -for i in $(seq 1 "${COUNT}"); do +for i in ${INSTANCES}; do NAME="${PREFIX}-${i}" log "removing runner ${NAME}"