Repository navigation
fix(ci): recover hosted macOS Docker runtime lost mid-scenario - #252
Merged
Merged
Conversation
The runtime and image phases already recover OrbStack, but nothing did once the scenario started. In main run 42971397902 the OrbStack VM stopped seven seconds into the LiteLLM schema bootstrap of `sidecars up`, and the job failed after five 2s backend probes. Scenario steps that drive Docker (exercise-sidecars, exercise-channels-runtime, exercise-browser-lab) now run through a runtime-loss guard on hosted macOS OrbStack runs with a recorded job budget. On failure a live probe decides whether the backend is gone; if so the runtime phase's OrbStack recovery runs inside min(scenario start + scenario reserve, job deadline - observer reserve), capped at the runtime recovery slice and leaving 600s for the retried step, then the step is retried once. Linux, local and budget-less runs keep failing fast; product code is unchanged. Diagnostics now keep the tail of OrbStack's own logs and widen the unified-log predicate to the Virtualization framework and kernel memorystatus kills, the evidence the failed run lacked. A CI-only drill, FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars, runs `orb stop` before the sidecars stack starts. It is reachable only from a manual Acceptance Tests dispatch off main and is re-checked by the helper.
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hosted macOS fresh-host scenarios now survive the OrbStack Docker backend dying mid-run: a failed Docker-backed step is probed, the runtime phase's OrbStack recovery runs inside the scenario's job budget, and the step is retried once. A CI-only fault drill proves it on a real runner.
Root cause: what failed in main run 42971397902
Job "macOS Fresh Host Sidecars (openclaw-default)", step "Run macOS fresh-host scenario":
sidecars upstarts its second compose call, the LiteLLM schema bootstrap (docker compose run --rm litellm --skip_server_startup) (job log)OrbStack Helpercancels its owncom.apple.Virtualization.VirtualMachineXPC connection (0x600001ca81e0, opened 06:21:21) (orbstack-unified.log)error during connect: Get ".../docker.sock/_ping": EOF(job log)service inactive: com.apple.Virtualization.VirtualMachine.7D08…(orbstack-unified.log)orbstack-unified.log)sidecars downgetsCommandError: Docker is installed but the backend is not reachable, then 5 backend probes fail and the job fails (job log)Post-mortem state:
orb status=Stopped, socket file gone,docker infofails with "no such file or directory".What the evidence shows: OrbStack shut down its own VM. The VM manager cancelled its Virtualization connection before the socket EOF and before launchd marked the VM inactive. That is a VM-manager-initiated stop, not the host killing the VM process from outside.
Ruled out:
vm_statcounters are cumulative since boot:Swapouts: 0,Swapins: 0, 97 compressions in total, ~4.7 GiB free. The host never swapped at any point in the job, including at 06:40:35.Not determinable from this run: why the guest or VM manager stopped (guest kernel panic, guest OOM of an OrbStack agent, or a VM manager fault). The unified-log predicate only matched host-side OrbStack processes, and OrbStack's own
~/.orbstack/log/vmgr.logandgui.logwere never collected. So no memory-limit or disk change is made: nothing in the evidence supports one. Instead, diagnostics now collect that evidence next time. This was 1 failure in the last 60 CI runs; the same code passed on #251 and 7a4eb20.Design
recover_lost_runtime(in_hosted_docker/runtime.py) reuses fix(ci): bound hosted docker waits by job-budget deadlines #238's_recover_orbstack_runtime. A probe extracted from_classify_runtime_failuredecides whether the backend is gone. A step failing with, say, "timed out waiting for compose service" while Docker answers is passed through untouched.scenario_phase_deadline= min(scenario start + scenario reserve, job deadline − observer reserve), which equals the scenario step's own timeout. Recovery gets a slice of it, capped at the runtime phase's 360s recovery slice and always leavingSCENARIO_RETRY_RESERVE_SECONDS= 600s for the retried step and later phases. The slowest observed exercise-sidecars was 411s, plus 41s of later phases._fresh_host/runtime_loss.py). It wraps exercise-sidecars, exercise-channels-runtime and exercise-browser-lab. On recovery it retries once. The channels retry restarts the sidecars stack, because the lost VM took it down. It is armed only for macOS + OrbStack + a recorded job budget. Linux, local and budget-less runs keep their current fail-fast behavior. Product code insrc/clawopsis unchanged and still raises its clear error.diagnostics/runtime-loss/<step>/{report.json,attempt-NN/{before,after},recovery-command.txt}. Diagnostics also now copy the tail of each~/.orbstack/log/*.logand widen the unified-log predicate tocom.apple.Virtualization*and kernelmemorystatuskills.FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecarsrunsorb stoponce, right before the sidecars stack starts. A marker in diagnostics makes it one-shot and timestamped. It is gated three ways:workflow_dispatchonly, not onworkflow_call;workflow_dispatchand a ref other thanrefs/heads/main;GITHUB_EVENT_NAME/GITHUB_REFand ignores the request otherwise.Fault-injection proof
Run: https://github.com/jsugg/strongclaw/actions/runs/42982265106. It was dispatched on this branch with
fault_injection=orbstack-stop-during-sidecarsand is green (attempt 2).Attempt 1's hypermemory Sidecars job hit "The hosted runner lost communication with the server" during
actions/setup-node, about 15s into the job and before any helper code ran. Only that job was rerun, with the same inputs.macOS Fresh Host Sidecars (openclaw-default), attempt 1. The first lines reproduce the original failure exactly; the run then recovers:
macOS Fresh Host Sidecars (hypermemory), attempt 2:
Artifacts contain:
diagnostics/runtime-loss/exercise-sidecars/report.json:recovered: true,trigger_reason: orbstack_socket_missing,deadline_limited_by: runtime-loss-cap, commands[["orb","start"]].attempt-01/{before,after}diagnostics.fault-injection/orbstack-stop-during-sidecars.jsonmarker.orbstack-logs/vmgr*.log. The rotatedvmgr.1.logrecords the stop asWait for services to exit→reboot: Power down→vmgr [VM] stopped, the evidence that was missing for run 42971397902.Normal PR CI (switch off): https://github.com/jsugg/strongclaw/actions/runs/42982302893, all green. The sidecars job shows
FRESH_HOST_FAULT_INJECTION:(empty) andRuntime-loss recovery armed: 2100s scenario window, with no fault and no loss.Tests
tests/suites/unit/ci/test_fresh_host_runtime_loss.py(fake clock): arming (macOS+budget, no budget, Linux), observer-reserve disarm, pass-through without a window, Linux pass-through with a budget, a reachable backend is not runtime loss, socket-EOF recovery and a single retry, missing socket usesorb start, retry action, refusal when only the retry reserve is left, exhaustion mid-recovery, failed recovery with attempt evidence, unrecoverable reason, and an end-to-end sidecars phase with an injectedorb stop.tests/suites/unit/ci/test_fresh_host_faults.py: event/ref gating, off values, unknown fault, one-shot firing, inert on push to main.test_fresh_host_budget.py: scenario deadline cap, late start, exhaustion.test_hosted_docker.py: OrbStack log tails and the widened predicate.test_fault_injection_cannot_be_enabled_on_push_to_main.Local gates (raw):
make lint(ruff: all checks passed),make typecheck(mypy: no issues in 378 files; pyright: 0 errors),make imports,make test-unit(1220 passed),make test-contracts(150 passed),actionlintclean.