Repository navigation
fix(ops): fail sidecar health waits fast when Docker is lost - #253
Merged
Merged
Conversation
A Docker backend that died inside a sidecars health wait was polled for the rest of that wait (LiteLLM 300s, Neo4j 600s, Postgres 180s twice on hosted macOS) before the #252 runtime-loss recovery could start. Each failed `docker compose ps` inside a health wait now probes the backend. A missing DOCKER_HOST unix socket, or `docker info` failing three probes in a row, ends the wait at once; a healthy backend with a slow container keeps waiting exactly as before. - clawops: `_wait_for_compose_service` raises DockerBackendLostError, a CommandError subclass, so exit codes and the --json payload shape are unchanged, and emits clawops.ops.sidecars.wait.backend_lost. `sidecars up` no longer recreates Postgres after the backend is lost. - CI: verify_compose_services_running (sidecars, channels retry, browser-lab) fails fast the same way. A hung compose ps is now a failed poll capped at 30s, not an uncaught TimeoutExpired that the runtime-loss guard could not catch. - Drill: FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars-health runs `orb stop` once while `sidecars up` waits on sidecar health. It is gated like the existing drill: manual dispatch off main only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #252. A Docker backend that dies inside a sidecars, channels or browser-lab health wait now ends that wait within seconds instead of polling out its timeout, so the #252 runtime-loss recovery starts right away.
Problem
#252 left this gap open: "a VM dying inside the 300s health check would burn that wait before recovery starts". Both health-wait layers polled
docker compose psblindly. On a failure they slept 2s and tried again until the deadline:_wait_for_compose_service(insideclawops ops sidecars up)src/clawops/strongclaw_ops.pyverify_sidecar_services_running/verify_compose_services_runningtests/utils/helpers/_fresh_host/shell.pyThe CI wait had a second defect. Each poll's subprocess timeout was the whole wait, and a hung
docker compose psraisedsubprocess.TimeoutExpired. That is not aFreshHostError, so the #252 guard could not catch it.Design
Fail-fast in the product, shared with CI. A user running
clawops ops sidecars upagainst a dead backend hits the same 300-600s wait, so the fix lives inclawops.strongclaw_runtime, next toensure_docker_backend_ready:probe_docker_backend(env, cwd)checks theDOCKER_HOSTunix socket first. A missing socket is definitive. Otherwise it runsdocker infowith a 15s timeout.DockerBackendWatchis called only after a status poll fails. It confirms loss on a definitive probe, or after 3 faileddocker infoprobes in a row. Any successful poll or probe resets the streak.compose pswhiledocker infoanswers also keeps waiting and still times out with the old message.Classification. #252 decides runtime loss with a live probe, not by matching error text. The fail-fast only needs to end the step while the backend is still down. A unit test runs the real CI wait inside
run_with_runtime_recoveryand checks the whole path: the wait fails after 4s of fake time against a 300s timeout, the report saysdocker_socket_eof, the stack recovers, and the single retry passes.User-facing change (
clawops ops sidecars up)Docker backend became unreachable while waiting for compose service '<svc>' to reach running/healthy: <detail>; last status inspection error: <...>. Restart the Docker runtime, then rerun the operation.DockerBackendLostErrorsubclassesCommandError, so the exit code (1) and the--jsonpayload (ok: false,error,failedService,command.exitCode: 1) keep their shape.clawops.ops.sidecars.wait.backend_lost(service,target,detail,elapsed_ms), documented inOBSERVABILITY.md.--force-recreate postgresstep.Fault drill
FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars-healtharms a watcher thread aroundsidecars up. On two consecutive 1s sightings of the full sidecars stack running with a required service not yet healthy, which is the product health wait, it runsorb stoponce. The watcher is joined before the guard's recovery runs, so recovery never races an in-flightorb stop. It is gated like the #252 drill: a dispatch-only input on Acceptance Tests, the core env expression, and the helper re-check. A contract test checks that every known fault is inert on push to main.Fault-injection proof
Run: https://github.com/jsugg/strongclaw/actions/runs/42996741189, dispatched on this branch with
fault_injection=orbstack-stop-during-sidecars-health. It is green on attempt 2.macOS Fresh Host Sidecars (hypermemory), attempt 1:
macOS Fresh Host Sidecars (openclaw-default), attempt 2 (only that job was rerun, with the same inputs):
The health wait now ends 10-16s after
orb stopstarts, and most of that isorb stopitself. Before, the LiteLLM wait would have polled for up to 300s.Attempt 1 of the openclaw-default lane failed, and the failure was in the budget, not the fail-fast. The runner was unusually slow:
setuptook 14 min against about 6 min on the other lane, and the LiteLLM schema bootstrap timed out 4 times. The fault therefore fired 25 min into the 2100s scenario window. The fail-fast still worked:orb stopran at 10:35:03 andDockerBackendLostErrorwas raised at 10:35:29. With only about 549s left, below the 600sSCENARIO_RETRY_RESERVE_SECONDS, the #252 guard correctly refused withbudget_exhausted: runtime_loss recovery ran out of time. Without this change the job would have failed the same way, just later.Normal PR CI (switch off): https://github.com/jsugg/strongclaw/actions/runs/42996743683, all green. The Sidecars job shows
FRESH_HOST_FAULT_INJECTION:(empty) and no injection or loss lines.Tests
test_strongclaw_runtime.py: socket-missing probe without running docker,docker infook/failed/timeout classification, watch streak/reset/definitive/validation.test_strongclaw_ops.py(fake clock): socket loss fails with 0 sleeps; 3 failed probes fail at 4s vs a 300s timeout; a slow LiteLLM on a healthy backend waits 100 polls with 0 probes; a failingcompose pswith an answering backend still times out with the old message; no Postgres recreate after loss;--jsonpayload.test_fresh_host.py(fake clock): the same matrix for the CI wait, plus a hungcompose pstreated as a failed poll capped at 30s.test_fresh_host_faults.py: health-pending detection, watcher fires on the second sighting, never fires after stop, inert unless requested, one-shot, and "did not fire" reporting.test_fresh_host_runtime_loss.py: health-wait fail-fast into recovery plus one retry.test_fault_injection_cannot_be_enabled_on_push_to_maincovers every fault.Local gates (raw)
make lint(ruff: all checks passed),make imports,make typecheck(mypy: no issues in 378 files; pyright: 0 errors),make test-unit(1253 passed),make test-contracts(150 passed),black --check(380 files unchanged). The commit hooks also ran actionlint, attribution and conventional-commit checks, and all passed.