fix(docker): retry transient image-pull failures with backoff

[r3y3r53]

Problem

pullImage() made one docker pull attempt. On failure it handled only the no-matching-manifest case (falling back to linux/amd64) and rethrew everything else:

} catch (err: unknown) {
  const msg = String(eObj.stderr || eObj.message || '');
  if (!msg.includes('no matching manifest') && !msg.includes('no match for platform')) {
    throw err;   // ← 503, rate limit, TLS timeout all die here
  }
}

Docker Hub is intermittently unreliable (5xx during their incidents, unauthenticated pull rate limits, TLS/connection timeouts on CI runners). Any of those aborted the run during container setup with no retry — a benchmark that had already spent real model spend and wall-clock time was thrown away by a transient registry hiccup.

Change

Classify the failure, then retry only what retrying can actually fix.

src/lib/docker.ts

export function classifyPullError(err: unknown): 'manifest' | 'transient' | 'fatal'

Class	Signals	Behaviour
manifest	no matching manifest, no match for platform	Skip retries — deterministic. Go straight to the linux/amd64 fallback.
transient	503/502/504, 429/toomanyrequests, connection refused/reset, TLS handshake timeout, i/o timeout, unexpected EOF, Gateway Time-out	Retry up to DOCKER_PULL_MAX_ATTEMPTS (3) with exponential backoff from DOCKER_PULL_BASE_DELAY_MS (5s) → 5s, 10s
fatal	auth failures, missing repos, unknown manifest, anything unrecognized	Fail fast — a bad image reference is not hammered three times

Two deliberate design decisions:

1. A transient failure is never masked behind a platform fallback. If retries are exhausted the original error is rethrown. Falling back to linux/amd64 on a 503 would silently hide the real cause and make the run slower under emulation for no reason.
2. Manifest errors are not retried. They are deterministic — retrying the native pull can only waste time.

Backoff is injectable via options.sleep so tests run instantly, retries are surfaced through onProgress (and an optional onRetry hook), and pullAndStartContainers() now reports retry activity for both images.

Backwards compatibility

• pullImage(image, onProgress?) still works with no third argument.
• The manifest → linux/amd64 fallback contract and its boolean return are unchanged.
• PullRetryOptions is entirely optional.

New exports: classifyPullError, PullRetryOptions, DOCKER_PULL_MAX_ATTEMPTS, DOCKER_PULL_BASE_DELAY_MS.

Tests

tests/unit/docker-retry.test.ts (22 new tests) covers:

• classification of manifest / transient / fatal signals, including 5xx, rate limits, TLS timeouts and EOF
• retry-then-succeed with no platform fallback
• exact exponential backoff delays ([1000, 2000])
• exhaustion rethrowing the transient error after exactly DOCKER_PULL_MAX_ATTEMPTS
• fatal errors failing on attempt 1
• onRetry arguments, custom maxAttempts
• transient failure during the amd64 fallback retrying and succeeding
• preservation of the original manifest-fallback contract and progress messages

One existing test in tests/unit/docker.test.ts needed updating: its "non-manifest error" example was 'connection refused', which is now correctly classified as transient and retried. The test's actual intent — no amd64 fallback for non-manifest errors — still holds and is now asserted directly (every attempt is a native pull), with a no-op sleep injected so it doesn't wait.

Verification

npm run typecheck   # clean
npm run test        # 441 passed (17 files)
npm run build       # clean

Before: 421 tests. After: 441 tests, all passing.

Risk

Low, and opt-out-free by design. Retries only add bounded delay (max ~15s of backoff) in cases that previously failed outright. Fatal errors still surface immediately on the first attempt, so misconfigured images fail as fast as before. The one behavioural change to be aware of: a transient error now appears after ~15s and 3 attempts instead of instantly.

Closes #76