Skip to content

fix(workers): close the four worker-observability gaps (#248) - #266

Merged
linhdmn merged 1 commit into
mainfrom
feat/obs248-watchdog
Sep 9, 2026
Merged

fix(workers): close the four worker-observability gaps (#248)#266
linhdmn merged 1 commit into
mainfrom
feat/obs248-watchdog

Conversation

@linhdmn

@linhdmn linhdmn commented Sep 9, 2026

Copy link
Copy Markdown
Member

What changed

Fixes all four observability gaps from #248 (evidence: loop 176, run d19381cb, issue #239).

1. WatchdogSink wired on the direct spawn path (Q34 rows for every spawn)

  • internal/workers/spawnstream.go:68watchdogLedgerSink is the production sink: one best-effort ledger.AppendWatchdogHealthRecord into the ledger events file at WatchdogLedger.RepoPath, mirroring the herdr-pane writer (internal/herdr/herdr.go:461).
  • spawnstream.go:93-105watchdogSinkFor: caller-injected sink wins; nil resolves to the default ledger sink when a WatchdogLedgerContext is present. The old TODO(FR-GO-05 #190) at spawnstream.go:45-47 is closed — no production callsite changes needed because PrepareWorkerSpawn copies the options struct verbatim (internal/workers/sandbox.go:355-366), so the sink rides every adapter launch.
  • Gate unchanged: rows require armed clock + ledger context; no-context probes write nothing. herdr-pane rows byte-identical (herdr tests green untouched).

2. events populated for successful NDJSON omp runs

  • internal/workers/omp.go:185-188 — the NDJSON walk now collects session/turn_end/message_end/result records into OmpOutcome.Events (omp.go:100).
  • omp.go:278-321ompFinalize no longer returns Events: [] on success-without-result-envelope: the lifecycle records are the events evidence. On TimedOut the count reports whatever the partial stream delivered (0 for a silent burn, > 0 when the session streamed before the kill) while ResultText stays empty — the turn was never confirmed complete.
  • Test pins events > 0 on the real fixture internal/workers/testdata/omp-smoke-2026-08-30.jsonl (omp_test.go: TestOmpFinalize_NdjsonSuccessHasEvents), plus timeout/partial-stream and zero-burn pins.

3. Stream metrics threaded; productive wall-kill gets its own classification

  • internal/workers/types.go:71-79WorkerResult gains WatchdogFired/ClockResets/MeaningfulBytes; SpawnCliResult carries them from the streaming path (spawnstream.go:121-140); all five adapters stamp them onto the final result (adapter_common.go: stampStreamMetrics, omp additionally coalesces across retries).
  • internal/pipeline/implstage.go:519-543impMeaningfulBytesThreshold (1024) + impIsProductiveWallKill: a wall kill with clockResets > 0 or meaningfulBytes >= threshold is NOT infra-transient. impIsInfraTransient (implstage.go:546-560) keeps cold-start and zero-progress burns transient.
  • implstage.go:414-448 — the attempt-finished ledger row now carries watchdogFired/coldStart/clockResets/meaningfulBytes on kills, and a productive wall-kill logs its own row: "wall-timeout (productive): attempt N streamed X clock resets / Y meaningful bytes before the kill; retrying with resume". Retry semantics unchanged (same backoff, same -c resume continuation, infra-retry budget); ledger schema untouched.
  • Bonus mechanism fix for finding 4's retry side: launchBudgetMs (omp.go:439-458, wired into all five adapter prepare blocks) caps each launch's wall at the remaining run deadline so a late retry cannot restart with a fresh full timeout.

4. Post-kill drain bounded

  • spawnstream.go:499-516waitPipeDrain(wg, waitMs): named const DefaultPostKillDrainWaitMs = 3000 (spawnstream.go:62), PostKillDrainWaitMs option (0 = default, negative = legacy unbounded), replacing the bare drainWG.Wait() after cmd.Wait (spawnstream.go:397). A grandchild holding the pipes can no longer push the finish past the wall (the +716s overshoot mechanism).
  • Regression tests: TestWaitPipeDrain_Bounded/_UnboundedOptIn/_DefaultCapSentinel (deterministic; a never-releasing WaitGroup is the pipe-hold model — the fake-process variant TestSpawnCliStreaming_PostKillDrainBound also ships and stays green on darwin, but the process-level hold cannot be forced deterministically there because Go's pipe poller EOFs the read side on child exit, so the WaitGroup-level tests are the load-bearing ones).

Evidence

go test ./internal/workers/ ./internal/pipeline/ ./internal/orchestrator/ -count=1
ok  github.com/FreePeak/devagent/internal/workers    11.1s
ok  github.com/FreePeak/devagent/internal/pipeline   14.4s
ok  github.com/FreePeak/devagent/internal/orchestrator 4.3s

Also green: go build ./..., go vet on both packages, gofmt -l clean, internal/herdr + internal/ledger suites (herdr-pane row parity).

New/updated tests: TestOmpFinalize_NdjsonSuccessHasEvents, TestOmpFinalize_TimeoutKeepsPartialStreamEvents, TestOmpFinalize_ThreadStreamMetrics, TestLaunchBudgetMs, TestSpawnCliStreaming_DefaultSinkAppendsLedgerRow, TestSpawnCliStreaming_NoContextNoRow, TestSpawnCliStreaming_PostKillDrainBound, TestWaitPipeDrain_{Bounded,UnboundedOptIn,DefaultCapSentinel}, TestImpRetryLoopProductiveWallKillIsNotInfraTransient, TestImpIsInfraTransient_WallKillClasses.

Closes #248

- Q34 watchdog-health rows now flow on the direct (non-herdr) spawn
  path: a nil WatchdogSink resolves to the default ledger-append sink
  at WatchdogLedger.RepoPath (closes the FR-GO-05 #190 TODO at
  spawnstream.go). herdr-pane rows unchanged.
- omp events metric populated for every outcome class: NDJSON success
  without a result envelope synthesizes events from the lifecycle
  records (session/turn_end/message_end/result); wall/watchdog kills
  report the partial stream (0 for a silent burn). TimedOut keeps
  ResultText empty. Pinned on the real 2026-08-30 fixture.
- Stream evidence (watchdogFired/clockResets/meaningfulBytes) threads
  from spawnstream through SpawnCliResult into WorkerResult for all
  five adapters. impIsInfraTransient no longer files a productive
  wall-kill as 'Transient infra failure': impIsProductiveWallKill
  (clockResets>0 or meaningfulBytes>=1024) emits its own
  'wall-timeout (productive)' row and retries through the same resume
  path; zero-progress burns keep the transient class.
- Post-kill drain bounded at DefaultPostKillDrainWaitMs (3s, named
  const, PostKillDrainWaitMs option; negative = legacy unbounded) so
  a pipe-holding grandchild cannot push the finish past the wall;
  adapter retry loops arm each launch with the remaining wall budget
  (launchBudgetMs) instead of a fresh full timeout.

Closes #248
@linhdmn
linhdmn merged commit 0bb033f into main Sep 9, 2026
7 checks passed
@linhdmn
linhdmn deleted the feat/obs248-watchdog branch September 9, 2026 08:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant