fix(workers): close the four worker-observability gaps (#248) - #266
Merged
Conversation
- Q34 watchdog-health rows now flow on the direct (non-herdr) spawn path: a nil WatchdogSink resolves to the default ledger-append sink at WatchdogLedger.RepoPath (closes the FR-GO-05 #190 TODO at spawnstream.go). herdr-pane rows unchanged. - omp events metric populated for every outcome class: NDJSON success without a result envelope synthesizes events from the lifecycle records (session/turn_end/message_end/result); wall/watchdog kills report the partial stream (0 for a silent burn). TimedOut keeps ResultText empty. Pinned on the real 2026-08-30 fixture. - Stream evidence (watchdogFired/clockResets/meaningfulBytes) threads from spawnstream through SpawnCliResult into WorkerResult for all five adapters. impIsInfraTransient no longer files a productive wall-kill as 'Transient infra failure': impIsProductiveWallKill (clockResets>0 or meaningfulBytes>=1024) emits its own 'wall-timeout (productive)' row and retries through the same resume path; zero-progress burns keep the transient class. - Post-kill drain bounded at DefaultPostKillDrainWaitMs (3s, named const, PostKillDrainWaitMs option; negative = legacy unbounded) so a pipe-holding grandchild cannot push the finish past the wall; adapter retry loops arm each launch with the remaining wall budget (launchBudgetMs) instead of a fresh full timeout. Closes #248
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Fixes all four observability gaps from #248 (evidence: loop 176, run d19381cb, issue #239).
1. WatchdogSink wired on the direct spawn path (Q34 rows for every spawn)
internal/workers/spawnstream.go:68—watchdogLedgerSinkis the production sink: one best-effortledger.AppendWatchdogHealthRecordinto the ledger events file atWatchdogLedger.RepoPath, mirroring the herdr-pane writer (internal/herdr/herdr.go:461).spawnstream.go:93-105—watchdogSinkFor: caller-injected sink wins; nil resolves to the default ledger sink when a WatchdogLedgerContext is present. The oldTODO(FR-GO-05 #190)at spawnstream.go:45-47 is closed — no production callsite changes needed becausePrepareWorkerSpawncopies the options struct verbatim (internal/workers/sandbox.go:355-366), so the sink rides every adapter launch.2.
eventspopulated for successful NDJSON omp runsinternal/workers/omp.go:185-188— the NDJSON walk now collectssession/turn_end/message_end/resultrecords intoOmpOutcome.Events(omp.go:100).omp.go:278-321—ompFinalizeno longer returnsEvents: []on success-without-result-envelope: the lifecycle records are the events evidence. OnTimedOutthe count reports whatever the partial stream delivered (0 for a silent burn, > 0 when the session streamed before the kill) whileResultTextstays empty — the turn was never confirmed complete.events > 0on the real fixtureinternal/workers/testdata/omp-smoke-2026-08-30.jsonl(omp_test.go: TestOmpFinalize_NdjsonSuccessHasEvents), plus timeout/partial-stream and zero-burn pins.3. Stream metrics threaded; productive wall-kill gets its own classification
internal/workers/types.go:71-79—WorkerResultgainsWatchdogFired/ClockResets/MeaningfulBytes;SpawnCliResultcarries them from the streaming path (spawnstream.go:121-140); all five adapters stamp them onto the final result (adapter_common.go: stampStreamMetrics, omp additionally coalesces across retries).internal/pipeline/implstage.go:519-543—impMeaningfulBytesThreshold(1024) +impIsProductiveWallKill: a wall kill with clockResets > 0 or meaningfulBytes >= threshold is NOT infra-transient.impIsInfraTransient(implstage.go:546-560) keeps cold-start and zero-progress burns transient.implstage.go:414-448— the attempt-finished ledger row now carrieswatchdogFired/coldStart/clockResets/meaningfulByteson kills, and a productive wall-kill logs its own row: "wall-timeout (productive): attempt N streamed X clock resets / Y meaningful bytes before the kill; retrying with resume". Retry semantics unchanged (same backoff, same -c resume continuation, infra-retry budget); ledger schema untouched.launchBudgetMs(omp.go:439-458, wired into all five adapter prepare blocks) caps each launch's wall at the remaining run deadline so a late retry cannot restart with a fresh full timeout.4. Post-kill drain bounded
spawnstream.go:499-516—waitPipeDrain(wg, waitMs): named constDefaultPostKillDrainWaitMs = 3000(spawnstream.go:62),PostKillDrainWaitMsoption (0 = default, negative = legacy unbounded), replacing the baredrainWG.Wait()aftercmd.Wait(spawnstream.go:397). A grandchild holding the pipes can no longer push the finish past the wall (the +716s overshoot mechanism).TestWaitPipeDrain_Bounded/_UnboundedOptIn/_DefaultCapSentinel(deterministic; a never-releasing WaitGroup is the pipe-hold model — the fake-process variantTestSpawnCliStreaming_PostKillDrainBoundalso ships and stays green on darwin, but the process-level hold cannot be forced deterministically there because Go's pipe poller EOFs the read side on child exit, so the WaitGroup-level tests are the load-bearing ones).Evidence
Also green:
go build ./...,go veton both packages,gofmt -lclean, internal/herdr + internal/ledger suites (herdr-pane row parity).New/updated tests: TestOmpFinalize_NdjsonSuccessHasEvents, TestOmpFinalize_TimeoutKeepsPartialStreamEvents, TestOmpFinalize_ThreadStreamMetrics, TestLaunchBudgetMs, TestSpawnCliStreaming_DefaultSinkAppendsLedgerRow, TestSpawnCliStreaming_NoContextNoRow, TestSpawnCliStreaming_PostKillDrainBound, TestWaitPipeDrain_{Bounded,UnboundedOptIn,DefaultCapSentinel}, TestImpRetryLoopProductiveWallKillIsNotInfraTransient, TestImpIsInfraTransient_WallKillClasses.
Closes #248