From af68874db3d16bb4d5e570a124f1285f8d6a789a Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 12:35:32 -0400 Subject: [PATCH 01/95] fix(engine): close two goal-supervision longevity gaps (#187) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): resume the goal worker through a provider-exhausted wall Box bx-01m0x8996, a real long-running goal-supervised session, logged "engine: goal worker turn parked after 1 permanent-tier attempt(s): [permanent] anthropic: You have reached your specified API usage limits. You will regain access on ." and never resumed without an operator DELETE + re-register. promptTurnWithRetry's fail-fast permanent branch treated the account-level usage wall exactly like a structurally malformed request (an orphaned tool_use, NEP-5272): one attempt, no backoff, immediate park. That is correct for a malformed request, which fails identically on every retry, but wrong for a quota wall, which lifts on its own, unchanged, the moment the provider's own clock rolls over. PR #174 added provider.ErrKindProviderExhausted / AsProviderExhausted for exactly this shape, giving task CHILDREN a distinct, resumable outcome instead of a generic failure. The goal worker predates that work and was never wired to it — provider.AsPermanent(err) still caught an exhausted error first, since the anthropic adapter wraps it provider.MarkPermanent for ordinary HTTP-retry purposes (no short backoff schedule outlives a monthly quota). The fix reclassifies a provider-exhausted error, in both promptTurnWithRetry and PursueGoal's worker-turn handling, as a weather-shaped failure rather than a doomed one: it is excluded from the permanent fail-fast branch and instead rides its own budget (goalProviderExhaustedMaxAttempts, sized and scheduled identically to the existing overloaded/rate_limited/server_error tier via goalRetryableBackoff) with its own attempt counter, so a concurrent overload spell and an account wall in the same turn can never share or steal from one another's budget. The provider's own RecoverHint ("you regain access on ") is deliberately never parsed into a wait duration — its format varies by provider and by plan (see provider.Error.RecoverHint's doc comment) — so this tier rides the same jittered schedule ordinary weather uses rather than guessing an exact wake time. Both call sites share one classifyProviderExhausted helper rather than duplicating the provider.AsProviderExhausted check and override, so they cannot independently drift. An alternative considered and rejected: retrying in-loop for the account wall's own duration without ever parking. Rejected because this package's own Round 7 rework (GitHub issue #61) already found that shape unsafe for open-ended weather — it pins the run slot to the parked loop for the whole outage, so a queued prompt can only ever be injected mid-turn into a doomed attempt, never dispatched as its own turn. A quota wall can run far longer than ordinary weather (hours to days, not minutes), so the same argument applies more strongly here. A wall that clears within the bounded budget resumes automatically with no operator action; a wall that outlasts it still parks — freeing the slot — but the classification is now honest ("provider account usage limit exhausted the retry budget", surfaced via a new goalClassProviderExhausted marker folded into the existing retryable/class bookkeeping) rather than "permanent provider error and cannot succeed on retry", which was actively misleading for a condition that resolves on its own. The same honest rendering also replaces recordGoalStalled's reason for this class: err.Error() for a provider-exhausted error still reads "[permanent] ...", which would otherwise self-contradict that SAME goal.stalled record's own Retryable:true/RetryableClass:"provider_exhausted" fields. Semantic change: a provider-exhausted worker-turn error retried and resumed automatically is no longer distinguishable, from goal.stalled/ goal.parked's perspective, from ordinary overload/rate-limit weather, except by its dedicated RetryableClass value. Every other classified error (malformed request, context overflow, deterministic failure) keeps its exact prior behavior. Verified: TestPursueGoalProviderExhaustedRetriesThenRecovers reverted against the pre-fix code reproduces the live incident's exact error text, "goal worker turn parked after 1 permanent-tier attempt(s)", confirming the test guards the named defect; its goal.stalled reason assertions were separately red-verified against a reverted recordGoalStalled, reproducing the raw "[permanent] ..." text. go test -race ./... and go vet ./... are clean. * fix(engine): bound the goal evaluator's transcript to its own context window Box bx-01m0x8996, a real long-running goal-supervised session, logged "engine: goal evaluator failed at 5 consecutive turn boundaries: context exhausted: prompt 245332 tokens > limit ..." and goal supervision went silently inert. runEvaluator built its CONVERSATION TRANSCRIPT field from renderConversation(s.History()) with no bound of any kind — it grows with the entire session transcript, forever. The MAIN session is protected by automatic compaction (maybeAutoCompact/engine/compact.go), armed off the main model's own context window; the evaluator has always been a second, independent model call with no such protection, so a session that never crosses the main model's (possibly much larger) compaction threshold can still blow straight through the evaluator's own, smaller limit. The fix adds renderConversationBounded, called from runEvaluator in place of the unconditional renderConversation(s.History()). The budget is derived from the EVALUATOR model's own context window via goalEvaluatorTranscriptBudgetBytes, which reads modelmeta's table DIRECTLY through the modelContextWindowLookup test seam, not through engine.resolveContextWindow: that function's minAutoContextWindowTokens floor exists to decide whether automatic compaction should ARM, so it folds a genuinely unrecognized model and a real, small, KNOWN model (gpt-4's documented 8,192-token window is modelmeta's own example) into the identical (0, disabled) result. Calling it here would have handed a real 8,192-token evaluator model a budget derived from the 16k fallback — roughly DOUBLE its actual window, the exact overflow class this fix exists to close. goalEvaluatorTranscriptBudgetBytes trusts any positive, known window from the table however small, and falls back to goalEvaluatorFallbackContextWindowTokens (mirroring minAutoContextWindowTokens's value) only for a genuine no-entry miss. It also reserves headroom for the system prompt and MaxTokens' output budget, and applies a conservative 0.5 fraction on top of the same crude ~4-bytes-per-token estimate (bytesPerTokenEstimate) compaction's own resilience fallback already uses. Design alternative considered: have the evaluator request and cache its own summary of the transcript prefix, independent of compaction. Rejected in favor of reusing what compaction already maintains: Compact (engine/compact.go) splices its own summary message directly into s.history in place of whatever range it folded, tagged with the compactionSummaryIDTag prefix (isCompactionSummaryID). So renderConversationBounded simply walks history from the newest message backward, accumulating rendered blocks until the budget (in BYTES — renamed from an earlier budgetRunes that measured with len, not RuneCountInString) would be exceeded, and stops the INSTANT it includes a compaction-summary message even with budget still unspent — that message already is the compacted record of everything before it. Whenever automatic compaction has run at all, the evaluator gets "the latest compaction summary plus every raw message after it, bounded to what fits" for free: no second summarization call, no new stored field, no new coupling beyond the one ID-prefix helper compaction already exports to this package. The newest message is always kept regardless of budget (an empty transcript could never be assessed); a truncated transcript is prefixed with goalEvaluatorTruncationNotice so the evaluator, and an operator reading a later goal.eval record, never mistakes a bounded view for the whole session. Semantic change: the evaluator's CONVERSATION TRANSCRIPT field is now always bounded to a budget derived from the evaluator model, never the whole session history. Nothing about verdict parsing, the worker-turn loop, or compaction itself changes; a session small enough to fit the budget renders identically to before. Verified: TestPursueGoalEvaluatorPromptBoundedForHugeTranscript seeds a 900KB synthetic transcript and asserts the evaluator's actual received request stays under a fixed 100KB safety ceiling while still achieving the goal; reverted against the pre-fix code it observes an unbounded ~903KB prompt instead, reproducing the shape of the live incident. TestGoalEvaluatorTranscriptBudgetBytesUsesRealWindowBelowFloor separately red-verified against the earlier resolveContextWindow-based version, reproducing the exact conflation described above (both a known 8,192-token model and a genuinely unknown one collapsed to the same 23,808-byte budget). go test -race ./... and go vet ./... are clean. --- AGENTS.md | 82 ++++- engine/goal.go | 405 +++++++++++++++++++++++-- engine/goal_evaluator_bound_test.go | 213 +++++++++++++ engine/goal_provider_exhausted_test.go | 203 +++++++++++++ 4 files changed, 870 insertions(+), 33 deletions(-) create mode 100644 engine/goal_evaluator_bound_test.go create mode 100644 engine/goal_provider_exhausted_test.go diff --git a/AGENTS.md b/AGENTS.md index d22262b9..e4fff88a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -334,8 +334,50 @@ loop also emits `goal.*` engine events so the server journals them. Config `goal_evaluator_model` supplies the evaluator for `harness run -goal` and `POST /session/{id}/goal`. +The evaluator's own `CONVERSATION TRANSCRIPT` field is BOUNDED, independent +of whatever context window the MAIN session model has and independent of +whether automatic compaction (below) has fired at all. +`renderConversationBounded` (`engine/goal.go`, called from `runEvaluator`) +replaced the old unconditional `renderConversation(s.History())`: box +bx-01m0x8996, a real long session, died with "engine: goal evaluator failed +at 5 consecutive turn boundaries: context exhausted: prompt 245332 tokens > +limit ..." because the evaluator's prompt grows with the WHOLE session +transcript forever — unlike the main session, which automatic compaction +protects, the evaluator had no bound of its own at all. The budget comes +from `goalEvaluatorTranscriptBudgetBytes`, which resolves the EVALUATOR +model's own context window via `modelContextWindowLookup` +(`modelmeta.ContextWindow`) — the same table automatic compaction's +`resolveContextWindow` (`engine/context_window.go`) reads, called +DIRECTLY rather than through `resolveContextWindow` itself: that +function's `minAutoContextWindowTokens` floor answers "should automatic +compaction ARM for this window," so a real, small, KNOWN window (gpt-4's +documented 8,192 tokens) reports identically to a genuinely UNRECOGNIZED +model (0, disabled) — conflating them would give a real small-window +evaluator a budget roughly double its actual limit, the exact overflow +class this fix closes. `goalEvaluatorTranscriptBudgetBytes` trusts ANY +positive, known window from the table, however small, and falls back to +`goalEvaluatorFallbackContextWindowTokens` (mirroring +`minAutoContextWindowTokens`'s value) only when the model has NO entry at +all. It also reserves headroom for the system prompt and MaxTokens' output +budget, and applies a conservative fraction (`goalEvaluatorContextBudgetFraction`, +0.5) on top of the same crude ~4-bytes-per-token estimate +(`bytesPerTokenEstimate`) automatic compaction's own resilience fallback +uses — reused, not reinvented. +`renderConversationBounded` walks history from the NEWEST message backward, +accumulating rendered blocks until the budget would be exceeded, and +prefers "summary + tail" for free rather than summarizing a second time: +Compact (`engine/compact.go`) already splices its own summary message in +place of whatever range it folded, tagged with the `compactionSummaryIDTag` +prefix, so the backward walk simply STOPS the instant it includes such a +message — everything before it is already captured there. The newest +message is always kept regardless of budget (an empty transcript can never +be assessed); a truncated transcript is prefixed with +`goalEvaluatorTruncationNotice` so the evaluator, and an operator reading a +later `goal.eval` record, never mistakes a bounded view for the whole +session. + A worker-turn error (`s.Prompt` failing) is retried by `promptTurnWithRetry` -on one of THREE independent budgets, chosen by classification via +on one of FOUR independent budgets, chosen by classification via `provider.AsRetryable` — never by matching error text. One class skips every budget. Before it selects a budget, @@ -353,6 +395,24 @@ select a more accurate classified reason and tier name (`classifyGoalWorkerError`, `goalWorkerParkedError`), so an operator can tell a single-attempt park from `goalWorkerRetries`+1 identical attempts. +One shape is wrapped `provider.MarkPermanent` by the adapter but is +DELIBERATELY EXCLUDED from this fail-fast branch: `provider.AsProviderExhausted` +— an ACCOUNT-level usage/quota wall (PR #174's `provider.ErrKindProviderExhausted`, +originally added for task-child resumability; see `engine/session_manager.go`'s +`FailKindProviderExhausted`). An adapter marks it permanent for ordinary +HTTP-retry purposes (no short backoff schedule outlives a monthly quota), +but a wall lifts on its own, unchanged, so treating it as a doomed malformed +request silently kills goal supervision on the very first usage-limit +rejection. Live evidence: box bx-01m0x8996 parked after "1 permanent-tier +attempt(s)" on "You have reached your specified API usage limits" and never +resumed without an operator `DELETE` + re-register. `promptTurnWithRetry` +and `PursueGoal`'s worker-turn handling both check +`provider.AsProviderExhausted` explicitly and fold a positive result into +their local `retryable`/`class` bookkeeping (`class` set to the dedicated +marker `goalClassProviderExhausted`, never one of `provider.RetryableClass`'s +real values) — reusing the existing stall/park recording machinery rather +than adding a fourth field throughout. + A deterministic failure (not classified retryable, not permanent) gets `goalWorkerRetries` (2) additional attempts on the short schedule (~5s total: 1s, then 4s). A provider error @@ -366,9 +426,25 @@ error to classify from (see the idle-stream watchdog below) — gets its own deterministic tier uses (~5s total): truncation is retryable, but it is not weather — waiting longer never raises a stream ceiling, and every retry re-prompts a full turn at full input cost — so it must ride neither the fast -deterministic budget nor the long weather-tier one. Every attempt records a +deterministic budget nor the long weather-tier one. A `provider.AsProviderExhausted` +failure gets its OWN `goalProviderExhaustedMaxAttempts` budget (equal in +size to `goalRetryableMaxAttempts`, on the identical jittered schedule) — +never the ordinary weather counter, so a concurrent overload spell and an +account wall in the same turn can never silently share or steal from one +another's budget. It deliberately rides the SAME schedule ordinary weather +uses rather than computing a wait from the provider's own `RecoverHint` +("you regain access on "): `RecoverHint`'s format varies by provider +and by plan (see `provider.Error.RecoverHint`'s doc comment), so it is +never parsed into a duration, only ever quoted verbatim to a model-visible +caller. A wall that clears within the budget (a burst rate limit that +reached this classification, or a short-lived cap) resumes the worker turn +— and the whole goal — with NO operator action; a wall measured in hours or +days still exhausts the budget and parks, but honestly classified via +`classifyGoalWorkerError`'s dedicated branch ("provider account usage limit +exhausted the retry budget"), never as a permanent, unretriable request. +Every attempt records a `goal.stalled` record regardless of tier, so the loop is never silent. -Exhausting ANY of the three budgets — or the non-idempotency gate stopping +Exhausting ANY of the four budgets — or the non-idempotency gate stopping retries early once a tool has already executed this attempt — PARKS the goal instead of clearing it: `PursueGoal` exits, journals a durable, CLASSIFIED `goal.parked` record (never raw provider error text — the same leak rule diff --git a/engine/goal.go b/engine/goal.go index e9d4314d..50c0e5af 100644 --- a/engine/goal.go +++ b/engine/goal.go @@ -582,6 +582,90 @@ const ( goalRetryableMaxAttempts = 12 ) +// goalProviderExhaustedMaxAttempts bounds a single turn's provider-exhausted +// attempts (see provider.AsProviderExhausted, provider/errors.go) before +// promptTurnWithRetry gives up and PursueGoal parks the turn — the same +// weather-shaped budget goalRetryableMaxAttempts uses for ordinary +// overloaded/rate_limited/server_error weather, kept as its own named +// constant (and its own attempt counter — see promptTurnWithRetry) so a +// concurrent overload spell and an account wall in the same turn never +// share, and silently steal from, one another's budget. +// +// Before this constant existed, a provider-exhausted error was wrapped +// provider.MarkPermanent by the adapter (see provider/anthropic/anthropic.go: +// an account-level usage-limit rejection has no distinct HTTP status or wire +// error type, so the adapter can only classify it after regex-matching the +// message) and promptTurnWithRetry's fail-fast permanent branch treated it +// exactly like a structurally malformed request: ONE attempt, no backoff, +// immediate park. That is correct for a malformed request (retrying an +// identical request fails identically forever) but wrong here — an account +// wall lifts on its own, unchanged, the moment the provider's own clock +// rolls over (see provider.ErrKindProviderExhausted's doc comment) — so +// fail-fast silently killed goal supervision on the very first usage-limit +// rejection instead of giving the wall any chance to clear. Live evidence: +// box bx-01m0x8996 parked after "1 permanent-tier attempt(s)" on "You have +// reached your specified API usage limits" and never resumed without an +// operator DELETE + re-register. +// +// RecoverHint (the provider's own "you regain access on " statement) +// is deliberately NEVER parsed into a wait duration — see +// provider.Error.RecoverHint's doc comment: the format varies by provider +// and by plan, so computing an exact wake time from it would be guessing +// dressed as precision. This tier instead rides the exact same jittered +// backoff schedule (goalRetryableBackoff/waitGoalRetryableBackoff) ordinary +// weather uses. goalRetryableMaxAttempts' own doc comment already argues +// this shape correctly: it trades a bounded, generous wait (~30 minutes +// worst case) against the alternative of an unbounded, unattended hold on +// the run slot for however long a quota happens to be spent — a wall that +// clears in seconds (a burst rate limit that reached this classification) +// or minutes resumes automatically within this budget; a wall measured in +// hours or days still exhausts it and parks, honestly classified (see +// goalClassProviderExhausted/classifyGoalWorkerError) rather than either +// pinning the run slot for the outage's full, unknown duration (the exact +// GitHub issue #61 shape this package's Round 7 rework rejected — see +// goalRetryableExhaustedError's doc comment) or silently dying on attempt +// one as it did before this fix. +const goalProviderExhaustedMaxAttempts = goalRetryableMaxAttempts + +// classifyProviderExhausted re-derives retryable/class for a +// provider-exhausted error into the goal loop's local classification +// bookkeeping (see goalClassProviderExhausted's doc comment below), shared +// by promptTurnWithRetry and PursueGoal's worker-turn error handling so the +// two sites can never independently drift on what counts as +// provider-exhausted or which class value marks it — a review finding on +// the fix that introduced this tier: the override was duplicated verbatim +// at both call sites. retryable/class are the caller's own +// provider.AsRetryable(err) result, passed through unchanged when err is +// not provider-exhausted; providerExhausted reports which branch fired, for +// callers (promptTurnWithRetry) that need it for their own dispatch beyond +// just retryable/class. +func classifyProviderExhausted(err error, retryable bool, class provider.RetryableClass) (newRetryable bool, newClass provider.RetryableClass, providerExhausted bool) { + if _, ok := provider.AsProviderExhausted(err); ok { + return true, goalClassProviderExhausted, true + } + return retryable, class, false +} + +// goalClassProviderExhausted is the local provider.RetryableClass marker +// used to record a provider-exhausted worker-turn failure through the +// EXISTING retryable/class bookkeeping (goalWorkerParkedError, +// recordGoalParked, classifyGoalWorkerError, the goal.stalled/goal.parked +// records and their matching events) instead of adding a fourth boolean or +// a new record field throughout this file. provider.AsRetryable(err) itself +// never returns this — a provider-exhausted error is wrapped +// provider.MarkPermanent, not provider.MarkRetryable (see +// provider.ErrKindProviderExhausted's doc comment: adapters mark it +// permanent for ordinary HTTP-retry purposes, since no short backoff +// schedule outlives a monthly quota) — but for THIS package's purposes it +// behaves like weather, not a doomed request, so promptTurnWithRetry and +// PursueGoal's worker-turn handling both fold it into their local +// retryable/class variables explicitly (see the "provider-exhausted" branch +// in each). It is not one of provider/retryable.go's real RetryableClass +// values, so a reader of a goal.stalled/goal.parked record's +// RetryableClass field sees it clearly labeled apart from +// overloaded/rate_limited/server_error/stream_truncated. +const goalClassProviderExhausted provider.RetryableClass = "provider_exhausted" + // goalRetryableDelay returns the base (pre-jitter) backoff for the given // 1-indexed retryable-class attempt that just failed, doubling each time up // to goalRetryableBackoffCap — the same shape as goalRetryDelay, just a @@ -712,8 +796,10 @@ func IsGoalEvaluatorExhausted(err error) bool { } // goalWorkerParkedError is returned by PursueGoal when a worker turn -// exhausts either exhaustion tier — deterministic (goalWorkerRetries) or -// retryable-class (goalRetryableMaxAttempts) — and the loop exit-parks +// exhausts any exhaustion tier — deterministic (goalWorkerRetries), +// retryable-class (goalRetryableMaxAttempts), stream-truncated +// (goalStreamTruncatedMaxAttempts), or provider-exhausted +// (goalProviderExhaustedMaxAttempts) — and the loop exit-parks // instead of clearing the goal. See PursueGoal's doc comment and the // package doc's "Round 7" section: unlike goalEvaluatorExhaustedError // above, reaching this sentinel is NOT a durable "give up" terminal — the @@ -748,6 +834,8 @@ type goalWorkerParkedError struct { func (e *goalWorkerParkedError) Error() string { tier := "deterministic" switch { + case e.class == goalClassProviderExhausted: + tier = "provider-exhausted" case e.permanent: tier = "permanent" case e.retryable: @@ -793,6 +881,14 @@ func IsGoalWorkerParked(err error) bool { // only one ever did. Only ever true when retryable is false. func classifyGoalWorkerError(retryable, permanent bool, class provider.RetryableClass) string { switch { + case class == goalClassProviderExhausted: + // Named explicitly, ahead of the generic retryable case below, so + // an operator reading goal.parked never confuses this with ordinary + // overload/rate-limit weather: this is an account-level usage/quota + // wall (see goalProviderExhaustedMaxAttempts' doc comment), still + // resumable, just parked longer than this turn's budget could ride + // out. + return "provider account usage limit exhausted the retry budget" case retryable: return fmt.Sprintf("provider %s errors exhausted the retry budget", class) case permanent: @@ -803,7 +899,7 @@ func classifyGoalWorkerError(retryable, permanent bool, class provider.Retryable } // recordGoalParked records goal.parked: the terminal PursueGoal reaches -// when a worker turn exhausts either exhaustion tier without clearing the +// when a worker turn exhausts any exhaustion tier without clearing the // goal (see PursueGoal's exit-park branches and classifyGoalWorkerError for // why this record's Reason is classified rather than the raw error text // goal.stalled/goal.eval_failed carry). Deliberately does NOT touch @@ -1171,6 +1267,19 @@ func (s *Session) PursueGoal(ctx context.Context, condition string, opts GoalOpt // accurately, exactly as goal.stalled already does for the same // failing attempt (see promptTurnWithRetry). class, retryable := provider.AsRetryable(err) + // classifyProviderExhausted (shared with promptTurnWithRetry's + // own call below, so the two sites can never drift): by the time + // promptTurnWithRetry returns an error here for a + // provider-exhausted failure, its own tier budget + // (goalProviderExhaustedMaxAttempts) has already been spent + // retrying it, so this IS a genuine exhaustion, not the + // fail-fast-on-attempt-one shape the pre-fix permanent branch + // produced. Reclassifying it as retryable/goalClassProviderExhausted + // here — rather than leaving it to fall into the permanent branch + // below — keeps the resulting goal.parked record and + // classifyGoalWorkerError reason honest: "provider capacity + // exhausted", never "permanent provider error". + retryable, class, _ = classifyProviderExhausted(err, retryable, class) if !retryable && provider.IsContextOverflow(err) { // Issue #62, layer 1: a deterministic context/prompt // overflow gets its own distinct clear reason instead of a @@ -1380,16 +1489,20 @@ func (s *Session) PursueGoal(ctx context.Context, condition string, opts GoalOpt // later — there is no bound on how many times this can recur short of // Prompt gaining a resumable, sub-turn checkpoint, which it does not have. // -// # Three independent budgets, chosen by error classification +// # Four independent budgets, chosen by error classification // // Every failed attempt is first classified via provider.AsRetryable (never -// by matching error text — see provider/retryable.go). A DETERMINISTIC -// failure (not classified retryable) runs the fast path exactly as -// described above: goalWorkerRetries additional attempts, goalRetryDelay's -// short backoff. A RETRYABLE failure runs one of two further loops, -// depending on its class, and neither increments the deterministic counter, -// so surviving either kind of failure costs a turn nothing against -// goalWorkerRetries: +// by matching error text — see provider/retryable.go), then re-checked via +// provider.AsProviderExhausted (see goalClassProviderExhausted's doc +// comment) since an exhausted error is wrapped provider.MarkPermanent, not +// provider.MarkRetryable, and needs its own local override to be treated as +// weather rather than a doomed request. A DETERMINISTIC failure (not +// classified retryable, not provider-exhausted) runs the fast path exactly +// as described above: goalWorkerRetries additional attempts, goalRetryDelay's +// short backoff. A RETRYABLE (or provider-exhausted) failure runs one of +// three further loops, depending on its class, and none of them increments +// the deterministic counter, so surviving any of them costs a turn nothing +// against goalWorkerRetries: // // - provider.RetryableStreamTruncated (a response stream that died before // its terminal event) runs its OWN, much smaller loop, up to @@ -1398,25 +1511,34 @@ func (s *Session) PursueGoal(ctx context.Context, condition string, opts GoalOpt // doc comment for why: waiting longer can never raise a stream ceiling, // so this class must not be allowed to ride the long weather schedule // below. +// - a provider-exhausted failure (an account-level usage/quota wall) runs +// its own loop, up to goalProviderExhaustedMaxAttempts attempts, on the +// SAME long jittered schedule (goalRetryableBackoff) ordinary weather +// uses — see goalProviderExhaustedMaxAttempts' doc comment for why it +// needs its own counter rather than sharing retryableAttempt below. // - every other retryable class runs its own loop, up to // goalRetryableMaxAttempts attempts, spaced by goalRetryableBackoff's // much longer jittered schedule (see the doc comment on that function). // -// If either retryable budget is exhausted, this function returns a -// *goalRetryableExhaustedError wrapping the last error, which PursueGoal -// recognizes and parks the turn instead of clearing the goal (see -// PursueGoal's doc comment and goalRetryableExhaustedError's). +// If any of the three budgets is exhausted, this function returns an error +// PursueGoal recognizes and parks the turn instead of clearing the goal (see +// PursueGoal's doc comment): the ordinary retryable and stream-truncated +// tiers wrap it in *goalRetryableExhaustedError first (see that type's doc +// comment); the provider-exhausted tier returns the bare err, since +// PursueGoal's caller reclassifies it directly via +// provider.AsProviderExhausted rather than needing a dedicated wrapper. // // The non-idempotency gate below (stop retrying once a tool has executed -// this attempt) applies identically to BOTH budgets: retrying after a tool -// call ran is unsafe regardless of why the subsequent call failed. +// this attempt) applies identically to all three retry-shaped budgets: +// retrying after a tool call ran is unsafe regardless of why the subsequent +// call failed. // // gen is the calling turn's goalSnapshot generation, threaded straight // through to recordGoalStalled so a stall record for an attempt is never // journaled once an UpdateGoal has moved the goal past this turn's // generation — see recordGoalStalled and PursueGoal's stale-discard handling. func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, turn int, gen uint64) (attempts int, err error) { - var deterministicAttempt, retryableAttempt, truncatedAttempt int + var deterministicAttempt, retryableAttempt, truncatedAttempt, exhaustedAttempt int // anchorID identifies the message directiveReuseEligible and // dropUnansweredDirective both measure their tail from — the point // immediately before whichever directive is CURRENTLY this turn's live, @@ -1488,11 +1610,28 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur return attempts, err } class, retryable := provider.AsRetryable(err) + // A provider-exhausted error (an account-level usage/quota wall, + // provider.ErrKindProviderExhausted) is wrapped provider.MarkPermanent + // by the adapter, not provider.MarkRetryable — provider.AsRetryable + // above already returned false for it — but for the GOAL LOOP it + // behaves like weather, not a doomed request: the wall lifts on its + // own (see goalProviderExhaustedMaxAttempts' doc comment for the + // live incident this closes). classifyProviderExhausted (shared with + // PursueGoal's own identical call above, so the two sites can never + // drift) folds it into the local retryable/class variables here, + // rather than adding a fourth classification threaded separately + // through every branch below, reusing the exact same tier-dispatch, + // stall-recording, and park-recording machinery the other three + // tiers already exercise. + retryable, class, providerExhausted := classifyProviderExhausted(err, retryable, class) // Stream truncation is retryable-CLASS (it parks on exhaustion, // carries its class on every stall record, and never spends the // deterministic budget) but runs its OWN, much smaller budget on // the SHORT schedule — see goalStreamTruncatedMaxAttempts for why - // it must ride neither of the other two tiers. + // it must ride neither of the other two tiers. Provider-exhausted + // gets its own budget for the same reason: it must never share a + // counter with, and be silently starved or padded by, ordinary + // overload/rate-limit weather in the same turn. truncated := class == provider.RetryableStreamTruncated // exhausted decides, for a retryable failure, whether THIS attempt // is the one that exhausts its tier's budget — computed before the @@ -1500,7 +1639,8 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur // "one more than the retries already spent, including this one, // would meet or exceed the ceiling". exhausted := retryable && ((truncated && truncatedAttempt+1 >= goalStreamTruncatedMaxAttempts) || - (!truncated && retryableAttempt+1 >= goalRetryableMaxAttempts)) + (providerExhausted && exhaustedAttempt+1 >= goalProviderExhaustedMaxAttempts) || + (!truncated && !providerExhausted && retryableAttempt+1 >= goalRetryableMaxAttempts)) // The tool-execution gate is evaluated BEFORE the stall is // journaled so the record's waiting flag tells the truth: an // attempt that ran a tool and then failed is about to stop @@ -1525,7 +1665,7 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur // comment). return attempts, err } - if provider.AsPermanent(err) { + if !providerExhausted && provider.AsPermanent(err) { // NEP-5272 defect 1: a provider error classified permanent (an // HTTP 400 invalid_request_error naming a structurally // malformed request — e.g. an orphaned tool_use left over from @@ -1548,6 +1688,14 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur // PursueGoal's error handling: retryable is false and // IsContextOverflow is false, so this reaches the "every // remaining case parks" branch unmodified). + // + // providerExhausted is excluded from this branch even though + // provider.AsPermanent(err) is ALSO true for it (see + // goalClassProviderExhausted's doc comment: the adapter wraps + // it provider.MarkPermanent too) — an account wall is not a + // malformed request, and must not fail fast on attempt one. It + // falls through to the tier-dispatch section below instead, + // where the providerExhausted branch handles it. return attempts, err } if toolGateStops { @@ -1595,6 +1743,31 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur } continue } + if providerExhausted { + exhaustedAttempt++ + if exhausted { + // Budget exhausted: return the bare underlying err (never + // wrapped) so PursueGoal's caller — which classifies + // directly via provider.AsProviderExhausted(err), the same + // as it does for provider.AsRetryable(err) — sees straight + // through to the real classification without needing a + // dedicated sentinel type. goalRetryableExhaustedError exists + // only so promptTurnWithRetry can signal its OWN budget gave + // out to its own tests (see that type's doc comment); + // provider-exhausted's own tests can assert directly on + // exhaustedAttempt/attempts instead. + return attempts, err + } + // Same long jittered schedule ordinary weather uses (see + // goalProviderExhaustedMaxAttempts' doc comment for why: an + // account wall is worth waiting out, exactly like overload/ + // rate-limit weather, and RecoverHint is deliberately never + // parsed into an exact wake time). + if werr := waitGoalRetryableBackoff(ctx, exhaustedAttempt); werr != nil { + return attempts, werr + } + continue + } retryableAttempt++ if exhausted { return attempts, &goalRetryableExhaustedError{err: err, class: class} @@ -1835,6 +2008,17 @@ func isInterruptedToolResultMessage(m message.Message) bool { // still within its budget ("waiting out provider weather") and false for // the final retryable stall that reports the budget exhausted (the turn is // about to park — see PursueGoal's doc comment). +// +// Reason is err.Error() verbatim for every class except +// goalClassProviderExhausted — a review finding on the fix that introduced +// that class: err.Error() for a provider-exhausted error starts with +// "[permanent] ..." (the adapter wraps it provider.MarkPermanent — see that +// constant's doc comment), which reads as self-contradicting next to this +// SAME record's own Retryable:true/RetryableClass:"provider_exhausted" +// fields. That one class instead renders through classifyGoalWorkerError, +// the same classified rendering recordGoalParked already uses, so a +// goal.stalled record for this class reads consistently with its own +// classification fields instead of echoing raw permanent-branch text. func (s *Session) recordGoalStalled(err error, turn, attempt int, retryable bool, class provider.RetryableClass, waiting bool, gen uint64) bool { s.mu.Lock() if !s.goalActive || s.goalGen != gen { @@ -1842,6 +2026,9 @@ func (s *Session) recordGoalStalled(err error, turn, attempt int, retryable bool return false } reason := err.Error() + if class == goalClassProviderExhausted { + reason = classifyGoalWorkerError(retryable, false, class) + } s.persistGoalLocked(recGoalStalled, goalRecord{ Reason: reason, Turn: turn, @@ -2242,7 +2429,11 @@ func (s *Session) runEvaluator(ctx context.Context, condition string, evaluator if err != nil { return "", err } - content := "GOAL CONDITION:\n" + condition + "\n\nCONVERSATION TRANSCRIPT:\n" + renderConversation(s.History()) + transcript, truncated := renderConversationBounded(s.History(), goalEvaluatorTranscriptBudgetBytes(evaluator)) + if truncated { + transcript = goalEvaluatorTruncationNotice + transcript + } + content := "GOAL CONDITION:\n" + condition + "\n\nCONVERSATION TRANSCRIPT:\n" + transcript req := &provider.Request{ Model: evaluator, System: []string{systemPrompt}, @@ -2355,21 +2546,175 @@ func goalGuidance(condition, reason string) string { } // renderConversation renders history compactly for the evaluator: each message -// role-labeled, each part rendered as text and capped at goalPartCap. +// role-labeled, each part rendered as text and capped at goalPartCap. This +// has no length bound of its own — see renderConversationBounded, which +// runEvaluator actually calls, for the evaluator-model-sized budget. func renderConversation(history []message.Message) string { var b strings.Builder for _, m := range history { - b.WriteString(strings.ToUpper(string(m.Role))) - b.WriteString(":\n") - for _, p := range m.Parts { - b.WriteString(truncateForGoal(renderPart(p))) - b.WriteByte('\n') - } - b.WriteByte('\n') + b.WriteString(renderMessageBlock(m)) } return strings.TrimSpace(b.String()) } +// renderMessageBlock renders one message exactly as renderConversation's +// loop body did before this function was split out — role-labeled, each +// part capped at goalPartCap — so both renderConversation and +// renderConversationBounded share one rendering rule instead of drifting. +func renderMessageBlock(m message.Message) string { + var b strings.Builder + b.WriteString(strings.ToUpper(string(m.Role))) + b.WriteString(":\n") + for _, p := range m.Parts { + b.WriteString(truncateForGoal(renderPart(p))) + b.WriteByte('\n') + } + b.WriteByte('\n') + return b.String() +} + +// goalEvaluatorTruncationNotice prefixes a bounded transcript +// (renderConversationBounded) whenever it actually dropped earlier +// messages, so the evaluator — and an operator reading a goal.eval record +// later — never mistakes a truncated transcript for the whole session. +const goalEvaluatorTruncationNotice = "[earlier conversation omitted to fit the evaluator's context budget]\n\n" + +// renderConversationBounded is renderConversation's budget-aware sibling: +// the fix for the live incident on box bx-01m0x8996, whose evaluator died +// with "context exhausted: prompt 245332 tokens > limit ..." because +// renderConversation(s.History()) has no bound at all — it grows with the +// WHOLE session transcript forever, while the main session is protected by +// automatic compaction (engine/compact.go) and the evaluator never was. +// +// It walks history from the NEWEST message backward, accumulating rendered +// blocks (renderMessageBlock — the exact same per-part goalPartCap rendering +// renderConversation uses, so nothing here changes how one message renders, +// only how many are kept) until the next block would push the running total +// past budgetBytes, then stops and reverses the kept slice back into +// chronological order. +// +// "Prefer summary + tail" falls out of this walk for free rather than +// needing a second summarization path of its own: Compact (engine/ +// compact.go) splices its summary message directly into s.history in place +// of the range it folded, tagged with the compactionSummaryIDTag prefix +// (isCompactionSummaryID). The backward walk here stops the INSTANT it +// includes such a message — even with budget still unspent — because that +// message already IS the compacted record of everything before it walking +// further back would just render already-summarized content a second time. +// So whenever automatic compaction has run at all, the evaluator naturally +// gets exactly "the latest compaction summary plus every raw message after +// it, bounded to what fits" with no new summarization call, no new stored +// field, and no coupling to compaction's internals beyond the one ID-prefix +// helper it already exports to this package. +// +// The newest message is always kept, however large, rather than dropped +// outright: an evaluator call with an empty transcript could never assess +// anything. A single oversized message still gets its own per-part cap from +// renderMessageBlock/truncateForGoal, so this is bounded too, just not by +// budgetBytes. +func renderConversationBounded(history []message.Message, budgetBytes int) (transcript string, truncated bool) { + if len(history) == 0 { + return "", false + } + kept := make([]message.Message, 0, len(history)) + used := 0 + for i := len(history) - 1; i >= 0; i-- { + block := renderMessageBlock(history[i]) + if len(kept) > 0 && budgetBytes > 0 && used+len(block) > budgetBytes { + break + } + kept = append(kept, history[i]) + used += len(block) + if isCompactionSummaryID(history[i].ID) { + break + } + } + for l, r := 0, len(kept)-1; l < r; l, r = l+1, r-1 { + kept[l], kept[r] = kept[r], kept[l] + } + return renderConversation(kept), len(kept) < len(history) +} + +// goalEvaluatorReserveTokens sets aside room, in the evaluator model's OWN +// token budget, for everything in the request besides the transcript: the +// system prompt (goalEvaluatorSystem/goalEvaluatorStrictSystem, both well +// under 700 bytes), the "GOAL CONDITION" preamble and condition text, and +// the 256-token MaxTokens output reply. Deliberately generous relative to +// those actual sizes — goalEvaluatorContextBudgetFraction below is what +// does the real safety work; this constant only keeps a degenerate tiny +// window (goalEvaluatorFallbackContextWindowTokens) from computing a +// negative or implausibly small transcript budget. +const goalEvaluatorReserveTokens = 2048 + +// goalEvaluatorContextBudgetFraction bounds the evaluator transcript to this +// fraction of the evaluator model's context window, after +// goalEvaluatorReserveTokens is set aside — the same "stay well under the +// hard limit, don't cut it exactly at the edge" shape +// defaultCompactionThreshold (engine/compact.go) uses for the main session. +// A fraction well under 1.0 matters more here than it does there: +// bytesPerTokenEstimate's 4-bytes/token conversion (reused from compact.go, +// not reinvented — see goalEvaluatorTranscriptBudgetBytes) is a crude +// heuristic, not the provider's real tokenizer, and this budget has no +// second line of defense the way compaction's threshold-then-hard-overflow +// does: an evaluator call that still overflows has nothing left to retry +// into. +const goalEvaluatorContextBudgetFraction = 0.5 + +// goalEvaluatorFallbackContextWindowTokens is the transcript budget's floor +// for an evaluator model modelmeta has NO ENTRY for at all (an unrecognized +// ref, a custom gateway alias) — mirrors minAutoContextWindowTokens +// (engine/context_window.go), the exact same floor automatic compaction +// refuses to ARM below. A genuinely unrecognized evaluator model still gets +// a real, bounded budget from this floor instead of falling back to the +// fully unbounded renderConversation(s.History()) that produced the "prompt +// 245332 tokens > limit" evaluator failure on bx-01m0x8996 in the first +// place. +// +// This floor must NOT be reused as a stand-in for "the model's real window +// is small" — see goalEvaluatorTranscriptBudgetBytes's doc comment for why +// resolveContextWindow (which folds that case into this same floor, for +// automatic-compaction-ARMING purposes) is deliberately NOT what this +// function calls. +const goalEvaluatorFallbackContextWindowTokens = minAutoContextWindowTokens + +// goalEvaluatorTranscriptBudgetBytes returns the byte budget +// renderConversationBounded must fit the rendered CONVERSATION TRANSCRIPT +// field inside, derived from the EVALUATOR model's own context window — +// never the main session model's, which can be (and on bx-01m0x8996, was — +// a 1,000,000-token model against an evaluator whose own limit the incident +// error names) far larger than the evaluator's own. +// +// This calls modelContextWindowLookup (modelmeta.ContextWindow) DIRECTLY — +// deliberately NOT resolveContextWindow, despite that function existing +// for exactly this "look up a model's context window" job and this +// function's own earlier revision having called it. A review finding +// caught why that was wrong: resolveContextWindow's minAutoContextWindowTokens +// floor answers "should automatic compaction ARM for this window" — a +// window below the floor is treated as bogus/untrustworthy metadata and the +// function reports (0, disabled), identically to a model with NO metadata +// at all. Calling it here silently conflated two different evaluator +// models: a genuinely UNRECOGNIZED one (no table entry — this really +// should fall back to a floor) and a REAL, SMALL, KNOWN one (gpt-4's +// documented 8_192-token window is the table's own example of a legitimate +// entry under the 16k floor) — both funneled into the SAME +// goalEvaluatorFallbackContextWindowTokens (16k) fallback, so a real +// 8_192-token evaluator got a budget roughly TWICE its actual window: the +// exact overflow class this whole fix exists to close. Calling +// modelContextWindowLookup directly and trusting ANY positive, KNOWN +// window — however small — fixes that: the floor here applies only to a +// true "no entry at all" miss, never to "the real entry is small." +func goalEvaluatorTranscriptBudgetBytes(evaluator message.ModelRef) int { + windowTokens, ok := modelContextWindowLookup(evaluator) + if !ok || windowTokens <= 0 { + windowTokens = goalEvaluatorFallbackContextWindowTokens + } + budgetTokens := int(float64(windowTokens)*goalEvaluatorContextBudgetFraction) - goalEvaluatorReserveTokens + if budgetTokens < goalEvaluatorReserveTokens { + budgetTokens = goalEvaluatorReserveTokens + } + return budgetTokens * bytesPerTokenEstimate +} + func renderPart(p message.Part) string { switch v := p.(type) { case *message.Text: diff --git a/engine/goal_evaluator_bound_test.go b/engine/goal_evaluator_bound_test.go new file mode 100644 index 00000000..e898cf8c --- /dev/null +++ b/engine/goal_evaluator_bound_test.go @@ -0,0 +1,213 @@ +package engine + +import ( + "context" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// goalEvaluatorPromptCeilingBytes is a generous, hardcoded safety ceiling — +// deliberately NOT computed from goalEvaluatorTranscriptBudgetBytes or any +// other production constant, so this test still fails meaningfully if a +// future change quietly widens the budget back toward unbounded. It sits +// far below the synthetic transcript this test seeds (see +// hugeSyntheticHistory) and comfortably above the real evaluator budget +// (goalEvaluatorFallbackContextWindowTokens * goalEvaluatorContextBudgetFraction +// * bytesPerTokenEstimate, on the order of tens of KB for the "test/eval" +// fake model, which modelmeta has no entry for and therefore falls back to +// the floor), leaving headroom for the fixed "GOAL CONDITION"/"CONVERSATION +// TRANSCRIPT" wrapper text and the truncation notice. +const goalEvaluatorPromptCeilingBytes = 100_000 + +// hugeSyntheticHistory builds n messages of roughly size bytes each, +// alternating user/assistant, reproducing the shape of a real long-running +// session's transcript (see box bx-01m0x8996's live incident: "prompt +// 245332 tokens > limit") without needing an actual multi-hundred-turn run. +func hugeSyntheticHistory(n, size int) []message.Message { + history := make([]message.Message, 0, n) + filler := strings.Repeat("x", size) + for i := 0; i < n; i++ { + role := message.RoleUser + if i%2 == 1 { + role = message.RoleAssistant + } + history = append(history, message.Message{ + ID: newID("msg"), + Role: role, + Parts: message.Parts{&message.Text{Text: filler}}, + }) + } + return history +} + +// TestPursueGoalEvaluatorPromptBoundedForHugeTranscript is the red-first +// regression test for the first live-evidence defect on box bx-01m0x8996: +// "engine: goal evaluator failed at 5 consecutive turn boundaries: context +// exhausted: prompt 245332 tokens > limit ...". Before this fix, +// runEvaluator built its CONVERSATION TRANSCRIPT field from +// renderConversation(s.History()) with no bound at all — it grows with the +// entire session transcript forever, unlike the main session, which +// automatic compaction protects. +// +// This seeds a synthetic history far larger (300 messages * 3000 bytes == +// ~900KB, comfortably north of what any bounded evaluator budget should +// admit) than a goal evaluator call can safely fit, then drives the actual +// production entry point — PursueGoal, not renderConversationBounded +// directly — so the fix is proven on the path a real caller takes. The +// worker and evaluator are both scripted to succeed on the first turn, so +// nothing about retries or backoff is under test here — only whether the +// REQUEST the evaluator receives fits a sane bound. +func TestPursueGoalEvaluatorPromptBoundedForHugeTranscript(t *testing.T) { + prov := &goalProvider{ + worker: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "all done"}), + }, + eval: [][]provider.Event{ + evalTurn("MET: looks complete"), + }, + } + s := goalSession(t, prov, t.TempDir()) + s.history = hugeSyntheticHistory(300, 3000) + seededBytes := 0 + for _, m := range s.history { + seededBytes += len(m.Parts.Text()) + } + + res, err := s.PursueGoal(context.Background(), "cond", GoalOptions{Evaluator: evalModel}) + if err != nil { + t.Fatalf("PursueGoal error = %v, want nil (a huge transcript must not fail the evaluator)", err) + } + if !res.Achieved { + t.Fatalf("result = %+v, want achieved", res) + } + + var evalReq *provider.Request + for _, r := range prov.requests { + if len(r.Tools) == 0 { + evalReq = r + } + } + if evalReq == nil { + t.Fatal("no evaluator (tool-less) request recorded") + } + content := evalReq.Messages[0].Parts.Text() + if len(content) > goalEvaluatorPromptCeilingBytes { + t.Errorf("evaluator prompt = %d bytes, want <= %d (bounded to the evaluator's own context budget, not the whole %d-byte seeded transcript)", + len(content), goalEvaluatorPromptCeilingBytes, seededBytes) + } + if !strings.Contains(content, goalEvaluatorTruncationNotice) { + t.Error("evaluator prompt does not carry the truncation notice, want it present since the transcript was truncated") + } + // The newest message (the worker's own "all done" turn, plus the + // directive that started it) must survive truncation — an evaluator + // that cannot see what just happened cannot assess anything. + if !strings.Contains(content, "all done") { + t.Error("evaluator prompt lost the most recent turn, want the tail preserved") + } +} + +// TestRenderConversationBoundedPrefersSummaryPlusTail proves +// renderConversationBounded's "prefer summary + tail" behavior directly: a +// leading compaction-summary message (see isCompactionSummaryID, +// engine/compact.go — Compact splices its summary in place of the range it +// folded, tagged with the compactionSummaryIDTag prefix) followed by ordinary +// tail messages must render as exactly [summary, tail...], with the walk +// stopping at the summary rather than continuing further back into +// already-summarized history — even though a much older, oversized filler +// message sits before it that a naive byte-budget walk would otherwise have +// room to include. +func TestRenderConversationBoundedPrefersSummaryPlusTail(t *testing.T) { + oldFiller := message.Message{ + ID: newID("msg"), + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: "ancient message that predates compaction"}}, + } + summary := message.Message{ + ID: newID(compactionSummaryIDTag), + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: CompactionSummaryBanner + "earlier work: set up the repo and wrote tests"}}, + } + tailUser := message.Message{ID: newID("msg"), Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "please continue"}}} + tailAsst := message.Message{ID: newID("msg"), Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "continuing now"}}} + + history := []message.Message{oldFiller, summary, tailUser, tailAsst} + + got, truncated := renderConversationBounded(history, 1_000_000) // budget is not the limiting factor here + if !truncated { + t.Error("truncated = false, want true (the pre-summary filler message was correctly dropped)") + } + if strings.Contains(got, "ancient message") { + t.Errorf("rendered transcript = %q, must not include content from before the compaction summary", got) + } + if !strings.Contains(got, "earlier work: set up the repo") { + t.Errorf("rendered transcript = %q, want the compaction summary's own content present", got) + } + if !strings.Contains(got, "please continue") || !strings.Contains(got, "continuing now") { + t.Errorf("rendered transcript = %q, want both tail messages present", got) + } +} + +// TestRenderConversationBoundedKeepsNewestMessageEvenOverBudget proves the +// one deliberate exception: a budget too small for even one message never +// yields an empty transcript. renderPart/truncateForGoal's own per-part cap +// (goalPartCap) still bounds the single kept message, so this stays finite. +func TestRenderConversationBoundedKeepsNewestMessageEvenOverBudget(t *testing.T) { + history := []message.Message{ + {ID: newID("msg"), Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: strings.Repeat("y", 10_000)}}}, + } + got, truncated := renderConversationBounded(history, 10) // far too small for the message alone + if truncated { + t.Error("truncated = true, want false (a single message is never dropped, only per-part capped)") + } + if got == "" { + t.Error("rendered transcript is empty, want the newest message kept regardless of budget") + } +} + +// TestGoalEvaluatorTranscriptBudgetBytesUsesRealWindowBelowFloor is the +// red-first regression test for a review finding on this fix: +// goalEvaluatorTranscriptBudgetBytes originally called resolveContextWindow, +// which conflates two different things behind the same (0, disabled) +// result — a model with NO modelmeta entry at all, and a model with a REAL, +// KNOWN entry that merely sits below minAutoContextWindowTokens (gpt-4's +// documented 8_192-token window is modelmeta's own example of the latter). +// resolveContextWindow's floor exists to answer "should automatic +// compaction ARM for this window," which is the right question for THAT +// caller but the wrong one here: folding a real 8_192-token evaluator model +// into the SAME goalEvaluatorFallbackContextWindowTokens (16k) fallback a +// genuinely unrecognized model gets would hand it a budget roughly DOUBLE +// its actual context window — the exact overflow class this whole fix +// exists to close. +// +// Uses the modelContextWindowLookup test seam (engine/context_window.go) +// to register one small-but-real window and leave a second model +// genuinely unregistered, then asserts the two get DIFFERENT budgets: the +// known-small one derived from its real 8_192 window, the unknown one from +// the 16k floor — proving the fix no longer treats them as the same case. +func TestGoalEvaluatorTranscriptBudgetBytesUsesRealWindowBelowFloor(t *testing.T) { + orig := modelContextWindowLookup + t.Cleanup(func() { modelContextWindowLookup = orig }) + + small := message.ModelRef{Provider: "test", Model: "small-known"} + unknown := message.ModelRef{Provider: "test", Model: "genuinely-unrecognized"} + modelContextWindowLookup = func(m message.ModelRef) (int, bool) { + if m == small { + return 8_192, true // real, known, but below minAutoContextWindowTokens (16k) + } + return 0, false + } + + smallBudget := goalEvaluatorTranscriptBudgetBytes(small) + unknownBudget := goalEvaluatorTranscriptBudgetBytes(unknown) + + wantSmall := (int(8_192*goalEvaluatorContextBudgetFraction) - goalEvaluatorReserveTokens) * bytesPerTokenEstimate + if smallBudget != wantSmall { + t.Errorf("goalEvaluatorTranscriptBudgetBytes(known 8192-token model) = %d, want %d (derived from the model's REAL window)", smallBudget, wantSmall) + } + if smallBudget >= unknownBudget { + t.Errorf("known-small-window budget (%d bytes) must be strictly SMALLER than the genuinely-unknown-model fallback budget (%d bytes) — a real 8192-token model must never get the same or a larger budget than an unrecognized one", smallBudget, unknownBudget) + } +} diff --git a/engine/goal_provider_exhausted_test.go b/engine/goal_provider_exhausted_test.go new file mode 100644 index 00000000..d2374488 --- /dev/null +++ b/engine/goal_provider_exhausted_test.go @@ -0,0 +1,203 @@ +package engine + +import ( + "context" + "strings" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// providerExhaustedErr builds a fake provider error marked permanent AND +// classified provider.ErrKindProviderExhausted, as if an adapter had +// regex-matched an account-level usage-limit rejection (see +// provider/anthropic/anthropic.go's parseUsageExhaustion and apiError). +// Reproduces the live incident fingerprint verbatim (box bx-01m0x8996): +// "[permanent] anthropic: You have reached your specified API usage +// limits. You will regain access on ." +func providerExhaustedErr() error { + return provider.MarkPermanent(&provider.Error{ + Kind: provider.ErrKindProviderExhausted, + Raw: "anthropic: You have reached your specified API usage limits. You will regain access on 2026-09-01 at 00:00 UTC.", + RecoverHint: "2026-09-01 at 00:00 UTC", + }) +} + +// TestPursueGoalProviderExhaustedRetriesThenRecovers is the red-first +// regression test for the second live-evidence defect on box bx-01m0x8996: +// "engine: goal worker turn parked after 1 permanent-tier attempt(s): +// [permanent] anthropic: You have reached your specified API usage +// limits...". Before this fix, provider.AsProviderExhausted(err) was never +// consulted — an exhausted error is wrapped provider.MarkPermanent (see +// provider.ErrKindProviderExhausted's doc comment: adapters mark it +// permanent for ordinary HTTP-retry purposes, since no short backoff +// schedule outlives a monthly quota), so promptTurnWithRetry's permanent +// fail-fast branch caught it and parked after exactly ONE attempt, with no +// resume path short of an operator DELETE + re-register. +// +// This proves the fix: an account wall that CLEARS within the +// goalProviderExhaustedMaxAttempts budget lets the worker turn — and the +// whole goal — complete normally, with no operator action of any kind. The +// fake provider fails 3 times with the exhausted classification before +// succeeding (more than one attempt, proving retry actually happened; well +// under goalProviderExhaustedMaxAttempts, proving the budget is generous +// enough to ride out a real recovery). +func TestPursueGoalProviderExhaustedRetriesThenRecovers(t *testing.T) { + orig := goalJitterFunc + t.Cleanup(func() { goalJitterFunc = orig }) + goalJitterFunc = func(max time.Duration) time.Duration { return 0 } // deterministic: exactly half the base delay + + synctest.Test(t, func(t *testing.T) { + prov := &goalProvider{ + workerErrN: 3, + workerErr: providerExhaustedErr(), + worker: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "all done"}), + }, + eval: [][]provider.Event{ + evalTurn("MET: looks complete"), + }, + } + var evs []Event + s := goalSession(t, prov, t.TempDir()) + s.cfg.OnEvent = func(ev Event) { evs = append(evs, ev) } + + res, err := s.PursueGoal(context.Background(), "cond", GoalOptions{Evaluator: evalModel}) + if err != nil { + t.Fatalf("PursueGoal error = %v, want nil (a provider-exhausted wall that clears must resume on its own)", err) + } + if !res.Achieved || res.Turns != 1 { + t.Fatalf("result = %+v, want achieved in 1 turn after retrying past 3 provider-exhausted errors", res) + } + if prov.workerCall != 4 { + t.Errorf("worker provider calls = %d, want exactly 4 (3 failures + 1 success)", prov.workerCall) + } + + var stalled int + for _, ev := range evs { + switch ev.Type { + case EventGoalStalled: + stalled++ + if !ev.GoalRetryable { + t.Errorf("goal.stalled event %d: GoalRetryable = false, want true (a clearing account wall is retried, not fail-fast)", stalled) + } + if ev.GoalRetryableClass != string(goalClassProviderExhausted) { + t.Errorf("goal.stalled event %d: GoalRetryableClass = %q, want %q", stalled, ev.GoalRetryableClass, goalClassProviderExhausted) + } + if !ev.GoalWaiting { + t.Errorf("goal.stalled event %d: GoalWaiting = false, want true (budget not exhausted)", stalled) + } + // Review finding: err.Error() for a provider-exhausted + // error is "[permanent] anthropic: ..." (see + // providerExhaustedErr), which would self-contradict this + // SAME record's GoalRetryable:true/GoalRetryableClass: + // "provider_exhausted" fields above. The reason must + // instead read like classifyGoalWorkerError's honest + // classified text, exactly like goal.parked already does. + if strings.Contains(ev.GoalReason, "[permanent]") { + t.Errorf("goal.stalled event %d: GoalReason = %q, must not carry the raw [permanent]-tagged provider text — self-contradicts GoalRetryable=true", stalled, ev.GoalReason) + } + if ev.GoalReason != "provider account usage limit exhausted the retry budget" { + t.Errorf("goal.stalled event %d: GoalReason = %q, want the same honest classified reason goal.parked uses", stalled, ev.GoalReason) + } + case EventGoalParked: + t.Error("goal.parked emitted — a provider-exhausted wall that clears within budget must never park") + } + } + if stalled != 3 { + t.Errorf("goal.stalled events = %d, want 3 (one per failed attempt)", stalled) + } + if cond, ok := s.ActiveGoal(); ok { + t.Errorf("ActiveGoal = %q, active after achievement, want inactive", cond) + } + }) +} + +// TestPursueGoalProviderExhaustedBudgetExhaustedParksHonestly proves the +// other half of the fix: an account wall that outlasts the entire +// goalProviderExhaustedMaxAttempts budget (a quota that resets in days, not +// minutes) still must not pin the run slot forever — it parks, exactly like +// every other exhaustion tier (see the package doc's "Round 7" section) — +// but the classification must be HONEST: "provider account usage limit +// exhausted the retry budget", never "permanent provider error and cannot +// succeed on retry" (which the pre-fix single-attempt fail-fast produced, +// and which is actively wrong for a wall that lifts on its own). The goal +// stays fully active, ready for a later external resume, exactly like every +// other parked tier. +func TestPursueGoalProviderExhaustedBudgetExhaustedParksHonestly(t *testing.T) { + orig := goalJitterFunc + t.Cleanup(func() { goalJitterFunc = orig }) + goalJitterFunc = func(max time.Duration) time.Duration { return 0 } + + synctest.Test(t, func(t *testing.T) { + prov := &goalProvider{ + workerErrN: 1000, // never recovers within the test + workerErr: providerExhaustedErr(), + } + var evs []Event + s := goalSession(t, prov, t.TempDir()) + s.cfg.OnEvent = func(ev Event) { evs = append(evs, ev) } + + res, err := s.PursueGoal(context.Background(), "cond", GoalOptions{Evaluator: evalModel}) + if res != nil { + t.Fatalf("result = %+v, want nil (an exit-park returns an error, not a GoalResult)", res) + } + if !IsGoalWorkerParked(err) { + t.Fatalf("err = %v, want IsGoalWorkerParked", err) + } + if prov.workerCall != goalProviderExhaustedMaxAttempts { + t.Errorf("worker provider calls = %d, want exactly %d (the provider-exhausted budget)", prov.workerCall, goalProviderExhaustedMaxAttempts) + } + + if cond, ok := s.ActiveGoal(); !ok || cond != "cond" { + t.Errorf("ActiveGoal = %q, %v; want the goal left ACTIVE for resume, not cleared", cond, ok) + } + + var sawCleared bool + var parked int + for _, ev := range evs { + switch ev.Type { + case EventGoalCleared: + sawCleared = true + case EventGoalParked: + parked++ + if ev.GoalReason != "provider account usage limit exhausted the retry budget" { + t.Errorf("goal.parked GoalReason = %q, want the honest provider-exhausted reason, not a permanent-error one", ev.GoalReason) + } + if !ev.GoalRetryable { + t.Error("goal.parked GoalRetryable = false, want true (an account wall is not a malformed, unretriable request)") + } + if ev.GoalAttempts != goalProviderExhaustedMaxAttempts { + t.Errorf("goal.parked GoalAttempts = %d, want %d", ev.GoalAttempts, goalProviderExhaustedMaxAttempts) + } + } + } + if sawCleared { + t.Error("goal.cleared emitted — a provider-exhausted budget exhaustion must park, never clear") + } + if parked != 1 { + t.Fatalf("goal.parked events = %d, want exactly 1", parked) + } + }) +} + +// TestClassifyGoalWorkerErrorProviderExhaustedReason is a narrow unit check +// on classifyGoalWorkerError's new branch: the provider-exhausted class must +// render distinctly from both the generic retryable-weather message and the +// permanent-error one, regardless of what the permanent bool happens to be +// (mirrors the mutual-exclusion documented on goalWorkerParkedError.permanent +// — this class is always reached with permanent already false, but the +// function itself must not depend on the caller getting that right). +func TestClassifyGoalWorkerErrorProviderExhaustedReason(t *testing.T) { + got := classifyGoalWorkerError(true, false, goalClassProviderExhausted) + want := "provider account usage limit exhausted the retry budget" + if got != want { + t.Errorf("classifyGoalWorkerError(true, false, provider_exhausted) = %q, want %q", got, want) + } + if strings.Contains(got, "overloaded") || strings.Contains(got, string(provider.RetryableOverloaded)) { + t.Errorf("classifyGoalWorkerError provider-exhausted reason = %q, must not read like ordinary weather", got) + } +} From b855740ffe841ab98e009a1279282cfef8c97501 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 12:56:49 -0400 Subject: [PATCH 02/95] fix(engine): revive a Reap()-ed descendant for task verbs (#186) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): revive a Reap()-ed descendant for task verbs A live incident: a parent ran the documented `task send` re-drive-a- settled-child flow (PR #155) against a child that HAD in fact finished and had its result queued for delivery, and got back `task: no such session`. The child's session log was intact on disk, and the server's own GET /session/ served it 200 via its disk fallback — but the child had already been Reap()-ed from SessionManager's live tree (Reap collects a done/failed/canceled leaf the instant it settles), and CancelDescendant/DescendantInfo/DescendantTranscript/ SendToDescendant all resolved strictly against the live tree. Whether a follow-up to a settled child worked depended on internal reap timing the parent has no way to observe: a broken contract, not a race the caller could reasonably guard against. resolveOrReviveDescendantLocked is the shared fix: on a live-tree miss, it falls back to disk (LoadSession) and validates ancestry from the target's own DURABLE task_parent_id chain (durableAncestorChainHas), never from live state alone — that is exactly what Reap already erased. Only a target with no session log on disk either, or whose durable lineage does not reach the caller, still answers ErrUnknownSession/ ErrNotDescendant. The disk-bound half runs with m.mu released (mirroring recoverCrashedChildrenLocked's existing "cold-load outside the tree lock" convention) and re-validates m.nodes on reacquiring it, so a concurrent adopt of the same id — another Spawn, AdoptReloaded, or a second racing revival — always has exactly one winner, the same "already managed, defer to the winner" rule AdoptReloaded's existing callers already follow. The four verbs were given deliberately different revival behavior rather than one uniform rule: - `send` RE-ADOPTS the revived child via adoptReloadedLocked — the same adopt-on-first-sight path AdoptReloaded's public wrapper and handleSpawnChild's parent-lookup fallback already use for a cold reload resolved from a caller-supplied id — and re-runs it exactly like a settled-but-unreaped child. A second adoption path was considered and rejected: reusing the tested one is what makes budgetedByChild's existing Reap-survival guarantee (see its own doc comment) apply for free, so the revived child cannot double-credit its already-spent usage into the tree budget. - `status`/`log` serve the disk-loaded state directly (durableSnapshot/deriveSettledStatus, a new read-only sibling of restoreKnownStatusLocked's identical classification) WITHOUT re-adopting: a poll-shaped read verb should not have the side effect of pinning a reaped descendant back into memory. - `cancel` on a reaped target is a no-op success reporting its real terminal status, never StatusCanceled — mirroring cancelOneNodeLocked's own "already-terminal, status left untouched" rule for a still-live target: Reap only ever collects an already-finalized leaf, so there is provably nothing left in flight to interrupt. Semantic change: `task send`/`status`/`log`/`cancel` (and their SessionManager-level counterparts) now answer identically for a settled-but-unreaped descendant and a Reap()-ed one; only `send` and `cancel` differ in observable side effects (re-adopt vs. no-op) from before. Ancestry enforcement, unknown-id handling, and the live-tree fast path are unchanged. Verification: new tests in engine/task_verbs_revival_test.go cover send reviving a reaped child (a scripted provider proves a genuinely new turn, not a replay), status/log serving a reaped child without re-adopting it, cancel no-op'ing on one, unknown ids and ancestry violations still erroring after a reap, and usageByRoot crediting a two-turn child's usage exactly once across the reap+revive. The send case is red-verified: reverting resolveOrReviveDescendantLocked reproduces the exact incident error, "engine: unknown session id". The single-winner concurrency property (send revival racing a concurrent AdoptReloaded of the same id) passed 1000 runs at GOMAXPROCS=2 under -race. Full `go test -race ./...`, `go vet ./...`, and `gofmt -l .` are clean. * perf(engine): walk the durable ancestor chain by header reads alone Review found durableAncestorChainHas calling LoadSession per hop — a full log replay to read one header field, O(depth) whole-transcript parses in the unlocked revival window for deep chains. The header record is the first line of every log (ensureLog writes it first, in the same atomic buffer as the model record), so one bounded line read per hop suffices: new loadSessionTaskParent opens the file, reads the first line through a capped reader, and fails loudly if that line is not a recSession header rather than guessing. Verification: new TestLoadSessionTaskParentReadsHeaderOnly covers the happy path and the non-header first record; revival suite and full engine suite green under -race; vet and gofmt clean. --- AGENTS.md | 43 ++- engine/session_manager.go | 411 ++++++++++++++++++++++++++--- engine/task_verbs_revival_test.go | 419 ++++++++++++++++++++++++++++++ 3 files changed, 836 insertions(+), 37 deletions(-) create mode 100644 engine/task_verbs_revival_test.go diff --git a/AGENTS.md b/AGENTS.md index e4fff88a..493564d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1841,14 +1841,49 @@ stops reporting a wall it already got past; `finalizeTurn`'s keep snapshotting a classification no live cancellation sets and `restoreKnownStatusLocked` restores as empty. +A REAPED descendant is still resolvable, not "no such session." `Reap` +collects a done/failed/canceled LEAF the instant it settles (its own doc +comment), which a caller that spawned it has no way to observe before +asking about it again — a live incident hit exactly this: `task send` to +a settled child answered `no such session` depending on internal reap +timing the parent could not see. `resolveOrReviveDescendantLocked` +(`engine/session_manager.go`), the shared first step of all four verbs, +falls back to disk when a live-tree lookup misses: `LoadSession` the +target, then confirm ancestry from its own DURABLE `task_parent_id` +chain (`durableAncestorChainHas`) — never from live state alone, which is +exactly what `Reap` already erased. Only a target with no session log on +disk either, or whose durable lineage does not reach the caller, still +answers `ErrUnknownSession`/`ErrNotDescendant`. The four verbs then +diverge on what a successful disk resolution does: `send` RE-ADOPTS the +revived child into the tree (`adoptReloadedLocked`, the same +adopt-on-first-sight path `AdoptReloaded`/`handleSpawnChild`'s +parent-lookup fallback already use) and re-runs it exactly like a +settled-but-unreaped child — `budgetedByChild` surviving `Reap` by design +is what stops that re-adopt from double-crediting its already-spent +usage. `status`/`log` serve the disk-loaded state directly +(`durableSnapshot`/`deriveSettledStatus`) WITHOUT re-adopting: a +read-only, poll-shaped verb must not have the side effect of pinning a +reaped descendant back into memory. `cancel` on a reaped target is a +no-op success (nothing left to interrupt) reporting its real terminal +status, never `StatusCanceled`. The disk-bound half of this resolution +runs with `m.mu` released — one slow disk read must never stall every +other session's `Info`/`Reap`/`Spawn`/`Send` call — and re-validates +`m.nodes[targetID]` on reacquiring the lock, so a concurrent adopt of the +same id (another `Spawn`, `AdoptReloaded`, or a second racing revival) +always wins single-handedly: whichever adoption reaches `m.nodes` first +is authoritative, and the loser's own "already managed" is ignored, the +same rule `AdoptReloaded`'s existing callers already follow for that +race. + A parent can read a dead child's tail. The `task` tool's `log` verb (`runTaskLog`, `engine/task_tool.go`, over `SessionManager.DescendantTranscript`) returns the last N transcript entries of a descendant, LIVING OR DEAD, under the same ancestor gate -(`isDescendantLocked`) cancel/status/send use — a terminal node keeps its -`*Session`, history included, until `Reap`, so no reload and no disk read -is involved; a REAPED descendant answers "no such session" like every -other verb. It is bounded on three axes, because its output lands in the +(`isDescendantLocked`, or the disk-backed lineage check above once +`Reap` has removed the live node) cancel/status/send use — a terminal +node keeps its `*Session`, history included, until `Reap`, so no reload +and no disk read is involved for a still-tracked descendant. It is +bounded on three axes, because its output lands in the PARENT's context and replays on every later turn: `tail` (default 20, clamped at 100, a negative value is an error), a per-entry rune cap, and a total rune budget filled NEWEST-first so the messages nearest a death diff --git a/engine/session_manager.go b/engine/session_manager.go index 9cc92417..6ebc09ff 100644 --- a/engine/session_manager.go +++ b/engine/session_manager.go @@ -8,8 +8,11 @@ import ( "sync" "time" + "bufio" + "encoding/json" "github.com/majorcontext/harness/message" "github.com/majorcontext/harness/provider" + "os" ) // SessionStatus is a session's lifecycle state as tracked by a @@ -1200,6 +1203,81 @@ func (m *SessionManager) restoreKnownStatusLocked(n *sessionNode, s *Session) { m.usageByRoot[n.rootID] = u } +// deriveSettledStatus computes a session's terminal lifecycle facts — +// status, result, fail reason, fail kind — purely from ITS OWN durable +// state (committedTurnOutcome, or the HasHistoryOrSpawnedChildren/ +// settledSuccessResult legacy fallback), with NO SessionManager +// bookkeeping touched: no sessionNode field set, no budget map folded, no +// markChangedLocked. It is the read-only counterpart to +// restoreKnownStatusLocked's own identical classification (same two +// signals, same nodeStatusForOutcome/settledSuccessResult/ +// unknownLegacyOutcomeFailReason primitives, in the same preference +// order — see that method's own doc comment for the full reasoning behind +// each branch, which this function deliberately does not repeat), used by +// durableSnapshot to answer the task tool's status/log verbs for a +// descendant Reap has already removed from the live tree WITHOUT +// re-adopting it (see resolveOrReviveDescendantLocked's own doc comment +// for why a read-only verb must not mutate the tree the way send's own +// revival deliberately does). +// +// ok is false only when NEITHER signal proves s ever ran a turn at all — +// genuinely fresh, with nothing settled to report; each caller applies +// its own "genuinely fresh" default (StatusIdle) rather than this +// function guessing one on the caller's behalf. +func deriveSettledStatus(s *Session) (status SessionStatus, result, failReason, failKind string, ok bool) { + if committed, has := s.committedTurnOutcome(); has { + switch nodeStatusForOutcome(committed) { + case StatusCanceled: + // Mirrors restoreKnownStatusLocked's identical Canceled case: + // result/failReason/failKind stay empty. Restoring + // committed.FailReason ("canceled", the fixed text + // finalizeTurn's alreadyCanceled branch puts in the + // PARENT-facing notification) here would invent a value a + // live cancellation never actually sets. + return StatusCanceled, "", "", "", true + case StatusDone: + return StatusDone, committed.Result, "", "", true + default: + return StatusFailed, "", committed.FailReason, committed.FailKind, true + } + } + if s.HasHistoryOrSpawnedChildren() { + if result, has := s.settledSuccessResult(); has { + return StatusDone, result, "", "", true + } + return StatusFailed, "", unknownLegacyOutcomeFailReason, "", true + } + return "", "", "", "", false +} + +// durableSnapshot builds a read-only SessionNode directly from a +// disk-loaded, NOT tree-adopted *Session — sessionNode.snapshot()'s +// counterpart for a descendant that only exists on disk right now (see +// resolveOrReviveDescendantLocked). Children is durable-only +// (sess.SpawnedChildIDs()): there is no live half to union via +// mergeChildIDs, since sess was never adopted into m.nodes. +func durableSnapshot(id, parentID string, depth int, sess *Session) SessionNode { + children := sess.SpawnedChildIDs() + if children == nil { + children = []string{} + } + snap := SessionNode{ + ID: id, + ParentID: parentID, + Depth: depth, + Status: StatusIdle, + Children: children, + AgentType: sess.TaskAgentType(), + } + if status, result, failReason, failKind, ok := deriveSettledStatus(sess); ok { + snap.Status = status + snap.Result = result + snap.FailReason = failReason + snap.FailKind = failKind + } + return snap +} + // recoverCrashedChildrenLocked sweeps n's own durably-recorded children // (Session.SpawnedChildIDs, engine.go) for any whose turn crashed and was // never recovered, adopting (and thereby recovering) each one found — a @@ -3008,6 +3086,185 @@ func (m *SessionManager) isDescendantLocked(ancestorID, targetID string) bool { return false } +// durableAncestorChainHas reports whether callerID appears anywhere in a +// disk-resolved descendant's own durable TaskParentID chain, starting from +// startParentID (the descendant's own TaskParentID) and walking strictly +// through LoadSession — never the live tree, never m.mu — so it is safe to +// call from resolveOrReviveDescendantLocked's UNLOCKED window (see that +// method's own doc comment for why the disk-bound half of a revival must +// not hold m.mu). +// +// TaskParentID is set once, durably, at Spawn time and never changes +// afterward (Config.TaskParentID's own doc comment), so this answer is +// exactly as authoritative as isDescendantLocked's live-tree walk is for a +// still-tracked chain — it is simply reading the SAME lineage fact from +// disk instead of from memory, for the hops Reap has already erased from +// memory. +// +// maxHops bounds the walk at m.maxDepth: Spawn's own depth-limit gate +// (ErrDepthLimit) makes a genuine chain at most m.maxDepth hops long, so a +// walk that has not reached "" or callerID within that many hops cannot be +// a real lineage — the bound stops the function from following however a +// corrupted or adversarial TaskParentID chain might otherwise be shaped, +// rather than trusting disk content to be well-formed indefinitely. +func durableAncestorChainHas(cfg Config, startParentID, callerID string, maxHops int) bool { + parentID := startParentID + for hops := 0; parentID != "" && hops <= maxHops; hops++ { + if parentID == callerID { + return true + } + // Header-only read, not LoadSession: this walk needs exactly one + // field per hop (the next TaskParentID), and LoadSession replays + // the WHOLE log — O(chain depth) full-transcript parses in the + // unlocked window for deep chains (a review finding). The header + // record is the first line of every log (ensureLog writes it + // first, in the same atomic buffer as the model record), so one + // bounded line read per hop suffices. + next, err := loadSessionTaskParent(cfg, parentID) + if err != nil { + return false + } + parentID = next + } + return false +} + +// loadSessionTaskParent reads a session log's durable TaskParentID from its +// header record alone — the first line of the file — without replaying the +// log. A file whose first record is not a recSession header (impossible for +// a log this engine wrote, ensureLog's single-write ordering) reports an +// error rather than guessing. +func loadSessionTaskParent(cfg Config, id string) (string, error) { + if !ValidSessionID(id) { + return "", fmt.Errorf("%w: %q", ErrInvalidSessionID, id) + } + f, err := os.Open(sessionPath(cfg.SessionDir, id)) + if err != nil { + return "", err + } + defer f.Close() + // One bounded line: headers are small (a recSession record), but a + // generous cap keeps a pathological first line from slurping a huge + // log into memory. + r := bufio.NewReaderSize(f, 64*1024) + line, err := r.ReadString('\n') + if err != nil && line == "" { + return "", err + } + var rec record + if err := json.Unmarshal([]byte(line), &rec); err != nil { + return "", fmt.Errorf("session %s: unparseable header line: %w", id, err) + } + if rec.Type != recSession { + return "", fmt.Errorf("session %s: first record is %q, want %q", id, rec.Type, recSession) + } + return rec.TaskParentID, nil +} + +// resolveOrReviveDescendantLocked resolves targetID against m's live tree, +// falling back to disk when Reap has already collected it as a terminal +// leaf — Reap's own doc comment: a done/failed/canceled child is eligible +// the instant it settles, which races a caller (the `task` tool's own +// cancel/status/send/log verbs) that spawned it and asks about it again +// before it can observe that timing. Before this method existed, that race +// answered "no such session" — internal reap timing the caller has no way +// to observe, turning a documented "send a follow-up to a settled child" +// flow into a coin flip. Every one of the four verbs now shares this same +// "live miss -> disk fallback" step, so a caller cannot tell a +// settled-but-unreaped child apart from a reaped one by how the reply +// differs. +// +// callerID MUST already be confirmed tracked by the caller (every verb's +// own first check, unchanged) — this method assumes it and does not +// re-verify: callerID is definitionally the session driving THIS very +// tool call, StatusRunning for the call's whole duration, and Reap never +// removes a running node. +// +// # Two return shapes +// +// - node non-nil, loaded nil: targetID is live-tracked, exactly the +// unchanged fast path every existing caller/test already covers. The +// caller still owns the ancestry check (isDescendantLocked) — this +// method does not run it for a live node, so a live sibling or an +// unrelated live session is returned here too, same as before this +// fix, for the caller to reject. +// - node nil, loaded non-nil: targetID was NOT live-tracked and has been +// resolved from disk instead, with ancestry ALREADY confirmed against +// its own durable TaskParentID chain (durableAncestorChainHas) — the +// caller does not need to (and must not, for a read-only verb — see +// each verb's own doc comment) run isDescendantLocked again, since +// there is no live node to run it against. depth is the disk-resolved +// descendant's own depth, computed with the identical preference order +// adoptReloadedLocked's own doc comment documents (durable TaskDepth +// first, else a tracked live parent's depth+1, else the maxDepth +// refusal sentinel) — needed by the read-only verbs' durableSnapshot, +// and redundant-but-harmless for send's own adoptReloadedLocked call, +// which recomputes it the same way internally. +// +// # Locking — "Locked" despite not holding m.mu throughout +// +// Called with m.mu held; always RETURNS with m.mu held, so every caller +// treats this as an ordinary *Locked method regardless of which path it +// took — mirrors recoverCrashedChildrenLocked's own identical convention +// (see its doc comment's "Not actually held throughout" section) for the +// same reason: the live-tree fast path above does no I/O and never +// unlocks, but the disk fallback below calls LoadSession (real disk I/O, +// potentially several times across durableAncestorChainHas' own walk) and +// releases m.mu for that entire span, so one slow disk read cannot stall +// every other session's Info/Reap/Spawn/Send call in the process. +// +// # Single-winner under a concurrent adopt of the SAME id +// +// m.mu is re-acquired before this method returns, and targetID is +// re-checked against m.nodes immediately on reacquiring it: if some other +// path (a concurrent Spawn, AdoptReloaded, or another goroutine's own call +// into this same method for the same id) adopted targetID while this one +// was reading disk, THAT live node is authoritative and is returned +// instead of the disk copy this call just loaded — never both. This is the +// same "already managed... a concurrent adopt may have won the race, use +// its winner instead" rule AdoptReloaded's own callers already follow +// (handleSpawnChild, server/session_tree.go) rather than a second, bespoke +// race primitive: whichever adoption reaches m.nodes[targetID] first, by +// construction only ever one thing can occupy that map entry at a time, so +// there is no window left in which two different *Session objects could +// both end up backing the same on-disk log. +func (m *SessionManager) resolveOrReviveDescendantLocked(callerID, targetID string) (node *sessionNode, loaded *Session, depth int, err error) { + if n, ok := m.nodes[targetID]; ok { + return n, nil, 0, nil + } + cfg := m.nodes[callerID].session.configSnapshot() + m.mu.Unlock() + sess, lerr := LoadSession(cfg, targetID) + var ancestorOK bool + if lerr == nil { + ancestorOK = sess.hasTaskParent() && durableAncestorChainHas(cfg, sess.TaskParentID(), callerID, m.maxDepth) + } + m.mu.Lock() + if n, ok := m.nodes[targetID]; ok { + // Someone else adopted targetID while this call was reading disk — + // single winner: use it, exactly like AdoptReloaded's own callers + // treat this identical race (see this method's own doc comment). + return n, nil, 0, nil + } + if lerr != nil { + return nil, nil, 0, fmt.Errorf("%w: %s", ErrUnknownSession, targetID) + } + if !ancestorOK { + return nil, nil, 0, fmt.Errorf("%w: %s", ErrNotDescendant, targetID) + } + // Same depth-resolution preference order as adoptReloadedLocked's own + // doc comment documents — see this method's own doc comment for why + // duplicating it here (rather than only inside adoptReloadedLocked) is + // required: the read-only verbs need a depth WITHOUT adopting anything. + depth = m.maxDepth + if d := sess.TaskDepth(); d > 0 { + depth = d + } else if p, ok := m.nodes[sess.TaskParentID()]; ok { + depth = p.depth + 1 + } + return nil, sess, depth, nil +} + // CancelDescendant cancels targetID's entire subtree on callerID's // behalf, after confirming callerID is a live ancestor of targetID (see // isDescendantLocked) — the SessionManager side of the `task` tool's @@ -3041,17 +3298,45 @@ func (m *SessionManager) isDescendantLocked(ancestorID, targetID string) bool { // the outcome inside the same critical section that produced it closes // that gap by construction rather than papering over it with a guess. // -// Returns ErrUnknownSession if either id is not tracked, ErrNotDescendant -// if targetID is not callerID's descendant. +// A target Reap has already collected (see resolveOrReviveDescendantLocked) +// is a NO-OP success, never an error and never re-adopted: Reap only ever +// removes an ALREADY-terminal, ALREADY-finalized leaf (its own doc +// comment) — a live subtree with anything genuinely still running could +// never have been eligible in the first place — so "cancel" against a +// disk-revived id can never actually be interrupting real in-flight work. +// Reporting its real terminal status (never StatusCanceled — this call did +// not, in fact, cancel anything) mirrors cancelOneNodeLocked's own +// "already-terminal -> status left untouched" rule for a LIVE target (see +// its own doc comment): the disk-revived case is simply that same rule +// extended past the point Reap removed the node, and — like status/log, +// unlike send — needs no re-adoption to answer it: an operation whose +// whole point is "stop something" has nothing left to mutate once the +// target is confirmed to be sitting completely idle on disk. +// +// Returns ErrUnknownSession if either id is not tracked live and not +// resolvable from disk, ErrNotDescendant if targetID's durable lineage +// does not reach callerID. func (m *SessionManager) CancelDescendant(callerID, targetID string) (SessionStatus, error) { m.mu.Lock() defer m.mu.Unlock() if _, ok := m.nodes[callerID]; !ok { return "", fmt.Errorf("%w: %s", ErrUnknownSession, callerID) } - n, ok := m.nodes[targetID] - if !ok { - return "", fmt.Errorf("%w: %s", ErrUnknownSession, targetID) + n, loaded, _, err := m.resolveOrReviveDescendantLocked(callerID, targetID) + if err != nil { + return "", err + } + if n == nil { + status, _, _, _, ok := deriveSettledStatus(loaded) + if !ok { + // Reap can only ever have collected an already-finalized leaf, + // so this is unreachable in practice — kept as a defensive + // default (matching adoptLocked's own StatusIdle default for a + // node with nothing yet to report) rather than an assumption a + // future disk-resolution caller might rely on silently. + status = StatusIdle + } + return status, nil } if !m.isDescendantLocked(callerID, targetID) { return "", fmt.Errorf("%w: %s", ErrNotDescendant, targetID) @@ -3072,17 +3357,31 @@ func (m *SessionManager) CancelDescendant(callerID, targetID string) (SessionSta // already establishes for this package — so it reflects targetID's // current total, not a stale or partial snapshot. // -// Returns ErrUnknownSession if either id is not tracked, ErrNotDescendant -// if targetID is not callerID's descendant. +// A target Reap has already collected (see resolveOrReviveDescendantLocked) +// is served straight from its own disk-loaded state, WITHOUT re-adopting +// it into the tree: status is a read-only, poll-shaped verb, and a caller +// asking about a descendant's outcome must not have the side effect of +// pinning that descendant back into m.nodes (and, via the send-verb-only +// budget fold, extending its usageByRoot credit window) purely for having +// looked at it. Contrast SendToDescendant, which DOES re-adopt a revived +// target — see that method's own doc comment for why a send genuinely +// needs a live node to act through and a status read does not. +// +// Returns ErrUnknownSession if either id is not tracked live and not +// resolvable from disk, ErrNotDescendant if targetID's durable lineage +// does not reach callerID. func (m *SessionManager) DescendantInfo(callerID, targetID string) (SessionNode, provider.Usage, error) { m.mu.Lock() defer m.mu.Unlock() if _, ok := m.nodes[callerID]; !ok { return SessionNode{}, provider.Usage{}, fmt.Errorf("%w: %s", ErrUnknownSession, callerID) } - n, ok := m.nodes[targetID] - if !ok { - return SessionNode{}, provider.Usage{}, fmt.Errorf("%w: %s", ErrUnknownSession, targetID) + n, loaded, depth, err := m.resolveOrReviveDescendantLocked(callerID, targetID) + if err != nil { + return SessionNode{}, provider.Usage{}, err + } + if n == nil { + return durableSnapshot(targetID, loaded.TaskParentID(), depth, loaded), loaded.Usage(), nil } if !m.isDescendantLocked(callerID, targetID) { return SessionNode{}, provider.Usage{}, fmt.Errorf("%w: %s", ErrNotDescendant, targetID) @@ -3123,14 +3422,22 @@ func (m *SessionManager) DescendantInfo(callerID, targetID string) (SessionNode, // collects it, and its *Session — history included — stays with it. A // parent whose child died therefore reads the child's own last messages // through the same call it would use on a running one, with no session -// reload and no disk read. A REAPED descendant is gone from the tree and -// answers ErrUnknownSession, exactly as it does for every other verb; the -// child's durable log still exists on disk for an operator to read out of -// band. +// reload and no disk read. +// +// A REAPED descendant is resolved from disk instead (see +// resolveOrReviveDescendantLocked) and its history read straight off that +// disk-loaded *Session, WITHOUT re-adopting it into the tree — log is a +// read-only, poll-shaped verb exactly like status, and DescendantInfo's +// own doc comment explains why a read must not have the side effect of +// pinning a Reaped descendant back into m.nodes. The child's durable log +// existing on disk is what makes this possible at all; only the "which +// object reads it" story differs from the still-tracked case. // // History is read through n.session.History() (which takes s.mu) while // m.mu is still held — m.mu outer, session mu inner, the same nesting -// DescendantInfo's own Usage() read establishes. +// DescendantInfo's own Usage() read establishes — or, for a disk-revived +// target, straight off the freshly loaded *Session, which no other +// goroutine can yet observe. // // tail <= 0 returns no messages rather than the whole history: the caller // decides the bound, and a zero must never silently mean "everything" on @@ -3143,23 +3450,31 @@ func (m *SessionManager) DescendantInfo(callerID, targetID string) (SessionNode, // survived the bound — and a separate follow-up read could race the // descendant's own next turn appending to it. // -// Returns ErrUnknownSession if either id is not tracked, ErrNotDescendant -// if targetID is not callerID's descendant. +// Returns ErrUnknownSession if either id is not tracked live and not +// resolvable from disk, ErrNotDescendant if targetID's durable lineage +// does not reach callerID. func (m *SessionManager) DescendantTranscript(callerID, targetID string, tail int) (node SessionNode, msgs []message.Message, total int, err error) { m.mu.Lock() defer m.mu.Unlock() if _, ok := m.nodes[callerID]; !ok { return SessionNode{}, nil, 0, fmt.Errorf("%w: %s", ErrUnknownSession, callerID) } - n, ok := m.nodes[targetID] - if !ok { - return SessionNode{}, nil, 0, fmt.Errorf("%w: %s", ErrUnknownSession, targetID) + n, loaded, depth, rerr := m.resolveOrReviveDescendantLocked(callerID, targetID) + if rerr != nil { + return SessionNode{}, nil, 0, rerr } - if !m.isDescendantLocked(callerID, targetID) { - return SessionNode{}, nil, 0, fmt.Errorf("%w: %s", ErrNotDescendant, targetID) + var snap SessionNode + var history []message.Message + if n == nil { + snap = durableSnapshot(targetID, loaded.TaskParentID(), depth, loaded) + history = loaded.History() + } else { + if !m.isDescendantLocked(callerID, targetID) { + return SessionNode{}, nil, 0, fmt.Errorf("%w: %s", ErrNotDescendant, targetID) + } + snap = n.snapshot() + history = n.session.History() } - snap := n.snapshot() - history := n.session.History() total = len(history) if tail <= 0 { return snap, nil, total, nil @@ -3261,11 +3576,33 @@ func mergeChildIDs(durable, live []string) []string { // eliminating the window entirely rather than merely narrowing or // documenting it. // -// Returns ErrUnknownSession if either id is not tracked, ErrNotDescendant -// if targetID is not callerID's descendant, ErrSessionCanceled if -// targetID is canceled, or ErrConcurrencyLimit if the tree is already at -// its running-children cap (settled-target restart path only — a -// running target's enqueue never touches this budget). +// A target Reap has already collected is REVIVED, not refused: resolved +// from disk and re-adopted into the tree via adoptReloadedLocked — the +// SAME adopt-on-first-sight machinery AdoptReloaded's public wrapper and +// handleSpawnChild's own parent-lookup fallback already use for a cold +// reload resolved from a caller-supplied id (server/session_tree.go), not +// a second, bespoke adoption path — so a revived settled child re-runs +// through EXACTLY the settled-target restart path documented below, the +// same as a settled child Reap simply had not gotten to yet. Unlike the +// read-only status/log verbs (see DescendantInfo's own doc comment for +// why THEY must not do this), send has no read-only option: delivering a +// message means starting a turn, and a turn needs a live node to run +// through — reserveSendLocked, drainQueueAndPrompt, and finalizeTurn all +// operate on a *sessionNode, not a bare *Session. recover=true (matching +// AdoptReloaded's own public wrapper) restores the revived node's real +// terminal status/result from its own committed outcome +// (restoreKnownStatusLocked) and folds its already-spent usage into the +// tree budget through budgetedByChild, which survives Reap by design +// exactly so this fold cannot double-credit it (see that field's own doc +// comment) — the revived node is indistinguishable, from here on, from a +// settled child Reap simply had not collected yet. +// +// Returns ErrUnknownSession if either id is not tracked live and not +// resolvable from disk, ErrNotDescendant if targetID's durable lineage +// does not reach callerID, ErrSessionCanceled if targetID is canceled, or +// ErrConcurrencyLimit if the tree is already at its running-children cap +// (settled-target restart path only — a running target's enqueue never +// touches this budget). func (m *SessionManager) SendToDescendant(callerID, targetID, text string) (queued bool, err error) { // Validate and trim ONCE, before either delivery path — a review // finding: the running-target branch rejected blank text while the @@ -3284,12 +3621,20 @@ func (m *SessionManager) SendToDescendant(callerID, targetID, text string) (queu m.mu.Unlock() return false, fmt.Errorf("%w: %s", ErrUnknownSession, callerID) } - n, ok := m.nodes[targetID] - if !ok { + n, loaded, _, rerr := m.resolveOrReviveDescendantLocked(callerID, targetID) + if rerr != nil { m.mu.Unlock() - return false, fmt.Errorf("%w: %s", ErrUnknownSession, targetID) + return false, rerr } - if !m.isDescendantLocked(callerID, targetID) { + if n == nil { + // loaded != nil: a disk revival, ancestry already durably + // confirmed by resolveOrReviveDescendantLocked — adopt it directly + // via the *Locked variant (not the public AdoptReloaded, which + // would re-acquire m.mu) since resolveOrReviveDescendantLocked's + // own re-check on reacquiring m.mu already proved targetID is NOT + // currently tracked. + n = m.adoptReloadedLocked(loaded, true) + } else if !m.isDescendantLocked(callerID, targetID) { m.mu.Unlock() return false, fmt.Errorf("%w: %s", ErrNotDescendant, targetID) } diff --git a/engine/task_verbs_revival_test.go b/engine/task_verbs_revival_test.go new file mode 100644 index 00000000..6fa93cc3 --- /dev/null +++ b/engine/task_verbs_revival_test.go @@ -0,0 +1,419 @@ +// Tests for reviving a Reap()-ed descendant across the `task` tool's four +// verbs (cancel/status/send/log). Reap collects a done/failed/canceled +// LEAF the instant it settles (Reap's own doc comment, session_manager.go) +// — before this fix, a caller that spawned that child and asked about it +// again after that instant, but before observing Reap's own internal +// timing, got "no such session" for a descendant it plainly still owned. +// resolveOrReviveDescendantLocked closes that gap by falling back to a +// disk-backed resolution, validated against the descendant's own durable +// TaskParentID chain, whenever a live-tree lookup misses. See that +// method's own doc comment, and each of CancelDescendant/DescendantInfo/ +// DescendantTranscript/SendToDescendant's, for the full design. +package engine + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + "github.com/majorcontext/harness/provider" + "os" +) + +// settleAndReapChild spawns a child of parentID that immediately completes +// (scriptedTurns("child", doneTurn(...)) style provider registered under +// model "child"), waits for it to go StatusDone, force-reaps it via the +// production Reap() entry point (no special test hook needed — Reap is +// already exported and callable directly, exactly as +// engine/session_manager_test.go's own waitForReap does), and returns its +// id. cfg.SessionDir MUST already be set on mgr's root — LoadSession (the +// mechanism under test) requires it. +func settleAndReapChild(t *testing.T, mgr *SessionManager, parentID string, agentType string) string { + t.Helper() + childID, err := mgr.Spawn(SpawnOptions{ParentID: parentID, Prompt: "go", Model: modelFor("child"), AgentType: agentType}) + if err != nil { + t.Fatalf("Spawn: %v", err) + } + waitForStatus(t, mgr, childID, StatusDone, time.Second) + waitForReap(t, mgr, 1, time.Second, "settled child never became reapable") + if _, ok := mgr.Session(childID); ok { + t.Fatalf("child %s still tracked after Reap: test setup did not actually reap it", childID) + } + return childID +} + +// TestSendToDescendantRevivesReapedChild is the headline regression test: +// a settled child, reaped out of the live tree, gets a genuinely NEW turn +// from `send` — proven structurally (blockAfterFirstProvider's second call +// blocks until release, exactly like TestSendToDescendantSettledRelaunches +// Asynchronously's identical proof for the settled-but-unreaped case this +// generalizes), not merely a replay of the first. +// +// Red-verified: reverting resolveOrReviveDescendantLocked's disk fallback +// (making SendToDescendant answer ErrUnknownSession on a live-tree miss, +// the pre-fix behavior) turns this red with exactly the error the live +// incident reported — `engine: unknown session id`. +func TestSendToDescendantRevivesReapedChild(t *testing.T) { + dir := t.TempDir() + release := make(chan struct{}) + childProv := &blockAfterFirstProvider{name: "child", release: release} + cfg := managedConfig("root", scriptedTurns("root", nil), childProv) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + queued, err := mgr.SendToDescendant(root.ID, childID, "please redo this") + if err != nil { + t.Fatalf("SendToDescendant on a reaped child: %v", err) + } + if queued { + t.Error("SendToDescendant on a reaped (settled) child: queued = true, want false (a fresh re-run turn, not an enqueue)") + } + + // The child must be back in the live tree, genuinely running its + // second turn — not merely reporting an outcome from thin air. + info, ok := mgr.Info(childID) + if !ok { + t.Fatal("Info(childID) right after SendToDescendant returned: not tracked, want the revived node") + } + if info.Status != StatusRunning { + t.Errorf("revived child status right after SendToDescendant returned = %s, want %s", info.Status, StatusRunning) + } + if info.ParentID != root.ID { + t.Errorf("revived child ParentID = %q, want %q (root)", info.ParentID, root.ID) + } + if info.AgentType != AgentGeneralPurpose { + t.Errorf("revived child AgentType = %q, want %q", info.AgentType, AgentGeneralPurpose) + } + + close(release) + waitForStatus(t, mgr, childID, StatusDone, time.Second) +} + +// TestDescendantInfoServesReapedChildWithoutReadopting proves the `status` +// verb answers correctly for a reaped descendant AND that answering it has +// no side effect on the tree: status is read-only, so the child must stay +// exactly as absent from mgr.nodes after the call as it was before — see +// DescendantInfo's own doc comment for why re-adopting on a read would be +// wrong (pinning memory, and double-extending the child's budget credit +// window, purely for having been asked about). +func TestDescendantInfoServesReapedChildWithoutReadopting(t *testing.T) { + dir := t.TempDir() + usage := provider.Usage{InputTokens: 7, OutputTokens: 5, CacheReadTokens: 1} + cfg := managedConfig("root", scriptedTurns("root", nil), scriptedTurns("child", doneTurnWithUsage("the answer", usage))) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentExplore) + + node, gotUsage, err := mgr.DescendantInfo(root.ID, childID) + if err != nil { + t.Fatalf("DescendantInfo on a reaped child: %v", err) + } + if node.ID != childID { + t.Errorf("ID = %q, want %q", node.ID, childID) + } + if node.ParentID != root.ID { + t.Errorf("ParentID = %q, want %q", node.ParentID, root.ID) + } + if node.Depth != 1 { + t.Errorf("Depth = %d, want 1", node.Depth) + } + if node.Status != StatusDone { + t.Errorf("Status = %s, want done", node.Status) + } + if node.AgentType != AgentExplore { + t.Errorf("AgentType = %q, want %q", node.AgentType, AgentExplore) + } + if node.Result != "the answer" { + t.Errorf("Result = %q, want %q", node.Result, "the answer") + } + if gotUsage != usage { + t.Errorf("Usage = %+v, want %+v", gotUsage, usage) + } + + if _, ok := mgr.Session(childID); ok { + t.Error("child is tracked again after a read-only DescendantInfo call: status must not re-adopt a reaped descendant") + } +} + +// TestDescendantTranscriptServesReapedChild proves the `log` verb reads a +// reaped descendant's transcript straight off a disk-loaded *Session, +// without re-adopting it — the log-verb counterpart to +// TestDescendantInfoServesReapedChildWithoutReadopting. +func TestDescendantTranscriptServesReapedChild(t *testing.T) { + dir := t.TempDir() + cfg := managedConfig("root", scriptedTurns("root", nil), scriptedTurns("child", doneTurn("child's final answer"))) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + node, msgs, total, err := mgr.DescendantTranscript(root.ID, childID, 20) + if err != nil { + t.Fatalf("DescendantTranscript on a reaped child: %v", err) + } + if node.Status != StatusDone { + t.Errorf("Status = %s, want done", node.Status) + } + if total == 0 || len(msgs) == 0 { + t.Fatalf("DescendantTranscript returned no messages for a settled child (total=%d, len(msgs)=%d)", total, len(msgs)) + } + found := false + // Render via the same helper the `task` tool's log action itself uses, + // so this assertion exercises the identical rendering path a model + // would see. + entries := renderTaskLog(msgs) + for _, e := range entries { + if e.Text == "child's final answer" { + found = true + } + } + if !found { + t.Errorf("rendered log entries %+v do not contain the child's final answer", entries) + } + + if _, ok := mgr.Session(childID); ok { + t.Error("child is tracked again after a read-only DescendantTranscript call: log must not re-adopt a reaped descendant") + } +} + +// TestCancelDescendantNoOpsOnReapedChild proves `cancel` against a reaped +// descendant is a no-op success: it reports the descendant's real terminal +// status (never StatusCanceled — nothing was actually canceled) and does +// not re-adopt it — see CancelDescendant's own doc comment for why a +// reaped target can never have had genuine in-flight work to interrupt. +func TestCancelDescendantNoOpsOnReapedChild(t *testing.T) { + dir := t.TempDir() + cfg := managedConfig("root", scriptedTurns("root", nil), scriptedTurns("child", doneTurn("done"))) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + status, err := mgr.CancelDescendant(root.ID, childID) + if err != nil { + t.Fatalf("CancelDescendant on a reaped child: %v", err) + } + if status != StatusDone { + t.Errorf("CancelDescendant on a reaped, already-done child: status = %s, want %s (nothing was actually canceled)", status, StatusDone) + } + if _, ok := mgr.Session(childID); ok { + t.Error("child is tracked again after a no-op CancelDescendant call: cancel must not re-adopt a reaped descendant") + } +} + +// TestSendToDescendantUnknownIdStillErrors proves the disk fallback does +// not turn every unresolvable id into a silent success: an id with no +// live node AND no session log on disk (never existed at all) still +// answers ErrUnknownSession, exactly as it did before this fix, now +// exercising the code path that actually attempts (and fails) a +// LoadSession rather than skipping straight to the live-tree miss. +func TestSendToDescendantUnknownIdStillErrors(t *testing.T) { + dir := t.TempDir() + cfg := managedConfig("root", scriptedTurns("root", nil)) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + if _, err := mgr.SendToDescendant(root.ID, "ses_0000000000000000", "hi"); !errors.Is(err, ErrUnknownSession) { + t.Errorf("SendToDescendant(root, neverexisted): err = %v, want ErrUnknownSession", err) + } + if _, _, err := mgr.DescendantInfo(root.ID, "ses_0000000000000000"); !errors.Is(err, ErrUnknownSession) { + t.Errorf("DescendantInfo(root, neverexisted): err = %v, want ErrUnknownSession", err) + } +} + +// TestSendToDescendantAncestryViolationForReapedNonDescendant proves the +// disk fallback still enforces "only your own ancestors" for a reaped +// target: a child reaped under ONE root must still refuse an unrelated +// root's send/status/cancel, exactly as an ancestry violation refuses a +// LIVE non-descendant — the disk-resolved case is not a backdoor around +// isDescendantLocked's own rule. +func TestSendToDescendantAncestryViolationForReapedNonDescendant(t *testing.T) { + dir := t.TempDir() + cfg := managedConfig("root", + scriptedTurns("root", nil), + scriptedTurns("other", nil), + scriptedTurns("child", doneTurn("done")), + ) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + otherRoot := mgr.NewRoot(Config{Providers: cfg.Providers, Model: modelFor("other"), System: cfg.System, SessionDir: dir}) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + if _, err := mgr.SendToDescendant(otherRoot.ID, childID, "hi"); !errors.Is(err, ErrNotDescendant) { + t.Errorf("SendToDescendant from an unrelated root against a reaped non-descendant: err = %v, want ErrNotDescendant", err) + } + if _, _, err := mgr.DescendantInfo(otherRoot.ID, childID); !errors.Is(err, ErrNotDescendant) { + t.Errorf("DescendantInfo from an unrelated root against a reaped non-descendant: err = %v, want ErrNotDescendant", err) + } + if _, err := mgr.CancelDescendant(otherRoot.ID, childID); !errors.Is(err, ErrNotDescendant) { + t.Errorf("CancelDescendant from an unrelated root against a reaped non-descendant: err = %v, want ErrNotDescendant", err) + } + if _, ok := mgr.Session(childID); ok { + t.Error("child is tracked again after refused ancestry-violation calls: a refusal must not adopt anything") + } +} + +// TestSendToDescendantRevivalDoesNotDoubleCreditUsage proves requirement +// #3 (budgetedByChild survives Reap by design specifically so a re-adopt +// cannot double-credit it — see that field's own doc comment): reviving a +// reaped child via `send` and letting it complete a SECOND turn must fold +// only the second turn's OWN new usage into usageByRoot, never re-add the +// first turn's already-credited usage a second time. +func TestSendToDescendantRevivalDoesNotDoubleCreditUsage(t *testing.T) { + dir := t.TempDir() + firstUsage := provider.Usage{InputTokens: 100, OutputTokens: 50} + secondUsage := provider.Usage{InputTokens: 10, OutputTokens: 5} + // A two-turn script: the child's FIRST Spawn'd turn serves firstUsage, + // and its SECOND turn — the one `send` launches after reviving it — + // serves secondUsage. scriptedTurns serves one turn per call, in + // order, so no blocking/release machinery is needed to pin the two + // turns apart. + turns := append(doneTurnWithUsage("first", firstUsage), doneTurnWithUsage("second", secondUsage)...) + cfg := managedConfig("root", scriptedTurns("root", nil), scriptedTurns("child", turns)) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + mgr.mu.Lock() + before := mgr.usageByRoot[root.ID] + mgr.mu.Unlock() + if before != firstUsage { + t.Fatalf("usageByRoot[root] after the first turn = %+v, want %+v (the baseline this test's revival step must not double-count)", before, firstUsage) + } + + queued, err := mgr.SendToDescendant(root.ID, childID, "again") + if err != nil { + t.Fatalf("SendToDescendant on a reaped child: %v", err) + } + if queued { + t.Fatal("SendToDescendant on a reaped, settled child: queued = true, want false") + } + waitForStatus(t, mgr, childID, StatusDone, time.Second) + + mgr.mu.Lock() + after := mgr.usageByRoot[root.ID] + mgr.mu.Unlock() + + want := provider.Usage{ + InputTokens: firstUsage.InputTokens + secondUsage.InputTokens, + OutputTokens: firstUsage.OutputTokens + secondUsage.OutputTokens, + } + if after != want { + t.Errorf("usageByRoot[root] after the revived child's second turn = %+v, want %+v (firstUsage+secondUsage exactly once each — a mismatch means the reap+revive double- or under-credited)", after, want) + } +} + +// TestSendToDescendantRevivalSingleWinnerUnderConcurrentAdopt hammers the +// concurrency requirement: a `send`-driven revival racing a CONCURRENT, +// independent AdoptReloaded of the SAME reaped id (simulating, e.g., the +// server's own handleSpawnChild parent-lookup fallback touching the same +// id at the same moment) must produce exactly one live node for childID — +// never two competing *Session objects backing the same on-disk log, and +// never a corrupted parent.children list (a double-adopt would append +// childID to root's children twice). Run with `-race -count=1000 +// GOMAXPROCS=2` to hammer the race per AGENTS.md's concurrency-testing +// rule; a single run here still exercises the same code path +// deterministically enough to catch a gross ordering bug and, under +// -race, any unsynchronized access. +func TestSendToDescendantRevivalSingleWinnerUnderConcurrentAdopt(t *testing.T) { + dir := t.TempDir() + // Two scripted turns: the child's original Spawn'd turn, and the + // SECOND turn SendToDescendant's settled-target restart always + // launches, regardless of which of the two racing goroutines below + // happens to win the adopt itself — AdoptReloaded never launches a + // turn on its own, only SendToDescendant's own settled-target branch + // does, so exactly one fresh turn always follows the race, never two. + childTurns := append(doneTurn("first"), doneTurn("second")...) + cfg := managedConfig("root", scriptedTurns("root", nil), scriptedTurns("child", childTurns)) + cfg.SessionDir = dir + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID := settleAndReapChild(t, mgr, root.ID, AgentGeneralPurpose) + + loadCfg := root.configSnapshot() + + var wg sync.WaitGroup + wg.Add(2) + var sendErr error + go func() { + defer wg.Done() + _, sendErr = mgr.SendToDescendant(root.ID, childID, "revive via send") + }() + go func() { + defer wg.Done() + if loaded, err := LoadSession(loadCfg, childID); err == nil { + _ = mgr.AdoptReloaded(loaded) // "already managed" ignored on the loser, by design — see AdoptReloaded's own doc comment + } + }() + wg.Wait() + + if sendErr != nil { + t.Fatalf("SendToDescendant racing a concurrent AdoptReloaded: %v", sendErr) + } + + mgr.mu.Lock() + n, ok := mgr.nodes[childID] + var childCount int + if p, pok := mgr.nodes[root.ID]; pok { + for _, cid := range p.children { + if cid == childID { + childCount++ + } + } + } + mgr.mu.Unlock() + + if !ok { + t.Fatal("childID not tracked after the race: revival was lost entirely") + } + if childCount != 1 { + t.Errorf("root.children contains childID %d time(s), want exactly 1 (a double-adopt corrupted the tree)", childCount) + } + if n.parentID != root.ID { + t.Errorf("revived node ParentID = %q, want %q", n.parentID, root.ID) + } + + waitForStatus(t, mgr, childID, StatusDone, time.Second) +} + +// TestLoadSessionTaskParentReadsHeaderOnly proves the ancestry walk's +// header read returns the durable TaskParentID from the first line alone, +// and fails loudly on a file whose first record is not a header. +func TestLoadSessionTaskParentReadsHeaderOnly(t *testing.T) { + dir := t.TempDir() + cfg := Config{SessionDir: dir} + s := NewSession(Config{SessionDir: dir, TaskParentID: "ses_parent00000000000000000"}) + if err := s.ensureLog(); err != nil { + t.Fatal(err) + } + got, err := loadSessionTaskParent(cfg, s.ID) + if err != nil { + t.Fatal(err) + } + if got != "ses_parent00000000000000000" { + t.Fatalf("loadSessionTaskParent = %q", got) + } + // A non-header first line errors instead of guessing. + bad := s.ID[:len(s.ID)-1] + "x" + if err := os.WriteFile(sessionPath(dir, bad), []byte("{\"type\":\"model\"}\n"), 0o600); err != nil { + t.Fatal(err) + } + if _, err := loadSessionTaskParent(cfg, bad); err == nil { + t.Fatal("want error for non-header first record") + } +} From bc4429fc7f2d223d45da585637c5d074a3a5c854 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 13:12:45 -0400 Subject: [PATCH 03/95] feat(engine): emit per-turn latency and cache metrics to stdout (#188) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(engine): emit per-turn latency and cache metrics to stdout A box session and an equivalent Claude Code session can feel very different in speed, with no way to say why: harness had no per-call latency or prompt-cache signal anywhere, so "the box feels slow" had no data behind it. The nearest prior art, the 433k-token cache rewrite measured on a live box session, only existed because someone hand-instrumented one session by hand; there was no standing signal an operator could grep across the fleet. Box pods already ship stdout to a log pipeline (a fleet of `harness serve` processes, each collected into a central log store), so the natural fix is a structured line written to stdout, not a new transport: one `turn_metrics` line per completed model call, with send-to-first-byte and stream-duration timing split out, token and prompt-cache accounting, and request shape (system length, tool count, retry attempt). Design: streamTurn (engine/engine.go) is the one place that already walks a provider's event stream and sees EventActivity, the content deltas, and the terminal EventDone with its Usage — so it is also the one place that can time TTFT (request sent to first non-activity delta) and stream duration (first delta to EventDone) without a second pass over the stream. The metric is emitted only on a completed call (EventDone reached); a turn that errors or is interrupted mid-stream reports nothing, since there is no finished call to summarize. The emit is a new Config.OnTurnMetrics func(TurnMetrics) seam, mirroring the existing OnRequest/OnEvent/OnStorePhase seams in shape, but deliberately different in default behavior: every other On* callback is a no-op when nil, while OnTurnMetrics substitutes defaultTurnMetricsLog (engine/turn_metrics.go), a slog.NewJSONHandler line written to os.Stdout specifically. This repo's own convention is JSON-to-stderr for structured logs (see cmd/harness/main.go's serve/run loggers), but a per-turn line is telemetry a deployment's log pipeline scrapes off stdout, not a diagnostic a human tails alongside process logs on stderr — so this one line deliberately breaks that convention, on purpose, and says so in both the Config doc comment and AGENTS.md. An embedder that wants a different sink sets OnTurnMetrics directly; it never needs to suppress the default first. TurnMetrics.SessionID/Model/SystemLen/ToolsCount are computed to match the server's existing request.meta record exactly (same "\n"-joined system length, same tool count) so a turn_metrics stdout line and its request.meta record share a natural join key with no new ID threaded through the provider boundary — see request.meta's own construction in server/journal.go's OnRequest, which this design mirrors rather than duplicates. Attempt threads streamTurnWithRetry's 1-indexed attempt counter into streamTurn as a new parameter (its one call site, prompt_retry.go, already has the number); streamTurn otherwise needs no knowledge of the retry policy. Timing is read through a new Config.Now func() time.Time seam, scoped to this one measurement rather than a general engine clock — every other timestamp in the package still reads time.Now directly. This exists purely so a test can script an exact instant sequence: a scripted provider stream's events have no real wall-clock gap between them, so two real time.Now calls a nanosecond apart would compute a ~0 duration and prove nothing, and the repo's testing rules ban a real sleep in a test to manufacture one. Semantic change: streamTurn's signature gains an attempt int parameter (its single call site updated); a completed model call now always produces one turn_metrics stdout line, by default, with no config required. Nothing about request assembly, retry policy, or history is touched. Verification: new engine/turn_metrics_test.go covers latency/usage computation against the injected clock, cache-token pass-through, retry-attempt propagation, EventActivity exclusion from TTFT (via the firstDeltaAt latch), and the completed-call-only emit rule. TestTurnMetricsFirstDeltaLatches and TestTurnMetricsRecordsRetryAttempt were red-verified: reverting the firstDeltaAt latch reproduced the predicted wrong TTFT/stream/call-count (90ms/410ms/5 calls instead of 50ms/30ms/3), and hardcoding Attempt to 1 broke the retry-attempt assertion. Full go test -race ./..., go vet ./..., and gofmt are clean. * fix(engine): emit turn_metrics to stderr, keeping harness run's stdout clean Review caught the default sink polluting the one place it must not: harness run's stdout is the answer channel (textStreamPrinter), so a turn_metrics JSON line interleaves into captured output. Neither command sets OnTurnMetrics, so the default emitter is what ships. Stderr keeps every property the feature wants — Kubernetes captures both streams, so the Vector collector still delivers the line to BetterStack — and harness run's stdout stays exactly the model's output. Doc section updated to say stderr. Verification: turn-metrics suite green under -race; gofmt/build clean. * docs(engine): rewrite the stream-choice comments the stderr move mangled The stdout-to-stderr fix was applied as a find-replace and left four comments self-contradicting ("stderr, not the stderr"), citing a nonexistent architecture note, or still naming stdout. Rewrite them to state the real rationale coherently: stderr joins the repo's one structured-log stream and stays out of harness run's stdout answer channel; Kubernetes captures both streams so pipeline delivery is unchanged. Comment-only. Verification: turn-metrics suite green under -race; gofmt/build clean; no self-contradicting or stale stream references remain (grepped). --- AGENTS.md | 58 ++++++++ engine/engine.go | 76 +++++++++- engine/prompt_retry.go | 2 +- engine/turn_metrics.go | 108 ++++++++++++++ engine/turn_metrics_test.go | 288 ++++++++++++++++++++++++++++++++++++ 5 files changed, 530 insertions(+), 2 deletions(-) create mode 100644 engine/turn_metrics.go create mode 100644 engine/turn_metrics_test.go diff --git a/AGENTS.md b/AGENTS.md index 493564d5..9e8d4711 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -285,6 +285,64 @@ the outer weather tier ever counts it, so a goal loop rides a brief provider blip without spending an outer attempt. `PromptRetries` 0 disables the inner budget for a host that wants the outer tiers to be the only retry. +### Per-turn metrics + +`streamTurn` (`engine/engine.go`) emits one structured `turn_metrics` line per +COMPLETED model call — a stream that reached `EventDone`; a turn that errors +or is interrupted mid-stream (see `interruptedTurnError` above) reports +nothing, since there is no finished call to summarize. This is the box-fleet +answer to "why does this session feel slow": TTFT and stream duration, token +and prompt-cache accounting, and request shape, greppable straight off a +process's stderr. + +Fields: `session_id`, `model` (full `provider/model` ref), `ttft_ms` (elapsed +from just before `prov.Stream` to the first non-`EventActivity` stream event +— `EventActivity` carries no content, so a keep-alive ping or an in-progress +tool-argument chunk must never be mistaken for "first byte"; if `EventDone` +itself is the first event, `ttft_ms` covers the whole call and `stream_ms` is +0), `stream_ms` (first delta to `EventDone`), `input_tokens`/`output_tokens`/ +`cache_read_tokens`/`cache_write_tokens` (passed through from +`provider.Usage` verbatim), `system_len`/`tools_count`, and `retry` (the +1-indexed attempt number `streamTurnWithRetry` — `engine/prompt_retry.go` — +was on when this call completed; 1 for a turn that succeeded on its first +try). `system_len` is computed identically to the server's `request.meta` +record (`len(strings.Join(req.System, "\n"))`, see `server/journal.go`'s +`OnRequest`) — deliberately, not coincidentally: `session_id` + `model` + +`system_len` together are a natural join key between a `turn_metrics` stderr +line and the durable `request.meta` record for the same request, with no new +ID threaded through the provider boundary. + +`Config.OnTurnMetrics func(TurnMetrics)` is the seam. Unlike every other +`On*` callback in `Config` (`OnEvent`, `OnRequest`, `OnStorePhase`), nil is +NOT "disabled": `emitTurnMetrics` substitutes `defaultTurnMetricsLog` +(`engine/turn_metrics.go`), a `slog.NewJSONHandler` line written to +`os.Stderr` — the same stream every other structured log line in this repo +uses (see `cmd/harness/main.go`'s "Structured logging: JSON to stderr" +comment). Stderr keeps the line out of `harness run`'s stdout, which is +the model's answer channel, while a deployment's log pipeline (Kubernetes +captures both streams) scrapes it identically. A plain `harness run`/`harness serve` process +with no embedder wiring therefore still emits this line by default; an +embedder that wants a different sink (an OTel exporter, an in-memory test +recorder) sets `OnTurnMetrics` and never needs to suppress the default +first. + +`Config.Now func() time.Time` is the clock this measurement reads (nil +resolves to `time.Now` in `newSession`), scoped to this one seam rather than +a general engine clock — every other timestamp in the package still reads +`time.Now` directly. It exists so a test can script an exact instant sequence +instead of depending on real elapsed wall-clock time between two in-process +calls with nothing to wait on between them, per the Testing rule against real +sleeps. + +This was built for a deployment that ships a served process's stderr to a +log pipeline (a fleet of boxes running `harness serve`, each pod's stderr +collected by a Vector-style agent into BetterStack or an equivalent log +store). The intended query there filters `msg: "turn_metrics"` and groups by +`model`/`session_id` to compare TTFT and stream-duration distributions across +sessions — quantifying, with real numbers instead of a feeling, whether a +session "feels slow" because of provider latency, prompt-cache misses, or +something else entirely. + ### Goal loop `Session.PursueGoal(ctx, condition, GoalOptions)` drives the ordinary `Prompt` diff --git a/engine/engine.go b/engine/engine.go index ec7a79a8..ff3cba7b 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -446,6 +446,38 @@ type Config struct { // safely (see newSessionFn/loadSessionFn). OnRequest func(sessionID string, turn int, req *provider.Request) + // OnTurnMetrics, when non-nil, is invoked once per COMPLETED streamTurn + // call (a stream that reached EventDone; a turn that errors or is + // interrupted mid-stream emits nothing, since there is no finished call + // to report) with a TurnMetrics summarizing its latency and usage. Called + // synchronously from streamTurn, immediately after EventDone and before + // streamTurn returns — keep it fast, and never call back into the + // Session (same rule as OnEvent/OnStorePhase above). + // + // Unlike every other On* callback in this Config, nil is NOT "disabled": + // emitTurnMetrics (turn_metrics.go) substitutes defaultTurnMetricsLog, a + // stderr JSON slog line, so a plain `harness run`/`harness serve` process + // with no embedder wiring still emits per-turn telemetry by default. An + // embedder that wants a different sink (an in-memory recorder for a + // test, an OTel exporter) sets this field; it never needs to suppress + // the default first. + OnTurnMetrics func(TurnMetrics) + + // Now is the clock TurnMetrics timing reads: streamTurn calls it just + // before the provider call, at the first non-activity stream event, and + // at EventDone, to compute TTFTMillis/StreamMillis (see TurnMetrics). + // Nil (the default) resolves to time.Now in newSession. This exists so a + // test can inject a scripted sequence of instants instead of depending + // on real elapsed wall-clock time between two calls to a fake stream's + // Next() — the AGENTS.md testing rule against real sleeps in tests + // applies here exactly as it does to any other timer-dependent code, and + // a real duration between two in-process function calls with no actual + // I/O between them would otherwise round to ~0 and prove nothing. Scoped + // to this one measurement, not a general engine clock seam: every other + // timestamp in this package (CreatedAt, etc.) still reads time.Now + // directly. + Now func() time.Time + // Instructions controls project-instruction (AGENTS.md) injection into // the system prompt. A nil value is the default: auto-discover AGENTS.md // by walking up from WorkDir. See InstructionsConfig. @@ -1013,6 +1045,9 @@ func newSession(cfg Config) *Session { if cfg.BashTimeout <= 0 { cfg.BashTimeout = 2 * time.Minute } + if cfg.Now == nil { + cfg.Now = time.Now + } // contextWindowExplicit records whether the CALLER set // Config.ContextWindowTokens, captured from the ORIGINAL value before // resolveContextWindow below overwrites it with the effective (possibly @@ -2019,7 +2054,12 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // streamTurn makes one model call and returns the assembled assistant // message. -func (s *Session) streamTurn(ctx context.Context) (*message.Message, provider.StopReason, provider.Usage, error) { +// attempt is streamTurnWithRetry's 1-indexed attempt counter for this turn +// (1 on a turn's first call), threaded through purely so the turn_metrics +// emit at EventDone (see the end of this function and TurnMetrics.Attempt) +// can report whether this completed call was a retry, without streamTurn +// itself needing to know anything else about the retry budget or policy. +func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message, provider.StopReason, provider.Usage, error) { // Assembly order matters, and three steps are pinned: // // 1. chat.params runs first: it fixes params.Model, which both the @@ -2178,6 +2218,10 @@ func (s *Session) streamTurn(ctx context.Context) (*message.Message, provider.St // abort takes through the adapter's HTTP body read. ctx, watch, release := s.armIdleWatchdog(ctx) defer release() + // sentAt anchors TurnMetrics.TTFTMillis (see the EventDone case below): + // captured immediately before the provider dial, mirroring where + // OnRequest above already hands the same req to an observer. + sentAt := s.cfg.Now() stream, err := prov.Stream(ctx, req) if err != nil { return nil, "", provider.Usage{}, watch.explain(err) @@ -2191,6 +2235,14 @@ func (s *Session) streamTurn(ctx context.Context) (*message.Message, provider.St // why. var text strings.Builder var toolCalls []*message.ToolCall + // firstDeltaAt is set once, on the first non-EventActivity event this + // stream yields (see provider.EventActivity's doc comment: it carries no + // content, so it must not count as "first byte"). If EventDone is + // itself that first event — a provider that streams nothing before its + // terminal event — firstDeltaAt lands on EventDone too, and + // TurnMetrics.StreamMillis below comes out zero. + var firstDeltaAt time.Time + var gotFirstDelta bool for { ev, err := stream.Next() if err != nil { @@ -2211,6 +2263,10 @@ func (s *Session) streamTurn(ctx context.Context) (*message.Message, provider.St } } watch.kick() + if !gotFirstDelta && ev.Type != provider.EventActivity { + firstDeltaAt = s.cfg.Now() + gotFirstDelta = true + } switch ev.Type { case provider.EventTextDelta: text.WriteString(ev.Text) @@ -2228,6 +2284,24 @@ func (s *Session) streamTurn(ctx context.Context) (*message.Message, provider.St // or error still has this call's identity to work with. toolCalls = append(toolCalls, ev.ToolCall) case provider.EventDone: + doneAt := s.cfg.Now() + s.emitTurnMetrics(TurnMetrics{ + SessionID: s.ID, + Model: params.Model, + Attempt: attempt, + TTFTMillis: firstDeltaAt.Sub(sentAt).Milliseconds(), + StreamMillis: doneAt.Sub(firstDeltaAt).Milliseconds(), + InputTokens: ev.Usage.InputTokens, + OutputTokens: ev.Usage.OutputTokens, + CacheReadTokens: ev.Usage.CacheReadTokens, + CacheWriteTokens: ev.Usage.CacheWriteTokens, + // SystemLen mirrors server/journal.go's OnRequest computation + // exactly (len of the "\n"-joined system slice) so the two + // records join on session_id+model+system_len — see + // TurnMetrics's doc comment. + SystemLen: len(strings.Join(system, "\n")), + ToolsCount: len(tools), + }) return ev.Message, ev.StopReason, ev.Usage, nil } } diff --git a/engine/prompt_retry.go b/engine/prompt_retry.go index 6f59209a..d2a49d44 100644 --- a/engine/prompt_retry.go +++ b/engine/prompt_retry.go @@ -132,7 +132,7 @@ func waitBasePromptRetryBackoff(ctx context.Context, attempt int) error { // function's doc comment for why lastUsage is deliberately left untouched. func (s *Session) streamTurnWithRetry(ctx context.Context) (*message.Message, provider.StopReason, provider.Usage, error) { for attempt := 1; ; attempt++ { - asst, stop, usage, err := s.streamTurn(ctx) + asst, stop, usage, err := s.streamTurn(ctx, attempt) if err == nil { if !turnHasActionableContent(asst) { s.accumulateDiscardedTurnUsage(usage) diff --git a/engine/turn_metrics.go b/engine/turn_metrics.go new file mode 100644 index 00000000..44e197c0 --- /dev/null +++ b/engine/turn_metrics.go @@ -0,0 +1,108 @@ +package engine + +import ( + "log/slog" + "os" + + "github.com/majorcontext/harness/message" +) + +// TurnMetrics summarizes one completed streamTurn model call: request +// latency broken into time-to-first-token and stream duration, the token and +// prompt-cache accounting the provider reported, and the shape of the +// request that produced it. See Config.OnTurnMetrics. +// +// SessionID, Model, SystemLen, and ToolsCount are deliberately computed the +// same way the server's request.meta record computes them (see +// server/journal.go's OnRequest: SystemLen is len(strings.Join(req.System, +// "\n")), ToolsCount is len(req.Tools)) so a turn_metrics log line and the +// request.meta record for the same turn share a natural join key — +// session_id, model, and system_len together identify the same request on +// both sides, without threading a new shared ID through the provider +// boundary. +type TurnMetrics struct { + // SessionID is the firing session's own ID, mirroring OnRequest's + // sessionID parameter (see its doc comment for why this is passed + // explicitly rather than closed over: a Spawn'd child must report its + // own ID, never its parent's). + SessionID string + // Model is the full "provider/model" ref this turn was sent to. + Model message.ModelRef + // Attempt is streamTurnWithRetry's 1-indexed attempt counter: 1 for a + // turn that succeeded on its first try, 2+ when a prior attempt this + // turn failed retryably and was re-issued. Named "retry" on the wire + // (see defaultTurnMetricsLog) because that is the operator-facing + // question: did this completed call cost more than one model request. + Attempt int + // TTFTMillis is the elapsed time from just before the request was sent + // (prov.Stream) to the first non-activity stream event — the first + // event carrying content or, if the provider streams nothing before its + // terminal event, the usage-bearing EventDone itself. + TTFTMillis int64 + // StreamMillis is the elapsed time from that first delta to EventDone. + // Zero when EventDone was itself the first delta (TTFTMillis already + // covers the whole call in that case). + StreamMillis int64 + // InputTokens, OutputTokens, CacheReadTokens, and CacheWriteTokens are + // provider.Usage passed through verbatim from EventDone. + InputTokens int + OutputTokens int + CacheReadTokens int + CacheWriteTokens int + // SystemLen is the byte length of this request's joined system prompt + // (see the type doc comment above for why this must match request.meta's + // own computation exactly). + SystemLen int + // ToolsCount is the number of tools offered on this request. + ToolsCount int +} + +// defaultTurnMetricsStderr is the JSON handler every default turn_metrics +// emit writes through. It is a package-level var (never per-Session) so a +// test that swaps Config.OnTurnMetrics for a recorder pays nothing for it, +// and so every session in one process shares one handler exactly like +// slog.Default() would, but pinned to os.Stderr regardless of what +// slog.SetDefault has been called with elsewhere. +// +// Stderr is deliberate on BOTH counts. It joins the same stream every +// other structured log line in this repo uses (cmd/harness/main.go's +// "Structured logging: JSON to stderr"), so a deployment's log pipeline +// (Kubernetes captures stdout and stderr alike) scrapes it with no extra +// wiring — and it stays OFF stdout, which for `harness run` is the +// answer channel itself: a metrics line interleaved there would corrupt +// captured output (the review finding that moved this from stdout). +var defaultTurnMetricsStderr = slog.New(slog.NewJSONHandler(os.Stderr, nil)) + +// defaultTurnMetricsLog is Config.OnTurnMetrics's default when the embedder +// sets none: one structured "turn_metrics" line per completed model call. +// Field names match the wire vocabulary a log query (grep, a BetterStack/ +// Vector-style query) is expected to filter on. +func defaultTurnMetricsLog(m TurnMetrics) { + defaultTurnMetricsStderr.Info("turn_metrics", + "session_id", m.SessionID, + "model", m.Model.String(), + "ttft_ms", m.TTFTMillis, + "stream_ms", m.StreamMillis, + "input_tokens", m.InputTokens, + "output_tokens", m.OutputTokens, + "cache_read_tokens", m.CacheReadTokens, + "cache_write_tokens", m.CacheWriteTokens, + "system_len", m.SystemLen, + "tools_count", m.ToolsCount, + "retry", m.Attempt, + ) +} + +// emitTurnMetrics dispatches m to Config.OnTurnMetrics, or +// defaultTurnMetricsLog when the embedder configured none. Unlike OnRequest +// (nil means "no observer, skip the call entirely"), turn metrics always go +// somewhere: a box or CLI process with no embedder-supplied callback still +// gets the stderr line, which is the whole point of the seam — see the +// Config.OnTurnMetrics doc comment. +func (s *Session) emitTurnMetrics(m TurnMetrics) { + cb := s.cfg.OnTurnMetrics + if cb == nil { + cb = defaultTurnMetricsLog + } + cb(m) +} diff --git a/engine/turn_metrics_test.go b/engine/turn_metrics_test.go new file mode 100644 index 00000000..67963606 --- /dev/null +++ b/engine/turn_metrics_test.go @@ -0,0 +1,288 @@ +package engine + +import ( + "context" + "strings" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// stepClock returns a fixed, pre-scripted sequence of instants, one per +// call, panicking if exhausted. This is the fake clock Config.Now expects a +// test to inject (see Config.Now's doc comment): the events a scriptedStream +// yields have no real wall-clock gap between them, so real time.Now calls a +// nanosecond apart would compute a TTFT/stream duration of ~0 and prove +// nothing. Scripting the exact instants streamTurn's three Now() call sites +// (sentAt, firstDeltaAt, doneAt — see streamTurn in engine.go) observe lets a +// test assert an exact, non-zero TTFTMillis/StreamMillis. +type stepClock struct { + times []time.Time + i int +} + +func (c *stepClock) now() time.Time { + if c.i >= len(c.times) { + panic("stepClock: exhausted") + } + t := c.times[c.i] + c.i++ + return t +} + +// TestTurnMetricsComputesLatencyAndUsage is the red-first guard for the +// turn_metrics emit: one completed model call must report TTFTMillis (send +// to first content delta) and StreamMillis (first delta to EventDone) +// computed from the injected clock, plus every usage field (including +// prompt-cache read/write tokens) passed through verbatim from +// provider.Usage, and SystemLen/ToolsCount matching the exact request the +// provider received — the join key against the server's request.meta +// record (see TurnMetrics's doc comment). +// +// The Attempt/latch mechanisms this metric also depends on are red-verified +// by their own dedicated tests: TestTurnMetricsRecordsRetryAttempt (attempt +// propagation) and TestTurnMetricsFirstDeltaLatches (the firstDeltaAt gate) — +// see their doc comments for the exact reverts proven to fail. +func TestTurnMetricsComputesLatencyAndUsage(t *testing.T) { + base := time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC) + clock := &stepClock{times: []time.Time{ + base, // sentAt, just before prov.Stream + base.Add(120 * time.Millisecond), // firstDeltaAt, the text_delta event + base.Add(500 * time.Millisecond), // doneAt, EventDone + }} + + usage := provider.Usage{ + InputTokens: 321, + OutputTokens: 45, + CacheReadTokens: 200, + CacheWriteTokens: 12, + } + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + { + {Type: provider.EventTextDelta, Text: "hi"}, + { + Type: provider.EventDone, + Message: &message.Message{ID: "msg_a", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "hi"}}}, + StopReason: provider.StopEndTurn, + Usage: usage, + }, + }, + }} + + var recorded []TurnMetrics + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + System: []string{"base system prompt"}, + Now: clock.now, + OnTurnMetrics: func(m TurnMetrics) { recorded = append(recorded, m) }, + }) + + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt err = %v", err) + } + if len(recorded) != 1 { + t.Fatalf("OnTurnMetrics calls = %d, want 1", len(recorded)) + } + m := recorded[0] + + if m.SessionID != s.ID { + t.Errorf("SessionID = %q, want %q", m.SessionID, s.ID) + } + if want := "test/m1"; m.Model.String() != want { + t.Errorf("Model = %q, want %q", m.Model.String(), want) + } + if m.Attempt != 1 { + t.Errorf("Attempt = %d, want 1", m.Attempt) + } + if m.TTFTMillis != 120 { + t.Errorf("TTFTMillis = %d, want 120", m.TTFTMillis) + } + if m.StreamMillis != 380 { + t.Errorf("StreamMillis = %d, want 380", m.StreamMillis) + } + if m.InputTokens != usage.InputTokens || m.OutputTokens != usage.OutputTokens || + m.CacheReadTokens != usage.CacheReadTokens || m.CacheWriteTokens != usage.CacheWriteTokens { + t.Errorf("usage fields = %+v, want the provider.Usage fields passed through verbatim: %+v", m, usage) + } + + // SystemLen/ToolsCount must match the exact request the provider + // received — the join key against request.meta — not a hardcoded guess + // about what ambient segments this session assembles. + if len(prov.requests) != 1 { + t.Fatalf("provider requests = %d, want 1", len(prov.requests)) + } + wantSystemLen := len(strings.Join(prov.requests[0].System, "\n")) + if m.SystemLen != wantSystemLen { + t.Errorf("SystemLen = %d, want %d (len of the joined system the provider actually received)", m.SystemLen, wantSystemLen) + } + wantToolsCount := len(prov.requests[0].Tools) + if m.ToolsCount != wantToolsCount { + t.Errorf("ToolsCount = %d, want %d", m.ToolsCount, wantToolsCount) + } +} + +// TestTurnMetricsFirstDeltaLatches is the red-first guard for streamTurn's +// firstDeltaAt gate (engine.go): two activity events (a keep-alive ping, an +// in-progress tool-argument chunk — see provider.EventActivity's doc +// comment) precede the real content, and a SECOND text delta follows the +// first — TTFTMillis must land on the FIRST non-activity delta and never +// move again, including when a later delta arrives. +// +// This is deliberately checked by clock CALL COUNT, not just the final +// value: with a purely sequential fake clock, a bug that fires the gate on +// the wrong EVENT (say, the first activity instead of the first real delta) +// is unobservable in the output when the gate still fires exactly once — +// the Nth Now() call returns the same scripted instant regardless of which +// loop iteration asked for it. A bug that fires the gate MORE than once +// (forgetting the latch) is observable: it consumes an extra clock entry, so +// EventDone's own call is pushed onto a later, wrong-value slot. That is the +// mechanism this test actually red-verifies. +// +// Red-verify the NAMED mechanism: dropping the "!gotFirstDelta &&" half of +// streamTurn's gate (keeping only the EventActivity type check) makes the +// SECOND text delta re-fire the assignment — and, since EventDone itself is +// also != EventActivity, EventDone's own pre-switch pass re-fires it a +// THIRD time, stealing the value doneAt should get. Verified against that +// exact revert: TTFTMillis read 90 (not 50), StreamMillis read 410 (not +// 30), and clock.i read 5 (not 3) — every assertion below caught it. +func TestTurnMetricsFirstDeltaLatches(t *testing.T) { + base := time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC) + clock := &stepClock{times: []time.Time{ + base, // sentAt + base.Add(50 * time.Millisecond), // firstDeltaAt: the FIRST text delta + base.Add(80 * time.Millisecond), // doneAt (correct code stops here) + base.Add(90 * time.Millisecond), // never reached by correct code + base.Add(500 * time.Millisecond), // never reached by correct code + }} + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + { + {Type: provider.EventActivity}, + {Type: provider.EventActivity}, + {Type: provider.EventTextDelta, Text: "hi"}, + {Type: provider.EventTextDelta, Text: " there"}, + { + Type: provider.EventDone, + Message: &message.Message{ID: "msg_a", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "hi there"}}}, + StopReason: provider.StopEndTurn, + }, + }, + }} + var recorded []TurnMetrics + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + Now: clock.now, + OnTurnMetrics: func(m TurnMetrics) { recorded = append(recorded, m) }, + }) + + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt err = %v", err) + } + if len(recorded) != 1 { + t.Fatalf("OnTurnMetrics calls = %d, want 1", len(recorded)) + } + // Exactly 3 of the clock's 4 scripted instants are consumed — sentAt, + // the FIRST text delta, EventDone — proving neither the activity events + // nor the second text delta ever called Now(). + if clock.i != 3 { + t.Errorf("clock calls = %d, want 3", clock.i) + } + if recorded[0].TTFTMillis != 50 { + t.Errorf("TTFTMillis = %d, want 50 (the first delta, not a later one)", recorded[0].TTFTMillis) + } + if recorded[0].StreamMillis != 30 { + t.Errorf("StreamMillis = %d, want 30", recorded[0].StreamMillis) + } +} + +// TestTurnMetricsOnlyEmitsOnCompletedCall proves a turn that never reaches +// EventDone (a Stream dial error) emits no turn_metrics line at all — the +// deliverable is one line per COMPLETED model call, not one per attempt. +func TestTurnMetricsOnlyEmitsOnCompletedCall(t *testing.T) { + prov := &flakyProvider{name: "test", failN: 100, err: retryableServerErr()} + var recorded []TurnMetrics + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + OnTurnMetrics: func(m TurnMetrics) { recorded = append(recorded, m) }, + }) + + if _, err := s.Prompt(context.Background(), "go"); err == nil { + t.Fatal("Prompt err = nil, want the permanent Stream failure to surface") + } + if len(recorded) != 0 { + t.Errorf("OnTurnMetrics calls = %d, want 0 (no completed call ever happened)", len(recorded)) + } +} + +// TestTurnMetricsRecordsRetryAttempt is the red-first guard for +// streamTurnWithRetry's attempt number reaching the emitted TurnMetrics. +// Attempt 1 fails retryably (a Stream dial error, so streamTurn never enters +// its event loop and never emits metrics for that attempt — see +// TestTurnMetricsOnlyEmitsOnCompletedCall); attempt 2 succeeds and must +// report Attempt == 2, not 1. +// +// Red-verify the NAMED mechanism: with streamTurn's TurnMetrics literal +// hardcoded to Attempt: 1 (dropping the attempt parameter), this test's +// m.Attempt != 2 assertion is the only one that fails — TestPromptRetries* +// still pass because none of them inspect TurnMetrics. +func TestTurnMetricsRecordsRetryAttempt(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + base := time.Date(2026, 8, 26, 12, 0, 0, 0, time.UTC) + clock := &stepClock{times: []time.Time{ + base, // attempt 1 sentAt (Stream fails before any event) + base.Add(1 * time.Second), // attempt 2 sentAt + base.Add(1200 * time.Millisecond), // attempt 2 firstDeltaAt == doneAt (EventDone is the only event) + base.Add(1200 * time.Millisecond), // attempt 2 doneAt + }} + prov := &flakyProvider{ + name: "test", + failN: 1, + err: retryableServerErr(), + ok: asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + } + var recorded []TurnMetrics + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + PromptRetries: 2, + Now: clock.now, + OnTurnMetrics: func(m TurnMetrics) { recorded = append(recorded, m) }, + }) + + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt err = %v, want nil (the retry masks the blip)", err) + } + if len(recorded) != 1 { + t.Fatalf("OnTurnMetrics calls = %d, want 1 (only the successful attempt completes)", len(recorded)) + } + if recorded[0].Attempt != 2 { + t.Errorf("Attempt = %d, want 2 (attempt 1 failed and was retried)", recorded[0].Attempt) + } + if recorded[0].TTFTMillis != 200 { + t.Errorf("TTFTMillis = %d, want 200", recorded[0].TTFTMillis) + } + }) +} + +// TestDefaultTurnMetricsLogDoesNotPanic is a minimal smoke test for +// Config.OnTurnMetrics's default (see emitTurnMetrics/defaultTurnMetricsLog, +// turn_metrics.go): a session built with no OnTurnMetrics callback must +// still complete a turn without panicking, proving the stderr slog default +// is actually wired rather than left nil. +func TestDefaultTurnMetricsLogDoesNotPanic(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + }) + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt err = %v", err) + } +} From 28dc486b22d9d20b3479d1fb011e7f7e8f9cbfff Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 14:10:41 -0400 Subject: [PATCH 04/95] feat(anthropic): emit defer_loading and the bm25 tool search tool (#189) harness defers MCP tool schemas with its own client-side mechanism: a name-only catalog in the system prompt plus the mcp tool's search and select actions. That is portable -- it works on every provider -- and it is what Anthropic's own docs call a custom tool search implementation. On models that support Anthropic's SERVER-side tool search there is a better answer available, and this is the transcoder half of taking it: the API keeps deferred definitions out of the context window itself, runs the search server-side, and expands what the model discovers, with no round-trip through a harness tool at all. provider.ToolDef gains DeferLoading, a request for provider-native deferral. Only an adapter with a native mechanism acts on it; every other adapter ignores it, which is the safe default, since a deferred tool with no way to be discovered would be unreachable. The engine therefore sets it only for a route it knows can honor the request -- that selection is the next slice, so nothing sets it yet and no behaviour changes on any route. The anthropic transcoder sends the BM25 variant rather than regex. Both search the same fields and ship on the same models, so the choice is about how the model expresses a query. A malformed pattern is a real failure mode the regex variant owns -- the doc's own invalid_tool_input example is "missing ) at position 1", and a pattern is capped at 200 characters -- while a natural-language query cannot be syntactically invalid. harness's own client-side search already ranks a natural-language query over exactly those fields, so BM25 keeps one mental model for the same task across routes instead of making the query language depend on which provider a session happens to run. The search tool is prepended only when something is actually deferred, so a session that defers nothing serializes the tool array exactly as it did before this commit. It goes first because it is a fixed two-field entry: the array's leading bytes stay identical across requests, and the caller's own group order is preserved behind it, which is what the byte-stability rule needs. Deferral survives that rule -- the extended test asserts repeated transcodes of the same tool state are byte-identical. One documented 400 is handled by degrading rather than failing: the API rejects a request whose tools are all deferred, and the search tool does not count as the non-deferred one. A caller that defers everything gets its tools sent eagerly instead of a broken turn. modelmeta.SupportsToolSearch is the gate, from the doc's compatibility table, and it fails safe in two directions. Only provider "anthropic" can be true, because the openai and openai-compat routes reach Chat Completions surfaces with no tool_search at all. And a BEDROCK-STYLE anthropic ref is false whatever model it names: server-side tool search on Bedrock is InvokeModel-only, nothing in a ref says which Bedrock API sits in front of it, and guessing wrong costs a rejected request on every turn while guessing off costs a catalog segment that already works. Building that table found a real gap in the existing one: claude-mythos-5 is in the tool-search compatibility table but has no models.dev entry at all (checked live) and no route this repo talks to serves it, so no context window can be sourced for it. Rather than invent one, it is exempted by a SELF-RETIRING list -- the cross-table test fails if the exemption ever outlives its reason, and fails again if the list names a model that is not a tool-search model. Verification: four guards red-verified against the exact mechanism each names -- emitting the search tool, the all-deferred fallback, keeping the default path clean, and keeping cache breakpoints out of the tools array. go test -race ./... green, go vet and gofmt clean. Co-authored-by: andybons --- modelmeta/modelmeta.go | 76 ++++++++++ modelmeta/tool_search_test.go | 100 +++++++++++++ provider/anthropic/tool_search_test.go | 186 +++++++++++++++++++++++++ provider/anthropic/transcode.go | 69 ++++++++- provider/provider.go | 14 ++ 5 files changed, 443 insertions(+), 2 deletions(-) create mode 100644 modelmeta/tool_search_test.go create mode 100644 provider/anthropic/tool_search_test.go diff --git a/modelmeta/modelmeta.go b/modelmeta/modelmeta.go index 89790120..2831c786 100644 --- a/modelmeta/modelmeta.go +++ b/modelmeta/modelmeta.go @@ -270,3 +270,79 @@ func stripBedrockAnthropicPrefix(model string) (suffix string, isAnthropic bool) } return rest, true } + +// anthropicToolSearchModels is the set of first-party Anthropic model IDs +// that support the SERVER-side tool search tool +// (tool_search_tool_regex_20251119 / tool_search_tool_bm25_20251119), from +// the model-compatibility table in +// platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool. +// Both variants ship on exactly the same models, so one set answers for +// either. +// +// A set, not a table of versions: harness picks the variant (see +// provider/anthropic), so the only question this package answers is whether +// the ref can do server-side tool search at all. Claude Opus 4.1 and +// earlier cannot. +// +// Undated aliases are keyed alongside the dated IDs for the same reason +// anthropicContextWindows keys both: a config may name either form. +var anthropicToolSearchModels = map[string]bool{ + "claude-fable-5": true, + // claude-mythos-5 is in the tool-search compatibility table but has no + // models.dev entry (checked live: the anthropic provider's model map + // has no mythos key at all) and no route this repo talks to serves it, + // so anthropicContextWindows cannot key it either. Keeping it here is + // the accurate answer if such a ref ever appears, and costs nothing + // meanwhile; see TestToolSearchModelsAreKnownToContextWindow for the + // self-retiring exemption that keeps the two tables honest. + "claude-mythos-5": true, + "claude-haiku-4-5": true, + "claude-haiku-4-5-20251001": true, + "claude-opus-4-5": true, + "claude-opus-4-5-20251101": true, + "claude-opus-4-6": true, + "claude-opus-4-7": true, + "claude-opus-4-8": true, + "claude-opus-5": true, + "claude-sonnet-4-5": true, + "claude-sonnet-4-5-20250929": true, + "claude-sonnet-4-6": true, + "claude-sonnet-5": true, +} + +// SupportsToolSearch reports whether ref can use Anthropic's server-side +// tool search tool. It is the gate provider/anthropic gives native +// delegation: a ref this answers false for keeps harness's own client-side +// deferral (the catalog segment plus the mcp tool's search/select actions), +// which works on every provider. +// +// Two deliberate fail-safe-off rules, both of which make an unknown ref +// keep the client-side mechanism rather than emit a tool the route may +// reject: +// +// - Only ref.Provider "anthropic" can be true. The openai and +// openai-compat routes reach a Chat Completions surface, which has no +// tool_search at all (OpenAI's own tool search is Responses-API only), +// and a gateway that proxies Anthropic under some other provider name +// is not something this package can recognize. +// - A BEDROCK-STYLE anthropic ref is false, whatever model it names. +// Server-side tool search on Amazon Bedrock is available only through +// InvokeModel, not the Converse API, and nothing in a ref says which +// API the gateway in front of it uses. Guessing wrong costs a rejected +// request on every turn; guessing off costs a catalog segment that +// already works. Same conservative posture bedrockAnthropicContextWindows +// takes for a family its snapshot does not key. +// +// The Bifrost routing-namespace segment is stripped exactly as +// ContextWindow strips it (lastPathSegment), so a boxes ref like +// "anthropic/claude-opus-5" resolves. +func SupportsToolSearch(ref message.ModelRef) bool { + if ref.Provider != "anthropic" { + return false + } + model := lastPathSegment(ref.Model) + if _, isBedrockStyle := stripBedrockAnthropicPrefix(model); isBedrockStyle { + return false + } + return anthropicToolSearchModels[model] +} diff --git a/modelmeta/tool_search_test.go b/modelmeta/tool_search_test.go new file mode 100644 index 00000000..4adc86ba --- /dev/null +++ b/modelmeta/tool_search_test.go @@ -0,0 +1,100 @@ +package modelmeta + +import ( + "testing" + + "github.com/majorcontext/harness/message" +) + +// TestSupportsToolSearch pins the gate native delegation runs on. Every +// "true" model here is one the Anthropic tool-search doc's compatibility +// table lists; every "false" case is one where emitting the tool search +// tool would risk a rejected request on every turn, so the gate fails safe +// and the session keeps harness's own client-side deferral. +func TestSupportsToolSearch(t *testing.T) { + tests := []struct { + name string + ref message.ModelRef + want bool + }{ + // The documented table, in both the dated and undated forms a + // config may name. + {name: "opus 5", ref: message.ModelRef{Provider: "anthropic", Model: "claude-opus-5"}, want: true}, + {name: "fable 5", ref: message.ModelRef{Provider: "anthropic", Model: "claude-fable-5"}, want: true}, + {name: "mythos 5", ref: message.ModelRef{Provider: "anthropic", Model: "claude-mythos-5"}, want: true}, + {name: "sonnet 4.5 dated", ref: message.ModelRef{Provider: "anthropic", Model: "claude-sonnet-4-5-20250929"}, want: true}, + {name: "sonnet 4.5 undated", ref: message.ModelRef{Provider: "anthropic", Model: "claude-sonnet-4-5"}, want: true}, + {name: "haiku 4.5 dated", ref: message.ModelRef{Provider: "anthropic", Model: "claude-haiku-4-5-20251001"}, want: true}, + {name: "opus 4.5 dated", ref: message.ModelRef{Provider: "anthropic", Model: "claude-opus-4-5-20251101"}, want: true}, + {name: "opus 4.6", ref: message.ModelRef{Provider: "anthropic", Model: "claude-opus-4-6"}, want: true}, + {name: "opus 4.8", ref: message.ModelRef{Provider: "anthropic", Model: "claude-opus-4-8"}, want: true}, + + // A Bifrost boxes ref carries a routing-namespace segment ahead of + // the model ID; ContextWindow strips it and so must this. + {name: "bifrost namespaced", ref: message.ModelRef{Provider: "anthropic", Model: "anthropic/claude-opus-5"}, want: true}, + + // Opus 4.1 and earlier are named in the doc as unsupported. + {name: "opus 4.1", ref: message.ModelRef{Provider: "anthropic", Model: "claude-opus-4-1"}, want: false}, + {name: "unknown model", ref: message.ModelRef{Provider: "anthropic", Model: "claude-something-9"}, want: false}, + {name: "empty model", ref: message.ModelRef{Provider: "anthropic"}, want: false}, + + // Bedrock-style refs: server-side tool search is InvokeModel-only + // there, and a ref cannot say which Bedrock API is in front of it. + {name: "bedrock-style anthropic ref", ref: message.ModelRef{Provider: "anthropic", Model: "anthropic.claude-opus-5-v1:0"}, want: false}, + {name: "bedrock-style regional", ref: message.ModelRef{Provider: "anthropic", Model: "us.anthropic.claude-sonnet-4-5-20250929-v1:0"}, want: false}, + {name: "bedrock provider", ref: message.ModelRef{Provider: "bedrock", Model: "anthropic.claude-opus-5"}, want: false}, + + // Other providers have no tool_search on the surface we speak. + {name: "openai", ref: message.ModelRef{Provider: "openai", Model: "gpt-5.4"}, want: false}, + {name: "openai-compat gateway", ref: message.ModelRef{Provider: "bifrost", Model: "anthropic/claude-opus-5"}, want: false}, + {name: "empty ref", ref: message.ModelRef{}, want: false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := SupportsToolSearch(tc.ref); got != tc.want { + t.Fatalf("SupportsToolSearch(%+v) = %v, want %v", tc.ref, got, tc.want) + } + }) + } +} + +// toolSearchModelsWithoutContextWindow are the tool-search models this +// package deliberately cannot size. models.dev publishes no entry for them +// (checked live), and no route this repo talks to serves them, so inventing +// a context window would be worse than admitting the gap: a wrong window +// arms compaction at the wrong threshold. +// +// The exemption is SELF-RETIRING — see the test below, which fails once a +// model here does gain a context-window entry, so the list cannot quietly +// outlive its reason. +var toolSearchModelsWithoutContextWindow = map[string]bool{ + "claude-mythos-5": true, +} + +// TestToolSearchModelsAreKnownToContextWindow keeps the two tables honest +// about the same model set: every ref this package says can do tool search +// is one it also knows a context window for. A name in one table and not +// the other is a typo in whichever was edited last. +func TestToolSearchModelsAreKnownToContextWindow(t *testing.T) { + for model := range anthropicToolSearchModels { + ref := message.ModelRef{Provider: "anthropic", Model: model} + _, ok := ContextWindow(ref) + exempt := toolSearchModelsWithoutContextWindow[model] + switch { + case !ok && !exempt: + t.Errorf("%q supports tool search but has no context-window entry", model) + case ok && exempt: + t.Errorf("%q now has a context-window entry: drop it from toolSearchModelsWithoutContextWindow", model) + } + } +} + +// TestToolSearchExemptionsAreToolSearchModels stops the exemption list +// drifting into naming models that are not in the tool-search table at all. +func TestToolSearchExemptionsAreToolSearchModels(t *testing.T) { + for model := range toolSearchModelsWithoutContextWindow { + if !anthropicToolSearchModels[model] { + t.Errorf("%q is exempted but is not a tool-search model", model) + } + } +} diff --git a/provider/anthropic/tool_search_test.go b/provider/anthropic/tool_search_test.go new file mode 100644 index 00000000..2965076a --- /dev/null +++ b/provider/anthropic/tool_search_test.go @@ -0,0 +1,186 @@ +package anthropic + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// tsTool is a client tool def, deferred or not. +func tsTool(name string, defer_ bool) provider.ToolDef { + return provider.ToolDef{ + Name: name, + Description: "does " + name, + InputSchema: json.RawMessage(`{"type":"object"}`), + DeferLoading: defer_, + } +} + +func tsRequest(tools ...provider.ToolDef) *provider.Request { + return &provider.Request{ + Model: message.ModelRef{Provider: "anthropic", Model: "claude-opus-5"}, + System: []string{"sys"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + Tools: tools, + } +} + +// TestDeferredToolsEmitSearchToolAndDeferLoading is the wire shape from the +// tool-search doc: the search tool entry, every definition still sent, and +// defer_loading only on the tools the caller deferred. +func TestDeferredToolsEmitSearchToolAndDeferLoading(t *testing.T) { + out, err := transcodeRequest(tsRequest( + tsTool("bash", false), + tsTool("mcp__github__create_issue", true), + tsTool("mcp__github__list_issues", true), + ), DefaultCacheTTL) + if err != nil { + t.Fatal(err) + } + + // The search tool leads, so the array's opening bytes are stable. + if len(out.Tools) != 4 { + t.Fatalf("got %d tools, want the search tool plus all 3 definitions", len(out.Tools)) + } + if out.Tools[0].Type != toolSearchToolType || out.Tools[0].Name != toolSearchToolName { + t.Fatalf("first tool = %+v, want the bm25 search tool", out.Tools[0]) + } + // A server tool entry carries type+name only. + if out.Tools[0].Description != "" || len(out.Tools[0].InputSchema) != 0 || out.Tools[0].DeferLoading { + t.Fatalf("search tool entry carries client-tool fields: %+v", out.Tools[0]) + } + + // Every definition is still sent, deferred or not — the API needs them + // server-side to run the search and expand references. + byName := map[string]apiToolDef{} + for _, tool := range out.Tools[1:] { + byName[tool.Name] = tool + if len(tool.InputSchema) == 0 { + t.Errorf("%q was sent without its input schema", tool.Name) + } + } + if byName["bash"].DeferLoading { + t.Error("a non-deferred tool was marked defer_loading") + } + for _, name := range []string{"mcp__github__create_issue", "mcp__github__list_issues"} { + if !byName[name].DeferLoading { + t.Errorf("%q was not marked defer_loading", name) + } + } +} + +// TestNoDeferredToolsEmitsNoSearchTool keeps the default path byte-identical +// to what it was before native delegation existed. +func TestNoDeferredToolsEmitsNoSearchTool(t *testing.T) { + out, err := transcodeRequest(tsRequest(tsTool("bash", false), tsTool("read_file", false)), DefaultCacheTTL) + if err != nil { + t.Fatal(err) + } + if len(out.Tools) != 2 { + t.Fatalf("got %d tools, want exactly the 2 client tools", len(out.Tools)) + } + for _, tool := range out.Tools { + if tool.Type != "" { + t.Fatalf("a server tool entry appeared with nothing deferred: %+v", tool) + } + } + raw, err := json.Marshal(out.Tools) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(raw), "defer_loading") { + t.Fatalf("defer_loading reached the wire with nothing deferred: %s", raw) + } +} + +// TestAllToolsDeferredFallsBackToEager guards the one documented 400: "At +// least one tool must have defer_loading=false. All tools cannot be +// deferred." A caller that defers everything gets eager tools rather than a +// failed turn. +func TestAllToolsDeferredFallsBackToEager(t *testing.T) { + out, err := transcodeRequest(tsRequest(tsTool("a", true), tsTool("b", true)), DefaultCacheTTL) + if err != nil { + t.Fatal(err) + } + if len(out.Tools) != 2 { + t.Fatalf("got %d tools, want the 2 client tools with no search tool", len(out.Tools)) + } + for _, tool := range out.Tools { + if tool.DeferLoading { + t.Fatalf("%q stayed deferred with no non-deferred tool to satisfy the API: %+v", tool.Name, tool) + } + if tool.Type != "" { + t.Fatalf("a search tool was emitted with every tool deferred: %+v", tool) + } + } +} + +// TestDeferredToolsCarryNoCacheControl is the cache-composition guard. +// +// harness places its cache breakpoints on the last SYSTEM block and the last +// message block, never inside the tools array (apiToolDef has no +// cache_control field at all), so "a deferred tool carrying a breakpoint" is +// unreachable by construction rather than by convention. That matters +// because Anthropic's caching doc puts the tool-definitions breakpoint on +// the last tool in the array — which, once tools are deferred, is a deferred +// tool. This test pins the property so a future per-tool breakpoint cannot +// land there without failing here first. +// +// Deferral does not weaken the caching harness does have: the API excludes +// deferred definitions from the system-prompt prefix, so the prefix the +// system breakpoint covers is untouched. +func TestDeferredToolsCarryNoCacheControl(t *testing.T) { + out, err := transcodeRequest(tsRequest( + tsTool("bash", false), + tsTool("mcp__github__create_issue", true), + ), CacheTTL1h) + if err != nil { + t.Fatal(err) + } + raw, err := json.Marshal(out.Tools) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(raw), "cache_control") { + t.Fatalf("a cache breakpoint reached the tools array: %s", raw) + } + // The breakpoint harness does set is still on the last system block, + // and the deferred tool did not disturb it. + if n := len(out.System); n == 0 || out.System[n-1].CacheControl == nil { + t.Fatalf("system breakpoint missing with tools deferred: %+v", out.System) + } + if out.System[len(out.System)-1].CacheControl.TTL != CacheTTL1h { + t.Fatalf("system breakpoint TTL = %q, want %q", out.System[len(out.System)-1].CacheControl.TTL, CacheTTL1h) + } +} + +// TestToolArrayByteStableWithDeferral is the #164 property, extended to the +// native path: two requests that change no tool state must serialize the +// same tools array, search tool included. +func TestToolArrayByteStableWithDeferral(t *testing.T) { + tools := []provider.ToolDef{tsTool("bash", false), tsTool("mcp__a__x", true), tsTool("mcp__a__y", true)} + first, err := transcodeRequest(tsRequest(tools...), DefaultCacheTTL) + if err != nil { + t.Fatal(err) + } + want, err := json.Marshal(first.Tools) + if err != nil { + t.Fatal(err) + } + for i := 0; i < 10; i++ { + out, err := transcodeRequest(tsRequest(tools...), DefaultCacheTTL) + if err != nil { + t.Fatal(err) + } + got, err := json.Marshal(out.Tools) + if err != nil { + t.Fatal(err) + } + if string(got) != string(want) { + t.Fatalf("call %d differs:\nwant %s\ngot %s", i, want, got) + } + } +} diff --git a/provider/anthropic/transcode.go b/provider/anthropic/transcode.go index 38238968..2775ba45 100644 --- a/provider/anthropic/transcode.go +++ b/provider/anthropic/transcode.go @@ -119,11 +119,47 @@ type apiSource struct { } type apiToolDef struct { + // Type names a SERVER-side tool (e.g. the tool search tool); it is + // omitted for an ordinary client tool, which the API identifies by + // name plus schema. A server tool entry carries Type and Name only -- + // Description and InputSchema stay empty, which is why both are + // omitempty here. + Type string `json:"type,omitempty"` Name string `json:"name"` - Description string `json:"description"` - InputSchema json.RawMessage `json:"input_schema"` + Description string `json:"description,omitempty"` + InputSchema json.RawMessage `json:"input_schema,omitempty"` + // DeferLoading keeps this tool's definition out of the model's context + // until it discovers the tool through tool search. The definition is + // still sent: the API needs it server-side to run the search and to + // expand the tool_reference block it returns. + // + // Never set on the tool search tool itself, and never on every tool -- + // the API rejects a request whose tools are all deferred ("At least one + // tool must have defer_loading=false"). + DeferLoading bool `json:"defer_loading,omitempty"` } +// Tool search tool identifiers, from +// platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool. +// +// harness sends the BM25 variant. Both variants search the same fields +// (tool names, descriptions, argument names, argument descriptions) and +// ship on the same models, so the choice is only about how the model +// expresses a query: regex makes it construct a pattern, BM25 lets it write +// a natural-language query. Two reasons BM25 wins here. A malformed pattern +// is a real failure mode the regex variant owns -- the doc's own +// invalid_tool_input example is "missing ) at position 1", and a pattern is +// also capped at 200 characters -- while a natural-language query cannot be +// syntactically invalid. And harness's own client-side search, which stays +// the mechanism on every other provider, already ranks a natural-language +// query over exactly those fields (see engine/mcp_search.go), so BM25 keeps +// one mental model for the same task across routes rather than making the +// query language depend on which provider a session happens to run. +const ( + toolSearchToolType = "tool_search_tool_bm25_20251119" + toolSearchToolName = "tool_search_tool_bm25" +) + // apiCacheControl marks a prompt-cache breakpoint. TTL is the cache lifetime: // empty means the API default (5 minutes) and omits the field; "1h" selects the // extended TTL, which requires the extendedCacheTTLBeta header on the request. @@ -272,11 +308,40 @@ func transcodeRequest(req *provider.Request, ttl string) (*apiRequest, error) { out.System[n-1].CacheControl = cacheControl(ttl) } + // Tool array. A deferred tool is sent in full, exactly like any other + // -- defer_loading controls what enters the model's CONTEXT, not what + // the request carries -- and its presence is what makes the tool search + // tool useful, so the search tool is prepended whenever anything is + // deferred. + // + // The search tool goes FIRST, and that position is a prompt-cache + // decision as much as a readability one: it is a fixed two-field entry, + // so the array's leading bytes stay identical across requests, and the + // caller's own group order (built-ins, then MCP, then plugins -- see + // engine.Session.toolDefs) is preserved behind it. + var deferred int + for _, t := range req.Tools { + if t.DeferLoading { + deferred++ + } + } + if deferred > 0 && deferred < len(req.Tools) { + // The guard is deferred < len(req.Tools), not deferred > 0 alone: + // the API rejects a request whose tools are ALL deferred, and the + // search tool itself does not count as the non-deferred one for + // that rule. A caller that defers everything gets its tools sent + // eagerly rather than a 400 -- degrading to today's behaviour beats + // failing the turn. + out.Tools = append(out.Tools, apiToolDef{Type: toolSearchToolType, Name: toolSearchToolName}) + } for _, t := range req.Tools { out.Tools = append(out.Tools, apiToolDef{ Name: t.Name, Description: t.Description, InputSchema: t.InputSchema, + // Only marked when the search tool went out with it: a deferred + // tool with no way to be discovered is an unreachable tool. + DeferLoading: t.DeferLoading && deferred < len(req.Tools), }) } diff --git a/provider/provider.go b/provider/provider.go index fb130e20..2a86c81b 100644 --- a/provider/provider.go +++ b/provider/provider.go @@ -21,6 +21,20 @@ type ToolDef struct { Name string Description string InputSchema json.RawMessage // JSON Schema + + // DeferLoading asks the PROVIDER to keep this tool's schema out of the + // model's context until the model discovers it, instead of loading it + // up front. The definition is still sent on every request: deferral + // controls what enters the context window, not what the wire carries. + // + // Only an adapter whose API has a native deferral mechanism acts on + // this; provider/anthropic emits defer_loading plus a server-side tool + // search tool (see its transcoder). Every other adapter IGNORES the + // field, which is the safe default -- a tool with no way to be + // discovered would otherwise be unreachable -- so the engine sets it + // only for a route it knows can honor it, and keeps its own + // client-side deferral everywhere else. + DeferLoading bool } // Request is one model call. System and Messages are canonical; the adapter From cdb09acf7720a5775c58f175aa43a57f0875af27 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 16:28:25 -0400 Subject: [PATCH 05/95] test(plugin): prove concurrent RPCs to one plugin are id-multiplexed (#190) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(plugin): prove one connection carries concurrent RPCs The engine is about to execute the tool calls of one assistant message as a concurrent batch. Those calls dispatch tool.execute.before and tool.execute.after to the same plugin, over one stdio pipe, at the same time. Nobody had ever verified that the pipe can carry two in-flight requests. If it paired a response with a request by ARRIVAL ORDER, or if two writers could interleave the bytes of two frames, a parallel batch would cross the results of two unrelated tool calls. That is a silent, data-corrupting failure, and it would have shipped as a surprise inside a performance change. The answer is that the connection is already id-multiplexed, so this change adds no production code. conn.call takes a unique id from nextID, parks its own channel in pending, and the read loop routes each response by that id; conn.write holds wmu across the whole frame. The same conn type serves both ends, and an incoming request gets its own goroutine, so a Go SDK plugin answers concurrent requests concurrently. instance.start holds inst.mu across the whole dial-and-handshake, so a concurrent first dispatch spawns one process. plugin.Host is a box-scoped singleton shared by every session, so two sessions have always been able to dispatch to one plugin at once — this is not a new kind of concurrency. A per-plugin serialization mutex was the alternative, and it is rejected. It would throttle a transport that is already correct, and it would make one slow hook block every other tool call in a batch. Four mechanisms now have tests. TestConcurrentCallsMultiplexByID holds one request open, sends a second, and answers the second first; each caller must get its own result. TestConcurrentWritesNeverInterleaveFrames writes from 16 goroutines over a transport that tears every Write into 16-byte chunks — net.Pipe alone hides the defect, because it serializes a whole Write itself. TestHostConcurrentToolHooksStayIndependent and TestHostConcurrentExecuteToolStaysIndependent drive the production Host API with two dispatches genuinely in flight, held at a rendezvous until both have arrived. TestConcurrentFirstDispatchSpawnsOnce counts dials. Every test is deterministic. The rendezvous tests run in a synctest bubble with a one-hour hook deadline, so a transport that serialized requests makes the fake clock jump to that deadline and fail at once, with no wall-clock cost. Verification. Each mechanism was red-verified against its own name. Pairing responses by arrival order instead of by id: both calls report "response crossed with the other call". Removing wmu from conn.write: "frame is not one whole JSON-RPC message". Serializing conn.call to one in-flight request: the hook deadline fires and the dispatch fails open with unrewritten args, and the ExecuteTool test reports a bubble deadlock. Removing instance.start's mutex and started guard: -race reports a data race. Every break was reverted. go build, go vet and go test -race ./... are clean. The five tests ran 1000 times at GOMAXPROCS=2 under -race with no failure. * docs(plugin): specify the connection concurrency contract PROTOCOL.md said nothing about how many requests may be in flight on one connection. A plugin author in any language had to guess, and the safe guess — strict request-response, answer in arrival order — is wrong: the harness already dispatches to one plugin from several sessions at once, and parallel tool execution will soon do the same within one session. A plugin built on the wrong guess corrupts its own pipe, because a frame whose bytes interleave with another frame loses both. The contract is now written down where a plugin author reads it. The rules are: the harness may keep several requests in flight; id is the only correlation; a plugin may answer in any order; a plugin that writes from more than one thread must serialize its writes so every frame is one whole line; and a plugin whose handlers are not reentrant may serialize internally, which leaves the harness correct and only throttled. The hook/event FIFO guarantee is unchanged and is called out as unchanged, because concurrency applies to requests only. The Go SDK's own position needed stating too, because it is the opposite default: Serve gives each incoming request its own goroutine, so a Go plugin's hook and tool functions must be safe for concurrent use. That now sits on the Hooks type, where an author writing a hook will see it. conn.call carries the matching note for a harness-side reader. This documents an existing wire property. No payload shape changes, so ProtocolVersion stays 1, and the Versioning section needs no entry. * docs(plugin): state within-session hook overlap as future work Review of #190 found the Concurrency section, the conn.call comment and the concurrency_test.go header all describe parallel tool execution in the present tense. The engine does not do it yet — Session.runToolCalls still runs one call at a time — so a plugin author reading PROTOCOL.md as the binding spec would expect within-session overlap that cannot happen today. Cross-session overlap IS real today, through the box-scoped Host, so the two claims now stand apart: one as current fact, one as future work. The same review found TestConcurrentWritesNeverInterleaveFrames leaks the runNotifications goroutine. newConn starts it, and only conn.close, through conn.fail, closes the channel it exits on; the test closed the pipes alone. The test runs outside a synctest bubble, so nothing reported the leak. Its cleanup now closes the conn. --------- Co-authored-by: andybons --- plugin/PROTOCOL.md | 40 ++++ plugin/concurrency_test.go | 463 +++++++++++++++++++++++++++++++++++++ plugin/protocol.go | 12 + plugin/sdk.go | 8 + 4 files changed, 523 insertions(+) create mode 100644 plugin/concurrency_test.go diff --git a/plugin/PROTOCOL.md b/plugin/PROTOCOL.md index 83fa9bab..ba2ed3f1 100644 --- a/plugin/PROTOCOL.md +++ b/plugin/PROTOCOL.md @@ -149,6 +149,46 @@ on the fire-and-forget `event` hook needs a throttling/coalescing design first so a slow plugin can't fall arbitrarily far behind or amplify RPC volume. Not in this vocabulary yet. +## Concurrency + +The harness MAY keep **several requests in flight on one connection at the +same time**. A plugin must not assume request/response lockstep. + +The rules, for a plugin in any language: + +- **`id` is the only correlation.** Match a response to its request by the + request `id`, never by arrival order. Both sides number their own requests + independently, so an id is unique only within one direction. +- **Answer in any order.** A plugin may reply to a later request first. The + harness demultiplexes by `id` (`conn.call`, `conn.dispatch`), so a slow + handler never blocks a fast one. +- **Write each frame as one whole line, atomically.** A plugin that writes + from more than one thread MUST serialize its writes. Two frames whose + bytes interleave are both lost, and the connection carries responses for + every other in-flight request too. The Go SDK holds one write mutex for + the life of the connection (`conn.write`). +- **Reentrancy is the plugin's choice.** A plugin whose handlers are not + safe to run at the same time MAY serialize them internally. The harness + stays correct — it is throttled, not broken. The Go SDK does the opposite + by default: it serves each incoming request on its own goroutine, so a Go + plugin's hook handlers must be safe for concurrent use. +- **The `hook/event` FIFO guarantee is unchanged.** Notifications still + arrive in emit order, one at a time, per plugin (see "Chaining semantics" + above). Concurrency applies to REQUESTS only. + +Concurrency is not new to this version. `plugin.Host` is a box-scoped +singleton shared by every session, so two sessions have always been able to +dispatch a hook to one plugin at the same time. Parallel tool execution in +the engine will add the same concurrency WITHIN one session: the tool calls +of one assistant message will run as a batch, so their +`tool.execute.before` and `tool.execute.after` dispatches will overlap. +Today the engine still runs those calls one at a time +(`Session.runToolCalls`), so a plugin sees within-session overlap only +after that change lands. + +This section documents an existing wire property. It changes no payload +shape, so `ProtocolVersion` stays 1. + ## Message content Tool outputs and generate results use the canonical `message.Parts` encoding: diff --git a/plugin/concurrency_test.go b/plugin/concurrency_test.go new file mode 100644 index 00000000..4c219971 --- /dev/null +++ b/plugin/concurrency_test.go @@ -0,0 +1,463 @@ +package plugin + +import ( + "bufio" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" +) + +// This file is the specification for ONE property: a harness may keep +// several requests in flight on one plugin connection at the same time. +// +// The property is load-bearing for parallel tool execution in the engine, +// which is not implemented yet: Session.runToolCalls still runs one call at +// a time. A batch of tool calls that runs concurrently will dispatch the +// tool.execute.before / tool.execute.after hooks concurrently too, all over +// the same plugin pipe. If that pipe paired requests with responses by +// ARRIVAL ORDER, or if two writers could interleave the bytes of two +// frames, a parallel batch would cross the results of two unrelated tool +// calls — a silent, data-corrupting failure. +// +// The tests below prove the four mechanisms that make the pipe safe: +// +// 1. TestConcurrentCallsMultiplexByID — conn.call correlates a response by +// its JSON-RPC id, never by arrival order. +// 2. TestConcurrentWritesNeverInterleaveFrames — conn.write's wmu keeps +// every frame whole on a transport that tears a Write into chunks. +// 3. TestHostConcurrentToolHooksStayIndependent and +// TestHostConcurrentExecuteToolStaysIndependent — the same property +// through the production Host API, with two hooks genuinely in flight. +// 4. TestConcurrentFirstDispatchSpawnsOnce — a concurrent first dispatch +// spawns one plugin process, not two. +// +// See PROTOCOL.md, "Concurrency", for the contract these tests hold. + +// rendezvousTimeout is the hook deadline for the Host-level tests below. It +// is deliberately huge: every one of those tests runs inside a synctest +// bubble, so a hook that can never complete makes the bubble's fake clock +// jump straight to this deadline and the test fails at once, with no +// wall-clock cost. A small value would let a healthy dispatch look like a +// timeout instead. +const rendezvousTimeout = time.Hour + +// TestConcurrentCallsMultiplexByID proves conn.call correlates a response +// with its request by the JSON-RPC id, not by the order the peer answers. +// +// The fake peer holds the FIRST request open until the SECOND request has +// arrived, then answers the second one first. Both calls are therefore in +// flight together, and the responses come back in the opposite order. Each +// caller must still get its own result. +// +// Determinism: net.Pipe is synchronous, so a request line is on the wire +// only after the test reads it. The test reads request 1 before it starts +// call 2, and answers only after it has read both. No sleep, and no +// deadline, gates any step. +func TestConcurrentCallsMultiplexByID(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + peer, connSide := net.Pipe() + c := newConn(connSide, func(context.Context, string, json.RawMessage) (any, error) { + return nil, errors.New("plugin: test peer sends no requests") + }) + go c.run() //nolint:errcheck // stream end is expected at cleanup + t.Cleanup(func() { + _ = peer.Close() + _ = connSide.Close() + }) + + r := bufio.NewReader(peer) + w := bufio.NewWriter(peer) + + type outcome struct { + method string + got string + err error + } + done := make(chan outcome, 2) + call := func(method string) { + var got string + err := c.call(context.Background(), method, map[string]string{"method": method}, &got) + done <- outcome{method: method, got: got, err: err} + } + + readRequest := func() rpcMessage { + t.Helper() + line, err := r.ReadBytes('\n') + if err != nil { + t.Fatalf("reading request: %v", err) + } + var msg rpcMessage + if err := json.Unmarshal(line, &msg); err != nil { + t.Fatalf("unmarshaling request %q: %v", line, err) + } + if msg.ID == nil { + t.Fatalf("request %q carries no id", line) + } + return msg + } + respond := func(msg rpcMessage, result string) { + t.Helper() + raw, err := json.Marshal(result) + if err != nil { + t.Fatal(err) + } + out, err := json.Marshal(rpcMessage{JSONRPC: "2.0", ID: msg.ID, Result: raw}) + if err != nil { + t.Fatal(err) + } + if _, err := w.Write(append(out, '\n')); err != nil { + t.Fatalf("writing response: %v", err) + } + if err := w.Flush(); err != nil { + t.Fatalf("flushing response: %v", err) + } + } + + // Call "first" goes out and stays unanswered. + go call("first") + req1 := readRequest() + if req1.Method != "first" { + t.Fatalf("request 1 method = %q, want %q", req1.Method, "first") + } + + // Call "second" goes out while "first" is still in flight. Reading + // its line proves the connection accepted a second concurrent + // request instead of serializing behind the first. + go call("second") + req2 := readRequest() + if req2.Method != "second" { + t.Fatalf("request 2 method = %q, want %q", req2.Method, "second") + } + if *req1.ID == *req2.ID { + t.Fatalf("both requests carry id %d: ids must be unique per call", *req1.ID) + } + + // Answer in REVERSE order. Correlation by id is the only thing that + // can route these two responses correctly. + respond(req2, "result-second") + respond(req1, "result-first") + + results := make(map[string]string, 2) + for range 2 { + o := <-done + if o.err != nil { + t.Fatalf("call %q: %v", o.method, o.err) + } + results[o.method] = o.got + } + if got := results["first"]; got != "result-first" { + t.Errorf("call \"first\" got %q, want %q (response crossed with the other call)", got, "result-first") + } + if got := results["second"]; got != "result-second" { + t.Errorf("call \"second\" got %q, want %q (response crossed with the other call)", got, "result-second") + } + }) +} + +// tearingConn wraps a stream and splits every Write into small chunks, with +// a scheduling point between them. A real pipe does this: a write above +// PIPE_BUF is not atomic. conn.write holds wmu across its whole rwc.Write +// call, so the chunks of one frame stay together; without that lock two +// writers interleave their chunks and produce garbage lines. +// +// net.Pipe alone cannot show this defect, because net.Pipe serializes a +// whole Write internally (its own wrMu). tearingConn removes that cover. +type tearingConn struct { + io.ReadWriteCloser + chunk int +} + +func (t *tearingConn) Write(p []byte) (int, error) { + written := 0 + for len(p) > 0 { + n := min(t.chunk, len(p)) + got, err := t.ReadWriteCloser.Write(p[:n]) + written += got + if err != nil { + return written, err + } + p = p[n:] + } + return written, nil +} + +// TestConcurrentWritesNeverInterleaveFrames proves conn.write serializes +// whole frames. Many goroutines write at once over a transport that tears +// every Write into 16-byte chunks. Every line the peer reads must be one +// complete, well-formed JSON-RPC message, and the peer must see EXACTLY the +// frames that were sent — no missing frame and no extra one. +func TestConcurrentWritesNeverInterleaveFrames(t *testing.T) { + const writers = 16 + + peer, connSide := net.Pipe() + c := newConn(&tearingConn{ReadWriteCloser: connSide, chunk: 16}, func(context.Context, string, json.RawMessage) (any, error) { + return nil, errors.New("plugin: test peer sends no requests") + }) + // Close the conn itself, not just the pipes: newConn starts the + // runNotifications goroutine, and only conn.close (through conn.fail) + // closes the channel that goroutine exits on. Closing the pipes alone + // leaks it. + t.Cleanup(func() { + _ = c.close() + _ = peer.Close() + }) + + // Pad each frame well past the chunk size, so an unserialized write is + // certain to be cut apart and mixed with another writer's bytes. + pad := strings.Repeat("x", 512) + var wg sync.WaitGroup + start := make(chan struct{}) + for i := range writers { + wg.Add(1) + go func() { + defer wg.Done() + <-start + _ = c.notify("hook/event", map[string]any{"writer": i, "pad": pad}) + }() + } + close(start) + + r := bufio.NewReader(peer) + seen := make(map[int]int, writers) + for range writers { + line, err := r.ReadBytes('\n') + if err != nil { + t.Fatalf("reading frame: %v", err) + } + var msg rpcMessage + if err := json.Unmarshal(line, &msg); err != nil { + t.Fatalf("frame is not one whole JSON-RPC message: %v\nline: %q", err, line) + } + var params struct { + Writer int `json:"writer"` + Pad string `json:"pad"` + } + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + t.Fatalf("frame params are damaged: %v\nline: %q", err, line) + } + if params.Pad != pad { + t.Fatalf("frame from writer %d carries a damaged payload (%d bytes, want %d)", + params.Writer, len(params.Pad), len(pad)) + } + seen[params.Writer]++ + } + wg.Wait() + + // The surplus direction: every writer appears exactly once, and no + // writer appears twice. + if len(seen) != writers { + t.Fatalf("saw %d distinct writers, want %d: %v", len(seen), writers, seen) + } + for i := range writers { + if seen[i] != 1 { + t.Errorf("writer %d appeared %d times, want exactly 1", i, seen[i]) + } + } +} + +// TestHostConcurrentToolHooksStayIndependent drives the PRODUCTION Host API +// (Host.ToolExecuteBefore), not a hand-built conn, and proves two hook +// dispatches to ONE plugin run at the same time and never cross results. +// +// The fake plugin is a rendezvous: each invocation announces itself on +// arrived and then waits. The test reads BOTH announcements before it +// releases either. A transport that served one request at a time could +// never produce the second announcement, so the bubble would run out of +// runnable goroutines, jump to rendezvousTimeout, and fail. +func TestHostConcurrentToolHooksStayIndependent(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + arrived := make(chan string, 2) + proceed := make(chan struct{}) + release := make(chan struct{}) + t.Cleanup(func() { close(release) }) + + p := testPlugin(t, "rendezvous", &Hooks{ + ToolExecuteBefore: func(_ context.Context, _ *Client, req *ToolExecuteBeforeRequest) (*ToolExecuteBeforeResponse, error) { + arrived <- req.CallID + select { + case <-proceed: + case <-release: + } + // Echo the call id back through the rewritten args. A + // crossed response shows up as the wrong id here. + return &ToolExecuteBeforeResponse{ + Args: json.RawMessage(fmt.Sprintf(`{"echo":%q}`, req.CallID)), + }, nil + }, + }) + h := newTestHost(t, Options{HookTimeout: rendezvousTimeout}, p) + + type outcome struct { + callID string + args string + deny string + } + done := make(chan outcome, 2) + for _, id := range []string{"call-a", "call-b"} { + go func() { + args, deny := h.ToolExecuteBefore(context.Background(), &ToolExecuteBeforeRequest{ + SessionID: "s1", CallID: id, Tool: "bash", Args: json.RawMessage(`{}`), + }) + done <- outcome{callID: id, args: string(args), deny: deny} + }() + } + + // Both hooks must be in flight together before either may finish. + first, second := <-arrived, <-arrived + if first == second { + t.Fatalf("both dispatches reported call id %q: the two calls were not independent", first) + } + close(proceed) + + for range 2 { + o := <-done + if o.deny != "" { + t.Fatalf("call %s was denied: %q", o.callID, o.deny) + } + want := fmt.Sprintf(`{"echo":%q}`, o.callID) + if o.args != want { + t.Errorf("call %s got args %s, want %s (result crossed with the other call)", o.callID, o.args, want) + } + } + }) +} + +// TestHostConcurrentExecuteToolStaysIndependent is the same proof for +// Host.ExecuteTool: two plugin TOOL calls in flight at once over one +// connection, each getting its own output. +func TestHostConcurrentExecuteToolStaysIndependent(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + arrived := make(chan string, 2) + proceed := make(chan struct{}) + release := make(chan struct{}) + t.Cleanup(func() { close(release) }) + + p := testPlugin(t, "tooler", &Hooks{ + Tools: []Tool{{ + Def: ToolDef{Name: "echo", Description: "echoes its argument"}, + Execute: func(_ context.Context, _ *Client, args json.RawMessage) (message.Parts, error) { + var in struct { + Want string `json:"want"` + } + if err := json.Unmarshal(args, &in); err != nil { + return nil, err + } + arrived <- in.Want + select { + case <-proceed: + case <-release: + } + return message.Parts{&message.Text{Text: in.Want}}, nil + }, + }}, + }) + h := newTestHost(t, Options{HookTimeout: rendezvousTimeout}, p) + + type outcome struct { + want string + got string + err error + } + done := make(chan outcome, 2) + for _, want := range []string{"alpha", "beta"} { + go func() { + resp, err := h.ExecuteTool(context.Background(), &ToolExecuteRequest{ + SessionID: "s1", CallID: want, Tool: "echo", + Args: json.RawMessage(fmt.Sprintf(`{"want":%q}`, want)), + }) + o := outcome{want: want, err: err} + if err == nil && len(resp.Output) == 1 { + if txt, ok := resp.Output[0].(*message.Text); ok { + o.got = txt.Text + } + } + done <- o + }() + } + + first, second := <-arrived, <-arrived + if first == second { + t.Fatalf("both tool calls reported %q: the two calls were not independent", first) + } + close(proceed) + + for range 2 { + o := <-done + if o.err != nil { + t.Fatalf("tool call %q: %v", o.want, o.err) + } + if o.got != o.want { + t.Errorf("tool call %q returned %q (output crossed with the other call)", o.want, o.got) + } + } + }) +} + +// TestConcurrentFirstDispatchSpawnsOnce proves a plugin process is spawned +// exactly once when several dispatches reach a not-yet-started instance at +// the same time. instance.start holds inst.mu across the whole +// dial-plus-handshake, so the losers of the race wait and then reuse the +// one connection. +// +// Two claims, checked separately. The sequential claim is deterministic on +// its own. The concurrent claim depends on goroutine scheduling, so the +// hammer run (-count=1000 -cpu=2 -race) is what makes it a real guard; +// -race also reports the unsynchronized field writes a missing lock causes. +func TestConcurrentFirstDispatchSpawnsOnce(t *testing.T) { + const dispatchers = 8 + + var dials atomic.Int64 + hooks := &Hooks{ + ToolExecuteBefore: func(_ context.Context, _ *Client, req *ToolExecuteBeforeRequest) (*ToolExecuteBeforeResponse, error) { + return &ToolExecuteBeforeResponse{}, nil + }, + } + spec := Spec{ + Manifest: Manifest{Name: "counted", ProtocolVersion: ProtocolVersion, Hooks: hooks.hookList()}, + dial: func() (io.ReadWriteCloser, error) { + dials.Add(1) + hostSide, pluginSide := net.Pipe() + go serve(pluginSide, Manifest{Name: "counted"}, hooks) //nolint:errcheck + return hostSide, nil + }, + } + h := newTestHost(t, Options{HookTimeout: rendezvousTimeout}, spec) + + var wg sync.WaitGroup + start := make(chan struct{}) + for range dispatchers { + wg.Add(1) + go func() { + defer wg.Done() + <-start + h.ToolExecuteBefore(context.Background(), &ToolExecuteBeforeRequest{ + SessionID: "s1", CallID: "c", Tool: "bash", Args: json.RawMessage(`{}`), + }) + }() + } + close(start) + wg.Wait() + if got := dials.Load(); got != 1 { + t.Fatalf("%d concurrent dispatches dialed the plugin %d times, want exactly 1", dispatchers, got) + } + + // A later dispatch reuses the same process too: a started instance is + // never re-dialed. + h.ToolExecuteBefore(context.Background(), &ToolExecuteBeforeRequest{ + SessionID: "s1", CallID: "c", Tool: "bash", Args: json.RawMessage(`{}`), + }) + if got := dials.Load(); got != 1 { + t.Fatalf("a dispatch after the race dialed again: %d dials, want 1", got) + } +} diff --git a/plugin/protocol.go b/plugin/protocol.go index 785aae7f..aa2abf60 100644 --- a/plugin/protocol.go +++ b/plugin/protocol.go @@ -245,6 +245,18 @@ func (c *conn) serveRequest(msg rpcMessage) { // call sends a request and decodes the peer's result into result (which may // be nil to discard it). +// +// Concurrent calls on one conn are safe and expected: each call takes its +// own id from nextID, parks its own channel in pending, and the read loop +// (dispatch) routes each response by that id. Response ORDER is therefore +// irrelevant — a peer may answer a later request first. conn.write holds +// wmu across the whole frame, so two callers can never interleave bytes on +// the stream. Two sessions already dispatch hooks to one plugin this way, +// because Host is a box-scoped singleton. Parallel tool execution in the +// engine will depend on the same property within ONE session: one assistant +// message's tool calls will dispatch their tool.execute.before/after hooks +// to one plugin at the same time. See PROTOCOL.md, "Concurrency", and +// plugin/concurrency_test.go. func (c *conn) call(ctx context.Context, method string, params, result any) error { id := c.nextID.Add(1) ch := make(chan rpcMessage, 1) diff --git a/plugin/sdk.go b/plugin/sdk.go index eb125550..07ba184e 100644 --- a/plugin/sdk.go +++ b/plugin/sdk.go @@ -18,6 +18,14 @@ import ( // // Plugin processes stay warm for the session, so module-level caches (token // TTL caches, compiled matchers, per-session state) are expected and fine. +// +// A hook function must be SAFE FOR CONCURRENT USE. The harness keeps several +// requests in flight on one connection (see PROTOCOL.md, "Concurrency"), and +// this SDK serves each incoming request on its own goroutine, so two calls of +// the same hook — or of two different hooks — can run at the same time. Guard +// any shared cache. A plugin that cannot be made reentrant may serialize its +// own handlers with a mutex; the harness stays correct and is only throttled. +// Tool functions in Tools below follow the same rule. type Hooks struct { Event func(ctx context.Context, c *Client, events []Event) ChatParams func(ctx context.Context, c *Client, req *ChatParamsRequest) (*ChatParamsResponse, error) From 02742f8a0c87eb7178bde2450aee5b2e9e0bd07b Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 16:54:28 -0400 Subject: [PATCH 06/95] fix(engine): refuse write_file overwrite of an unread existing file (#192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): refuse write_file overwrite of an unread existing file write_file overwrote any existing file unconditionally, with no protection at all: a model that never opened a file, or that read it long enough ago for another actor to change it, could destroy its content in one call with no recovery path. edit_file never had this hole — its old_string exact-match requirement is safe by construction, since it cannot replace content it did not first quote correctly — but write_file's full-content overwrite had nothing playing the same role. Claude Code refuses Write on an unread existing file, and opencode raises a literal "You must read file X before overwriting it" error; harness had no equivalent for either of its two write tools. The fix tracks, per live session and in memory only, every path read_file has read or write_file/edit_file has written, keyed on the resolved absolute path (never the raw relative tool argument, so two arguments resolving to the same file are not tracked as two paths) and mapped to the sha256 hash of that path's raw on-disk bytes at the moment of that read or write. write_file on a path that resolves to an existing regular file now requires, in order: the path is in this session's read set, and hashing the path's CURRENT on-disk bytes still matches what was recorded. A path that does not resolve to an existing regular file is unguarded, since creation is write_file's main job and there is nothing to protect. A content hash, not just a boolean "was it ever read," is what closes the second failure mode: a session that read a file once and then overwrites it much later, after another actor changed it, would otherwise pass a same-path check while still destroying content the model never actually saw. read_file hashes the complete raw bytes it already reads off disk for its own image/text classification, never the offset/limit-sliced text it returns to the model — read_file always reads the whole file regardless of what window it displays, so hashing anything less would misrecord what the session actually saw on an out-of-range offset request. A successful write_file or edit_file also updates the hash for the path it just wrote, so an immediate follow-up write to the same path (the model overwriting its own just-written content, or an edit_file followed by a write_file) never has to read_file again first — the session already knows exactly what landed on disk because it just put it there. The read set is deliberately runtime-only: never persisted, never folded by LoadSession, and never copied by configSnapshot (it lives on Session state, not Config, so a spawned child session starts with its own empty set, correctly, since it has read nothing yet). A reloaded session therefore starts with an empty read set and must read_file a path again before write_file can overwrite it, even a path read in a prior process life. This is conservative by design: the guard exists to stop an overwrite of content the model never actually saw THIS session, and a resumed model has no live memory of raw file bytes from a prior process either, only whatever text made it into the persisted transcript. Tool calls execute strictly sequentially today (Session.runToolCalls), so the read set's own mutex-guarded map access is sufficient protection now. A parallel tool executor, when it lands, will need to serialize concurrent write_file/edit_file calls against the same resolved path — matching edit_file's existing same-path safety property — since the map's per-operation lock alone does not make the check-current-hash-then-write sequence atomic against a concurrent writer to the same path; this is noted at the field's declaration rather than solved here, since no parallel executor exists yet to serialize against. bash writes are explicitly out of scope: harness has no way to classify an arbitrary shell command as a file write versus anything else it might do, so this guard covers only the two built-in tools that make a structured, typed claim to write a file. AGENTS.md gains a "write_file read-before-overwrite guard" section documenting the design, per the rule that a behavior change updates AGENTS.md in the same commit. Verification: go test -race ./... is green across every package, go vet and gofmt are clean. TestWriteFileUnreadExistingFileErrors, TestWriteFileChangedSinceReadErrors, and TestReloadClearsReadGuardSet were red-verified by disabling the write_file guard block and confirming all three fail for the reason their names claim, then re-enabling it and confirming green. * fix(engine): harden the read-guard against failed reads and stat errors Cross-model review of the read-before-overwrite guard (PR #192) found two authorization holes: 1. read_file recorded the guard hash before validating the requested offset, so a read that returned ONLY an error (offset past end-of-file) still unlocked a full overwrite of content the model never received. The hash is now recorded only at a return that hands the model content: the image return, the empty-file return, and the windowed text return. A failed read records nothing. 2. write_file treated EVERY stat failure as proof that no protected file exists, falling through to unguarded creation. Only fs.ErrNotExist falls through now; any other stat error (permission, transient metadata failure) refuses the write, because it cannot prove there is nothing there to protect. The review's remaining findings — check-to-write is not atomic against external writers, bash and edit_file are unguarded routes, whole-file hash authorizes more than the displayed window — are accepted as the guard's documented advisory posture, matching the Claude Code and opencode guards this PR mirrors: it prevents the accidental blind overwrite, it is not a security boundary against a determined writer. The comment prose no longer over-claims otherwise. Verification: two new tests (failed read does not unlock; non-not-exist stat error refuses); red-verified by reverting each mechanism. * docs: align the guard section with the hardened stat and record semantics * fix(engine): gate the guard on regular files only, as documented The guard checked !info.IsDir(), which also matched FIFOs, sockets, and device files — write_file to /dev/null was refused as an unread existing file, a regression from unconditional pre-guard behavior and a contradiction of the code's own regular-file wording. The gate is now info.Mode().IsRegular(). New test writes /dev/null unguarded. --- AGENTS.md | 76 +++++++++++++ engine/engine.go | 20 ++++ engine/filetools.go | 93 ++++++++++++++- engine/filetools_test.go | 237 ++++++++++++++++++++++++++++++++++++++- 4 files changed, 421 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9e8d4711..57a17087 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -209,6 +209,82 @@ issue #129. Until it lands, a model with no vision support receives the image Blob exactly as any vision-capable model does; how it handles that block is between the model and its provider. +### write_file read-before-overwrite guard + +`write_file` (`engine/filetools.go`) refuses to overwrite an EXISTING file +the session has not read. Before this guard, `write_file` overwrote any +existing file unconditionally — a model could destroy a file it never +opened, with no recovery path. `edit_file` never had this hole: its +`old_string` match is exact-content-required by construction, so it cannot +blindly clobber unseen content. Claude Code and opencode both close the +same gap on their own write tools (opencode's is a literal `"You must read +file X before overwriting it"` error); this guard gives `write_file` the +same property. + +`Session.recordRead`/`readHashFor` (`engine/filetools.go`) track, per live +session and in memory only, every path `read_file` has read or +`write_file`/`edit_file` has written, keyed on the RESOLVED absolute path +(`s.resolvePath`'s output, never the raw tool argument — two different +relative arguments that resolve to the same file must not be tracked as two +separate paths), mapped to the sha256 hash of that path's raw on-disk bytes +at the moment of that read or write. `read_file` hashes the complete raw +file bytes it already read off disk for its own content classification +(`readPathContent`'s `TextData`/`ImageData`) — never the offset/limit-sliced +text it returns to the model. This matches the reference guards (Claude +Code, opencode): the guard authorizes per successful OPEN, not per byte +displayed — a windowed read of a large file authorizes replacing the whole +file. The hash is recorded only at a `read_file` return that hands the +model content; a read that errors (offset past end-of-file) records +nothing. + +`write_file` on a path that `os.Stat` resolves to an existing regular file +requires, in order: (1) the path is present in this session's read set — +absent means `write_file: %s exists and has not been read this session; +read it first (or use edit_file)`; (2) hashing the path's CURRENT on-disk +bytes fresh (never trusting the recorded hash's age) matches the recorded +hash — a mismatch means `write_file: %s changed on disk since it was read; +read it again before overwriting`. A path that does not exist +(`fs.ErrNotExist`) is unguarded — creation is `write_file`'s main job, and +there is no prior content to protect. Any OTHER stat failure (permission, +transient metadata error) refuses the write with `write_file: cannot stat +%s to check the read-before-overwrite guard` — a failed stat cannot prove +no protected file exists there. A +successful `write_file` or `edit_file` records/updates the written path's +hash to the new content's hash, so a write immediately followed by another +write to the SAME path (the assistant overwriting its own just-written +content, or an `edit_file` followed by a `write_file` on the same path) +never spuriously re-triggers the guard — the session already knows exactly +what is on disk because it just put it there. + +The read set is runtime-only: never persisted, never folded by +`LoadSession`. A reloaded session starts with an EMPTY read set, so a +resumed session must `read_file` a path again before `write_file` can +overwrite it — even a path the session genuinely read in a prior process +life. This is deliberately conservative and matches the guard's purpose: +the guard exists to stop an overwrite of content the model never actually +saw THIS session, and a resumed model has no live memory of raw file bytes +from a prior process either, only whatever text happened to land in the +persisted transcript. The read set lives on `Session` state, not `Config`, +so `configSnapshot` (used to seed a spawned child's config) never copies +it — a spawned child starts with its own empty read set, correctly, since +it is a different session that has read nothing yet. + +Tool calls execute strictly sequentially today +(`Session.runToolCalls`/`runToolCall`, `engine/engine.go`), so the read +set's own `sync.Mutex`-guarded map access is sufficient protection now. +A future parallel tool executor must serialize concurrent +`write_file`/`edit_file` calls against the SAME resolved path — matching +`edit_file`'s existing same-path safety requirement — rather than relying +on the map's per-operation lock alone, which does not cover the +check-current-hash-then-write sequence as one atomic unit against a +concurrent writer to the same path. + +`bash` writes (a model redirecting output to an existing file via a shell +command) are explicitly OUT of scope: harness cannot classify an arbitrary +shell command as a file write versus anything else it might do, so this +guard covers only the two built-in tools that make a structured, typed +claim to write a file. + ### Base loop retry The base interactive `Prompt` loop retries a transient provider error itself, diff --git a/engine/engine.go b/engine/engine.go index ff3cba7b..fdddcd80 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -6,6 +6,7 @@ package engine import ( "context" + "crypto/sha256" "encoding/json" "errors" "fmt" @@ -995,6 +996,24 @@ type Session struct { // rather than a per-process one. Guarded by mu. toolResultBytes int + // readHashes backs the write_file read-before-overwrite guard (see the + // "write_file read-before-overwrite guard" section of AGENTS.md and + // writeFileTool's doc comment in filetools.go). It maps a resolved + // absolute path (s.resolvePath's output, never a raw relative tool + // argument) to the sha256 hash of that path's raw on-disk bytes as of + // the last time this session read it (read_file) or wrote it + // (write_file/edit_file). write_file refuses to overwrite an existing + // file whose path is absent here, or whose current on-disk hash no + // longer matches the recorded one. Deliberately in-memory and + // per-live-Session only: never persisted, never folded by LoadSession, + // and never copied by configSnapshot (it lives on Session state, not + // Config, so a spawned child starts with its own empty set) — a + // reloaded or spawned session has read nothing yet, so starting empty + // is the conservative, correct default rather than a gap. Guarded by + // mu, like toolResults above; see filetools.go's recordRead/readHashFor + // for the only accessors. + readHashes map[string][sha256.Size]byte + // taskNotifications is this session's pending queue of child-completion // signals not yet checked out for an in-flight turn attempt (see // taskdelivery.go): SessionManager.finalizeTurn appends to it from @@ -1068,6 +1087,7 @@ func newSession(cfg Config) *Session { contextWindowSource: contextWindowSource, toolResultNextID: 1, toolResults: make(map[string]toolResultMeta), + readHashes: make(map[string][sha256.Size]byte), } for _, t := range []Tool{bashTool(cfg.BashTimeout, cfg.BashOutputCap), readFileTool(), writeFileTool(), editFileTool(), sessionInfoTool(), globTool(), grepTool(), lsTool()} { s.tools[t.Def.Name] = t diff --git a/engine/filetools.go b/engine/filetools.go index 55b0b250..7747471e 100644 --- a/engine/filetools.go +++ b/engine/filetools.go @@ -3,13 +3,16 @@ package engine import ( "bytes" "context" + "crypto/sha256" "encoding/json" + "errors" "fmt" "image" _ "image/gif" // register GIF decoder for image.DecodeConfig _ "image/jpeg" // register JPEG decoder for image.DecodeConfig _ "image/png" // register PNG decoder for image.DecodeConfig "io" + "io/fs" "net/http" "os" "path/filepath" @@ -178,6 +181,42 @@ func (s *Session) resolvePath(path string) string { return filepath.Join(s.cfg.WorkDir, path) } +// recordRead records that this session has seen resolvedPath's raw on-disk +// bytes hash to hash, either because read_file just read them or because +// write_file/edit_file just wrote them. resolvedPath must already be an +// s.resolvePath output — every caller in this file passes one, so the map +// never keys on a raw, unresolved tool argument. See the readHashes field +// doc comment (engine.go) and the "write_file read-before-overwrite guard" +// section of AGENTS.md for the full design. +func (s *Session) recordRead(resolvedPath string, hash [sha256.Size]byte) { + s.mu.Lock() + defer s.mu.Unlock() + s.readHashes[resolvedPath] = hash +} + +// readHashFor reports the hash last recorded for resolvedPath and whether +// this session has ever read or written it at all. +func (s *Session) readHashFor(resolvedPath string) (hash [sha256.Size]byte, everRead bool) { + s.mu.Lock() + defer s.mu.Unlock() + hash, everRead = s.readHashes[resolvedPath] + return hash, everRead +} + +// hashFileContent returns the sha256 hash of path's complete current bytes, +// read fresh from disk. write_file uses it to detect whether an +// already-read file changed on disk since this session last saw it (a +// concurrent writer, an external process, a `bash` command) — the recorded +// hash alone is not enough, since a stale record would let a write through +// against content the session's last read no longer describes. +func hashFileContent(path string) ([sha256.Size]byte, error) { + data, err := os.ReadFile(path) + if err != nil { + return [sha256.Size]byte{}, err + } + return sha256.Sum256(data), nil +} + func readFileTool() Tool { return Tool{ Def: provider.ToolDef{ @@ -215,7 +254,19 @@ func readFileTool() Tool { if err != nil { return nil, fmt.Errorf("read_file: %s: %w", path, err) } + rawBytes := content.TextData if content.IsImage { + rawBytes = content.ImageData + } + // The read-before-overwrite guard's hash comes from the RAW + // bytes readPathContent read off disk, never the offset/limit- + // sliced window below — matching the reference guards (Claude + // Code, opencode), which authorize per successful OPEN, not per + // byte displayed. It is recorded only at a return that hands the + // model content: a read that errors (offset past end-of-file) + // records nothing, so a failed read cannot unlock an overwrite. + if content.IsImage { + s.recordRead(path, sha256.Sum256(rawBytes)) summary := fmt.Sprintf("image (%s), %d bytes, %dx%d pixels", content.MediaType, len(content.ImageData), content.Width, content.Height) return message.Parts{ &message.Text{Text: summary}, @@ -240,11 +291,13 @@ func readFileTool() Tool { limit = readFileDefaultLimit } if total == 0 { + s.recordRead(path, sha256.Sum256(rawBytes)) return message.Parts{&message.Text{Text: "(empty file)"}}, nil } if offset > total { return nil, fmt.Errorf("read_file: offset %d is past end of file (%d lines)", offset, total) } + s.recordRead(path, sha256.Sum256(rawBytes)) end := offset + limit - 1 if end > total { end = total @@ -271,7 +324,7 @@ func writeFileTool() Tool { return Tool{ Def: provider.ToolDef{ Name: "write_file", - Description: "Write content to a file, creating parent directories as needed and overwriting any existing file. Prefer this over shell redirection or heredocs for creating and rewriting files. Relative paths resolve against the session working directory.", + Description: "Write content to a file, creating parent directories as needed. Overwriting an existing file requires having read it first with read_file this session, with no changes on disk since — use edit_file for a targeted change, or read_file then write_file to intentionally replace it. Relative paths resolve against the session working directory.", InputSchema: json.RawMessage(`{ "type": "object", "properties": { @@ -290,12 +343,43 @@ func writeFileTool() Tool { return nil, fmt.Errorf("write_file: missing path or content argument") } path := s.resolvePath(in.Path) + + // Read-before-overwrite guard: only an EXISTING regular file is + // gated — creation is write_file's main job, so a path that + // does not exist falls straight through unguarded. Any OTHER + // stat failure refuses the write: it cannot prove no protected + // file exists there. See AGENTS.md's "write_file + // read-before-overwrite guard" section for the full design. + info, statErr := os.Stat(path) + if statErr != nil && !errors.Is(statErr, fs.ErrNotExist) { + return nil, fmt.Errorf("write_file: cannot stat %s to check the read-before-overwrite guard: %v", path, statErr) + } + if statErr == nil && info.Mode().IsRegular() { + recorded, everRead := s.readHashFor(path) + if !everRead { + return nil, fmt.Errorf("write_file: %s exists and has not been read this session; read it first (or use edit_file)", path) + } + current, err := hashFileContent(path) + if err != nil { + return nil, fmt.Errorf("write_file: %w", err) + } + if current != recorded { + return nil, fmt.Errorf("write_file: %s changed on disk since it was read; read it again before overwriting", path) + } + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { return nil, fmt.Errorf("write_file: %w", err) } if err := os.WriteFile(path, []byte(*in.Content), 0o644); err != nil { return nil, fmt.Errorf("write_file: %w", err) } + // The bytes just written are now, by definition, what this + // session has "seen" at this path — record/update the hash so + // an immediate follow-up write to the SAME path (this session + // overwriting its own just-written content) never spuriously + // re-triggers the guard above. + s.recordRead(path, sha256.Sum256([]byte(*in.Content))) s.emitFileEdited(path) return message.Parts{&message.Text{Text: fmt.Sprintf("wrote %d bytes to %s", len(*in.Content), path)}}, nil }, @@ -354,6 +438,13 @@ func editFileTool() Tool { if err := os.WriteFile(path, []byte(content), 0o644); err != nil { return nil, fmt.Errorf("edit_file: %w", err) } + // Update the read-before-overwrite guard's hash to the new + // content: edit_file's own exact-match requirement already + // proves the model saw the pre-edit content, and the file now + // holds exactly what this session just wrote — a later + // write_file to this same path must not have to read_file + // again to learn what edit_file already put there. + s.recordRead(path, sha256.Sum256([]byte(content))) s.emitFileEdited(path) return message.Parts{&message.Text{Text: fmt.Sprintf("replaced %d occurrence(s) in %s", replaced, path)}}, nil }, diff --git a/engine/filetools_test.go b/engine/filetools_test.go index 278f165d..1be19c68 100644 --- a/engine/filetools_test.go +++ b/engine/filetools_test.go @@ -455,15 +455,180 @@ func TestWriteFileCreatesNestedDirs(t *testing.T) { } } -func TestWriteFileOverwrites(t *testing.T) { +// TestWriteFileUnreadExistingFileErrors is the red-verified regression guard +// for the core defect this feature closes: before the read-before-overwrite +// guard existed, write_file overwrote ANY existing file unconditionally — a +// model could destroy a file it never opened. Reverting the os.Stat/ +// readHashFor block in writeFileTool's Run (engine/filetools.go) makes this +// test fail (write_file silently succeeds and "new" clobbers "old"), +// confirming the guard, not something else, is what this test exercises. +func TestWriteFileUnreadExistingFileErrors(t *testing.T) { dir := t.TempDir() writeTestFile(t, filepath.Join(dir, "a.txt"), "old") - if _, err := runTool(t, writeFileTool(), dir, `{"path":"a.txt","content":"new"}`); err != nil { - t.Fatal(err) + + _, err := runTool(t, writeFileTool(), dir, `{"path":"a.txt","content":"new"}`) + if err == nil { + t.Fatal("want error overwriting an existing file never read this session") + } + wantSubstr := "exists and has not been read this session; read it first (or use edit_file)" + if !strings.Contains(err.Error(), wantSubstr) { + t.Errorf("error = %q, want it to contain %q", err, wantSubstr) + } + got, _ := os.ReadFile(filepath.Join(dir, "a.txt")) + if string(got) != "old" { + t.Errorf("content = %q, want unchanged %q", got, "old") + } +} + +// TestWriteFileReadThenWriteSucceeds is the happy path the guard must not +// block: read_file the existing content, then write_file succeeds and +// overwrites it. runTool builds a fresh session per call, so both tool +// invocations must share the SAME session for the guard to see the read. +func TestWriteFileReadThenWriteSucceeds(t *testing.T) { + dir := t.TempDir() + writeTestFile(t, filepath.Join(dir, "a.txt"), "old") + s := NewSession(Config{WorkDir: dir}) + + if _, err := readFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt"}`)); err != nil { + t.Fatalf("read_file: %v", err) + } + if _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","content":"new"}`)); err != nil { + t.Fatalf("write_file: %v", err) } got, _ := os.ReadFile(filepath.Join(dir, "a.txt")) if string(got) != "new" { - t.Errorf("content = %q", got) + t.Errorf("content = %q, want %q", got, "new") + } +} + +// TestWriteFileChangedSinceReadErrors proves the guard's second check: even +// with a recorded read, write_file refuses to overwrite a file that changed +// on disk since that read (a concurrent writer, another tool, an external +// process) — trusting only the "was it ever read" bit would let a write +// through against content the session's last read no longer describes. +func TestWriteFileChangedSinceReadErrors(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "a.txt") + writeTestFile(t, path, "old") + s := NewSession(Config{WorkDir: dir}) + + if _, err := readFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt"}`)); err != nil { + t.Fatalf("read_file: %v", err) + } + // Simulate an external change landing after the read, bypassing every + // tool this session tracks. + writeTestFile(t, path, "externally changed") + + _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","content":"new"}`)) + if err == nil { + t.Fatal("want error overwriting a file changed on disk since it was read") + } + wantSubstr := "changed on disk since it was read; read it again before overwriting" + if !strings.Contains(err.Error(), wantSubstr) { + t.Errorf("error = %q, want it to contain %q", err, wantSubstr) + } + got, _ := os.ReadFile(path) + if string(got) != "externally changed" { + t.Errorf("content = %q, want unchanged %q", got, "externally changed") + } +} + +// TestWriteFileCreateNewUnguarded proves the guard is scoped to EXISTING +// files only: creating a brand-new path never requires a prior read_file, +// since there is no existing content to protect. TestWriteFileCreatesNestedDirs +// above already covers this same path with a fresh runTool session per +// call; this test makes the "never read this session" condition explicit. +func TestWriteFileCreateNewUnguarded(t *testing.T) { + dir := t.TempDir() + s := NewSession(Config{WorkDir: dir}) + + if _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"new.txt","content":"hello"}`)); err != nil { + t.Fatalf("write_file on a new path: %v", err) + } + got, _ := os.ReadFile(filepath.Join(dir, "new.txt")) + if string(got) != "hello" { + t.Errorf("content = %q, want %q", got, "hello") + } +} + +// TestEditFileUpdatesReadGuardHash proves requirement 3 of the guard design: +// a successful edit_file updates the tracked hash to the post-edit content, +// so an edit-then-write sequence on the SAME path never has to read_file +// again — edit_file's own exact-match requirement already proves the model +// saw the pre-edit content, and the file now holds exactly what this +// session just wrote. +func TestEditFileUpdatesReadGuardHash(t *testing.T) { + dir := t.TempDir() + writeTestFile(t, filepath.Join(dir, "a.txt"), "hello world\n") + s := NewSession(Config{WorkDir: dir}) + + if _, err := readFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt"}`)); err != nil { + t.Fatalf("read_file: %v", err) + } + if _, err := editFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","old_string":"world","new_string":"there"}`)); err != nil { + t.Fatalf("edit_file: %v", err) + } + // No read_file call between edit_file and write_file: the guard must + // trust edit_file's own hash update, not require a fresh read. + if _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","content":"replaced entirely"}`)); err != nil { + t.Fatalf("write_file after edit_file with no intervening read_file: %v", err) + } + got, _ := os.ReadFile(filepath.Join(dir, "a.txt")) + if string(got) != "replaced entirely" { + t.Errorf("content = %q, want %q", got, "replaced entirely") + } +} + +// TestReloadClearsReadGuardSet proves requirement 4: the read set is +// runtime-only and never persisted. A session that read a path, then was +// persisted and reloaded via LoadSession (a fresh process resuming a +// session, or this same process after a restart), must NOT remember that +// read — write_file on the same path in the reloaded session requires a +// fresh read_file, exactly as if the path had never been read at all. +func TestReloadClearsReadGuardSet(t *testing.T) { + dir := t.TempDir() + sessionDir := t.TempDir() + writeTestFile(t, filepath.Join(dir, "a.txt"), "old") + + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg := Config{ + WorkDir: dir, + SessionDir: sessionDir, + Providers: provider.Registry{prov.name: prov}, + Model: message.ModelRef{Provider: prov.name, Model: "m1"}, + } + s := NewSession(cfg) + // A real turn so LoadSession below has a persisted log to find. + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatal(err) + } + if err := s.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + + if _, err := readFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt"}`)); err != nil { + t.Fatalf("read_file: %v", err) + } + // Confirm the guard is actually satisfied on the live session before + // reloading, so the reload assertion below means what it claims. + if _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","content":"live session can overwrite"}`)); err != nil { + t.Fatalf("write_file on live session after read_file: %v", err) + } + writeTestFile(t, filepath.Join(dir, "a.txt"), "old again") // restore for the reloaded check below + + reloaded, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatal(err) + } + _, err = writeFileTool().Run(context.Background(), reloaded, json.RawMessage(`{"path":"a.txt","content":"new"}`)) + if err == nil { + t.Fatal("want error: reloaded session's read set must be empty") + } + wantSubstr := "exists and has not been read this session" + if !strings.Contains(err.Error(), wantSubstr) { + t.Errorf("error = %q, want it to contain %q", err, wantSubstr) } } @@ -558,3 +723,67 @@ func TestFileToolsOfferedToProvider(t *testing.T) { } } } + +// TestWriteFileFailedReadDoesNotUnlock proves a read_file that ERRORED +// (offset past end-of-file) does not authorize an overwrite: the guard +// records a hash only at a return that handed the model content. +func TestWriteFileFailedReadDoesNotUnlock(t *testing.T) { + dir := t.TempDir() + writeTestFile(t, filepath.Join(dir, "a.txt"), "line1\nline2\n") + s := NewSession(Config{WorkDir: dir}) + + if _, err := readFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","offset":99}`)); err == nil { + t.Fatal("want read_file error for offset past end of file") + } + _, err := writeFileTool().Run(context.Background(), s, json.RawMessage(`{"path":"a.txt","content":"new"}`)) + if err == nil { + t.Fatal("want write_file refusal: the only read of this file errored") + } + if !strings.Contains(err.Error(), "has not been read this session") { + t.Errorf("error = %q, want the unread-file refusal", err) + } + got, _ := os.ReadFile(filepath.Join(dir, "a.txt")) + if string(got) != "line1\nline2\n" { + t.Errorf("content = %q, want unchanged", got) + } +} + +// TestWriteFileStatErrorRefuses proves a stat failure OTHER than not-exist +// refuses the write: an unreadable path cannot prove no protected file +// exists there, so falling through to unguarded-create would be a hole. +func TestWriteFileStatErrorRefuses(t *testing.T) { + if os.Getuid() == 0 { + t.Skip("permission-based stat failure cannot be produced as root") + } + dir := t.TempDir() + locked := filepath.Join(dir, "locked") + if err := os.Mkdir(locked, 0o755); err != nil { + t.Fatal(err) + } + writeTestFile(t, filepath.Join(locked, "a.txt"), "old") + if err := os.Chmod(locked, 0o000); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chmod(locked, 0o755) }) + + _, err := writeFileTool().Run(context.Background(), NewSession(Config{WorkDir: dir}), json.RawMessage(`{"path":"locked/a.txt","content":"new"}`)) + if err == nil { + t.Fatal("want write_file refusal on a stat error that is not not-exist") + } + if !strings.Contains(err.Error(), "cannot stat") { + t.Errorf("error = %q, want the cannot-stat refusal", err) + } +} + +// TestWriteFileSpecialFileUnguarded proves the guard's scope is REGULAR +// files, as documented: writing to a device file like /dev/null (a common +// discard idiom) needs no prior read. +func TestWriteFileSpecialFileUnguarded(t *testing.T) { + if _, err := os.Stat("/dev/null"); err != nil { + t.Skip("/dev/null unavailable") + } + dir := t.TempDir() + if _, err := writeFileTool().Run(context.Background(), NewSession(Config{WorkDir: dir}), json.RawMessage(`{"path":"/dev/null","content":"discard"}`)); err != nil { + t.Fatalf("write_file to /dev/null: %v", err) + } +} From 60783fdbef60e5b2d86d309aec4a82574ad876e3 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 17:10:42 -0400 Subject: [PATCH 07/95] fix(engine): auto-continue a turn cut off by max_tokens instead of parking it (#193) * fix(engine): auto-continue a turn cut off by max_tokens instead of parking it Box harness-parallel-tools stalled today: the model emitted a large tool call, the provider stopped mid-emission with stop reason "max_tokens", the engine synthesized the usual unexecuted-tool-call result (appendUnexecutedToolCallResults, NEP-5272), and runAgenticLoop's `if stop != provider.StopToolUse { return asst, nil }` early return then settled the turn. Nothing ever prompted the session again -- a silent work stoppage on a fleet that expects sessions to keep making progress on their own. A human had to notice and re-prompt it by hand. A max_tokens stop means the provider cut the model off; the model never chose to stop. Treating it the same as a clean end_turn is what let the session go idle. The fix: when the stop reason is max_tokens, append the synthetic unexecuted-tool-call result exactly as before (the truncated call must still never execute -- its arguments can be invalid mid-JSON), then re-issue a real follow-up model call in the SAME Prompt loop instead of returning. The follow-up carries a one-shot nudge -- rendered through the existing message.EngineContext/ withAmbientStatus idiom every other ambient status segment already uses (process, MCP, goal-parked, identity) -- telling the model its last turn hit the ceiling and to continue in smaller pieces. This applies identically to a pure-text max_tokens stop with no tool call at all: Claude Code's own behavior there is to let the turn end and rely on the user to re-prompt, but an unattended harness session has no user to do that, so it gets the same one bounded auto-continue. The bound is the load-bearing part. A model can pathologically re-emit oversized output turn after turn, so auto-continue cannot run unbounded -- that would trade a silent stall for a silent (and billed) infinite loop. runAgenticLoop tracks maxTokensStreak, a count of CONSECUTIVE max_tokens stops; Config.MaxTokensContinuations bounds it (default 3, following PromptRetries' own unset-vs-zero config idiom: a bare embedder engine.Config's zero value disables auto-continue entirely, matching every existing test that constructs one directly; config.Config.MaxTokensContinuationsValue supplies the product default via the new max_tokens_continuations key). Exhausting the bound settles the turn with a *maxTokensContinuationExhaustedError naming the bound and emits session.error -- the same "honest terminal, never a silent success" shape emptyTurnError's own retry exhaustion already uses -- rather than parking silently or looping forever. The streak resets to zero the moment any turn in the loop completes with a different stop reason, so an isolated max_tokens stop can never count against a later, unrelated one. An alternative considered: only auto-continue when a tool call was truncated, and leave pure-text truncation alone (Claude Code's behavior). Rejected for the same reason the whole fix exists -- harness sessions run unattended, and a truncated answer with no further call is just a slower version of the same stall. A task child runs through this identical runAgenticLoop (a child Session is a full NewSession(childCfg), and childCfg comes from configSnapshot, a whole-struct copy of the parent's engine.Config), so MaxTokensContinuations reaches a child with no separate wiring; TestTaskChildAutoContinuesMaxTokens asserts against the child's own provider request count and history to confirm this rather than assume it. Verification: five new engine tests plus one task-child test, all red-verified against the pre-fix mechanism (reverted the runAgenticLoop wiring, left the rest in place, confirmed the four auto-continue tests fail for exactly the stalled-session reason -- one provider request, no error, no further call -- then restored the fix and confirmed green). Two new config tests cover the max_tokens_continuations unset/zero/override/merge matrix, mirroring prompt_retries' own test shape. Full go test -race ./... is green, go vet is clean, gofmt reports no diffs. * fix(engine): close five review findings in max_tokens auto-continue An adversarial cross-model review of PR #193 (auto-continue a turn cut off by max_tokens) posted six findings. Five described real defects in the auto-continue mechanism; the sixth asked for three missing test cases. This commit fixes all six. 1. A tool call the provider cut off mid-emission can carry invalid, truncated JSON in Arguments (e.g. `{"comm`). runAgenticLoop kept that call in history and synthesized an is_error result for it, which the model then saw replayed as a hollow, argument-less call it never actually finished -- indistinguishable from one it genuinely intended with no arguments. Fix: dropInvalidPartialToolCall removes a StopMaxTokens turn's trailing ToolCall part in place, before it is ever appended, when its Arguments do not parse as JSON, and no synthetic result is generated for it -- the model simply re-issues the call, complete, once it continues. This supersedes an earlier, deliberate, incident-tested design (message.Message.Normalize clearing Arguments but keeping call identity, still correct and unchanged for every OTHER invalid- Arguments producer) specifically for this mid-emission shape; see TestPersistTruncatedToolCallArguments's updated doc comment for the full history of that design. 2. The continuation nudge was glued onto the newest EXISTING user message via withAmbientStatus, but by the time a continuation request is built that message is no longer the newest thing in the conversation -- the truncated assistant turn (and its synthetic tool result) is. The canonical request ended in RoleAssistant or RoleTool, which Anthropic serializes as assistant prefill: some models 400 outright, and even an accepting model saw a "continue" instruction that chronologically preceded the output it referred to. Fix: appendContinuationNudgeMessage appends a genuine NEW RoleUser message to the end of the request instead, so the request always ends exactly where a real continuation turn should. It still never touches s.history, preserving the one-shot, non-persisted property every ambient segment has. 3. runAgenticLoop's counter reset to zero on every StopToolUse, including a denied, unknown, or failing tool call -- none of which touch toolExecCount. A model could alternate max_tokens and tool_use indefinitely, spending an unbounded number of continuations without ever tripping Config.MaxTokensContinuations. Fix: the counter (renamed maxTokensUsed) is now a per-Prompt budget that only increments and never resets within one runAgenticLoop call, including across an intervening tool_use round. 4. Prompt-queue injection ran only at the tool-call boundary (StopToolUse). The max_tokens continuation branch looped straight back to streamTurnWithRetry with no drain, so an operator message queued during a long truncated response sat undelivered for the whole continuation chain. Fix: both call sites now share drainQueuedPromptsIntoHistory, run before every follow-up request either path issues. 5. maxTokensContinuationExhaustedError was an ordinary error, so goal.go's promptTurnWithRetry retried it through the deterministic goalWorkerRetries budget -- with the default bound of 3, one already-exhausted worker attempt (4 billed calls) could be retried up to 3 times, for 12 calls total on one goal boundary. Fix: the error is now wrapped provider.MarkPermanent at its one construction site, so promptTurnWithRetry's existing fail-fast branch stops after one attempt, the same as any other permanent-classified worker error. It still parks (not clears) the goal, since the condition might not recur on resume. 6. Extended the test suite: the nudge is present on request 2 and absent again on request 3 within the same Prompt call; LoadSession never sees the nudge in the durable log; a transient retry inside the continuation's own streamTurnWithRetry call carries the nudge on both attempts, and a second, unrelated Prompt call afterward does not resurrect it. Every new test is red-verified: each was run against the code with its one named fix reverted (the drop call disabled, the old withAmbientStatus call restored, the streak-reset line restored, the queue-drain call removed, the MarkPermanent wrap removed, the nudge-clear line disabled) and confirmed to fail for the reason its name claims, then re-verified green with the fix restored. A new goal-loop-level test, TestPursueGoalMaxTokensExhaustionFailsFastForGoalRetry, proves finding 5's fix end to end: exactly 4 worker calls for one parked goal boundary, not 12; red-verified the same way (5s of real retry backoff observed with the fix reverted, 0 attempts wasted with it applied). AGENTS.md's "Max-tokens auto-continue" section is rewritten to document all five mechanism changes in the same commit. Verification: go build ./..., go vet ./..., gofmt -l . (clean), and go test -race -count=1 ./engine/ ./provider/... all green. * fix(engine): rebut finding 1 with evidence, revert its implementation The previous commit on this branch implemented adversarial review finding 1 (a StopMaxTokens turn's trailing tool call can carry truncated, invalid partial_json in Arguments, e.g. `{"comm`) by dropping that call entirely from the replayed assistant message and skipping its synthetic result. That PR comment's summary flagged the change as touching an existing, incident-tested invariant (TestPersistTruncatedToolCallArguments, engine/tool_call_poison_test.go) without being able to confirm the finding's own premise reproduced. Checked the premise directly: it does not hold. message.Message.Normalize (Session.append's appendWithUsage, run on every append) already coerces invalid ToolCall.Arguments to nil in place -- through the same *ToolCall pointer the caller's asst still holds, not a copy -- before any continuation request is ever built. This is not incidental; it is the deliberate, already-shipped fix for a real production defect (two goal sessions dead at "json: error calling MarshalJSON for type json.RawMessage: unexpected end of JSON input"), and TestPersistTruncatedToolCallArguments is the regression guard for it. Finding 1's claim that the continuation request "fails JSON marshaling before the request reaches the provider" is therefore rebutted, not merely unconfirmed. This commit: - Removes dropInvalidPartialToolCall and its call site from runAgenticLoop (engine/engine.go), restoring the pre-finding-1 behavior: a genuinely mid-emission tool call keeps its identity (CallID, Name), has only its unusable Arguments cleared by Normalize, and gets the same synthetic is_error result via appendUnexecutedToolCallResults every other unexecuted call gets. - Restores TestPersistTruncatedToolCallArguments (engine/tool_call_poison_test.go) to its original assertions verbatim, and removes the now-obsolete TestMaxTokensDropsInvalidPartialToolCall (engine/max_tokens_continue_test.go), which asserted the reverted "drop entirely" behavior. - Adds TestMaxTokensPartialJSONMarshalsThroughRealTranscoder (engine/max_tokens_wire_test.go): the test that pins the rebuttal so finding 1 cannot be silently re-raised. Unlike an in-memory assertion, it drives a genuine *anthropic.Client (provider/anthropic) against a real httptest HTTP server through an actual Session.Prompt call: a truncated partial_json tool_use block, a max_tokens stop, and the resulting auto-continuation. It proves the wire request the client actually sends decodes cleanly server-side, with the truncated call's identity preserved and Arguments cleared to an empty object, paired with its is_error tool_result -- calling the real production entry point (Client.Stream -> transcodeRequest -> json.Marshal -> HTTP POST), not a hand-rolled marshal check. - Rewrites the corresponding paragraph in AGENTS.md's "Max-tokens auto-continue" section to record finding 1 as rebutted with evidence rather than fixed, naming the pinning test. Findings 2-6 are unchanged from the prior commit on this branch. Red-verified TestMaxTokensPartialJSONMarshalsThroughRealTranscoder two ways: (1) with message.Message.Normalize's ToolCall branch disabled, Prompt returns the exact production error text ("unexpected end of JSON input") and the test fails, confirming it would have caught the original incident; (2) with a temporary drop-entirely shim reintroduced in runAgenticLoop, the test's tool_use/tool_result pairing assertions fail, confirming it would catch finding 1's rejected approach if it is ever reintroduced. Both reverted before this commit. Verification: go build ./..., go vet ./..., gofmt -l . (clean), and go test -race -count=1 ./engine/ ./provider/... all green. --- AGENTS.md | 138 ++++++ cmd/harness/main.go | 2 + config/config.go | 33 ++ config/config_test.go | 68 +++ engine/engine.go | 350 ++++++++++++-- engine/max_tokens_continue_test.go | 711 +++++++++++++++++++++++++++++ engine/max_tokens_wire_test.go | 167 +++++++ 7 files changed, 1440 insertions(+), 29 deletions(-) create mode 100644 engine/max_tokens_continue_test.go create mode 100644 engine/max_tokens_wire_test.go diff --git a/AGENTS.md b/AGENTS.md index 57a17087..83be60b4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -361,6 +361,144 @@ the outer weather tier ever counts it, so a goal loop rides a brief provider blip without spending an outer attempt. `PromptRetries` 0 disables the inner budget for a host that wants the outer tiers to be the only retry. +### Max-tokens auto-continue + +A turn whose stop reason is `provider.StopMaxTokens` means the provider cut +the model off mid-emission — it did not choose to stop. Before this existed, +`runAgenticLoop`'s `if stop != provider.StopToolUse` branch +(`engine/engine.go`) treated every non-`tool_use` stop reason alike: append +`asst`, synthesize an is_error result for any orphaned `ToolCall` part via +`appendUnexecutedToolCallResults` (NEP-5272, see "Wire normalization" +above), and return. For `max_tokens` specifically that return settles the +session idle with nothing further ever prompting it. Incident: box +harness-parallel-tools — the model emitted a large tool call, the provider +stopped mid-emission with `max_tokens`, the engine synthesized the +unexecuted-call result exactly as designed, and the session then sat idle +until a human noticed and re-prompted it. On an autonomous fleet a silent +work stoppage is as bad as a crash. + +`runAgenticLoop` now branches on `stop == provider.StopMaxTokens` inside +that same `if`: `maybeAutoContinueMaxTokens` decides whether to `continue` +the loop (issuing a real follow-up model call in this SAME `Prompt` call) +instead of returning. This applies identically whether or not `asst` carried +a `ToolCall` part — a pure-text `max_tokens` truncation (Claude Code's own +behavior is to let the turn end and rely on the user re-prompting) gets the +same auto-continue an autonomous harness session needs, not a human-facing +half-answer. + +**A genuinely mid-emission tool call keeps its identity; only its Arguments +are cleared.** A cross-model adversarial review of this PR raised a +CRITICAL finding: a `StopMaxTokens` turn's trailing `ToolCall` can carry +non-empty but syntactically invalid `Arguments` — the raw, truncated +`partial_json` Anthropic's `assembledBlock.toolCall` +(`provider/anthropic/anthropic.go`) leaves behind when `max_tokens` lands +before the block's own `content_block_stop`, e.g. `{"comm` — and claimed +replaying that into the continuation request fails `json.Marshal` before it +reaches the provider. That premise does NOT hold: `message.Message.Normalize` +(`Session.append`'s `appendWithUsage`, run on every append) already coerces +the identical invalid-Arguments shape to nil in place, through the SAME +`*ToolCall` pointer `asst.Parts` already holds — so by the time the +continuation request is built, `Arguments` is already safe. This is not +incidental; it is the deliberate, incident-tested fix for a real production +defect (two goal sessions dead at "json: error calling MarshalJSON... "; see +`TestPersistTruncatedToolCallArguments`, `engine/tool_call_poison_test.go`). +The finding is REBUTTED WITH EVIDENCE, not implemented: no code change. See +`TestMaxTokensPartialJSONMarshalsThroughRealTranscoder` +(`engine/max_tokens_wire_test.go`), which drives a genuinely truncated +`partial_json` tool call through a REAL `anthropic.Client` (`provider/anthropic`, +via an `httptest` server, not a hand-rolled stand-in) and proves the +continuation request the client actually sends decodes cleanly server-side, +with the truncated call's identity preserved and its `Arguments` cleared — +this is the test that pins the rebuttal and must go red before any future +"drop the call entirely" change lands unchallenged. + +**The continuation nudge is a genuine new turn, never assistant prefill.** +The follow-up call carries the synthetic unexecuted-tool-call result (for +any COMPLETE call the engine chose not to execute) plus a one-shot nudge — +`s.pendingContinuationNudge`/`continuationNudgeSegment`. Unlike every other +ambient status segment (process, MCP, goal-parked, identity, task +notifications — see "Ambient engine context" above), this one is NOT glued +onto an existing message via `withAmbientStatus`: +`appendContinuationNudgeMessage` appends a genuine NEW `message.RoleUser` +message, carrying the nudge as its own `*message.EngineContext` part, to the +END of `streamTurn`'s own throwaway per-request message copy. +`withAmbientStatus` scans backward for the newest EXISTING `RoleUser` +message, which by the time a continuation request is built is an EARLIER +message than the just-truncated assistant turn (and its synthetic tool +result, if any) — leaving the canonical request ending in `RoleAssistant` or +`RoleTool`. Anthropic serializes that as assistant PREFILL: some models +reject it with a permanent 400, and even an accepting model sees a +"continue" instruction that chronologically precedes the output it refers +to. `appendContinuationNudgeMessage` never touches `s.history` — same as +every ambient segment — so a session reload, or any later unrelated +request, never sees the nudge. It still rides every attempt a +transient-error retry makes for that ONE follow-up call (mirroring +`checkoutTaskNotificationsSegment`'s idempotent-reread shape, +`taskdelivery.go`) and is cleared by `runAgenticLoop` the instant that whole +`streamTurnWithRetry` call returns, so it never bleeds into a later, +unrelated turn. + +**Queued operator input is drained before every continuation, not only at +tool-call boundaries.** `drainQueuedPromptsIntoHistory` (`engine/engine.go`) +is the shared implementation behind the tool-call-boundary drain (after a +`StopToolUse` round actually runs a tool) and the max_tokens continuation +branch (right before looping back for another follow-up call) — both are +points where `runAgenticLoop` is about to issue another provider request in +the SAME `Prompt` call, so both are valid mid-turn steering opportunities. +An operator prompt queued while a long truncated response streams is +delivered on the very next continuation request, not left undelivered for +the whole continuation chain. + +Loop safety is the critical part: `runAgenticLoop`'s local `maxTokensUsed` +is a PER-PROMPT BUDGET, spent by every continuation issued in the loop and +NEVER reset — including across an intervening `StopToolUse` round. (An +earlier version of this counter, `maxTokensStreak`, DID reset on any +non-`max_tokens` stop, which let a model alternate `max_tokens` and +`tool_use` — including denied, unknown, or failing tool calls, none of +which touch `toolExecCount` — indefinitely inside one `Prompt` call, +spending an unbounded number of continuations without ever tripping +`Config.MaxTokensContinuations`.) `maxTokensUsed` is bounded by +`Config.MaxTokensContinuations`: `Config.MaxTokensContinuations+1` +max_tokens stops used within the loop trips the bound. +`maybeAutoContinueMaxTokens` then returns a +`*maxTokensContinuationExhaustedError` naming the bound, wrapped +`provider.MarkPermanent`, instead of arming yet another doomed attempt; +`runAgenticLoop` emits `session.error` and returns it, the same "honest +terminal, never a silent success" shape `emptyTurnError`'s own budget +exhaustion uses (see "Base loop retry" above). + +**A goal-loop retry must never re-run an already-exhausted continuation +chain.** With the default bound of 3, one worker attempt that exhausts the +budget already makes 4 completed, fully billed `max_tokens` calls before +`*maxTokensContinuationExhaustedError` is even returned. +`maybeAutoContinueMaxTokens` wraps every value of that type +`provider.MarkPermanent` at its one construction site, so +`promptTurnWithRetry`'s existing `provider.AsPermanent` fail-fast branch +(`engine/goal.go`) stops after that ONE attempt instead of re-running the +whole exhausted chain up to `goalWorkerRetries` (2) additional times — which +would otherwise multiply 4 calls into 12 for one goal boundary. Like every +other permanent-classified worker error, this PARKS the goal (stays +resumable) rather than clearing it: the condition that produced the +exhaustion might not recur on a later resume. + +`Config.MaxTokensContinuations` follows `PromptRetries`'s own +unset-vs-zero config idiom: the engine field's zero value DISABLES +auto-continue entirely (a bare embedder-built `engine.Config`, and every +test that constructs one directly, keeps the exact pre-fix behavior — the +turn ends immediately on the first `max_tokens` stop). The config/CLI layer +(`config.Config.MaxTokensContinuations *int`, key +`max_tokens_continuations`, resolved via `MaxTokensContinuationsValue`) +supplies the product default of 3; an explicit `0` disables it the same way +`prompt_retries: 0` disables base-loop retry. + +A task child runs its turn through this exact same `runAgenticLoop` — a +child `Session` is a full `NewSession(childCfg)` +(`SessionManager.Spawn`, `engine/session_manager.go`), and `childCfg` comes +from `configSnapshot`, a whole-struct copy of the parent's `engine.Config` +— so `MaxTokensContinuations` (and every other engine.Config field) reaches +a child with no separate wiring. A child that hits `max_tokens` +auto-continues under the identical bound the root does. + ### Per-turn metrics `streamTurn` (`engine/engine.go`) emits one structured `turn_metrics` line per diff --git a/cmd/harness/main.go b/cmd/harness/main.go index f79add86..a082bc45 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -671,6 +671,7 @@ func runCmd(args []string) error { ContextWindowTokens: cfg.ContextWindowTokens, StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), + MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), CompactionThreshold: cfg.CompactionThreshold, CompactionKeepTurns: cfg.CompactionKeepTurns, // Tool-result retention (config keys tool_result_inline_bytes / @@ -1383,6 +1384,7 @@ func serveCmd(args []string) error { ContextWindowTokens: cfg.ContextWindowTokens, StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), + MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), CompactionThreshold: cfg.CompactionThreshold, CompactionKeepTurns: cfg.CompactionKeepTurns, // Tool-result retention, same keys and defaults as runCmd diff --git a/config/config.go b/config/config.go index 08ae22fd..56bb6458 100644 --- a/config/config.go +++ b/config/config.go @@ -115,6 +115,18 @@ type Config struct { // PromptRetriesValue. It is deliberately small and short — see // engine.Config.PromptRetries and streamTurnWithRetry. PromptRetries *int `json:"prompt_retries,omitempty"` + // MaxTokensContinuations sets engine.Config.MaxTokensContinuations: how + // many CONSECUTIVE times the base interactive Prompt loop auto-continues + // a turn that stopped with provider reason "max_tokens" (the provider + // cut the model off mid-emission) instead of settling the turn — see the + // engine field's own doc comment for the box harness-parallel-tools + // incident this closes. A nil value (the field omitted) leaves the + // product default of 3 in place; an explicit 0 disables auto-continue + // entirely, reverting to the pre-fix behavior. A *int distinguishes + // "unset" (3) from "0" (off) across the project-config merge, exactly + // like PromptRetries above. Resolve it with + // MaxTokensContinuationsValue. + MaxTokensContinuations *int `json:"max_tokens_continuations,omitempty"` // StreamIdleTimeoutS sets engine.Config.StreamIdleTimeout (in seconds) // for every session this process creates: how long a streamed response // may go without a delta before the engine's idle-stream watchdog aborts @@ -897,6 +909,9 @@ func merge(base, over *Config) *Config { if over.PromptRetries != nil { out.PromptRetries = over.PromptRetries } + if over.MaxTokensContinuations != nil { + out.MaxTokensContinuations = over.MaxTokensContinuations + } if over.StreamIdleTimeoutS != 0 { out.StreamIdleTimeoutS = over.StreamIdleTimeoutS } @@ -1135,6 +1150,24 @@ func (c *Config) PromptRetriesValue() int { return *c.PromptRetries } +// defaultMaxTokensContinuations is the product default for how many +// consecutive max_tokens stops the base interactive Prompt loop +// auto-continues when `max_tokens_continuations` is unset (see +// MaxTokensContinuationsValue and engine.Config.MaxTokensContinuations). +const defaultMaxTokensContinuations = 3 + +// MaxTokensContinuationsValue reports the base interactive Prompt loop's +// max_tokens auto-continuation budget. The default is +// defaultMaxTokensContinuations (3): only an explicit +// `max_tokens_continuations: 0` disables auto-continue. A nil receiver (no +// config) uses the default too. +func (c *Config) MaxTokensContinuationsValue() int { + if c == nil || c.MaxTokensContinuations == nil { + return defaultMaxTokensContinuations + } + return *c.MaxTokensContinuations +} + // defaultToolResultInlineBytes / defaultToolResultRetainedBytes are the // product defaults for the tool-result retention keys (see the // ToolResultInlineBytes/ToolResultRetainedBytes fields and package engine's diff --git a/config/config_test.go b/config/config_test.go index c0ced303..dda69ab4 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -788,6 +788,74 @@ func TestPromptRetries(t *testing.T) { }) } +// TestMaxTokensContinuations covers the max_tokens_continuations config +// field: a *int so unset means the product default +// (MaxTokensContinuationsValue -> 3), an explicit 0 disables auto-continue, +// and a project value overrides the user layer -- the same shape +// TestPromptRetries covers for its sibling field. +func TestMaxTokensContinuations(t *testing.T) { + t.Run("unset uses default 3", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"model": "anthropic/claude-fable-5"}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.MaxTokensContinuations != nil { + t.Errorf("MaxTokensContinuations = %v, want nil (unset)", c.MaxTokensContinuations) + } + if got := c.MaxTokensContinuationsValue(); got != 3 { + t.Errorf("MaxTokensContinuationsValue = %d, want 3 (default)", got) + } + }) + t.Run("explicit zero disables", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"max_tokens_continuations": 0}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.MaxTokensContinuations == nil || *c.MaxTokensContinuations != 0 { + t.Fatalf("MaxTokensContinuations = %v, want explicit 0", c.MaxTokensContinuations) + } + if got := c.MaxTokensContinuationsValue(); got != 0 { + t.Errorf("MaxTokensContinuationsValue = %d, want 0 (disabled)", got) + } + }) + t.Run("explicit value", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"max_tokens_continuations": 5}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if got := c.MaxTokensContinuationsValue(); got != 5 { + t.Errorf("MaxTokensContinuationsValue = %d, want 5", got) + } + }) + t.Run("nil receiver uses default", func(t *testing.T) { + var c *Config + if got := c.MaxTokensContinuationsValue(); got != 3 { + t.Errorf("nil MaxTokensContinuationsValue = %d, want 3", got) + } + }) + t.Run("project overrides user", func(t *testing.T) { + zero := 0 + base := &Config{MaxTokensContinuations: intPtr(3)} + merged := merge(base, &Config{MaxTokensContinuations: &zero}) + if merged.MaxTokensContinuations == nil || *merged.MaxTokensContinuations != 0 { + t.Errorf("merged = %v, want project override 0", merged.MaxTokensContinuations) + } + }) + t.Run("unset project inherits user", func(t *testing.T) { + base := &Config{MaxTokensContinuations: intPtr(3)} + merged := merge(base, &Config{}) + if merged.MaxTokensContinuations == nil || *merged.MaxTokensContinuations != 3 { + t.Errorf("merged = %v, want inherited 3", merged.MaxTokensContinuations) + } + }) +} + // TestStreamIdleTimeoutS is the red-first test for the stream_idle_timeout_s // config field: 0/omitted means "engine default", negative means "disable // the watchdog", and project non-zero values override the user layer, same diff --git a/engine/engine.go b/engine/engine.go index fdddcd80..c0b4933c 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -639,6 +639,59 @@ type Config struct { // provider.AsPermanent malformed-request shape, or an interruptedTurnError // is never retried regardless of this value — see streamTurnWithRetry. PromptRetries int + // MaxTokensContinuations bounds how many times, TOTAL within one + // Prompt call, runAgenticLoop auto-continues a turn whose stop reason + // was provider.StopMaxTokens — the provider cut the model off + // mid-emission instead of letting it choose to stop. Zero (the zero + // value) DISABLES auto-continue: a max_tokens turn ends exactly as it + // did before this field existed (asst is appended, any orphaned + // ToolCall parts get their usual synthetic is_error result via + // appendUnexecutedToolCallResults, and runAgenticLoop returns), which + // keeps a bare embedder-built engine.Config unchanged. The config/CLI + // wiring sets the product default of 3 (config key + // `max_tokens_continuations`, config.Config.MaxTokensContinuationsValue) + // — the same unset-vs-zero split PromptRetries uses. + // + // Unlike PromptRetries (a transient-error budget for ONE model call), + // this bounds a CHAIN of otherwise-successful calls: each continuation + // re-issues a fresh model call in the SAME Prompt loop, carrying + // forward whatever synthetic tool results the truncated turn produced + // plus a one-shot message.EngineContext nudge (see + // maybeAutoContinueMaxTokens and the maxTokensContinuationNudgeFmt + // constant) telling the model its last turn hit the ceiling and it + // should continue in smaller pieces. The bound exists because a model + // can pathologically re-emit oversized output turn after turn: without + // it, auto-continue would retry forever and never surface a failure. + // + // This is a PER-PROMPT BUDGET, not a consecutive streak: runAgenticLoop's + // local counter (maxTokensUsed) is spent by every continuation issued + // in the loop and NEVER resets partway through, including across an + // intervening StopToolUse round. An earlier version of this counter did + // reset on any non-max_tokens stop, which let a model alternate + // max_tokens and tool_use (including denied, unknown, or failing tool + // calls — none of which need touch toolExecCount) indefinitely inside + // one Prompt call, spending an unbounded number of continuations + // without ever tripping the bound (an adversarial review finding on + // the PR that introduced this field). Exhausting the budget — + // MaxTokensContinuations+1 max_tokens stops used within the loop — + // synthesizes a *maxTokensContinuationExhaustedError naming the bound, + // wrapped provider.MarkPermanent so a goal-loop retry + // (promptTurnWithRetry, goal.go) fails fast on attempt 1 instead of + // re-running the whole exhausted chain, emits session.error, and + // returns, rather than looping forever or settling silently idle. + // + // Fixes the box harness-parallel-tools incident: a provider stopped + // mid-tool-call-emission with stop reason "max_tokens", + // appendUnexecutedToolCallResults synthesized the usual unexecuted-call + // result, and the session then went idle with no further model + // call — a silent work stoppage on an autonomous fleet, requiring a + // human to notice and re-prompt. Applies equally to a task child's own + // turn loop: a child Session runs this exact same runAgenticLoop (see + // SessionManager's configSnapshot, which copies the whole engine.Config + // — including this field — into childCfg), so a child that hits + // max_tokens auto-continues under the identical bound, with no separate + // wiring. + MaxTokensContinuations int // ContextWindowTokens is the model's context window size, in tokens, as // an EXPLICIT operator override. A caller passing a positive value here // pins the window for the session's lifetime, immune to a later model @@ -837,6 +890,24 @@ type Session struct { turn int lastSystem []string + // pendingContinuationNudge is the one-shot max_tokens auto-continuation + // nudge text (see maybeAutoContinueMaxTokens and + // maxTokensContinuationNudgeFmt), set by runAgenticLoop right before it + // re-issues streamTurnWithRetry after deciding to continue a max_tokens + // stop. continuationNudgeSegment reads it on every streamTurn call + // inside that streamTurnWithRetry call (so a transient-error retry + // inside the SAME call keeps re-rendering the identical nudge, mirroring + // checkoutTaskNotificationsSegment's idempotent-reread shape, + // taskdelivery.go); runAgenticLoop clears it once that whole + // streamTurnWithRetry call returns, so it never bleeds into a later, + // unrelated turn. Not mu-guarded: unlike taskNotifications (which + // Spawn can mutate from another session's goroutine), this field is + // only ever touched from within this session's own single in-flight + // Prompt call — the same single-caller property turn/lastSystem rely + // on for their own correctness, just without session_info's need to + // read it concurrently. + pendingContinuationNudge string + // skills is the structured catalog discovered on the first Prompt (name + // absolute SKILL.md path), used by the session_info tool. The advertised // prompt segment lives in skillsSeg below; this is the same catalog, kept @@ -1955,6 +2026,20 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) s.emitStatus("busy") defer s.emitStatus("idle") + // maxTokensUsed counts every max_tokens auto-continuation issued in THIS + // loop — i.e. across one Prompt call — never across a whole Session's + // lifetime. It is a PER-PROMPT BUDGET, not a consecutive streak: unlike + // an earlier version of this counter, it does NOT reset on a StopToolUse + // turn. A model can alternate max_tokens and tool_use stops + // indefinitely, including denied, unknown, or failing tool calls that + // never touch toolExecCount; a counter that reset on every StopToolUse + // let that alternation spend an unbounded number of max_tokens + // continuations inside one Prompt call, defeating + // Config.MaxTokensContinuations as a bound on the loop (an adversarial + // review finding on the PR that introduced this counter). See + // maybeAutoContinueMaxTokens and Config.MaxTokensContinuations. + var maxTokensUsed int + for { // streamTurnWithRetry is a drop-in for streamTurn that smooths a // one-off retryable provider error (a momentary HTTP 5xx/429/529 or a @@ -1965,6 +2050,12 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // here), a context.Canceled, or an exhausted/non-retryable failure — // so everything below is unchanged. See its doc comment. asst, stop, usage, err := s.streamTurnWithRetry(ctx) + // The pending continuation nudge (if any) rode every attempt this + // call just made — see continuationNudgeSegment and + // pendingContinuationNudge's doc comment — and must not bleed into + // whatever streamTurnWithRetry call comes next, success or failure + // alike. + s.pendingContinuationNudge = "" if err != nil { // Whatever this attempt's own streamTurn calls checked out // (checkoutTaskNotificationsSegment, engine.go's streamTurn) @@ -2008,6 +2099,54 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // synthetic results before Prompt ever returns, closing the // hole instead of leaving them orphaned in history. s.appendUnexecutedToolCallResults(asst, stop) + if stop == provider.StopMaxTokens { + // The provider cut this turn off mid-emission rather than + // letting the model choose to stop -- see the box + // harness-parallel-tools incident (Config.MaxTokensContinuations' + // doc comment). Left alone, this early return would settle + // the session idle with nothing further ever prompting it: + // a silent work stoppage on an autonomous fleet. Ask to + // continue instead of returning immediately. + cont, cerr := s.maybeAutoContinueMaxTokens(&maxTokensUsed) + if cerr != nil { + // maxTokensUsed exceeded Config.MaxTokensContinuations: + // a model pathologically re-emitting oversized output + // must not loop forever. Settle with a loud, classified + // error instead -- the same "honest terminal, never a + // silent success" shape emptyTurnError's own budget + // exhaustion uses. Classified provider.MarkPermanent so + // a goal-loop retry (promptTurnWithRetry, goal.go) fails + // fast on attempt 1 instead of re-running the whole + // exhausted continuation chain: see + // maybeAutoContinueMaxTokens' doc comment. + s.emitSessionError(cerr) + return nil, cerr + } + if cont { + // maybeAutoContinueMaxTokens armed + // pendingContinuationNudge for the next + // streamTurnWithRetry call. Drain any operator prompt + // queued while this max_tokens turn was in flight + // FIRST, exactly like the tool-call-boundary drain + // below -- this loop is about to issue another + // provider request in the SAME Prompt call, which is + // the identical mid-turn steering opportunity a + // StopToolUse round already gets; without this, an + // operator message queued during a long truncated + // response could sit undelivered for the entire + // continuation chain (an adversarial review finding on + // the PR that introduced auto-continue). Then loop back + // around instead of returning, so that call issues a + // real follow-up model request in this SAME Prompt + // loop. + s.drainQueuedPromptsIntoHistory() + continue + } + // cont is false with no error only when + // Config.MaxTokensContinuations is 0 (auto-continue + // disabled): fall through to the pre-fix return below, + // unchanged. + } return asst, nil } results := s.runToolCalls(ctx, asst) @@ -2040,35 +2179,48 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // turn that ends with no tool calls never reaches this point at // all (see the two early returns above) — that path is unchanged // and left entirely to the server's tail drain / the goal loop's - // own turn-boundary drain. - // - // DequeueAllPrompts drains the ENTIRE queue, FIFO, in one locked - // operation and journals every prompt.dequeued(injected) record - // BEFORE this method returns — so, exactly like the goal-boundary - // drain in goal.go, a crash between that journal write and this - // append can never double-deliver: the prompt is simply gone from - // the queue on replay. The rendered content is the same labeled - // "OPERATOR MESSAGES" block goal-turn-boundary injection uses - // (operatorMessagesBlock, queue.go), differing only in the - // trailing clause: this call site passes operatorContextTask, not - // operatorContextGoal, since this loop has no goal directive to - // hand back to — even when it happens to be driving a goal loop's - // worker turn (see operatorMessagesBlock's doc comment). - // - // This appends a REAL, durable user message straight into history - // (never an ephemeral segment like the managed-processes status - // block near streamTurn below) — appending only, never touching an - // earlier message, so any provider's prompt-cache prefix stays - // intact exactly per the managed-processes precedent, except this - // one really is delivered mail, not a disposable status line. - if queued := s.DequeueAllPrompts("injected"); len(queued) > 0 { - s.append(message.Message{ - ID: newID("msg"), - Role: message.RoleUser, - Parts: message.Parts{&message.Text{Text: strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n")}}, - CreatedAt: time.Now().UTC(), - }) - } + // own turn-boundary drain. See drainQueuedPromptsIntoHistory's own + // doc comment for the mechanism (shared with the max_tokens + // auto-continue branch above, which needs the identical mid-turn + // steering opportunity for the same reason). + s.drainQueuedPromptsIntoHistory() + } +} + +// drainQueuedPromptsIntoHistory drains the ENTIRE prompt queue, FIFO, in one +// locked DequeueAllPrompts("injected") call and, if it returned anything, +// appends it as a REAL, durable RoleUser message straight into history +// (never an ephemeral segment like the managed-processes status block near +// streamTurn below) — appending only, never touching an earlier message, so +// any provider's prompt-cache prefix stays intact exactly per the +// managed-processes precedent, except this one really is delivered mail, +// not a disposable status line. DequeueAllPrompts journals every +// prompt.dequeued(injected) record BEFORE this method returns, so a crash +// between that journal write and this append can never double-deliver: the +// prompt is simply gone from the queue on replay. The rendered content is +// the same labeled "OPERATOR MESSAGES" block goal-turn-boundary injection +// uses (operatorMessagesBlock, queue.go); this call site always passes +// operatorContextTask, never operatorContextGoal, since runAgenticLoop has +// no goal directive to hand back to — even when it happens to be driving a +// goal loop's worker turn (see operatorMessagesBlock's doc comment). +// +// Two call sites in runAgenticLoop share this exact drain: the tool-call +// boundary (after a StopToolUse round actually runs a tool) and the +// max_tokens auto-continuation branch (right before looping back for +// another follow-up call) — both are points where this SAME Prompt call is +// about to issue another provider request, so both are valid mid-turn +// steering opportunities. Before the max_tokens call site existed, an +// operator prompt queued while a long truncated response was streaming +// went undelivered for the entire continuation chain — this closes that gap +// by reusing the identical mechanism rather than adding a second one. +func (s *Session) drainQueuedPromptsIntoHistory() { + if queued := s.DequeueAllPrompts("injected"); len(queued) > 0 { + s.append(message.Message{ + ID: newID("msg"), + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n")}}, + CreatedAt: time.Now().UTC(), + }) } } @@ -2187,6 +2339,17 @@ func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message if seg := s.checkoutTaskNotificationsSegment(); seg != "" { messages = withAmbientStatus(messages, seg) } + // The max_tokens auto-continuation nudge (see continuationNudgeSegment, + // maybeAutoContinueMaxTokens): present only on the follow-up call(s) + // runAgenticLoop issues right after a max_tokens stop it decided to + // continue, absent on every ordinary turn. Deliberately NOT + // withAmbientStatus, unlike every segment above: see + // appendContinuationNudgeMessage's doc comment for why this one needs a + // genuine new trailing user message instead of gluing onto an existing + // one. + if seg := s.continuationNudgeSegment(); seg != "" { + messages = appendContinuationNudgeMessage(messages, seg) + } req := &provider.Request{ Model: params.Model, System: system, @@ -2535,6 +2698,135 @@ func (s *Session) appendUnexecutedToolCallResults(asst *message.Message, stop pr s.append(syntheticUnexecutedToolResults(asst, fmt.Sprintf(unexecutedToolCallStopReasonTextFmt, stop))) } +// maxTokensContinuationNudgeFmt is the max_tokens auto-continuation nudge +// text for the follow-up request runAgenticLoop issues right after a +// max_tokens stop it decided to auto-continue (see +// maybeAutoContinueMaxTokens). It rides a genuine new user-role message as a +// *message.EngineContext part — see appendContinuationNudgeMessage's doc +// comment for why a new message, not an existing one — the same trusted, +// unforgeable PART TYPE every other ambient status segment uses (see +// message.EngineContext), so neither a user- nor a tool-authored string can +// ever spoof it. %d/%d are the 1-indexed continuation attempt and +// Config.MaxTokensContinuations' bound, so the model (and anyone reading +// the transcript) can see how much budget is left. +const maxTokensContinuationNudgeFmt = "[continuation: your previous turn was cut off because it reached the max_tokens output limit (auto-continue %d of %d). Continue exactly where you left off. Produce your output in smaller pieces so this does not happen again.]" + +// continuationNudgeSegment returns the pending max_tokens auto-continuation +// nudge, if any — the underlying state (pendingContinuationNudge) is +// written by maybeAutoContinueMaxTokens rather than recomputed from live +// external state, the one deliberate exception among the ambient segments +// streamTurn assembles (see that function). A deliberate READ, not a +// checkout: streamTurnWithRetry can call streamTurn more than once for one +// logical attempt (a transient-error retry), and every one of those +// attempts must see the identical nudge — runAgenticLoop is what clears +// pendingContinuationNudge, once, after the WHOLE streamTurnWithRetry call +// returns. See pendingContinuationNudge's own doc comment (Session struct) +// for the full checkout/clear split, mirroring +// checkoutTaskNotificationsSegment's reasoning in taskdelivery.go. +func (s *Session) continuationNudgeSegment() string { + return s.pendingContinuationNudge +} + +// appendContinuationNudgeMessage appends a genuine NEW message.RoleUser +// message carrying seg as a *message.EngineContext part to the END of +// messages, and returns the result. It is streamTurn's own throwaway, +// per-request copy being extended (messages == s.History(), never +// s.history itself), so — exactly like every other ambient segment — the +// nudge never touches the durable log: a session reload, or any later, +// unrelated request built from the same durable history, never sees it. +// +// This does NOT reuse withAmbientStatus, unlike every other ambient +// segment. withAmbientStatus glues its segment onto the NEWEST EXISTING +// RoleUser message, scanning backward from the end of messages — for the +// process/MCP/goal/identity/task-notification segments that message really +// is the newest thing in the conversation. It is not here: by the time a +// max_tokens continuation request is built, the newest messages are the +// truncated assistant turn and (if it carried a tool call) +// appendUnexecutedToolCallResults' synthetic tool-role result, so +// withAmbientStatus's scan lands on an EARLIER user message, still ending +// the canonical request with RoleAssistant or RoleTool. Anthropic +// serializes a request shaped that way as assistant PREFILL: some models +// reject it outright with a permanent 400, and even a model that accepts it +// sees a "continue" instruction that precedes, chronologically, the very +// output it refers to (an adversarial review finding on the PR that +// introduced auto-continue). Appending a whole new trailing user message +// instead makes the request end exactly where a real continuation turn +// should — genuinely after the truncated output — regardless of what +// stands before it. +func appendContinuationNudgeMessage(messages []message.Message, seg string) []message.Message { + return append(messages, message.Message{ + ID: newID("msg"), + Role: message.RoleUser, + Parts: message.Parts{&message.EngineContext{Text: seg}}, + CreatedAt: time.Now().UTC(), + }) +} + +// maxTokensContinuationExhaustedError is runAgenticLoop's synthetic, +// terminal error for a session that used Config.MaxTokensContinuations +// max_tokens auto-continuations within one Prompt call (see maxTokensUsed +// in runAgenticLoop — a per-Prompt budget, not a consecutive streak; see +// its own doc comment for why). A model pathologically re-emitting +// oversized output must not loop forever; this settles the turn with a +// classified, session.error-visible record instead — never a silent, +// parked-idle success. Mirrors emptyTurnError's exhaustion shape: the +// discarded attempts already appended real (truncated) content and billed +// real tokens, but the loop itself must stop and say so plainly, naming the +// bound that tripped. +// +// provider.MarkPermanent wraps every value of this type at its one +// construction site (maybeAutoContinueMaxTokens) — never left for a caller +// to classify — so goal.go's promptTurnWithRetry fails fast on attempt 1 +// via its existing provider.AsPermanent branch instead of re-running the +// goalWorkerRetries budget's worth of already-exhausted continuation +// chains: with the default bound of 3, one worker attempt already makes 4 +// completed, fully billed max_tokens calls before this error is even +// returned, and goalWorkerRetries (2 additional attempts) would otherwise +// multiply that to 12 for one goal boundary (an adversarial review finding +// on the PR that introduced auto-continue). Unlike a context-overflow +// error, a permanent classification here does not clear the goal — the +// condition that produced 3+1 consecutive max_tokens stops might not +// recur on a later resume — it only stops THIS attempt from being retried; +// see promptTurnWithRetry's provider.AsPermanent branch for the shared +// parking behavior every other permanent-classified worker error already +// gets. +type maxTokensContinuationExhaustedError struct { + bound int +} + +func (e *maxTokensContinuationExhaustedError) Error() string { + return fmt.Sprintf("max_tokens auto-continue exhausted: %d consecutive turns stopped with reason \"max_tokens\" (bound %d); the model may be pathologically re-emitting oversized output", e.bound+1, e.bound) +} + +// maybeAutoContinueMaxTokens decides whether runAgenticLoop should re-issue +// a follow-up model call after a turn stopped with provider.StopMaxTokens, +// rather than settling the turn immediately — see Config.MaxTokensContinuations' +// doc comment for the box harness-parallel-tools incident this exists to +// close. +// +// *used is runAgenticLoop's maxTokensUsed, incremented here on every call: +// a session with Config.MaxTokensContinuations == 0 returns (false, nil) +// without touching it at all, preserving the exact pre-fix behavior for a +// bare embedder engine.Config (auto-continue disabled entirely). A positive +// bound increments the count and compares it against the bound: at or +// under, it arms pendingContinuationNudge (naming this attempt number and +// the bound) and returns (true, nil) so the caller loops back around; over +// the bound, it returns (false, a provider.MarkPermanent-wrapped +// *maxTokensContinuationExhaustedError) so the caller settles the turn with +// a loud, classified failure instead of arming yet another doomed attempt. +func (s *Session) maybeAutoContinueMaxTokens(used *int) (bool, error) { + bound := s.cfg.MaxTokensContinuations + if bound <= 0 { + return false, nil + } + *used++ + if *used > bound { + return false, provider.MarkPermanent(&maxTokensContinuationExhaustedError{bound: bound}) + } + s.pendingContinuationNudge = fmt.Sprintf(maxTokensContinuationNudgeFmt, *used, bound) + return true, nil +} + // turnHasActionableContent reports whether asst carries content a caller // can act on: a *message.Text part with non-empty Text, or a // *message.ToolCall part. A message holding only a *message.Reasoning part diff --git a/engine/max_tokens_continue_test.go b/engine/max_tokens_continue_test.go new file mode 100644 index 00000000..c0d22ad8 --- /dev/null +++ b/engine/max_tokens_continue_test.go @@ -0,0 +1,711 @@ +package engine + +import ( + "context" + "errors" + "strings" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" + "github.com/majorcontext/harness/provider" +) + +// engineContextTexts returns the text of every *message.EngineContext part +// on the newest RoleUser message in req.Messages, in order. Several ambient +// segments (process/MCP/identity/task-notification/continuation-nudge) can +// stack onto that one message (see withAmbientStatus), so a test asserting +// on one of them must scan every part rather than assume it is the last — +// unlike lastUserText (process_ambient_test.go), which is only safe when +// the caller controls exactly which single segment is present. +func engineContextTexts(req *provider.Request) []string { + for i := len(req.Messages) - 1; i >= 0; i-- { + if req.Messages[i].Role != message.RoleUser { + continue + } + var texts []string + for _, p := range req.Messages[i].Parts { + if ec, ok := p.(*message.EngineContext); ok { + texts = append(texts, ec.Text) + } + } + return texts + } + return nil +} + +func containsSubstring(texts []string, substr string) bool { + for _, t := range texts { + if strings.Contains(t, substr) { + return true + } + } + return false +} + +// TestMaxTokensWithToolCallAutoContinues reproduces the box +// harness-parallel-tools incident: the provider stops mid-tool-call +// emission with stop reason "max_tokens". Before this fix, +// appendUnexecutedToolCallResults synthesized the usual unexecuted-call +// result and runAgenticLoop returned -- the session then sat idle with no +// further model call, a silent work stoppage on an autonomous fleet. With +// Config.MaxTokensContinuations enabled, the loop must instead issue a +// follow-up model call carrying the synthetic unexecuted-tool-call result +// plus a continuation nudge, in the SAME Prompt call. +func TestMaxTokensWithToolCallAutoContinues(t *testing.T) { + tc := toolCall("tc1", "bash", `{"command":"echo hi"}`) + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, tc), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success (the loop should auto-continue)", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + if len(prov.requests) != 2 { + t.Fatalf("provider requests = %d, want 2 (the auto-continue must issue a real follow-up model call)", len(prov.requests)) + } + + h := s.History() + if len(h) != 4 { + t.Fatalf("history len = %d, want 4 (user, assistant(tool_call), synthetic tool result, assistant(done)): %+v", len(h), h) + } + if h[2].Role != message.RoleTool { + t.Fatalf("h[2].Role = %s, want tool (the synthetic unexecuted-call result)", h[2].Role) + } + tr, ok := h[2].Parts[0].(*message.ToolResult) + if !ok { + t.Fatalf("h[2].Parts[0] = %T, want *message.ToolResult", h[2].Parts[0]) + } + if tr.CallID != "tc1" || !tr.IsError { + t.Errorf("synthetic result = %+v, want CallID=tc1 IsError=true", tr) + } + if !strings.Contains(tr.Content.Text(), `"max_tokens"`) { + t.Errorf("synthetic result text = %q, want it to name the max_tokens stop reason", tr.Content.Text()) + } + if got := s.toolExecutions(); got != 0 { + t.Errorf("toolExecutions() = %d, want 0 (a truncated call must never actually run)", got) + } + + // The follow-up request must carry a continuation nudge naming the + // attempt and the bound, so the model knows why it is being asked to + // continue and how much budget remains. + texts := engineContextTexts(prov.requests[1]) + if !containsSubstring(texts, "[continuation:") { + t.Errorf("second request ambient segments = %v, want a [continuation: ...] nudge", texts) + } + if !containsSubstring(texts, "max_tokens") { + t.Errorf("second request ambient segments = %v, want the nudge to name max_tokens", texts) + } + if !containsSubstring(texts, "1 of 3") { + t.Errorf("second request ambient segments = %v, want the nudge to report attempt 1 of 3", texts) + } + + // The nudge must never leak into a THIRD request -- there is none here, + // but the first request (the original turn) must not have carried it + // either, since the nudge only exists once a max_tokens continuation has + // actually been decided. + if containsSubstring(engineContextTexts(prov.requests[0]), "[continuation:") { + t.Errorf("first request carried a continuation nudge before any max_tokens stop occurred") + } +} + +// TestMaxTokensPureTextContinuesOnce covers a max_tokens stop with NO tool +// calls at all -- pure text truncation. Unlike Claude Code (which lets the +// turn end and relies on the user re-prompting), an autonomous harness +// session must not require a human, so this must also auto-continue. +func TestMaxTokensPureTextContinuesOnce(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "partial output "}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "rest of output"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + if final.Parts.Text() != "rest of output" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "rest of output") + } + if len(prov.requests) != 2 { + t.Fatalf("provider requests = %d, want 2 (a pure-text max_tokens stop must also auto-continue)", len(prov.requests)) + } + + h := s.History() + if len(h) != 3 { + t.Fatalf("history len = %d, want 3 (user, assistant(partial), assistant(rest)) -- no synthetic tool message since no ToolCall was ever emitted: %+v", len(h), h) + } + if h[1].Role != message.RoleAssistant || h[2].Role != message.RoleAssistant { + t.Fatalf("h[1]/h[2] roles = %s/%s, want assistant/assistant", h[1].Role, h[2].Role) + } + + texts := engineContextTexts(prov.requests[1]) + if !containsSubstring(texts, "1 of 3") { + t.Errorf("second request ambient segments = %v, want the nudge to report attempt 1 of 3", texts) + } +} + +// TestMaxTokensBoundTripsAndRecordsHonestError proves the loop-safety bound: +// a model pathologically re-emitting oversized output must not loop +// forever. With MaxTokensContinuations=3, the 4th consecutive max_tokens +// stop must trip the bound, settle with a classified, named error instead +// of arming a further doomed attempt, and emit exactly one session.error. +func TestMaxTokensBoundTripsAndRecordsHonestError(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "chunk1"}), + asstTurn(provider.StopMaxTokens, &message.Text{Text: "chunk2"}), + asstTurn(provider.StopMaxTokens, &message.Text{Text: "chunk3"}), + asstTurn(provider.StopMaxTokens, &message.Text{Text: "chunk4"}), + }} + hooks := &fakeHooks{} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + Hooks: hooks, + }) + + _, err := s.Prompt(context.Background(), "go") + if err == nil { + t.Fatal("Prompt = nil error, want the bound to trip with a named error") + } + var exhausted *maxTokensContinuationExhaustedError + if !errors.As(err, &exhausted) { + t.Fatalf("err = %T (%v), want *maxTokensContinuationExhaustedError", err, err) + } + if exhausted.bound != 3 { + t.Errorf("exhausted.bound = %d, want 3", exhausted.bound) + } + if !strings.Contains(err.Error(), "3") || !strings.Contains(err.Error(), "max_tokens") { + t.Errorf("err.Error() = %q, want it to name the bound (3) and max_tokens", err.Error()) + } + + if len(prov.requests) != 4 { + t.Fatalf("provider requests = %d, want 4 (initial attempt plus 3 continuations, no 5th doomed attempt)", len(prov.requests)) + } + + h := s.History() + if len(h) != 5 { + t.Fatalf("history len = %d, want 5 (user + 4 assistant turns, every one of them kept): %+v", len(h), h) + } + + var errEvents []plugin.Event + for _, ev := range hooks.events { + if ev.Type == plugin.EventSessionError { + errEvents = append(errEvents, ev) + } + } + if len(errEvents) != 1 { + t.Fatalf("session.error events = %d, want exactly 1: %+v", len(errEvents), hooks.events) + } +} + +// TestMaxTokensBudgetSpansToolUseRounds proves MaxTokensContinuations is a +// single PER-PROMPT budget that persists across an intervening tool_use +// round, not a counter that resets on one: with a bound of 2, two isolated +// max_tokens stops separated by a normal tool_use turn each consume one +// unit of the SAME budget and both still get their continuation (2 used, +// 2 allowed). +func TestMaxTokensBudgetSpansToolUseRounds(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "a"}), + asstTurn(provider.StopToolUse, toolCall("tc1", "bash", `{"command":"echo hi"}`)), + asstTurn(provider.StopMaxTokens, &message.Text{Text: "b"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 2, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success (both max_tokens stops fit the shared budget of 2)", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + if len(prov.requests) != 4 { + t.Fatalf("provider requests = %d, want 4 (both isolated max_tokens stops got their continuation)", len(prov.requests)) + } +} + +// TestMaxTokensBudgetDoesNotResetOnToolUse is the red-first guard for +// adversarial review finding 3: an earlier version of this counter +// (maxTokensStreak) reset to zero on ANY StopToolUse, including a denied, +// unknown, or failing tool call that never touches toolExecCount -- which +// let a model alternate max_tokens and tool_use indefinitely inside one +// Prompt call, spending an unbounded number of continuations without ever +// tripping Config.MaxTokensContinuations. With a bound of 1, this proves +// the SECOND max_tokens stop -- separated from the first by a genuine +// tool_use round -- does NOT get a fresh continuation: the budget was +// already spent by the first one and must stay spent for the rest of this +// Prompt call. +// +// Red-verify: against the pre-fix runAgenticLoop (maxTokensStreak reset to +// 0 on the StopToolUse branch), this exact sequence succeeds with 4 +// requests and no error -- see TestMaxTokensBudgetSpansToolUseRounds above, +// which is that old behavior preserved at a bound wide enough to still +// legitimately allow both stops. +func TestMaxTokensBudgetDoesNotResetOnToolUse(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "a"}), + asstTurn(provider.StopToolUse, toolCall("tc1", "bash", `{"command":"echo hi"}`)), + asstTurn(provider.StopMaxTokens, &message.Text{Text: "b"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + hooks := &fakeHooks{} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 1, + Hooks: hooks, + }) + + _, err := s.Prompt(context.Background(), "go") + if err == nil { + t.Fatal("Prompt = nil error, want the budget to stay spent across the intervening tool_use round") + } + var exhausted *maxTokensContinuationExhaustedError + if !errors.As(err, &exhausted) { + t.Fatalf("err = %T (%v), want *maxTokensContinuationExhaustedError", err, err) + } + if exhausted.bound != 1 { + t.Errorf("exhausted.bound = %d, want 1", exhausted.bound) + } + if !provider.AsPermanent(err) { + t.Errorf("err = %v, want provider.AsPermanent (finding 5: fail-fast for goal retry)", err) + } + + // Requests in order: initial (max_tokens "a"), continuation (consumes + // the bound-of-1 budget, tool_use "tc1"), a THIRD request after the + // tool ran that hits max_tokens again ("b") and finds the budget + // already spent -- no 4th request is ever issued. + if len(prov.requests) != 3 { + t.Fatalf("provider requests = %d, want 3 (no continuation granted for the second max_tokens stop)", len(prov.requests)) + } +} + +// TestMaxTokensContinuationDisabledPreservesOldBehavior proves +// Config.MaxTokensContinuations' zero value keeps the exact pre-fix +// behavior: a max_tokens stop with an orphaned tool call gets its +// synthetic unexecuted-call result and the turn ends immediately, with no +// further model call -- unchanged for a bare embedder engine.Config. +func TestMaxTokensContinuationDisabledPreservesOldBehavior(t *testing.T) { + tc := toolCall("tc1", "bash", `{"command":"echo hi"}`) + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, tc), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + // MaxTokensContinuations left at its zero value: disabled. + }) + + asst, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + if asst == nil { + t.Fatal("Prompt returned nil message") + } + if len(prov.requests) != 1 { + t.Fatalf("provider requests = %d, want 1 (auto-continue disabled: no follow-up call)", len(prov.requests)) + } + + h := s.History() + if len(h) != 3 || h[2].Role != message.RoleTool { + t.Fatalf("history = %+v, want [user, assistant, tool(synthetic)]", h) + } + if got := s.toolExecutions(); got != 0 { + t.Errorf("toolExecutions() = %d, want 0", got) + } +} + +// TestTaskChildAutoContinuesMaxTokens confirms the fix applies equally to a +// task child's own turn loop, not only a root session's -- the incident's +// own emphasis. A child Session runs through the identical runAgenticLoop +// (SessionManager.configSnapshot copies the whole parent engine.Config, +// including MaxTokensContinuations, into childCfg -- see Spawn), so this +// asserts against the CHILD's own provider request count and history, +// never the root's. +func TestTaskChildAutoContinuesMaxTokens(t *testing.T) { + rootProv := &scriptedProvider{name: "root"} + childProv := &scriptedProvider{name: "child", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "partial"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "child done"}), + }} + cfg := managedConfig("root", rootProv, childProv) + cfg.MaxTokensContinuations = 3 + + mgr := NewSessionManager(context.Background(), 0, 0) + root := mgr.NewRoot(cfg) + + childID, err := mgr.Spawn(SpawnOptions{ + ParentID: root.ID, + Prompt: "go", + Model: modelFor("child"), + AgentType: AgentGeneralPurpose, + }) + if err != nil { + t.Fatalf("Spawn: %v", err) + } + waitForStatus(t, mgr, childID, StatusDone, time.Second) + + if len(childProv.requests) != 2 { + t.Fatalf("child provider requests = %d, want 2 (the child's own turn loop must auto-continue too)", len(childProv.requests)) + } + child, ok := mgr.Session(childID) + if !ok { + t.Fatal("child session not found") + } + h := child.History() + if len(h) != 3 { + t.Fatalf("child history len = %d, want 3 (user, assistant(partial), assistant(child done)): %+v", len(h), h) + } + if h[2].Parts.Text() != "child done" { + t.Errorf("child final text = %q, want %q", h[2].Parts.Text(), "child done") + } +} + +// sequencedProvider serves a fixed sequence of outcomes, one per Stream call +// in order -- either a completed turn (events) or an error. It generalizes +// scriptedProvider (every call succeeds) and flakyProvider (a fixed prefix +// of failures, then one fixed turn repeated) for a test that needs an +// ARBITRARY mix of failures and successes across a longer call sequence. +type sequencedProvider struct { + name string + outcomes []sequencedOutcome + call int + requests []*provider.Request +} + +type sequencedOutcome struct { + err error + events []provider.Event +} + +func (p *sequencedProvider) Name() string { return p.name } + +func (p *sequencedProvider) Stream(_ context.Context, req *provider.Request) (provider.Stream, error) { + p.requests = append(p.requests, req) + o := p.outcomes[p.call] + p.call++ + if o.err != nil { + return nil, o.err + } + return &scriptedStream{events: o.events}, nil +} + +// TestMaxTokensContinuationAppendsGenuineNewUserMessage is the red-first +// guard for adversarial review finding 2: the continuation nudge must +// arrive as a genuine NEW user-role message appended AFTER the truncated +// assistant turn (and its synthetic tool result, if any) -- ending the +// canonical request with RoleUser -- never glued onto an earlier existing +// user message via withAmbientStatus. The old shape left the request +// ending in RoleAssistant/RoleTool: Anthropic serializes that as assistant +// PREFILL, which some models reject outright with a permanent 400, and even +// an accepting model saw a "continue" instruction that chronologically +// precedes the very output it refers to. +// +// Red-verify: against the pre-fix continuationNudgeSegment call site +// (withAmbientStatus(messages, seg), scanning backward for the newest +// EXISTING RoleUser message), the continuation request's trailing message +// is the synthetic tool-role result, not a new RoleUser message -- the +// first assertion below fails. +func TestMaxTokensContinuationAppendsGenuineNewUserMessage(t *testing.T) { + tc := toolCall("tc1", "bash", `{"command":"echo hi"}`) + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, tc), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + }) + + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + + req := prov.requests[1] + last := req.Messages[len(req.Messages)-1] + if last.Role != message.RoleUser { + t.Fatalf("continuation request's trailing message role = %s, want user (a genuine new turn, not assistant/tool prefill)", last.Role) + } + if last.ID == req.Messages[0].ID { + t.Fatalf("nudge landed on the original user message %q, want a distinct new trailing message", last.ID) + } + var nudgeText string + for _, p := range last.Parts { + if ec, ok := p.(*message.EngineContext); ok && strings.Contains(ec.Text, "[continuation:") { + nudgeText = ec.Text + } + } + if nudgeText == "" { + t.Fatalf("trailing user message parts = %+v, want the continuation nudge as its own EngineContext part", last.Parts) + } +} + +// TestMaxTokensContinuationDrainsQueuedPrompt is the red-first guard for +// adversarial review finding 4: an operator prompt queued while a +// max_tokens turn is in flight must be delivered on the very next +// continuation request -- the same mid-turn steering granularity the +// tool-call-boundary drain already gives a StopToolUse round -- rather than +// waiting undelivered for the whole continuation chain (or the whole Prompt +// call) to finish. +// +// Red-verify: against the pre-fix continuation branch (which loops back to +// streamTurnWithRetry with no drain call at all), the queued prompt is +// still sitting in the queue when the continuation request is built, so the +// "OPERATOR MESSAGES" assertion below fails and QueuedPrompts is non-empty +// after Prompt returns. +func TestMaxTokensContinuationDrainsQueuedPrompt(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "partial"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + }) + if _, err := s.EnqueuePrompt("steer now"); err != nil { + t.Fatalf("EnqueuePrompt = %v", err) + } + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + if pending := s.QueuedPrompts(); len(pending) != 0 { + t.Fatalf("QueuedPrompts after continuation = %+v, want drained", pending) + } + if len(prov.requests) != 2 { + t.Fatalf("provider requests = %d, want 2", len(prov.requests)) + } + + req := prov.requests[1] + if len(req.Messages) < 2 { + t.Fatalf("continuation request has %d messages, want at least 2 (the drained operator message plus the nudge)", len(req.Messages)) + } + // The durable operator drain (a real appended message) must precede the + // ephemeral nudge (appended after it, never persisted) -- the operator + // block is therefore the SECOND-TO-LAST message, the nudge the last. + operator := req.Messages[len(req.Messages)-2] + if operator.Role != message.RoleUser { + t.Fatalf("operator message role = %s, want user", operator.Role) + } + text := operator.Parts.Text() + if !strings.Contains(text, "OPERATOR MESSAGES") || !strings.Contains(text, "steer now") { + t.Fatalf("operator message = %q, want the labeled operator block with the queued text", text) + } + if !strings.Contains(text, "continue the task") { + t.Errorf("operator message = %q, want plain-turn wording (continue the task)", text) + } +} + +// TestMaxTokensNudgeAbsentOnThirdRequest extends +// TestMaxTokensWithToolCallAutoContinues (which only ever issues two +// requests, so it cannot show the nudge disappearing again) to a THIRD +// request within the same Prompt call -- a genuine tool_use round that +// follows the continuation. Closes adversarial review finding 6's first +// test gap: the nudge must be present on request 2 (the continuation) and +// absent again on request 3, proving pendingContinuationNudge is actually +// cleared once its one streamTurnWithRetry call returns, not merely never +// re-armed. +func TestMaxTokensNudgeAbsentOnThirdRequest(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, &message.Text{Text: "partial"}), + asstTurn(provider.StopToolUse, toolCall("tc1", "bash", `{"command":"echo hi"}`)), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + if len(prov.requests) != 3 { + t.Fatalf("provider requests = %d, want 3 (initial max_tokens, continuation, tool_use follow-up)", len(prov.requests)) + } + if !containsSubstring(engineContextTexts(prov.requests[1]), "[continuation:") { + t.Errorf("request 2 (the continuation) ambient segments = %v, want the nudge", engineContextTexts(prov.requests[1])) + } + if containsSubstring(engineContextTexts(prov.requests[2]), "[continuation:") { + t.Errorf("request 3 ambient segments = %v, want no continuation nudge (it must clear after request 2)", engineContextTexts(prov.requests[2])) + } +} + +// TestMaxTokensNudgeNotPersistedAcrossReload closes adversarial review +// finding 6's second test gap: LoadSession -- the production resume path, +// not a hand-built replay -- must never see the continuation nudge in the +// durable log. The nudge is appended only to streamTurn's own throwaway +// per-request message copy (see appendContinuationNudgeMessage), never to +// s.history, so a reloaded session's history must carry no +// *message.EngineContext part naming a continuation anywhere. +func TestMaxTokensNudgeNotPersistedAcrossReload(t *testing.T) { + dir := t.TempDir() + tc := toolCall("tc1", "bash", `{"command":"echo hi"}`) + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopMaxTokens, tc), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + cfg := persistCfg(dir, prov) + cfg.MaxTokensContinuations = 3 + s := NewSession(cfg) + + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt = %v, want success", err) + } + + loaded, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatalf("LoadSession = %v", err) + } + for _, m := range loaded.History() { + for _, p := range m.Parts { + if ec, ok := p.(*message.EngineContext); ok && strings.Contains(ec.Text, "[continuation:") { + t.Fatalf("reloaded history carries a persisted continuation nudge on message %s: %q", m.ID, ec.Text) + } + } + } + if loaded.pendingContinuationNudge != "" { + t.Errorf("loaded.pendingContinuationNudge = %q, want empty after reload", loaded.pendingContinuationNudge) + } +} + +// TestMaxTokensNudgeSurvivesTransientRetryButNotFutureTurn closes +// adversarial review finding 6's third test gap. It forces a genuine +// transient-error retry INSIDE the continuation's own streamTurnWithRetry +// call (attempt 1 fails with a classified retryable server_error, attempt 2 +// succeeds), proving the nudge rides both attempts of that one call, then +// issues a SECOND, wholly unrelated Prompt call on the same session and +// proves the nudge does not resurrect there. +func TestMaxTokensNudgeSurvivesTransientRetryButNotFutureTurn(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + prov := &sequencedProvider{name: "test", outcomes: []sequencedOutcome{ + {events: asstTurn(provider.StopMaxTokens, &message.Text{Text: "partial"})}, + {err: retryableServerErr()}, + {events: asstTurn(provider.StopEndTurn, &message.Text{Text: "done"})}, + {events: asstTurn(provider.StopEndTurn, &message.Text{Text: "second done"})}, + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + MaxTokensContinuations: 3, + PromptRetries: 1, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success (the transient retry must be masked)", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + if len(prov.requests) != 3 { + t.Fatalf("provider requests = %d, want 3 (initial max_tokens, continuation attempt 1 (fails), continuation attempt 2 (succeeds))", len(prov.requests)) + } + for _, reqIdx := range []int{1, 2} { + texts := engineContextTexts(prov.requests[reqIdx]) + if !containsSubstring(texts, "[continuation:") { + t.Errorf("request %d ambient segments = %v, want the nudge (it must ride every attempt of the same streamTurnWithRetry call)", reqIdx, texts) + } + } + + final2, err := s.Prompt(context.Background(), "go again") + if err != nil { + t.Fatalf("second Prompt = %v, want success", err) + } + if final2.Parts.Text() != "second done" { + t.Errorf("second final = %q, want %q", final2.Parts.Text(), "second done") + } + if len(prov.requests) != 4 { + t.Fatalf("provider requests = %d, want 4 after the second Prompt call", len(prov.requests)) + } + if containsSubstring(engineContextTexts(prov.requests[3]), "[continuation:") { + t.Errorf("second Prompt's request ambient segments = %v, want no continuation nudge (must not leak into a later, unrelated turn)", engineContextTexts(prov.requests[3])) + } + }) +} + +// TestPursueGoalMaxTokensExhaustionFailsFastForGoalRetry is the red-first +// guard for adversarial review finding 5, exercised at the goal-loop layer +// (not just engine.AsPermanent in isolation): with the default-shaped bound +// of 3, one worker attempt that exhausts Config.MaxTokensContinuations +// already makes bound+1 = 4 completed, fully billed max_tokens calls. +// Before maxTokensContinuationExhaustedError was classified +// provider.MarkPermanent, promptTurnWithRetry's deterministic +// goalWorkerRetries budget (2 additional attempts) retried the whole +// exhausted chain from scratch, multiplying 4 calls into +// (goalWorkerRetries+1)*4 = 12 for one goal boundary. This proves exactly 4 +// worker calls are made, not 12, and that the goal PARKS (stays resumable) +// rather than clears -- the same shape every other permanent-classified +// worker error already gets (see promptTurnWithRetry's provider.AsPermanent +// branch and TestPursueGoalPermanentWorkerErrorParksAfterOneAttempt in +// goal_permanent_error_test.go, whose shape this mirrors). +func TestPursueGoalMaxTokensExhaustionFailsFastForGoalRetry(t *testing.T) { + dir := t.TempDir() + // 12 consecutive max_tokens turns: enough to service every attempt the + // pre-fix (goalWorkerRetries+1)*4 = 12-call multiplication could burn + // through, so a regression that reintroduces it runs to completion + // (and this test's own worker-call assertion catches it) instead of + // the provider ever running dry mid-test. + var turns [][]provider.Event + for i := 0; i < 12; i++ { + turns = append(turns, asstTurn(provider.StopMaxTokens, &message.Text{Text: "chunk"})) + } + prov := &goalProvider{ + worker: turns, + eval: [][]provider.Event{evalTurn("MET: done")}, + } + s := goalSession(t, prov, dir) + s.cfg.MaxTokensContinuations = 3 + + _, err := s.PursueGoal(context.Background(), "cond", GoalOptions{Evaluator: evalModel}) + if err == nil { + t.Fatal("PursueGoal = nil error, want the exhausted continuation chain to park") + } + if !provider.AsPermanent(err) { + t.Errorf("err = %v, want provider.AsPermanent", err) + } + if !IsGoalWorkerParked(err) { + t.Fatalf("err = %v, want IsGoalWorkerParked", err) + } + if prov.workerCall != 4 { + t.Fatalf("worker provider calls = %d, want 4 (bound+1 = 3+1, exactly ONE worker attempt -- no goalWorkerRetries multiplication)", prov.workerCall) + } + + if cond, ok := s.ActiveGoal(); !ok || cond != "cond" { + t.Fatalf("ActiveGoal = %q, %v; want still active after a permanent-error park", cond, ok) + } +} diff --git a/engine/max_tokens_wire_test.go b/engine/max_tokens_wire_test.go new file mode 100644 index 00000000..b7aea4d2 --- /dev/null +++ b/engine/max_tokens_wire_test.go @@ -0,0 +1,167 @@ +package engine + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" + "github.com/majorcontext/harness/provider/anthropic" +) + +// TestMaxTokensPartialJSONMarshalsThroughRealTranscoder pins the rebuttal of +// adversarial review finding 1 on the PR that introduced max_tokens +// auto-continue. The finding claimed a StopMaxTokens turn's trailing +// ToolCall -- carrying raw, truncated partial_json Arguments like `{"comm`, +// the shape Anthropic's own protocol leaves behind when max_tokens lands +// before a tool_use block's content_block_stop -- gets replayed into the +// continuation request and fails json.Marshal before it ever reaches the +// provider. +// +// That does not hold: message.Message.Normalize (Session.append's +// appendWithUsage, run on every append) already coerces the identical +// invalid-Arguments shape to nil in place -- the deliberate, incident-tested +// fix for a real production defect (see TestPersistTruncatedToolCallArguments, +// engine/tool_call_poison_test.go, NEP-5272-adjacent) -- before the +// continuation request is ever built. This test proves that end to end +// through the REAL production entry point rather than a hand-rolled check: +// a genuine `*anthropic.Client` (provider/anthropic), talking to an httptest +// server over real HTTP, drives an actual Session.Prompt call through a +// truncated-partial_json max_tokens stop and its auto-continuation. If the +// wire request failed to marshal, Client.Stream would return an error before +// the second HTTP request is ever sent, and this test would see Prompt fail +// and the server receive only one request -- neither happens. +// +// This is the test that pins the rebuttal: red-verify it by reintroducing +// any change that drops the partial call instead of clearing its Arguments +// (the case this test's own tool_use/tool_result assertions below would +// then fail, since the dropped shape carries no tool_use block at all) or +// that bypasses message.Message.Normalize on the append path (which would +// resurface the original marshal failure and fail this test's Stream/Prompt +// calls directly). +func TestMaxTokensPartialJSONMarshalsThroughRealTranscoder(t *testing.T) { + var mu sync.Mutex + var reqCount int + var secondBody map[string]any + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + reqCount++ + n := reqCount + mu.Unlock() + + w.Header().Set("Content-Type", "text/event-stream") + if n == 1 { + // The real Anthropic wire shape for a tool_use block cut off + // mid-emission by max_tokens: content_block_stop still fires + // normally (see provider/anthropic/anthropic.go's doc comments + // on this exact incident), but the accumulated partial_json is + // truncated mid-token -- `{"comm`, never a complete + // `{"command":"echo hi"}`. + io.WriteString(w, "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"usage\":{\"input_tokens\":10}}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_1\",\"name\":\"bash\"}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"comm\"}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n") //nolint:errcheck + io.WriteString(w, "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"max_tokens\"},\"usage\":{\"output_tokens\":5}}\n\n") //nolint:errcheck + io.WriteString(w, "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n") //nolint:errcheck + return + } + // The continuation request: decode it server-side before responding + // -- a decode failure here means the client never sent a well-formed + // body, which is exactly what a marshal failure client-side would + // otherwise have prevented from arriving at all. + if err := json.NewDecoder(r.Body).Decode(&secondBody); err != nil { + t.Errorf("decode continuation request body: %v", err) + } + io.WriteString(w, "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_2\",\"usage\":{\"input_tokens\":10}}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"done\"}}\n\n") //nolint:errcheck + io.WriteString(w, "event: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0}\n\n") //nolint:errcheck + io.WriteString(w, "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":1}}\n\n") //nolint:errcheck + io.WriteString(w, "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n") //nolint:errcheck + })) + defer srv.Close() + + c := &anthropic.Client{APIKey: "test-key", BaseURL: srv.URL} + s := NewSession(Config{ + Providers: provider.Registry{anthropic.Family: c}, + Model: message.ModelRef{Provider: anthropic.Family, Model: "m"}, + MaxTokensContinuations: 3, + }) + + final, err := s.Prompt(context.Background(), "go") + if err != nil { + t.Fatalf("Prompt = %v, want success (the continuation request must marshal and send)", err) + } + if final.Parts.Text() != "done" { + t.Errorf("final = %q, want %q", final.Parts.Text(), "done") + } + + mu.Lock() + n := reqCount + mu.Unlock() + if n != 2 { + t.Fatalf("server received %d requests, want 2 (initial max_tokens turn, then the auto-continuation)", n) + } + if secondBody == nil { + t.Fatal("continuation request body was never decoded") + } + + msgs, ok := secondBody["messages"].([]any) + if !ok { + t.Fatalf("continuation request has no messages array: %+v", secondBody) + } + + // The truncated call's identity (id, name) survives the round trip + // through the real transcoder, with its Arguments cleared to an empty + // object -- the incident-tested behavior TestPersistTruncatedToolCallArguments + // protects -- and a paired is_error tool_result immediately follows it, + // so the wire request is fully valid, not merely non-crashing. + var foundToolUse, foundToolResult bool + for _, m := range msgs { + mm, ok := m.(map[string]any) + if !ok { + continue + } + content, _ := mm["content"].([]any) + for _, b := range content { + block, ok := b.(map[string]any) + if !ok { + continue + } + switch block["type"] { + case "tool_use": + if block["id"] != "toolu_1" || block["name"] != "bash" { + continue + } + foundToolUse = true + input, ok := block["input"].(map[string]any) + if !ok { + t.Fatalf("tool_use block's input = %#v, want a present empty object (the truncated Arguments cleared)", block["input"]) + } + if len(input) != 0 { + t.Errorf("tool_use block's input = %#v, want empty (truncated JSON is unusable)", input) + } + case "tool_result": + if block["tool_use_id"] != "toolu_1" { + continue + } + foundToolResult = true + if v, _ := block["is_error"].(bool); !v { + t.Errorf("tool_result block for toolu_1 is_error = %v, want true", block["is_error"]) + } + } + } + } + if !foundToolUse { + t.Fatalf("continuation request never replayed a tool_use block for toolu_1/bash: %+v", secondBody) + } + if !foundToolResult { + t.Fatalf("continuation request never carried the paired is_error tool_result for toolu_1: %+v", secondBody) + } +} From 0ea2daee61ec0786ca7cbf64da78251b78edccc7 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 17:29:15 -0400 Subject: [PATCH 08/95] feat(engine): run one assistant message's tool calls concurrently (#194) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(engine): execute one message's tool calls as a concurrent batch Session.runToolCalls executed an assistant message's tool calls strictly in order — its own doc comment said so. A turn that batches eight reads therefore paid the sum of eight round trips, when the model had already declared those reads independent by putting them in one message. Claude Code runs the same batch concurrently and lands 2-3x wall clock on a batched-read turn. The provider contract is the design. Every tool call inside ONE assistant message is a batch the model considers independent, so the engine runs it that way. Results still join in CALL order: tool_result order must match tool_use order on the wire, and a transcript must not depend on which call finished first. Three mechanisms keep that safe. A tool that mutates session state sets Serial and acts as a barrier: splitBatch cuts the batch into runs around it, so everything before it finishes first and nothing after it starts early. The mcp, model and goal tools set it, each naming the state it mutates. A tool that owns a resource sets Key: two calls with the same non-empty key run mutually exclusive, in call order, through an explicit baton hand-off rather than a plain mutex, because Go's Mutex is not FIFO under contention. read_file, write_file and edit_file share one path-keyed namespace, and the process tool keys on the process name. A bounded worker pool caps a segment at Config.ToolConcurrency calls in flight, default 8, so a twenty-call batch cannot fork-bomb a 2 vCPU box. The per-key invariant is written down in toolexec.go before the code, per the concurrency rule in AGENTS.md: a same-key call must never wait on a predecessor that has not yet been given a pool slot. Every baton is handed out synchronously, on the submitting goroutine, for the whole segment before any worker starts, so a waiter only ever blocks on a channel that already exists. maybeRetainToolResult moves to the JOIN and runs there in call order. It is internally locked, but writeRetainedToolResult mints trh_N handles from a counter and journals one record per retention, so concurrent retention would make handle numbers and journal order depend on completion order. Its retained-bytes ceiling check is also check-then-act across two separate s.mu sections, which two concurrent retentions can both pass. The join closes both. EventToolEnd already carried the pre-retention output, so no event changes. Config.ToolConcurrency is the only seam. The engine never reads an environment variable (see session_manager.go's rule), so the HARNESS_SEQUENTIAL_TOOLS and HARNESS_TOOL_CONCURRENCY operator knobs are resolved by cmd/harness in a follow-up PR. A value of 1 restores the pre-parallel path byte for byte. The full session-state audit — every built-in's runToolCall path, with a verdict and file-level evidence for each — is the package doc comment in toolexec.go. This commit carries the executor and the audit. Its test suite lands in the next commit on this branch. * test(engine): cover the parallel tool-call executor's core invariants Nine tests for toolexec.go: wall clock equals the longest call and not the sum, results join in call order under reversed completion, sequential mode never overlaps, the concurrency cap both bounds and is reached, a Serial tool is a barrier on both sides, same-key calls serialize in call order while a different key runs beside them, the three file tools share one path-key namespace (including the dot-dot and absolute aliases), two same-path edits apply in call order through the real tools, and one failing call never cancels its siblings. Concurrency is proved by rendezvous, not by timing: a fake tool blocks until a sibling has entered, so a one-at-a-time executor cannot make progress. Those tests run in a synctest bubble, where that is reported at once as a deadlock. The wall-clock test measures the bubble's fake clock, so its elapsed value is exact. Also closes two adversarial-review findings in the executor itself. admitAndRun is now the single admission gate: a call admitted after the turn is canceled does not run, and returns a synthesized canceled result. This deliberately changes the pre-parallel behavior, which kept starting remaining calls after an abort — a batch queues calls behind the cap, and write_file commits without consulting ctx, so 'the tool decides' was not a real gate. A done ledger plus a backfill pass then guarantees one result per call on every path, including one no path marked. filePathKey now cleans the resolved path, so 'a/../x' and 'x' key one file. resolvePath returns an absolute argument verbatim, so the dot-dot alias would otherwise bypass the key on an absolute path. * test(engine): cover batch cancellation, journal order, retention and hooks Five more tests for the concurrent executor. Cancellation is the orphan-pairing guard: a tool_use with no tool_result wedges a session forever, so both an already-canceled turn and one canceled with calls in flight must still return exactly one result per call, in call order. The already-canceled case also asserts no tool ran. The journal test drives a real Prompt turn and asserts both halves of the event contract at once: EventToolEnd arrives in completion order, which this test forces to be the reverse of call order, while the RoleTool message in history stays in call order. Its own OnEvent collector takes a lock, which documents the new Config.OnEvent contract. Retention gets two tests. The first proves handles mint in call order when the batch completes in reverse. The second is the adversarial-review finding on the retained-bytes ceiling: its check-then-act spans two s.mu sections, so concurrent retention could overshoot the cap by the concurrency factor. Running retention at the join removes the concurrency, and a batch whose results individually fit but whose sum exceeds the cap now stops at the cap. The hook test is the relayed finding from the plugin protocol review: dispatchChain folds responses into a shared request pointer, so each concurrent call must own its own request value. Two calls whose before-hooks are in flight together must each get their own rewritten args. The task test is the approved addendum's case: a batch of task spawn, read, task spawn overlaps and joins in call order. It needs a concurrency-safe provider fake, since two children stream at once and scriptedProvider mutates its fields with no lock. * test(engine): make the concurrency-cap test prove the cap The first version of TestBatchConcurrencyCapIsEnforced passed with the cap removed. It counted a running maximum while a wave barrier released each group of three, and an unbounded pool looks bounded under that shape: the early calls exit before the later ones enter, so the running maximum never reaches the true in-flight count. The test measured its own barrier, not the executor's cap. It now holds every call inside the tool and reads the count after synctest.Wait, which returns only once every goroutine in the bubble is durably blocked. That count IS the true in-flight maximum, so an unbounded pool cannot hide behind scheduling. Red-verified: with the cap ignored the test reports 9 calls inside at once, want 3. * fix(engine): close three cross-model review findings in the executor A cross-provider adversarial review of the batch executor raised three issues. All three are now fixed, each with its own red-verified test. A call waiting for its same-key predecessor no longer holds a pool slot while it waits. The first cut acquired the semaphore on the submitting goroutine and awaited the baton inside the worker, so a waiter sat on a slot it could not use and an unrelated later call was refused admission behind it — head-of-line blocking that grows with how many same-key calls a batch carries. The review called this a deadlock; it is not, because a predecessor is always submitted earlier and therefore always holds a slot before its successor exists, so the chain cannot invert. A batch that truly wedged would need one call to wait on another call in the SAME batch, which is exactly the independence the batch declares. The waiting order is still wrong, so each job now takes its goroutine at once, waits for its baton, and only then takes a slot. The pool bounds execution, not goroutine count, and a batch is bounded by one assistant message. A panicking tool no longer takes the process down. A panic in a worker goroutine cannot be recovered by the join, so it killed the process and left the assistant message's tool_use blocks unanswered for the life of the session — the exact orphan shape the pairing invariant exists to prevent. runOneGuarded now turns a panic into one ordinary error result. It wraps the sequential path too, deliberately: the guarantee must not depend on which execution mode a session runs in. A panicking Tool.Key gets the same treatment, and falls back to a shared per-tool key rather than to no key, so a tool's exclusion is never silently dropped. filePathKey now resolves to an ABSOLUTE path. Cleaning alone was not enough: resolvePath joins a relative argument onto Config.WorkDir, and WorkDir itself may be relative, so with WorkDir "." the argument "a.txt" and the absolute spelling of the same file took two different keys — and a write could race an edit on it. filepath.Abs resolves against the process working directory, which is the same directory the tools' own os.Open and os.WriteFile calls resolve against, so the key always names the file the tool actually touches. Red-verified. Admitting before the baton wait: the waiter test reports a bubble deadlock. Guard disabled: the panic test fails in both parallel and sequential mode. Clean instead of Abs: 'relative key "path:a.txt" != absolute key "path:/.../a.txt": one file must take one key'. * docs(engine): record two coupling notes the review surfaced A refused call emits no tool events. Both EventToolStart and EventToolEnd fire inside runToolCall, which the cancellation admission gate short-circuits, so an aborted batch's refused calls appear in the transcript as tool_result parts with no matching event pair. That is deliberate — the events describe an execution that never happened — and the transcript the provider validates still pairs every tool_use with a tool_result. It is a slightly broader observable change than 'EventToolEnd may interleave', so admitAndRun now says so. resolvePath is load-bearing beyond convenience. filePathKey builds the batch executor's per-file exclusion key from resolvePath's own output, so the key names the file a tool touches only while every file tool resolves the same way. A future file tool that resolved a path by another route would key one file under two names, and a concurrent write and edit on it would stop being serialized. resolvePath now states the requirement where a new tool's author will read it. * feat(cmd/harness): wire the tool-concurrency operator knobs Review of #194 found Config.ToolConcurrency's doc comment claiming, in the present tense, that cmd/harness resolves HARNESS_SEQUENTIAL_TOOLS and HARNESS_TOOL_CONCURRENCY into it. Nothing did. Every production session resolved 0 to the default of 8, so concurrent tool execution shipped on by default with no operator opt-out at all. That is worse than a stale comment. This PR's own answer to the hook-ordering review finding is that a deployment whose plugin depends on the pre-parallel cross-call hook order can set sequential mode. That mitigation is only real if the knob exists, so the knob belongs here rather than in the follow-up PR that carries the prompt sentence and the AGENTS.md text. cmd/harness gains toolConcurrency(), which both runCmd and serveCmd feed into Config. HARNESS_SEQUENTIAL_TOOLS=1 wins and returns 1, the kill switch. HARNESS_TOOL_CONCURRENCY= otherwise sets the cap through the existing envInt helper, so a malformed or non-positive value falls back to the engine default exactly like every other HARNESS_* integer knob. The engine still reads no environment variable. Three comment-accuracy fixes from the same review round. The sequential path is no longer described as the pre-parallel behavior 'byte for byte': it keeps the pre-parallel ORDER, but runOneGuarded and admitAndRun apply there too, so a panicking tool yields an error result and a canceled turn admits no further calls. Tool.Key's unparseable-args example named processTool, which returns no key; the fixed-fallback example is filePathKey, and the two shapes differ on purpose, so the comment now gives both. keyChain's mutex is removed: wait runs only on the submitting goroutine before any worker starts, so the lock implied a concurrency contract that does not exist. * docs(engine): record bash as a residual of the per-file key Review of #194 found the per-file exclusion key only half-covers the same-file hazard it advertises. filePathKey puts read_file, write_file and edit_file in one path namespace, but bashTool sets neither Key nor Serial and runs arbitrary shell, so 'echo x > f.txt' beside an edit_file on the same path races on disk. The strictly-sequential default never exposed that, which makes it a real behavior change for the default configuration, not a pre-existing quirk. Keying bash is not possible: its file targets are not statically knowable. Keying it pessimistically, under one global bash key, would serialize the most common parallel workload there is and give up most of the feature. bash therefore stays parallel by design — the model owns batching judgment, the same contract Claude Code ships — and HARNESS_SEQUENTIAL_TOOLS=1 is the operator's answer for a workload that cannot tolerate it. toolexec.go's package comment now carries a 'Residuals of the per-file key' section naming bash and the symlink/inode case together, and filePathKey points at it, so nobody reads the same-file guarantee as wider than it is. * fix(engine): resolve symlinks in the per-file key, state the hook contract Two findings from the latest adversarial round. The per-file exclusion key was bypassable through a valid filesystem alias. filePathKey canonicalized lexical and absolute spellings but left symlinks alone, so /work/real and /work/link -> real took two keys and a write_file could race an edit_file against one file's bytes. The earlier cut called that an accepted residual on the grounds that resolving it costs a syscall on the batch's hot path. That reasoning does not hold: a symlink is a valid way to name a file, not an exotic edge, and one lstat-walk is nothing beside a tool call that does real file I/O. canonicalFileKeyPath resolves the whole path first, which covers an existing file reached two ways, and falls back to resolving the PARENT directory and rejoining the base name, which covers a not-yet-created file inside a symlinked directory — a routine write_file target that a whole-path resolve cannot reach. A hard link still aliases and is now the only documented residual there, because closing it needs an inode comparison against every other key in the batch: quadratic, and still racing a file created mid-batch. The Hooks interface carried no concurrency contract. Config.OnEvent got one when the executor landed, but Hooks is where the plugin dispatches live, and those are what now overlap. It now states the whole contract: every method must be safe for concurrent use, cross-call order is not guaranteed and before(B) can precede before(A), per-call order IS guaranteed, and after-hooks across siblings are completion-ordered. It names the change from the pre-parallel engine, tells a hook that keeps cross-call state to key it by call id or serialize itself, points at plugin/PROTOCOL.md's Concurrency section for the out-of-process contract, and names HARNESS_SEQUENTIAL_TOOLS=1 for a deployment that cannot adapt. The audit also now records why a namespaced mcp__server__tool call is not a Serial barrier: mcp/conn.go is id-multiplexed like the plugin transport, and the MCPManager is a per-process singleton, so concurrent calls to one server already happen across sessions today. Red-verified: without symlink resolution, TestFilePathKeySeesThroughSymlinks reports the symlink and real keys differing, and the symlinked-directory case differing too. * fix(engine): key the task tool per descendant, close six doc gaps Review of #194 found the audit's biggest hole: the task tool. It is correctly not Serial — a spawn hands the child to the SessionManager and returns — but its verbs name a TARGET descendant, and unkeyed siblings run in completion order. A cancel(X) followed by a send(X) in one batch could execute as send-then-cancel: the message reaches a still-running child, then the child and the message die together. That is the reverse of what the model asked for, and SessionManager's own mutexes cannot help, because the defect is ORDER, not a race. taskToolKey keys on session_id, so two verbs naming one descendant run in call order while different targets and every spawn stay parallel. A malformed call takes no key, matching processToolKey: it cannot collide with a real session id, and runTaskTool rejects it before touching any session. HARNESS_TOOL_CONCURRENCY=-1 now means sequential. envInt folds every non-positive value into 0, so the negative branch of Config.ToolConcurrency's documented clamp was unreachable through the only operator-facing seam — an operator asking for sequential silently got parallel at 8. toolConcurrency() reads a negative value itself and returns 1. Zero, empty, and malformed still mean 'not set'. A parallel segment holding exactly one call now runs inline. That is the common shape — a turn with one tool call, or a lone call between two Serial ones — and a goroutine, a WaitGroup, a semaphore and a keyChain buy nothing when there is no sibling to run beside or exclude. Four comment fixes. The audit now covers task, read_tool_result and session_info, so its claim to cover every built-in is true. The admission gate no longer claims a canceled turn's results are discarded: they are appended to durable history and survive a resume, which makes the gate more important rather than less. filePathKey's residual paragraph no longer calls symlink aliasing unresolved, since canonicalFileKeyPath resolves it and only hard links remain. resolvePath had two stacked lead sentences. The synthesized-result constants said 'two' and declared three. Red-verified: without the Key, TestTaskToolKeysPerTargetDescendant reports 'a task verb naming a descendant has no key: cancel and send on one child could reorder'. * docs(engine): drop a forward reference to an unwritten spec section Review of #194 found the Hooks concurrency contract pointing at a plugin/PROTOCOL.md 'Concurrency' section that does not exist in this tree. That section belongs to the separate plugin-transport change, so the reference was true only in a future no reader of this tree can see, and a plugin author following it found nothing. The comment now states the binding fact directly instead of delegating it: an out-of-process plugin's hook dispatches ride the connection PROTOCOL.md specifies, that connection is id-multiplexed and already carries several requests in flight, so a plugin must never assume its hooks are called one at a time. That is durable wording — it stays correct whatever order the two changes land in, and it does not depend on a section name. * docs(engine): correct the operator-facing sequential-mode claim Config.ToolConcurrency still described 1 as 'the pre-parallel in-call-order path, byte-for-byte'. An earlier commit corrected exactly that claim in toolexec.go's sequential branch and missed this copy, so the two comments in one change contradicted each other — and this is the one an operator actually reads before setting HARNESS_SEQUENTIAL_TOOLS=1. Someone reaching for an exact revert got neither of the two deliberate differences: runOneGuarded turns a panicking tool into one error result instead of letting it unwind through Prompt, and admitAndRun refuses to start a call once the turn is canceled, where the old loop ran every remaining call. Both are intentional, because the one-result-per-call guarantee must hold identically however a batch executes. The field doc now says 'one call at a time, in call order', names both differences, and points at the toolexec.go branch that states the same thing, so the two cannot drift apart again unnoticed. * fix(engine): keep tool events balanced when a tool or hook panics runToolCall emits EventToolStart before anything can panic. Once runOneGuarded started recovering a panicking tool, the unwind skipped both end emits and reached the recover with the start already on the live event stream, so nothing ever closed it. History was fine — the recover still produced one error result — but a subscriber that pairs start and end by call id waits forever: the console renders a tool fold that never finishes, and the session monitor shows a tool running for the life of the box. The path is new with this change, because before the recover a tool panic killed the process and no session survived to render anything. The recover moves into runToolCall, which is the one frame that knows which events have already been emitted. runOneGuarded stays as the backstop for a panic outside that frame, so the one-result-per-call guarantee is unchanged and no result text is duplicated. Two flags, not one, because the two end events do not bracket the same region: emitToolExecuteEnd fires BEFORE the after-hook chain and EventToolEnd fires after it. A panic inside ToolExecuteAfter must emit only the second. Emitting both unconditionally would hand every plugin a duplicate tool.execute.end for one call, trading a dangling start for a phantom end. Red-verified both ways. Without the EventToolEnd emit the panicking-tool case reports 'EventToolEnd count = 0, want exactly 1 (a dangling start never closes)'; with the execute-end emit made unconditional the after-hook case reports 'tool.execute.end emitted 2 times, want exactly 1'. * fix(engine): a before-hook panic owes no tool.execute.end The previous commit balanced the event stream for a panicking tool and a panicking after-hook, and introduced the inverse imbalance for a panicking before-hook. emitToolExecuteStart fires AFTER the ToolExecuteBefore chain, so a panic inside that chain reached the recover having emitted no start, and the recover emitted an end anyway: a phantom tool.execute.end for a call that never executed. That also contradicted emitToolExecuteStart's own rule that a call denied by tool.execute.before fires neither event. The flag now records whether an end is OWED rather than whether one was already emitted, which is the only question the recover can answer correctly in both directions. The plugin-facing pair does not bracket the same region as the engine-facing one: the start fires after the before chain, the end fires before the after chain. A panic in the before chain owes nothing because the start never fired. A panic in the after chain owes nothing because the end already fired. Only a panic between them owes one. A boolean meaning 'the end was emitted' answered the first case wrong, which is exactly how the phantom end got in. TestPanicKeepsToolEventsBalanced gains the before-hook case and asserts both counts are zero, so neither half of the plugin-facing pair can reappear alone. Red-verified against the pre-fix semantics: 'tool.execute.end emitted 1 times, want 0: a phantom end with no start'. --------- Co-authored-by: andybons --- cmd/harness/main.go | 32 + cmd/harness/toolconcurrency_test.go | 37 + engine/engine.go | 215 +++- engine/filetools.go | 108 +- engine/goal_tool.go | 4 + engine/mcp_tool.go | 5 + engine/model_tool.go | 5 + engine/process.go | 22 + engine/task_tool.go | 30 + engine/toolexec.go | 632 ++++++++++++ engine/toolexec_test.go | 1444 +++++++++++++++++++++++++++ 11 files changed, 2498 insertions(+), 36 deletions(-) create mode 100644 cmd/harness/toolconcurrency_test.go create mode 100644 engine/toolexec.go create mode 100644 engine/toolexec_test.go diff --git a/cmd/harness/main.go b/cmd/harness/main.go index a082bc45..628951f5 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -306,6 +306,36 @@ func envInt(name string) int { return n } +// toolConcurrency resolves engine.Config.ToolConcurrency from the two +// operator knobs. The engine never reads an environment variable itself +// (see engine/session_manager.go), so this is where the variables become +// a value. +// +// HARNESS_SEQUENTIAL_TOOLS=1 wins: it is the kill switch that restores +// strictly one-at-a-time tool execution, for a box whose plugin depends +// on the pre-parallel cross-call hook order, or for any workload a +// concurrent batch upsets. HARNESS_TOOL_CONCURRENCY= otherwise sets +// the cap. Neither set leaves 0, which the engine resolves to its own +// default. +func toolConcurrency() int { + if os.Getenv("HARNESS_SEQUENTIAL_TOOLS") == "1" { + return 1 + } + // A NEGATIVE value is handled here rather than through envInt, which + // folds every non-positive value into 0 (the engine default). That + // fold would make engine.Config.ToolConcurrency's documented "a + // negative value is clamped to 1 (sequential)" unreachable through + // the only operator-facing seam: HARNESS_TOOL_CONCURRENCY=-1 would + // silently give parallel-at-8 to an operator who asked for the + // opposite. An explicit negative therefore means sequential, exactly + // as the field says. 0, empty, and a malformed value still mean "not + // set" and fall through to the engine default. + if n, err := strconv.Atoi(os.Getenv("HARNESS_TOOL_CONCURRENCY")); err == nil && n < 0 { + return 1 + } + return envInt("HARNESS_TOOL_CONCURRENCY") +} + // sessionDir resolves where session logs live, in precedence order: // -no-save (yields "", persistence disabled) > $HARNESS_SESSION_DIR > // configDir (config session_dir) > $HOME/.harness/sessions. Nothing is @@ -681,6 +711,7 @@ func runCmd(args []string) error { // engine checks itself. ToolResultInlineBytes: cfg.ToolResultInlineBytesValue(), ToolResultRetainedBytes: cfg.ToolResultRetainedBytesValue(), + ToolConcurrency: toolConcurrency(), // GoalTool mirrors serveCmd's mkCfg below: the `goal` session tool is // only useful once an evaluator is actually configured to drive a // goal loop against (-goal itself resolves and validates its own @@ -1392,6 +1423,7 @@ func serveCmd(args []string) error { // gets it unless an operator sets a non-positive inline value. ToolResultInlineBytes: cfg.ToolResultInlineBytesValue(), ToolResultRetainedBytes: cfg.ToolResultRetainedBytesValue(), + ToolConcurrency: toolConcurrency(), // GoalTool enables the `goal` session tool (status/set/adjust) // whenever an evaluator is configured to drive a goal loop // against — the same condition server.Options.GoalEvaluator diff --git a/cmd/harness/toolconcurrency_test.go b/cmd/harness/toolconcurrency_test.go new file mode 100644 index 00000000..089b50e0 --- /dev/null +++ b/cmd/harness/toolconcurrency_test.go @@ -0,0 +1,37 @@ +package main + +import "testing" + +// TestToolConcurrencyKnobs pins the operator kill switch. The engine never +// reads an environment variable, so this function is the only place the +// two knobs become a value — and engine.Config.ToolConcurrency 1 is what +// restores strictly sequential tool execution. +func TestToolConcurrencyKnobs(t *testing.T) { + for _, tc := range []struct { + name string + sequential string + cap string + want int + }{ + {"unset leaves the engine default", "", "", 0}, + {"sequential kill switch wins", "1", "16", 1}, + {"cap is honored", "", "4", 4}, + {"a non-one sequential value is ignored", "0", "4", 4}, + {"a bad cap falls back to the engine default", "", "nope", 0}, + {"a zero cap falls back to the engine default", "", "0", 0}, + // A negative value must reach the engine's documented clamp + // (Config.ToolConcurrency: "clamped to 1 (sequential)"). envInt + // alone folds it to 0, which would silently give parallel-at-8 to + // an operator who asked for the opposite. + {"a negative cap means sequential", "", "-1", 1}, + {"a large negative cap means sequential", "", "-5", 1}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("HARNESS_SEQUENTIAL_TOOLS", tc.sequential) + t.Setenv("HARNESS_TOOL_CONCURRENCY", tc.cap) + if got := toolConcurrency(); got != tc.want { + t.Errorf("toolConcurrency() = %d, want %d", got, tc.want) + } + }) + } +} diff --git a/engine/engine.go b/engine/engine.go index c0b4933c..c122b086 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -23,6 +23,28 @@ import ( // Hooks is the slice of the plugin host the engine uses. *plugin.Host // satisfies it; tests use fakes. A nil Hooks disables all hook dispatch. +// +// Every method MUST be safe for concurrent use, and CROSS-CALL ORDER IS +// NOT GUARANTEED. One assistant message's tool calls run as a concurrent +// batch (toolexec.go), so ToolExecuteBefore/ToolExecuteAfter/ExecuteTool +// for two different calls can be in flight at the same moment, and +// before(B) can precede before(A) for calls the model listed as A then B. +// +// What IS guaranteed is PER-CALL order: for any one call id, before runs, +// then the tool, then after. After-hooks across sibling calls are +// completion-ordered. +// +// This is a change from the pre-parallel engine, where call A's whole +// before/tool/after sequence finished before call B started. A hook +// implementation that keeps cross-call state — a running quota, an audit +// chain, a policy that reads the previous call — must key that state by +// call id or serialize itself. The same contract binds an out-of-process +// plugin: its hook dispatches ride the connection plugin/PROTOCOL.md +// specifies, which is id-multiplexed and already carries several requests +// in flight at once, so a plugin must never assume its hooks are called +// one at a time. A deployment that cannot adapt sets +// HARNESS_SEQUENTIAL_TOOLS=1, which restores one-at-a-time execution and +// with it the old hook order. type Hooks interface { ChatParams(ctx context.Context, req *plugin.ChatParamsRequest) plugin.ChatParams SystemTransform(ctx context.Context, req *plugin.SystemTransformRequest) []string @@ -39,9 +61,46 @@ type Hooks interface { } // Tool is a built-in (in-process) tool. +// +// Serial and Key both default to their Go zero value (false, nil), which is +// exactly today's behavior for every existing built-in and for every +// plugin/MCP tool (neither ever sets them) — see toolexec.go for the batch +// executor that reads them. type Tool struct { Def provider.ToolDef Run func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) + + // Serial marks this tool as a BARRIER inside one assistant message's + // batch of tool calls (see toolexec.go's splitBatch). A batch splits + // into runs around a Serial call: every call before it finishes before + // it starts, and it finishes before any call after it starts. Set true + // only for a tool that mutates session-wide state in a way a plain + // s.mu-guarded write cannot make safe to interleave with a sibling call + // — e.g. a model or goal swap mid-batch. See mcpTool, modelTool, + // goalTool for the three built-ins that set it, each with a one-line + // comment naming the state it mutates. + Serial bool + + // Key, when non-nil, computes a resource key from the running session + // and one call's raw arguments. Two calls in the same batch that + // produce the same non-empty key run mutually exclusive, in CALL + // order — never concurrently with each other, though still + // concurrently with calls that key differently or key empty. An empty + // string means "no key": the call runs alongside everything else in + // its segment, exactly as today. + // + // Key takes the Session (not just args) because a path-based key must + // resolve a relative path against the session's own working directory + // (s.resolvePath) to match an absolute path naming the same file — see + // editFileTool/writeFileTool/readFileTool. Key must never panic; a + // tool whose args fail to parse must return a key it has chosen + // deliberately, never a panic. The two built-in shapes differ on + // purpose: filePathKey returns a FIXED fallback key, so every + // unparseable file call serializes against every other one, while + // processToolKey returns "" (no key), because an unparseable process + // call cannot collide with a real process name and runProcessTool + // rejects it before touching any process anyway. + Key func(s *Session, args json.RawMessage) string } // Event is one entry in the session's event stream. Event types follow ACP @@ -383,6 +442,22 @@ type Config struct { // concurrent ClearGoal/evaluation race — so OnEvent must NEVER call back // into the Session that raised the event (Prompt, ClearGoal, ActiveGoal, // etc.), which would deadlock on that same mutex. + // + // The engine MAY call OnEvent from SEVERAL goroutines AT ONCE: one + // assistant message's batch of tool calls now runs concurrently (see + // toolexec.go), and EventToolStart/EventToolEnd fire from whichever + // goroutine is running that call, so two calls in one batch can each be + // inside OnEvent at the same moment. This was already true before + // toolexec.go existed, for a different reason — a `task` child's own + // background turn (SessionManager.Spawn) can call the SAME OnEvent + // concurrently with its parent's turn, since configSnapshot copies the + // func value by value into every child's Config (see + // TestRunOnEventHandlerSerializesConcurrentCallers, cmd/harness). A + // caller whose OnEvent is not naturally reentrant (writes a shared + // buffer, encodes to one io.Writer) MUST serialize its own body — see + // server.Publish/publishLive (routes every event through a session- + // keyed s.mu) and cmd/harness's newRunOnEventHandler (wraps its body in + // a mutex) for the two in-repo patterns. OnEvent func(Event) // OnStorePhase, when non-nil, receives one call per ENDED phase of the @@ -745,6 +820,41 @@ type Config struct { // the ceiling. Config key `tool_result_retained_bytes`, product default // 4194304. ToolResultRetainedBytes int + + // ToolConcurrency bounds how many of one assistant message's tool calls + // runToolCalls (toolexec.go) runs at once. Resolved ONCE, in newSession, + // into Session's own toolConcurrency field — see resolveToolConcurrency + // (toolexec.go), which follows resolveContextWindow's exact + // precedence shape (context_window.go). + // + // Precedence: 0 (the zero value, "unset") resolves to the package + // default, defaultToolConcurrency (8) — a batch runs in parallel, capped + // at 8 calls in flight. 1 resolves to strictly SEQUENTIAL execution: + // one call at a time, in call order, including for a serial tool (a + // barrier is a no-op when nothing ever runs beside it). A value above + // 1 is the cap verbatim. A negative value is clamped to 1 (sequential) + // — never treated as "unlimited". + // + // 1 restores the pre-parallel ORDER, not the pre-parallel behavior in + // every respect, and an operator reaching for it as an exact revert + // should know the two deliberate differences: runOneGuarded turns a + // panicking tool into one error result instead of letting it unwind + // through Prompt, and admitAndRun refuses to START a call once the + // turn is canceled, where the old loop ran every remaining call. + // Neither depends on the mode, by design — the one-result-per-call + // guarantee must hold identically however a batch executes. See the + // sequential branch in toolexec.go, which states the same thing. + // + // The engine itself never reads an environment variable (see + // session_manager.go's own "the engine itself never reads environment + // variables" rule) — this field is the ONLY seam. cmd/harness's + // toolConcurrency() turns the HARNESS_SEQUENTIAL_TOOLS=1 kill switch + // and the HARNESS_TOOL_CONCURRENCY= cap into this field before it + // builds Config, for both `harness run` and `harness serve`; this + // package neither reads nor knows about either variable. An embedder + // that builds Config itself sets the field directly and gets no + // environment handling at all. + ToolConcurrency int } // Session is one conversation: an in-memory history plus the agent loop. @@ -1006,6 +1116,15 @@ type Session struct { contextWindowExplicit bool contextWindowSource string + // toolConcurrency is the resolved (never-zero, never-negative) cap on + // how many of one batch's tool calls run at once — see + // Config.ToolConcurrency and resolveToolConcurrency (toolexec.go). Set + // once in newSession and never changed again for the session's + // lifetime; unlike the context window, no runtime event re-derives it. + // 1 means strictly sequential. Read-only after construction, so no + // lock is needed to read it from a running batch. + toolConcurrency int + // compactCount/lastCompactedAt track how many times this session has // been compacted and when the most recent one landed — durable via the // compact journal record (see store.go), so GET /session can show a UI @@ -1158,6 +1277,7 @@ func newSession(cfg Config) *Session { contextWindowSource: contextWindowSource, toolResultNextID: 1, toolResults: make(map[string]toolResultMeta), + toolConcurrency: resolveToolConcurrency(cfg.ToolConcurrency), readHashes: make(map[string][sha256.Size]byte), } for _, t := range []Tool{bashTool(cfg.BashTimeout, cfg.BashOutputCap), readFileTool(), writeFileTool(), editFileTool(), sessionInfoTool(), globTool(), grepTool(), lsTool()} { @@ -2998,50 +3118,74 @@ func (s *Session) toolDefsWithCatalog(ctx context.Context) ([]provider.ToolDef, return defs, plan.catalog } -// runToolCalls executes every tool call in an assistant message, in order, -// and returns the ToolResult parts. -// -// This is retention's single call site (maybeRetainToolResult, see -// toolresult.go): an oversized TEXT result is swapped for a preview plus a -// trh_N handle HERE, before the ToolResult is built and long before -// Session.append, message.NormalizeForWire, or any transcoder sees it. That -// placement is why tool-result handles need no wire-format change at all — -// every downstream layer still sees an ordinary ToolResult carrying ordinary -// Text parts. -// -// Retention runs on the post-hook output (runToolCall already applied -// ToolExecuteAfter), so a plugin that rewrites or enlarges a result has its -// final bytes measured, not the tool's originals. It is a total no-op when -// retention is disabled or the result is within the limit. +// runToolCalls executes every tool call in an assistant message and returns +// the ToolResult parts, one per call, in CALL order. See toolexec.go for the +// executor itself (batch splitting, concurrency, per-key ordering, and the +// join where retention runs). func (s *Session) runToolCalls(ctx context.Context, asst *message.Message) message.Parts { - var results message.Parts - for _, p := range asst.Parts { - tc, ok := p.(*message.ToolCall) - if !ok { - continue - } - out, isErr := s.runToolCall(ctx, tc) - results = append(results, &message.ToolResult{ - CallID: tc.CallID, - Content: s.maybeRetainToolResult(tc.Name, out), - IsError: isErr, - }) - } - return results + return s.runToolBatch(ctx, asst) } -func (s *Session) runToolCall(ctx context.Context, tc *message.ToolCall) (message.Parts, bool) { +// runToolCall runs one call end to end: the before-hook chain, the tool +// itself, the after-hook chain, and the four events that bracket them. +// +// It recovers a PANIC from the tool or from either hook chain. The recover +// lives here, rather than only in toolexec.go's runOneGuarded, because +// this is the one frame that knows WHICH events have already been emitted. +// EventToolStart fires before anything can panic, so a panic unwinding +// past this function would otherwise leave a dangling tool.start on the +// live event stream forever — a subscriber that pairs start and end by +// call id (the session monitor's reducer, an ACP tool node, a plugin +// audit trail) would wait on an end that never comes. History stayed +// correct either way, because runOneGuarded still produces one error +// result; the event stream is the half it cannot fix from outside. +// +// execEndOwed tracks whether a tool.execute.start is OUTSTANDING, which +// is the only question the recover can answer correctly in both +// directions. The plugin-facing pair does not bracket the same region as +// the engine-facing one: emitToolExecuteStart fires AFTER the before-hook +// chain and emitToolExecuteEnd fires BEFORE the after-hook chain. So a +// panic in ToolExecuteBefore owes no tool.execute.end — emitting one +// would be a phantom end for a call that never started, the inverse of +// the dangling start this recover exists to prevent, and it would +// contradict emitToolExecuteStart's own rule that a denied call fires +// neither. A panic in ToolExecuteAfter owes none either, because the end +// already fired; emitting a second would hand every plugin a duplicate +// for one call. Only a panic between the two owes one. A boolean that +// merely recorded "the end was emitted" got the first case wrong. +// +// This path is new with concurrent execution. Before runOneGuarded a tool +// panic killed the process, so "the session survives a panic" never +// existed and neither did the unbalanced pair. +func (s *Session) runToolCall(ctx context.Context, tc *message.ToolCall) (out message.Parts, isErr bool) { s.emit(Event{Type: EventToolStart, ToolCall: tc}) + execEndOwed, toolEndEmitted := false, false + defer func() { + r := recover() + if r == nil { + return + } + out = message.Parts{&message.Text{Text: fmt.Sprintf("%s: %v", toolCallPanicText, r)}} + isErr = true + if execEndOwed { + s.emitToolExecuteEnd(tc.Name, tc.CallID, false) + } + if !toolEndEmitted { + s.emit(Event{Type: EventToolEnd, ToolCall: tc, Output: out, IsError: true}) + } + }() + args := tc.Arguments if s.cfg.Hooks != nil { newArgs, deny := s.cfg.Hooks.ToolExecuteBefore(ctx, &plugin.ToolExecuteBeforeRequest{ SessionID: s.ID, CallID: tc.CallID, Tool: tc.Name, Args: args, }) if deny != "" { - out := message.Parts{&message.Text{Text: deny}} - s.emit(Event{Type: EventToolEnd, ToolCall: tc, Output: out, IsError: true}) - return out, true + denied := message.Parts{&message.Text{Text: deny}} + toolEndEmitted = true + s.emit(Event{Type: EventToolEnd, ToolCall: tc, Output: denied, IsError: true}) + return denied, true } if newArgs != nil { args = newArgs @@ -3049,18 +3193,21 @@ func (s *Session) runToolCall(ctx context.Context, tc *message.ToolCall) (messag } s.emitToolExecuteStart(tc.Name, tc.CallID) + execEndOwed = true s.mu.Lock() s.toolExecCount++ s.mu.Unlock() - out, isErr := s.executeTool(ctx, tc, args) + out, isErr = s.executeTool(ctx, tc, args) s.emitToolExecuteEnd(tc.Name, tc.CallID, !isErr) + execEndOwed = false if s.cfg.Hooks != nil { out = s.cfg.Hooks.ToolExecuteAfter(ctx, &plugin.ToolExecuteAfterRequest{ SessionID: s.ID, CallID: tc.CallID, Tool: tc.Name, Args: args, Output: out, }) } + toolEndEmitted = true s.emit(Event{Type: EventToolEnd, ToolCall: tc, Output: out, IsError: isErr}) return out, isErr } diff --git a/engine/filetools.go b/engine/filetools.go index 7747471e..ece8b4f4 100644 --- a/engine/filetools.go +++ b/engine/filetools.go @@ -172,8 +172,17 @@ func readPathContent(path string) (fileContent, error) { return fileContent{IsImage: true, ImageData: full, MediaType: mediaType, Width: cfg.Width, Height: cfg.Height}, nil } -// resolvePath resolves a tool path argument against the session working -// directory. Absolute paths pass through unchanged. +// resolvePath maps a tool argument to the path the tool opens, resolving +// a relative argument against the session working directory; an absolute +// argument passes through unchanged. Every file tool MUST route its path +// argument through it. +// +// That is load-bearing beyond convenience: filePathKey builds the batch +// executor's per-file exclusion key from resolvePath's own output (see +// filePathKey), so the key names the file a tool touches only while every +// tool resolves the same way. A new file tool that resolved a path by any +// other route would key one file under two names, and a concurrent write +// and edit on it would stop being serialized. func (s *Session) resolvePath(path string) string { if filepath.IsAbs(path) { return path @@ -181,6 +190,98 @@ func (s *Session) resolvePath(path string) string { return filepath.Join(s.cfg.WorkDir, path) } +// filePathKeyPrefix namespaces the resource key read_file, write_file, and +// edit_file share (see filePathKey). All three key on the same +// RESOLVED absolute path, so same-path calls across any of the three +// tools serialize together in call order, while different paths still run +// concurrently — a deliberate in-class widening of the approved +// "edit_file serializes per path" design: read_file and write_file join +// the same namespace because a read racing a concurrent write or edit to +// the SAME file is exactly the same same-file hazard class, just missing +// the "corruption" half (a stale read is still a wrong answer). See the +// Tool.Key field doc comment (engine.go) for the general contract. +const filePathKeyPrefix = "path:" + +// canonicalFileKeyPath resolves symlinks in abs, so two spellings that +// reach ONE file take one key. +// +// It tries the whole path first, which covers the dangerous case: an +// existing file reached both directly and through a symlink, where a +// write_file could otherwise race an edit_file on the same bytes. That +// fails for a path that does not exist yet — a write_file target +// routinely does not — so it then resolves the PARENT directory and +// rejoins the base name, which covers a not-yet-created file inside a +// symlinked directory. If neither resolves, the absolute path stands. +// +// Cost is one or two lstat-walks per keyed call, against a tool call that +// does real file I/O anyway. An earlier cut skipped this and documented +// the alias as an accepted residual; review pushed back that a valid +// filesystem alias is not a residual, and the syscall is cheap enough +// that the pushback is right. +// +// A HARD link still aliases: two names for one inode with no symlink to +// follow. Closing that needs a stat and an inode comparison against every +// other key in the batch, which is quadratic and still races a file +// created mid-batch. That one stays a documented residual. +func canonicalFileKeyPath(abs string) string { + if real, err := filepath.EvalSymlinks(abs); err == nil { + return real + } + if dir, err := filepath.EvalSymlinks(filepath.Dir(abs)); err == nil { + return filepath.Join(dir, filepath.Base(abs)) + } + return abs +} + +// filePathKey resolves a read_file/write_file/edit_file call's "path" +// argument against s's working directory (s.resolvePath), so a relative +// and an absolute argument naming the same file produce the same key. A +// call whose args do not parse, or carry no path, falls back to a fixed +// sentinel key rather than panicking — conservative because it still +// serializes every unparseable call against every other one, never +// against a well-formed call to an unrelated path. +func filePathKey(s *Session, args json.RawMessage) string { + var in struct { + Path string `json:"path"` + } + if err := json.Unmarshal(args, &in); err != nil || in.Path == "" { + return filePathKeyPrefix + "" + } + // Make the resolved path ABSOLUTE, then clean it, so every spelling of + // one file produces one key. + // + // Cleaning alone is not enough. resolvePath joins a relative argument + // onto Config.WorkDir, and WorkDir itself may be relative — an + // embedder sets that field directly. With WorkDir "." the argument + // "a.txt" resolves to "a.txt" while "/cwd/a.txt" resolves to itself, + // so one file would take two keys and a write could race an edit on + // it. filepath.Abs resolves against the process working directory, + // which is the same directory the tools' own os.Open/os.WriteFile + // calls resolve against, so the key always names the file the tool + // actually touches. Abs also cleans, which is what collapses the + // "a/../x" alias. + // + // Abs fails only when the process working directory cannot be read. + // Fall back to the cleaned path then: still correct whenever WorkDir + // is absolute, which is what cmd/harness always passes. + // resolvePath returns an absolute argument verbatim, and filepath.Join + // cleans only the relative branch, so the key would otherwise miss a + // dot-dot alias on an absolute path. The tools themselves open both + // spellings through the same resolvePath, so cleaning here can only + // merge two keys that name one file, never split one file into two. + // + // A SYMLINK alias is closed separately, by canonicalFileKeyPath below, + // which this function calls. A HARD link is the one remaining + // residual: two names for one inode with no link to follow. See + // canonicalFileKeyPath for why closing that is not worth it. + resolved := s.resolvePath(in.Path) + abs, err := filepath.Abs(resolved) + if err != nil { + return filePathKeyPrefix + filepath.Clean(resolved) + } + return filePathKeyPrefix + canonicalFileKeyPath(abs) +} + // recordRead records that this session has seen resolvedPath's raw on-disk // bytes hash to hash, either because read_file just read them or because // write_file/edit_file just wrote them. resolvedPath must already be an @@ -232,6 +333,7 @@ func readFileTool() Tool { "required": ["path"] }`), }, + Key: filePathKey, Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { var in struct { Path string `json:"path"` @@ -334,6 +436,7 @@ func writeFileTool() Tool { "required": ["path", "content"] }`), }, + Key: filePathKey, Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { var in struct { Path string `json:"path"` @@ -402,6 +505,7 @@ func editFileTool() Tool { "required": ["path", "old_string", "new_string"] }`), }, + Key: filePathKey, Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { var in struct { Path string `json:"path"` diff --git a/engine/goal_tool.go b/engine/goal_tool.go index 3c5c69e1..37e9d8e5 100644 --- a/engine/goal_tool.go +++ b/engine/goal_tool.go @@ -72,6 +72,10 @@ func goalTool() Tool { "required": ["action"] }`), }, + // Serial: set/adjust mutate the session's active goal (RegisterGoal/ + // UpdateGoal) — a barrier keeps a sibling call from racing a goal + // state change mid-batch. + Serial: true, Run: func(_ context.Context, s *Session, args json.RawMessage) (message.Parts, error) { return runGoalTool(s, args) }, diff --git a/engine/mcp_tool.go b/engine/mcp_tool.go index 4bf3fbf1..6dd5d9ca 100644 --- a/engine/mcp_tool.go +++ b/engine/mcp_tool.go @@ -116,6 +116,11 @@ func mcpTool(canDefer bool) Tool { } return Tool{ Def: def, + // Serial: a select/connect action mutates s.mcpSelected and the + // session's MCP registry state (see runMCPTool). A barrier keeps a + // sibling call in the same batch from reading tool definitions + // mid-mutation. + Serial: true, Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { return runMCPTool(ctx, s, args) }, diff --git a/engine/model_tool.go b/engine/model_tool.go index b02a82de..0215b907 100644 --- a/engine/model_tool.go +++ b/engine/model_tool.go @@ -75,6 +75,11 @@ func modelTool() Tool { "required": ["action"] }`), }, + // Serial: set swaps s.model via SetModel, which every later call in + // the batch (and every later request) must see consistently. A + // barrier keeps a sibling call from running against a model that + // is about to change mid-batch. + Serial: true, Run: func(_ context.Context, s *Session, args json.RawMessage) (message.Parts, error) { return runModelTool(s, args) }, diff --git a/engine/process.go b/engine/process.go index 15738d59..c1e6306c 100644 --- a/engine/process.go +++ b/engine/process.go @@ -79,12 +79,34 @@ func processTool(reg ProcessRegistry) Tool { "required": ["action"] }`), }, + // Key: serializes per process name — start/stop/restart/declare/ + // undeclare on the SAME name must run in call order, never + // concurrently with each other, while calls naming different + // processes (or "list", which names none) run alongside them. + Key: processToolKey, Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { return runProcessTool(ctx, reg, args) }, } } +// processToolKey resolves a process tool call's resource key: its "name" +// argument, or "" (no key) when args do not carry one — "list" and a call +// whose args fail to parse both fall through to "", which is a safe +// fallback here (unlike a path-keyed tool, an unparseable process call has +// no risk of colliding with a real process name) since runProcessTool's +// own json.Unmarshal error path already rejects it before touching any +// process. +func processToolKey(_ *Session, args json.RawMessage) string { + var in struct { + Name string `json:"name"` + } + if err := json.Unmarshal(args, &in); err != nil || in.Name == "" { + return "" + } + return "process:" + in.Name +} + // processToolDescription lists the config-declared roster (name, command, // dir) and explains runtime declaration — stable, cache-safe text (see the // package doc). An empty config roster still explains the tool's actions diff --git a/engine/task_tool.go b/engine/task_tool.go index e7f11a35..b857352e 100644 --- a/engine/task_tool.go +++ b/engine/task_tool.go @@ -176,9 +176,39 @@ func taskTool() Tool { Run: func(ctx context.Context, s *Session, args json.RawMessage) (message.Parts, error) { return runTaskTool(s, args) }, + // Key: serializes per TARGET descendant. Two calls in one batch + // that name the same session_id must run in call order — a + // cancel(X) followed by a send(X) that executed as send-then- + // cancel would deliver a message to a still-running child and + // then kill both, which is the reverse of what the model asked + // for. Calls naming DIFFERENT descendants still run side by side. + // + // A spawn carries no session_id and so takes no key: it is + // asynchronous and cheap, it hands the child to the + // SessionManager and returns, and two spawns are independent by + // construction. That is the approved design's "task spawn is + // already async-cheap — parallel is fine". + Key: taskToolKey, } } +// taskToolKey returns the per-descendant resource key for one `task` +// call: its session_id, or "" (no key) for a call that names none — a +// spawn, or a malformed call runTaskTool rejects before it touches any +// session. An unparseable call cannot collide with a real session id, so +// no key is the safe fallback here, the same reasoning processToolKey +// uses (and unlike filePathKey, which needs a fixed fallback because an +// unparseable path COULD name a file another call in the batch touches). +func taskToolKey(_ *Session, args json.RawMessage) string { + var in struct { + SessionID string `json:"session_id"` + } + if err := json.Unmarshal(args, &in); err != nil || in.SessionID == "" { + return "" + } + return "task-session:" + in.SessionID +} + // runTaskTool dispatches one `task` tool call against s by in.Action, // defaulting an omitted (empty-string) action to taskActionSpawn — the // original, pre-verbs argument shape (agent/prompt/model, no action diff --git a/engine/toolexec.go b/engine/toolexec.go new file mode 100644 index 00000000..ead200e8 --- /dev/null +++ b/engine/toolexec.go @@ -0,0 +1,632 @@ +// Parallel tool-call execution: one assistant message's batch of tool +// calls runs concurrently, bounded by a cap, instead of one at a time. +// Claude Code's own contract is the model: the model already treats every +// call inside one message as independent (that is the entire reason it +// batched them), so harness executes the batch that way too. Measured +// impact is 2-3x wall clock on a batched-read turn. +// +// # Audit: session-state assumptions runToolCall's callees make +// +// This section is Part A of the design: every piece of session state a +// built-in tool's Run touches, and whether it survives concurrent calls +// from one batch unchanged. Each finding is verified against source, not +// assumed. +// +// - s.emit (-> Config.OnEvent): now called from several goroutines at +// once for one batch (EventToolStart/EventToolEnd interleave across +// calls). s.emit itself does nothing but stamp SessionID and invoke +// the callback — no shared mutable state, so concurrent invocation is +// safe AT THIS LAYER. The obligation moves to the callback: see +// Config.OnEvent's doc comment (engine.go), which this feature updates +// to say the engine MAY call it from several goroutines at once. Both +// in-repo consumers already tolerate this: server.Publish/publishLive +// (server/journal.go) route every event through s.mu-guarded +// bookkeeping, and cmd/harness's newRunOnEventHandler already wraps +// its callback in a mutex — TestRunOnEventHandlerSerializesConcurrentCallers +// covers exactly this, pre-dating this feature (a `task` child's +// background goroutine could already call the same OnEvent concurrently +// with its parent's own turn). Neither needed a change for this PR. +// +// - emitToolExecuteStart/emitToolExecuteEnd (-> Config.Hooks.Emit) and +// the ToolExecuteBefore/ToolExecuteAfter/ExecuteTool hook dispatches: +// plugin.Host.Emit enqueues onto a per-plugin-instance channel (safe +// for concurrent senders) and a dedicated per-instance goroutine drains +// it in RECEIPT order — see Host.Emit's own doc comment. Receipt order +// across DIFFERENT call ids is whatever order concurrent callers +// enqueued in (i.e., completion order for that hook type), same +// acceptance as the mcp.tools_selected note below. Host.ExecuteTool and +// the dispatchChain-based hooks each open their own request over the +// shared conn, keyed by a fresh atomic request id (conn.call, protocol.go) +// and serialized only at the write (conn.wmu) — safe for concurrent +// calls, ordinary RPC client behavior. +// +// - s.toolExecCount++ (engine.go): already under s.mu — verified at the +// call site. No change needed. +// +// - maybeRetainToolResult (toolresult.go): MUST run at the JOIN, in call +// order, over the whole batch — not per-call. It is internally +// s.mu-guarded, but two of its effects are only correct when called in +// call order for one batch: +// 1. writeRetainedToolResult mints the next handle from +// s.toolResultNextID and journals one durable record per +// retention (toolresult.go). Concurrent retention would make +// handle numbers (trh_N) and their journal order depend on +// completion order instead of call order, an observable, +// nondeterministic transcript. +// 2. The per-session retained-bytes ceiling check +// (maybeRetainToolResult's `used+len(masked) > cap` branch) reads +// s.toolResultBytes and compares it OUTSIDE the lock that later +// writes it back (writeRetainedToolResult's own separate +// acquisition) — a check-then-act split across two s.mu sections. +// Two concurrent retentions can each observe the ceiling as not +// yet crossed and both proceed, when only one should have. +// Running retention at the join, sequentially in call order, closes +// both: handle numbering and journal order become call-order again, +// and the ceiling check-then-act is never concurrent with itself. +// This changes nothing observable about EventToolEnd: that event +// already carries the PRE-retention output today (retention happens +// in the old runToolCalls, one level above runToolCall — see git +// history), so moving retention's OWN call site later, to the join, +// is not a new event-ordering change. An intra-batch handle +// dependency (one call's arguments naming a handle another call in +// the SAME batch is about to mint) is structurally impossible: the +// model can only ever learn a handle from a PREVIOUS turn's tool +// result, which by definition is not part of the batch currently +// executing. +// +// - markMCPToolsSelected (mcp_lazy.go): already mutates and journals +// under s.mu — verified at the call site. Under a parallel batch, the +// only new nondeterminism is the ORDER of mcp.tools_selected records +// across sibling calls in one batch, which becomes completion-order +// instead of call-order. Accepted: each record names its own tool by +// value (there is no ordering-sensitive accumulation — see +// markMCPToolsSelected's `if s.mcpSelected[name] { continue }` check), +// so two records landing in either relative order describe the exact +// same eventual selected set. This is NOT the same class of bug as +// the retention ceiling above, where relative order changes the +// OUTCOME (which call wins the ceiling), not just the record order. +// +// - task (task_tool.go): NOT Serial, and correctly so — a spawn hands +// the child to the SessionManager and returns. But its verbs +// (cancel/send/status/log) name a TARGET descendant, and two calls +// naming one descendant reordered by completion order change the +// outcome: a cancel(X) then send(X) executed as send-then-cancel +// delivers a message to a still-running child and then kills both. +// Fixed by keying, not by a barrier: taskToolKey keys on session_id, +// so same-target calls run in call order while different targets and +// all spawns stay parallel. SessionManager's own state is +// independently mutex-guarded; this is about ORDER, not about a race. +// +// - read_tool_result (toolresult_tool.go): reads s.toolResults through +// lookupToolResult/openRetainedToolResult, both s.mu-guarded, and +// writes nothing. maybeRetainToolResult exempts its output from +// re-retention by tool name, which is a pure read of the name. Safe +// for concurrent calls, no key needed: a handle's sidecar file is +// immutable once written. +// +// - session_info (session_info.go): reads s.mu-guarded counters plus +// toolDefs and Plugins, and writes nothing. Concurrent with a batch +// it reports a mid-batch snapshot (a toolExecCount that a sibling +// call is still incrementing), which is inherent to asking a session +// about itself while it works and was already true of a `task` +// child's concurrent turn. Safe. +// +// - s.tools (the built-in tool registry map): written only by +// newSession, LoadSession, and SessionManager's adopt/spawn paths +// (store.go, session_manager.go) — every one of those runs BEFORE a +// session is exposed to any concurrent turn, never during one. +// Verified: no write site is reachable from runToolCall or anything +// it calls. Concurrent READS during a batch (executeTool's +// `s.tools[tc.Name]` lookup) are therefore safe with no lock — an +// unsynchronized read of a map nobody is writing. +// +// - s.cfg.WorkDir / s.resolvePath: WorkDir is set once at construction +// and never mutated; resolvePath (filetools.go) is a pure function of +// it plus the call's own argument. Safe to call concurrently. +// +// - s.cfg.MCP.CallTool (mcp.go): CallTool takes MCPManager's RWMutex only +// for the binding lookup (RLock) and delegates the actual call to +// callTool, which opens its own request over the connected client — +// ordinary concurrent-safe client usage. Safe for concurrent calls. +// Note that a namespaced mcp__server__tool call is NOT a Serial +// barrier: only the built-in `mcp` tool is, because only it mutates +// the session's own schema selection. Two calls to one MCP SERVER +// therefore overlap. That is safe on our side for the same two +// reasons the plugin transport is: mcp/conn.go is id-multiplexed +// (nextID plus a pending map, responses routed by id), and the +// MCPManager is a per-process singleton shared by every session +// (cmd/harness builds one), so concurrent calls to one server already +// happen today across sessions. A third-party server that cannot +// answer two requests at once is non-conformant with JSON-RPC's own +// id correlation; HARNESS_SEQUENTIAL_TOOLS=1 is the operator's answer +// if one turns up. +// +// # Design +// +// A batch is one assistant message's ToolCall parts, in order. splitBatch +// walks them and cuts a new segment at each Serial call (its own +// single-element segment) and at each run of non-Serial calls (a parallel +// segment). Segments run in order; a segment completes fully — every call +// in it has a result — before the next segment starts. This is the +// "barrier" semantic: everything before a Serial call has already +// finished, and nothing after it starts until it returns. +// +// Within one parallel segment, up to s.toolConcurrency calls run at once, +// via a bounded worker pool. Results land in a slice indexed by the call's +// position in the whole batch (not just its segment), so the join can +// walk the batch once, in order, regardless of which segment or worker +// produced which result. +// +// # Per-key mutual exclusion invariant +// +// Two calls in one batch that share a non-empty Key must never run +// concurrently, and must run in CALL order (the first one queued must be +// the first one to acquire the key) — a plain sync.Mutex does not +// guarantee the second property, since Go's Mutex is not FIFO under +// contention. keyChain implements an explicit hand-off baton instead: the +// Nth call for a key waits on a channel the (N-1)th call closes when it +// finishes, and creates the channel the (N+1)th call will wait on before +// releasing its own turn. This is built and wired up FRONT-TO-BACK, before +// any worker goroutine starts, precisely so waiting for a predecessor's +// baton can never itself block on the worker pool (see below). +// +// Invariant, written down before implementation per AGENTS.md: a same-key +// call's wait for its predecessor must NEVER depend on that predecessor +// having already acquired a worker-pool slot. If the wait were expressed +// as "block until the predecessor's goroutine starts running", and the +// predecessor is itself still queued behind the concurrency cap, a +// same-key successor that already holds a slot would block forever inside +// it — starving the pool of the one slot the predecessor needs to make +// progress, a classic self-deadlock. The fix: keyChain hands out every +// call's baton channel synchronously, on the submitting goroutine, for the +// WHOLE segment before any worker begins running calls. A worker that +// dequeues a call only waits on a channel that already exists and will be +// closed by whichever call — running now, or still queued — owns it; it +// never waits on a goroutine that has not been scheduled yet to CREATE +// that channel. Waiting for a predecessor's completion therefore never +// competes with that predecessor for a pool slot: the predecessor, once it +// does get a slot, runs and closes the channel independent of who else is +// waiting on it. +// +// # Residuals of the per-file key +// +// filePathKey covers read_file, write_file and edit_file. Two tool +// shapes sit OUTSIDE that namespace, and an operator relying on the +// same-file guarantee should know both. +// +// 1. bash. bashTool (bash.go) sets neither Key nor Serial, and it runs +// arbitrary shell: "echo x > f.txt" and "cat f.txt" touch files the +// engine cannot see. A batch that pairs a bash write with an +// edit_file on the same path, or two bash calls on one file, races on +// disk. A bash call's file targets are not statically knowable, so +// keying it generally is not possible, and keying it pessimistically +// (one global bash key) would serialize the most common parallel +// workload there is. bash stays parallel by design: the model owns +// batching judgment, which is the same contract Claude Code ships. +// The strictly-sequential default never exposed this, so it IS a +// behavior change for the default configuration, and +// HARNESS_SEQUENTIAL_TOOLS=1 is the operator's answer for a workload +// that cannot tolerate it. +// 2. A HARD link. filePathKey resolves symlinks (see +// canonicalFileKeyPath), so a symlinked alias keys correctly, but two +// hard links to one inode have no link to follow. Closing that needs +// an inode comparison against every other key in the batch, which is +// quadratic and still races a file created mid-batch. +// +// # Cancellation and the orphan-result invariant +// +// ctx cancellation (an aborted turn) cancels every in-flight call's own +// context, but every call still yields exactly one ToolResult — the +// NEP-5272 invariant (AGENTS.md's "empty tool result" rule, and the +// orphan tool_use rule generally) holds regardless of how the batch ends. +// A call whose ctx is already cancelled before it starts still runs +// (executeTool/runToolCall are unchanged; a cancelled context is not a +// license to skip a call, only a signal the call's own logic may check — +// bash, for instance, already turns ctx.Err() into a captured-output +// result, not a skip). This mirrors runToolCall's pre-existing contract: +// this package does not add a new cancellation check that could produce +// zero results for a call. +package engine + +import ( + "context" + "encoding/json" + "fmt" + "sync" + + "github.com/majorcontext/harness/message" +) + +// defaultToolConcurrency is Config.ToolConcurrency's zero-value default: +// the cap on how many of one batch's tool calls run at once when the +// operator has not set an explicit value. Chosen so a wide but ordinary +// batch (a handful of reads, a few greps) runs fully in parallel while a +// pathological one (20 globs) cannot fork-bomb a small (2 vCPU) box. +const defaultToolConcurrency = 8 + +// resolveToolConcurrency implements Config.ToolConcurrency's precedence: +// explicit value wins (clamped to a sane floor), zero (unset) is the +// package default. See Config.ToolConcurrency's doc comment for the full +// contract; this mirrors resolveContextWindow's shape (context_window.go). +func resolveToolConcurrency(explicit int) int { + switch { + case explicit == 0: + return defaultToolConcurrency + case explicit < 0: + // A negative value is not "unlimited" — clamp to the strictly + // sequential floor rather than silently misinterpreting the + // operator's intent. + return 1 + default: + return explicit + } +} + +// runToolBatch is runToolCalls' actual implementation (see engine.go), +// split out here to keep this package's concurrency logic in one file. +// It executes every ToolCall part of asst and returns their ToolResult +// parts, one per call, in CALL order — regardless of completion order, +// concurrency, or partial failure. +func (s *Session) runToolBatch(ctx context.Context, asst *message.Message) message.Parts { + calls := toolCallsOf(asst) + if len(calls) == 0 { + return nil + } + + outputs := make([]message.Parts, len(calls)) + errs := make([]bool, len(calls)) + // done is the ONE-RESULT-PER-CALL ledger. A result slot exists for + // every call before anything is scheduled, and every execution path + // marks its own slot. The backfill below then turns "no path marked + // this slot" into a real error result instead of an orphan tool_use. + // outputs[i] alone cannot serve as the ledger: a tool may legitimately + // return nil parts. + done := make([]bool, len(calls)) + + if s.toolConcurrency <= 1 { + // Sequential mode: one call at a time, in call order. A Serial + // tool's barrier is a no-op here since nothing ever runs beside + // it, and per-key exclusion is likewise moot with one call in + // flight at a time. + // + // This is the pre-parallel ORDER, not the pre-parallel behavior in + // every respect. Two guarantees this file adds apply here too, by + // design: runOneGuarded turns a panicking tool into one error + // result instead of killing the process, and admitAndRun refuses + // to start a call after the turn is canceled, where the old loop + // ran every remaining call unconditionally. An operator who sets + // ToolConcurrency 1 to escape concurrency gets exactly that — + // serial execution — not a revert of the whole change. + for i, tc := range calls { + outputs[i], errs[i] = s.runOneGuarded(ctx, tc) + done[i] = true + } + } else { + s.runToolBatchParallel(ctx, calls, outputs, errs, done) + } + + // Backfill: no call may leave this function without a result. See + // "Cancellation and the orphan-result invariant" in the package doc + // comment. Nothing reaches this loop today — every path above marks + // its slot — and that is the point: it is the structural guarantee, + // not a live code path. + for i := range calls { + if !done[i] { + outputs[i] = message.Parts{&message.Text{Text: toolCallNoResultText}} + errs[i] = true + } + } + + // The join. This is retention's single call site + // (maybeRetainToolResult, toolresult.go): an oversized TEXT result is + // swapped for a preview plus a trh_N handle HERE, before the + // ToolResult is built and long before Session.append, + // message.NormalizeForWire, or any transcoder sees it. That placement + // is why tool-result handles need no wire-format change at all — every + // downstream layer still sees an ordinary ToolResult carrying ordinary + // Text parts. Retention runs on the post-hook output (runToolCall + // already applied ToolExecuteAfter), so a plugin that rewrites or + // enlarges a result has its final bytes measured, not the tool's + // originals. It is a total no-op when retention is disabled or the + // result is within the limit. + // + // Running it here, on the join goroutine, in call order, is also what + // keeps maybeRetainToolResult SINGLE-THREADED. That matters beyond + // handle numbering: its per-session retained-bytes ceiling is a + // check-then-act split across two separate s.mu sections (it reads + // s.toolResultBytes, unlocks, writes the sidecar, then adds the bytes + // back under a second acquisition). Concurrent callers could each + // observe the ceiling as uncrossed and all proceed, overshooting the + // configured cap by up to the concurrency factor. The join removes the + // concurrency instead of re-locking the ceiling: with one caller there + // is no window to race. Do NOT move retention back inside the worker + // without first making that reserve-and-mint one atomic section. + results := make(message.Parts, len(calls)) + for i, tc := range calls { + results[i] = &message.ToolResult{ + CallID: tc.CallID, + Content: s.maybeRetainToolResult(tc.Name, outputs[i]), + IsError: errs[i], + } + } + return results +} + +// These are the three synthesized results the executor produces when a +// tool did not return a normal result: the call was refused after the +// turn was canceled, no execution path filled its slot, or the tool +// panicked. All three exist to hold the pairing invariant: a tool_use +// block with no tool_result wedges a session permanently (AGENTS.md, +// NEP-5272). +const ( + toolCallCanceledText = "tool call not started: the turn was canceled" + toolCallNoResultText = "tool call produced no result" + toolCallPanicText = "tool call failed: the tool panicked" +) + +// admitAndRun is the ONE admission gate every call passes through. A call +// admitted while ctx is already canceled does not run: it returns a +// synthesized canceled result instead. +// +// This deliberately CHANGES the pre-parallel behavior, which ran every +// remaining call after an abort. Two reasons. A batch has calls queued +// behind the concurrency cap, so an abort that arrives early would +// otherwise keep starting fresh work for as long as the batch is wide. +// And several built-ins commit their side effect without consulting ctx +// at all — write_file writes the file — so "the tool decides" is not a +// real gate for them. The turn is over, so the only thing still-starting +// work can add is a side effect nobody asked for. +// +// A canceled turn's results are NOT discarded, and an earlier version of +// this comment wrongly said they were. runToolCalls returns the +// synthesized results, runAgenticLoop appends the whole RoleTool message +// to DURABLE history, and they survive a resume. That makes the gate more +// important, not less: what lands in the log is a legible "not started" +// result for each refused call, instead of a side effect the operator +// aborted to prevent. +// +// A call ALREADY RUNNING when the abort lands is not interrupted here: it +// owns its own ctx and returns whatever it returns. This gate governs +// admission only. +// +// A refused call emits NO tool events. Both EventToolStart and +// EventToolEnd fire inside runToolCall (engine.go), which this gate +// short-circuits, so an aborted batch's refused calls appear in the +// transcript as tool_result parts with no matching event pair. That is +// deliberate — the events describe an execution that never happened — and +// it is safe because an aborted turn's event stream is already incomplete +// by definition. The transcript, which the provider validates, still +// pairs every tool_use with a tool_result. +func (s *Session) admitAndRun(ctx context.Context, tc *message.ToolCall) (message.Parts, bool) { + if ctx.Err() != nil { + return message.Parts{&message.Text{Text: toolCallCanceledText}}, true + } + return s.runToolCall(ctx, tc) +} + +// runOneGuarded is the single execution wrapper every path uses. It turns a +// PANIC inside a tool, or inside a hook the tool's dispatch calls, into one +// ordinary error result. +// +// Without it the one-result-per-call guarantee is not true. A panic in a +// worker goroutine cannot be recovered by the join, so it takes the whole +// process down and the batch's other results die with it — and the +// assistant message's tool_use blocks are already in history, so the next +// load of that session meets unanswered tool calls. A recovered panic +// keeps the session honest instead: the model gets a real error result for +// the call that failed, and its siblings still return their own. +// +// This also changes sequential mode, which previously let a tool panic +// unwind through Prompt. That is deliberate: the guarantee must not depend +// on which execution mode a session runs in. +func (s *Session) runOneGuarded(ctx context.Context, tc *message.ToolCall) (out message.Parts, isErr bool) { + defer func() { + if r := recover(); r != nil { + out = message.Parts{&message.Text{Text: fmt.Sprintf("%s: %v", toolCallPanicText, r)}} + isErr = true + } + }() + return s.admitAndRun(ctx, tc) +} + +// toolCallsOf extracts asst's ToolCall parts in order. +func toolCallsOf(asst *message.Message) []*message.ToolCall { + var calls []*message.ToolCall + for _, p := range asst.Parts { + if tc, ok := p.(*message.ToolCall); ok { + calls = append(calls, tc) + } + } + return calls +} + +// batchSegment is one contiguous run of calls executed as a unit: either a +// single Serial call, or a run of non-Serial calls that execute in +// parallel. idx holds each call's ORIGINAL index into the whole batch, so +// results can be written back to the right slot regardless of segment +// boundaries. +type batchSegment struct { + idx []int + calls []*message.ToolCall + serial bool +} + +// splitBatch walks calls in order and cuts a new segment at each Serial +// call — its own one-element segment — and at each run of non-Serial +// calls. See the package doc comment's "Design" section. +func (s *Session) splitBatch(calls []*message.ToolCall) []batchSegment { + var segs []batchSegment + var cur batchSegment + flush := func() { + if len(cur.calls) > 0 { + segs = append(segs, cur) + cur = batchSegment{} + } + } + for i, tc := range calls { + if s.toolIsSerial(tc.Name) { + flush() + segs = append(segs, batchSegment{idx: []int{i}, calls: []*message.ToolCall{tc}, serial: true}) + continue + } + cur.idx = append(cur.idx, i) + cur.calls = append(cur.calls, tc) + } + flush() + return segs +} + +// toolIsSerial reports whether name names a built-in Tool with Serial set. +// A plugin or MCP tool is never Serial — only a built-in Tool carries the +// flag (see the Tool struct's doc comment). +func (s *Session) toolIsSerial(name string) bool { + t, ok := s.tools[name] + return ok && t.Serial +} + +// toolKey computes name's resource key for one call's args, or "" if the +// tool has no Key func. See the Tool struct's Key field doc comment. +func (s *Session) toolKey(name string, args json.RawMessage) (key string) { + t, ok := s.tools[name] + if !ok || t.Key == nil { + return "" + } + // A panicking Key runs on the SUBMITTING goroutine, before any result + // slot is filled, so it would take down the batch with no results at + // all. Fall back to one shared per-tool key instead: conservative, + // because every call whose key could not be computed then serializes + // with every other one, exactly like filePathKey's own unparsed + // fallback. Never fall back to "" — that would silently drop the + // exclusion the tool asked for. + defer func() { + if r := recover(); r != nil { + key = "panicking-key:" + name + } + }() + return t.Key(s, args) +} + +// runToolBatchParallel executes calls' segments in order, filling outputs/ +// errs by original batch index. Caller has already checked +// s.toolConcurrency > 1. +func (s *Session) runToolBatchParallel(ctx context.Context, calls []*message.ToolCall, outputs []message.Parts, errs []bool, done []bool) { + for _, seg := range s.splitBatch(calls) { + if seg.serial { + i := seg.idx[0] + outputs[i], errs[i] = s.runOneGuarded(ctx, seg.calls[0]) + done[i] = true + continue + } + if len(seg.calls) == 1 { + // One-call fast path. A parallel segment of exactly one call + // is the COMMON shape — a turn with a single tool call, or a + // lone call between two Serial ones — and building a + // goroutine, a WaitGroup, a semaphore channel and a keyChain + // for it buys nothing: there is no sibling to run beside, and + // no sibling to exclude. Running it inline is what the serial + // branch above already does. + i := seg.idx[0] + outputs[i], errs[i] = s.runOneGuarded(ctx, seg.calls[0]) + done[i] = true + continue + } + s.runParallelSegment(ctx, seg, outputs, errs, done) + } +} + +// keyChain is the per-key hand-off chain for one segment: the channel a +// waiting call blocks on, closed by its predecessor when done, and the +// channel the NEXT same-key call will wait on in turn. +// It carries no lock. wait is called only from runParallelSegment's setup +// loop, which runs entirely on the submitting goroutine before any worker +// starts; a worker only closes its own channel and reads its own +// predecessor. A mutex here would imply a concurrency contract that does +// not exist. If a future change ever calls wait from a worker, add the +// lock back with it. +type keyChain struct { + tail map[string]chan struct{} +} + +// wait registers this call as the next holder of key and returns a +// release func to call when the work is done. It returns immediately +// (nil predecessor channel) for the first call on a key. See the package +// doc comment's "Per-key mutual exclusion invariant" for why this hand-off +// is built synchronously, before any worker runs. +func (c *keyChain) wait(key string) (predecessor <-chan struct{}, release func()) { + predecessor = c.tail[key] + mine := make(chan struct{}) + if c.tail == nil { + c.tail = make(map[string]chan struct{}) + } + c.tail[key] = mine + return predecessor, func() { close(mine) } +} + +// runParallelSegment runs one non-Serial segment's calls with up to +// s.toolConcurrency in flight, honoring per-key exclusion. Every call gets +// exactly one result, in outputs/errs at its ORIGINAL batch index, even if +// ctx is already cancelled. +func (s *Session) runParallelSegment(ctx context.Context, seg batchSegment, outputs []message.Parts, errs []bool, done []bool) { + // Baton hand-off is wired up front, on THIS goroutine, for every call + // in the segment before any worker starts — see the invariant in the + // package doc comment. + var chain keyChain + type job struct { + idx int + tc *message.ToolCall + key string + predecessor <-chan struct{} + release func() + } + jobs := make([]job, len(seg.calls)) + for i, tc := range seg.calls { + key := s.toolKey(tc.Name, tc.Arguments) + var pred <-chan struct{} + var rel func() + if key != "" { + pred, rel = chain.wait(key) + } + jobs[i] = job{idx: seg.idx[i], tc: tc, key: key, predecessor: pred, release: rel} + } + + // The pool bounds EXECUTION, not goroutine count. Each job gets its + // goroutine immediately; that goroutine waits for its baton FIRST and + // only then takes a pool slot. + // + // The order matters, and an earlier cut had it the other way round + // (slot taken on the submitting goroutine, baton awaited inside). That + // shape lets a same-key waiter sit on a slot it cannot use while an + // unrelated later call is refused admission — head-of-line blocking + // that scales with how many same-key calls a batch carries. It could + // not deadlock, because a predecessor is always submitted earlier and + // so always holds a slot before its successor is even created, but a + // slot held by a goroutine that is only waiting is pure waste. Waiting + // for the baton before admission removes the whole class. + // + // A batch is bounded by one assistant message, so the goroutine count + // is bounded too, and a parked goroutine costs a few kilobytes. + slots := min(s.toolConcurrency, len(jobs)) + sem := make(chan struct{}, slots) + var wg sync.WaitGroup + for _, j := range jobs { + wg.Add(1) + go func(j job) { + defer wg.Done() + // Release the baton on the way out, whatever happens inside: + // a same-key successor must never wait forever on a + // predecessor that died. Each call closes only its OWN + // channel, exactly once. + if j.release != nil { + defer j.release() + } + if j.predecessor != nil { + <-j.predecessor + } + sem <- struct{}{} + defer func() { <-sem }() + outputs[j.idx], errs[j.idx] = s.runOneGuarded(ctx, j.tc) + done[j.idx] = true + }(j) + } + wg.Wait() +} diff --git a/engine/toolexec_test.go b/engine/toolexec_test.go new file mode 100644 index 00000000..3094f794 --- /dev/null +++ b/engine/toolexec_test.go @@ -0,0 +1,1444 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" + "github.com/majorcontext/harness/provider" +) + +// Tests for the concurrent tool-call batch executor (toolexec.go). +// +// Concurrency is proved by RENDEZVOUS, never by timing: a fake tool blocks +// until a sibling has entered, so an executor that runs one call at a time +// cannot make progress. Those tests run inside a testing/synctest bubble, +// where "cannot make progress" is reported at once as a deadlock instead of +// hanging on the wall clock. Wall-clock claims are measured with the +// bubble's FAKE clock, where a time.Sleep costs nothing and elapsed time is +// exact. + +// batchTool builds a fake built-in tool named name whose Run calls fn. The +// tool echoes name back as its output, so a result can be traced to the +// call that produced it. +func batchTool(name string, fn func(ctx context.Context, call string)) Tool { + return Tool{ + Def: provider.ToolDef{ + Name: name, + Description: "test tool", + InputSchema: json.RawMessage(`{"type":"object"}`), + }, + Run: func(ctx context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + if fn != nil { + fn(ctx, in.Call) + } + return message.Parts{&message.Text{Text: in.Call}}, nil + }, + } +} + +// batchCall builds one ToolCall for tool name, tagged with call so the +// fake tool and the assertions can identify it. +func batchCall(name, call string) *message.ToolCall { + return &message.ToolCall{ + CallID: "tc_" + call, + Name: name, + Arguments: json.RawMessage(fmt.Sprintf(`{"call":%q}`, call)), + } +} + +func asstWith(calls ...*message.ToolCall) *message.Message { + parts := make(message.Parts, len(calls)) + for i, c := range calls { + parts[i] = c + } + return &message.Message{ID: "msg_a", Role: message.RoleAssistant, Parts: parts} +} + +// resultOrder returns each ToolResult's CallID, in the order the executor +// returned them. It also fails the test if any part is not a ToolResult. +func resultOrder(t *testing.T, parts message.Parts) []string { + t.Helper() + ids := make([]string, len(parts)) + for i, p := range parts { + tr, ok := p.(*message.ToolResult) + if !ok { + t.Fatalf("results[%d] = %T, want *message.ToolResult", i, p) + } + ids[i] = tr.CallID + } + return ids +} + +// wantOrder asserts the results pair one-to-one with calls, in call order. +// It checks BOTH directions: no call is missing a result, and no result is +// a duplicate or a surplus. +func wantOrder(t *testing.T, parts message.Parts, calls ...*message.ToolCall) { + t.Helper() + if len(parts) != len(calls) { + t.Fatalf("results = %d, want exactly %d (one per tool call)", len(parts), len(calls)) + } + got := resultOrder(t, parts) + seen := make(map[string]int, len(got)) + for i, c := range calls { + if got[i] != c.CallID { + t.Errorf("results[%d] call id = %q, want %q (results must join in CALL order)", i, got[i], c.CallID) + } + seen[got[i]]++ + } + for id, n := range seen { + if n != 1 { + t.Errorf("call id %q appears %d times in the results, want exactly 1", id, n) + } + } +} + +// TestBatchWallClockIsMaxNotSum proves the batch runs concurrently: six +// tools that each take three seconds finish the batch in three seconds, +// not eighteen. +// +// The measurement runs inside a synctest bubble, so time is FAKE and the +// elapsed value is exact rather than approximate — a sanctioned time +// mechanism, unlike a real sleep (see AGENTS.md, Testing). +func TestBatchWallClockIsMaxNotSum(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const n = 6 + const each = 3 * time.Second + + s := NewSession(Config{}) + s.tools["slow"] = batchTool("slow", func(context.Context, string) { + time.Sleep(each) + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("slow", fmt.Sprintf("c%d", i)) + } + + start := time.Now() + results := s.runToolCalls(context.Background(), asstWith(calls...)) + elapsed := time.Since(start) + + wantOrder(t, results, calls...) + if elapsed != each { + t.Fatalf("batch of %d took %v, want %v (the longest call, not the sum %v)", + n, elapsed, each, n*each) + } + }) +} + +// TestBatchResultsJoinInCallOrderUnderReversedCompletion proves the join is +// order-stable. The tools are released in REVERSE call order, so completion +// order is exactly the opposite of call order, and the results must still +// come back in call order. +// +// Reading every entry of entered before releasing anything also proves all +// five calls were in flight together. +func TestBatchResultsJoinInCallOrderUnderReversedCompletion(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const n = 5 + + entered := make(chan string, n) + release := make([]chan struct{}, n) + for i := range release { + release[i] = make(chan struct{}) + } + index := map[string]int{} + for i := range n { + index[fmt.Sprintf("c%d", i)] = i + } + + s := NewSession(Config{}) + s.tools["held"] = batchTool("held", func(_ context.Context, call string) { + entered <- call + <-release[index[call]] + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("held", fmt.Sprintf("c%d", i)) + } + + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + + for range n { + <-entered + } + var completion []string + for i := n - 1; i >= 0; i-- { + close(release[i]) + completion = append(completion, fmt.Sprintf("c%d", i)) + synctest.Wait() + } + synctest.Wait() + + if completion[0] != "c4" { + t.Fatalf("completion order = %v, want it reversed", completion) + } + wantOrder(t, results, calls...) + }) +} + +// TestBatchSequentialModeNeverOverlaps proves Config.ToolConcurrency 1 +// restores the pre-parallel path: no two calls are ever in flight at once, +// and they run in call order. +// +// This test deliberately uses NO rendezvous — a rendezvous is exactly what +// a sequential executor cannot satisfy. It observes the in-flight counter +// instead. +func TestBatchSequentialModeNeverOverlaps(t *testing.T) { + const n = 5 + + var mu sync.Mutex + inFlight, maxInFlight := 0, 0 + var order []string + + s := NewSession(Config{ToolConcurrency: 1}) + s.tools["counted"] = batchTool("counted", func(_ context.Context, call string) { + mu.Lock() + inFlight++ + if inFlight > maxInFlight { + maxInFlight = inFlight + } + order = append(order, call) + mu.Unlock() + mu.Lock() + inFlight-- + mu.Unlock() + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("counted", fmt.Sprintf("c%d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + + wantOrder(t, results, calls...) + if maxInFlight != 1 { + t.Errorf("max calls in flight = %d, want 1 in sequential mode", maxInFlight) + } + for i, call := range order { + if want := fmt.Sprintf("c%d", i); call != want { + t.Errorf("execution order[%d] = %q, want %q", i, call, want) + } + } +} + +// TestBatchConcurrencyCapIsEnforced proves the cap both BOUNDS the batch +// and is REACHED. Nine calls run with a cap of three, and every call +// blocks inside the tool. synctest.Wait then returns only once every +// goroutine in the bubble is durably blocked, so the number of calls +// inside the tool at that moment is the true in-flight maximum: exactly +// three. An executor that ignored the cap would have all nine inside. +// +// Counting a running maximum instead would NOT prove this. A barrier that +// releases each wave lets an unbounded pool look bounded, because the +// early calls can exit before the later ones enter. +func TestBatchConcurrencyCapIsEnforced(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const n, capacity = 9, 3 + + entered := make(chan string, n) + releaseAll := make(chan struct{}) + + s := NewSession(Config{ToolConcurrency: capacity}) + s.tools["capped"] = batchTool("capped", func(_ context.Context, call string) { + entered <- call + <-releaseAll + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("capped", fmt.Sprintf("c%d", i)) + } + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + + synctest.Wait() + if got := len(entered); got != capacity { + t.Fatalf("%d calls were inside the tool at once, want exactly %d (the cap)", got, capacity) + } + + close(releaseAll) + synctest.Wait() + + wantOrder(t, results, calls...) + if got := len(entered); got != n { + t.Errorf("%d calls ran in total, want %d", got, n) + } + }) +} + +// TestBatchSerialToolIsABarrierOnBothSides proves a Serial tool splits the +// batch. The batch is [p0, p1, barrier, p2, p3]. p0 and p1 rendezvous with +// each other, and p2 with p3, so each parallel run must genuinely overlap +// or the bubble deadlocks. The recorded log must then show both leading +// calls finished before the barrier started, and the barrier finished +// before either trailing call started. +func TestBatchSerialToolIsABarrierOnBothSides(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var mu sync.Mutex + var log []string + record := func(s string) { + mu.Lock() + log = append(log, s) + mu.Unlock() + } + + lead := make(chan struct{}, 2) + leadGo := make(chan struct{}) + trail := make(chan struct{}, 2) + trailGo := make(chan struct{}) + + s := NewSession(Config{ToolConcurrency: 8}) + s.tools["par"] = batchTool("par", func(_ context.Context, call string) { + record("enter " + call) + switch call { + case "p0", "p1": + lead <- struct{}{} + <-leadGo + default: + trail <- struct{}{} + <-trailGo + } + record("exit " + call) + }) + barrier := batchTool("barrier", func(context.Context, string) { + record("enter b") + record("exit b") + }) + barrier.Serial = true + s.tools["barrier"] = barrier + + calls := []*message.ToolCall{ + batchCall("par", "p0"), batchCall("par", "p1"), + batchCall("barrier", "b"), + batchCall("par", "p2"), batchCall("par", "p3"), + } + + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + + // Both leading calls must be inside before either may finish. + <-lead + <-lead + close(leadGo) + // Both trailing calls must be inside before either may finish. + <-trail + <-trail + close(trailGo) + synctest.Wait() + + wantOrder(t, results, calls...) + + mu.Lock() + defer mu.Unlock() + pos := func(entry string) int { + for i, e := range log { + if e == entry { + return i + } + } + t.Fatalf("log has no %q entry: %v", entry, log) + return -1 + } + for _, before := range []string{"exit p0", "exit p1"} { + if pos(before) > pos("enter b") { + t.Errorf("%q happened after the barrier started: %v", before, log) + } + } + for _, after := range []string{"enter p2", "enter p3"} { + if pos(after) < pos("exit b") { + t.Errorf("%q happened before the barrier finished: %v", after, log) + } + } + }) +} + +// TestBatchSameKeyCallsSerializeInCallOrder proves per-key exclusion. Three +// calls share one path key and one call uses a different path. The +// same-key calls must never overlap and must run in call order; the +// different-key call must overlap with the first same-key call, which the +// rendezvous forces (a wholly serialized executor deadlocks the bubble). +func TestBatchSameKeyCallsSerializeInCallOrder(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var mu sync.Mutex + inFlightSameKey := 0 + var keyOrder []string + overlap := make(chan struct{}) + otherIn := make(chan struct{}) + + keyed := func(name string) Tool { + tool := batchTool(name, nil) + tool.Key = filePathKey + return tool + } + + s := NewSession(Config{WorkDir: "/w", ToolConcurrency: 8}) + same := keyed("same") + same.Run = func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + mu.Lock() + inFlightSameKey++ + if inFlightSameKey > 1 { + t.Errorf("two same-key calls ran at once (%q)", in.Call) + } + keyOrder = append(keyOrder, in.Call) + first := len(keyOrder) == 1 + mu.Unlock() + if first { + // The first same-key call waits for the OTHER-key call, + // so the two keys must genuinely run side by side. + <-otherIn + } + mu.Lock() + inFlightSameKey-- + mu.Unlock() + return message.Parts{&message.Text{Text: in.Call}}, nil + } + s.tools["same"] = same + other := keyed("other") + other.Run = func(context.Context, *Session, json.RawMessage) (message.Parts, error) { + close(otherIn) + <-overlap + return message.Parts{&message.Text{Text: "other"}}, nil + } + s.tools["other"] = other + + mkCall := func(name, call, path string) *message.ToolCall { + return &message.ToolCall{ + CallID: "tc_" + call, + Name: name, + Arguments: json.RawMessage(fmt.Sprintf(`{"call":%q,"path":%q}`, call, path)), + } + } + calls := []*message.ToolCall{ + mkCall("same", "s0", "shared.txt"), + mkCall("same", "s1", "shared.txt"), + mkCall("other", "o0", "unrelated.txt"), + mkCall("same", "s2", "shared.txt"), + } + + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + + <-otherIn + close(overlap) + synctest.Wait() + + wantOrder(t, results, calls...) + mu.Lock() + defer mu.Unlock() + want := []string{"s0", "s1", "s2"} + if len(keyOrder) != len(want) { + t.Fatalf("same-key execution order = %v, want %v", keyOrder, want) + } + for i, c := range want { + if keyOrder[i] != c { + t.Errorf("same-key execution order = %v, want %v (call order)", keyOrder, want) + break + } + } + }) +} + +// TestFileToolsShareOnePathKeyNamespace is the production wiring check for +// the key itself: read_file, write_file and edit_file must resolve ONE key +// per file, so a read racing a write or an edit to that file cannot happen. +// It also pins the two normalization rules the key promises. +func TestFileToolsShareOnePathKeyNamespace(t *testing.T) { + s := NewSession(Config{WorkDir: "/work"}) + args := func(path string) json.RawMessage { + return json.RawMessage(fmt.Sprintf(`{"path":%q}`, path)) + } + + base := s.toolKey("edit_file", args("a.txt")) + if base == "" { + t.Fatal("edit_file has no resource key: same-path edits would run concurrently") + } + for _, tool := range []string{"read_file", "write_file"} { + if got := s.toolKey(tool, args("a.txt")); got != base { + t.Errorf("%s key = %q, want %q: all three file tools must share one namespace", tool, got, base) + } + } + if got := s.toolKey("edit_file", args("b.txt")); got == base { + t.Error("two different paths share a key: different files must run concurrently") + } + // Normalization: a dot-dot alias and an absolute spelling of the same + // file must key the same. + if got := s.toolKey("edit_file", args("sub/../a.txt")); got != base { + t.Errorf("key for %q = %q, want %q: a dot-dot alias must not bypass the key", "sub/../a.txt", got, base) + } + if got := s.toolKey("edit_file", args("/work/a.txt")); got != base { + t.Errorf("absolute key = %q, want %q", got, base) + } + // An unparseable call still gets a key, so it cannot slip past + // exclusion entirely. + if got := s.toolKey("edit_file", json.RawMessage(`{`)); got == "" { + t.Error("an unparseable file call has no key: it would run unserialized") + } +} + +// TestEditFileSamePathBatchAppliesInCallOrder drives the REAL file tools +// through the production entry point. Two edits chained on one file +// (a->b, then b->c) can only both succeed if they run in call order +// against each other's output. +func TestEditFileSamePathBatchAppliesInCallOrder(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "f.txt") + if err := os.WriteFile(path, []byte("a"), 0o644); err != nil { + t.Fatal(err) + } + + s := NewSession(Config{WorkDir: dir}) + calls := []*message.ToolCall{ + {CallID: "tc1", Name: "edit_file", Arguments: json.RawMessage(`{"path":"f.txt","old_string":"a","new_string":"b"}`)}, + {CallID: "tc2", Name: "edit_file", Arguments: json.RawMessage(`{"path":"f.txt","old_string":"b","new_string":"c"}`)}, + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + for i, p := range results { + if tr := p.(*message.ToolResult); tr.IsError { + t.Errorf("edit %d failed: %s", i, tr.Content.Text()) + } + } + got, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(got) != "c" { + t.Errorf("file = %q, want %q: the two same-path edits did not apply in call order", got, "c") + } +} + +// TestBatchPartialFailureLetsSiblingsFinish proves one failing call never +// cancels its siblings: every call still returns its own result, and the +// error lands on the right call id. The rendezvous forces the siblings to +// be in flight while the failing call is inside, so this is a real +// concurrent partial failure, not a sequential one. +func TestBatchPartialFailureLetsSiblingsFinish(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + entered := make(chan string, 3) + release := make(chan struct{}) + + s := NewSession(Config{ToolConcurrency: 8}) + s.tools["ok"] = batchTool("ok", func(_ context.Context, call string) { + entered <- call + <-release + }) + s.tools["boom"] = Tool{ + Def: provider.ToolDef{Name: "boom", Description: "fails", InputSchema: json.RawMessage(`{}`)}, + Run: func(context.Context, *Session, json.RawMessage) (message.Parts, error) { + entered <- "boom" + <-release + return nil, fmt.Errorf("deliberate failure") + }, + } + + calls := []*message.ToolCall{ + batchCall("ok", "c0"), + {CallID: "tc_boom", Name: "boom", Arguments: json.RawMessage(`{}`)}, + batchCall("ok", "c2"), + } + + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + for range 3 { + <-entered + } + close(release) + synctest.Wait() + + wantOrder(t, results, calls...) + for i, p := range results { + tr := p.(*message.ToolResult) + wantErr := tr.CallID == "tc_boom" + if tr.IsError != wantErr { + t.Errorf("results[%d] (%s) IsError = %v, want %v", i, tr.CallID, tr.IsError, wantErr) + } + } + }) +} + +// TestBatchCancellationStillYieldsOneResultPerCall is the orphan-pairing +// guard (AGENTS.md, NEP-5272): a tool_use block with no tool_result wedges +// a session forever, so an aborted turn must still produce exactly one +// result per call — never fewer, and never a duplicate. +// +// Two shapes. In "already canceled" no call runs at all. In "canceled in +// flight" every call is inside the tool when the abort lands, and the +// calls queued behind the cap are admitted after it. +func TestBatchCancellationStillYieldsOneResultPerCall(t *testing.T) { + const n = 6 + + t.Run("already canceled", func(t *testing.T) { + var ran int + var mu sync.Mutex + s := NewSession(Config{ToolConcurrency: 2}) + s.tools["never"] = batchTool("never", func(context.Context, string) { + mu.Lock() + ran++ + mu.Unlock() + }) + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("never", fmt.Sprintf("c%d", i)) + } + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + results := s.runToolCalls(ctx, asstWith(calls...)) + + wantOrder(t, results, calls...) + for i, p := range results { + tr := p.(*message.ToolResult) + if !tr.IsError { + t.Errorf("results[%d] is not an error result; a canceled call must say so", i) + } + if got := tr.Content.Text(); got != toolCallCanceledText { + t.Errorf("results[%d] = %q, want %q", i, got, toolCallCanceledText) + } + } + mu.Lock() + defer mu.Unlock() + if ran != 0 { + t.Errorf("%d calls ran after the turn was canceled, want 0", ran) + } + }) + + t.Run("canceled in flight", func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + entered := make(chan struct{}, 2) + + s := NewSession(Config{ToolConcurrency: 2}) + s.tools["waits"] = batchTool("waits", func(ctx context.Context, _ string) { + entered <- struct{}{} + <-ctx.Done() + }) + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("waits", fmt.Sprintf("c%d", i)) + } + + var results message.Parts + go func() { + results = s.runToolCalls(ctx, asstWith(calls...)) + }() + // Both calls the cap admits are inside before the abort. + <-entered + <-entered + cancel() + synctest.Wait() + + wantOrder(t, results, calls...) + }) + }) +} + +// TestBatchJournalCommitsInCallOrderWhileEventsInterleave drives a REAL +// turn through Session.Prompt and asserts the two halves of the approved +// event contract at once. EventToolEnd MAY arrive out of call order — +// events are keyed by call id, so interleaving is expected and correct — +// while the RoleTool message that lands in history MUST be in call order, +// because tool_result order has to match tool_use order on the wire. +func TestBatchJournalCommitsInCallOrderWhileEventsInterleave(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const n = 4 + + entered := make(chan string, n) + release := make([]chan struct{}, n) + for i := range release { + release[i] = make(chan struct{}) + } + index := map[string]int{} + calls := make([]*message.ToolCall, n) + for i := range n { + name := fmt.Sprintf("c%d", i) + index[name] = i + calls[i] = batchCall("held", name) + } + parts := make(message.Parts, n) + for i, c := range calls { + parts[i] = c + } + + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + {{Type: provider.EventDone, Message: &message.Message{ + ID: "msg_a", Role: message.RoleAssistant, Parts: parts, + }, StopReason: provider.StopToolUse}}, + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }} + + var evMu sync.Mutex + var toolEnds []string + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + // OnEvent is called from several goroutines at once now (see + // Config.OnEvent): this collector must lock, and the lock is + // part of what the test documents. + OnEvent: func(ev Event) { + if ev.Type != EventToolEnd { + return + } + evMu.Lock() + toolEnds = append(toolEnds, ev.ToolCall.CallID) + evMu.Unlock() + }, + }) + s.tools["held"] = batchTool("held", func(_ context.Context, call string) { + entered <- call + <-release[index[call]] + }) + + done := make(chan struct{}) + go func() { + defer close(done) + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Errorf("Prompt: %v", err) + } + }() + + for range n { + <-entered + } + // Finish in reverse call order. + for i := n - 1; i >= 0; i-- { + close(release[i]) + synctest.Wait() + } + <-done + + // History: user, assistant(tool calls), tool(results), assistant. + h := s.History() + if len(h) != 4 { + t.Fatalf("history len = %d, want 4: %+v", len(h), h) + } + if h[2].Role != message.RoleTool { + t.Fatalf("history[2] role = %q, want %q", h[2].Role, message.RoleTool) + } + wantOrder(t, h[2].Parts, calls...) + + // The events tell the other half of the story: they came back in + // completion order, which is the REVERSE of call order here. + evMu.Lock() + defer evMu.Unlock() + if len(toolEnds) != n { + t.Fatalf("EventToolEnd count = %d, want %d", len(toolEnds), n) + } + if toolEnds[0] != calls[n-1].CallID { + t.Fatalf("EventToolEnd order = %v; this test needs completion order to differ from call order", toolEnds) + } + }) +} + +// TestBatchRetentionMintsHandlesInCallOrder proves retention runs at the +// JOIN, in call order, and not inside the workers. Three oversized results +// complete in REVERSE call order; their trh_N handles must still be +// numbered by call position. Concurrent retention would number them by +// completion, which makes a transcript depend on scheduling. +func TestBatchRetentionMintsHandlesInCallOrder(t *testing.T) { + dir := t.TempDir() + synctest.Test(t, func(t *testing.T) { + const n = 3 + entered := make(chan string, n) + release := make([]chan struct{}, n) + for i := range release { + release[i] = make(chan struct{}) + } + index := map[string]int{"c0": 0, "c1": 1, "c2": 2} + + s := NewSession(Config{ + SessionDir: dir, + ToolResultInlineBytes: 200, + ToolResultRetainedBytes: 1 << 20, + ToolConcurrency: 8, + }) + body := strings.Repeat("x", 4000) + s.tools["big"] = Tool{ + Def: provider.ToolDef{Name: "big", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + entered <- in.Call + <-release[index[in.Call]] + return message.Parts{&message.Text{Text: in.Call + " " + body}}, nil + }, + } + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("big", fmt.Sprintf("c%d", i)) + } + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + for range n { + <-entered + } + for i := n - 1; i >= 0; i-- { + close(release[i]) + synctest.Wait() + } + synctest.Wait() + + wantOrder(t, results, calls...) + var handles []string + for i, p := range results { + text := p.(*message.ToolResult).Content.Text() + m := toolResultHandleInTextPattern.FindString(text) + if m == "" { + t.Fatalf("results[%d] carries no retention handle: %q", i, text) + } + handles = append(handles, m) + } + want := []string{"trh_1", "trh_2", "trh_3"} + for i := range want { + if handles[i] != want[i] { + t.Fatalf("handles = %v, want %v: retention must mint in CALL order, not completion order", handles, want) + } + } + }) +} + +// TestBatchRetentionCeilingHoldsAcrossOneBatch is the adversarial-review +// finding on the retained-bytes ceiling. Its check-then-act spans two +// separate s.mu sections, so concurrent retention could let several +// results each observe an uncrossed ceiling and all write. Running +// retention at the join removes the concurrency, so a batch whose results +// individually fit but whose SUM exceeds the cap stops at the cap. +func TestBatchRetentionCeilingHoldsAcrossOneBatch(t *testing.T) { + dir := t.TempDir() + const each = 4000 + const ceiling = 2 * each // only two of the four may be retained + + s := NewSession(Config{ + SessionDir: dir, + ToolResultInlineBytes: 200, + ToolResultRetainedBytes: ceiling, + ToolConcurrency: 8, + }) + s.tools["big"] = Tool{ + Def: provider.ToolDef{Name: "big", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + return message.Parts{&message.Text{Text: in.Call + strings.Repeat("y", each)}}, nil + }, + } + + calls := make([]*message.ToolCall, 4) + for i := range calls { + calls[i] = batchCall("big", fmt.Sprintf("c%d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + s.mu.Lock() + used, handles := s.toolResultBytes, len(s.toolResults) + s.mu.Unlock() + if used > ceiling { + t.Errorf("retained %d bytes, over the %d-byte ceiling", used, ceiling) + } + if handles == 0 || handles == len(calls) { + t.Errorf("retained %d of %d results; want some but not all, so the ceiling actually bound", handles, len(calls)) + } +} + +// TestBatchHookRequestsNeverCross is the relayed finding from the plugin +// protocol review. plugin.Host.dispatchChain folds each plugin's response +// into a SHARED *Req as it walks the chain, which is only correct while +// every concurrent tool call owns its OWN request value. Two calls in one +// batch, both passing through a tool.execute.before hook that rewrites +// args, must each get their own rewritten args. +func TestBatchHookRequestsNeverCross(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + arrived := make(chan struct{}, 2) + proceed := make(chan struct{}) + + hooks := &rewritingHooks{ + before: func(req *plugin.ToolExecuteBeforeRequest) json.RawMessage { + arrived <- struct{}{} + <-proceed + return json.RawMessage(fmt.Sprintf(`{"call":%q}`, "rw-"+req.CallID)) + }, + } + var mu sync.Mutex + seen := map[string]string{} + s := NewSession(Config{Hooks: hooks, ToolConcurrency: 8}) + s.tools["echo"] = Tool{ + Def: provider.ToolDef{Name: "echo", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + mu.Lock() + seen[in.Call] = in.Call + mu.Unlock() + return message.Parts{&message.Text{Text: in.Call}}, nil + }, + } + + calls := []*message.ToolCall{batchCall("echo", "a"), batchCall("echo", "b")} + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + // Both before-hooks must be in flight together. + <-arrived + <-arrived + close(proceed) + synctest.Wait() + + wantOrder(t, results, calls...) + mu.Lock() + defer mu.Unlock() + for _, c := range calls { + want := "rw-" + c.CallID + if _, ok := seen[want]; !ok { + t.Errorf("call %s did not receive its own rewritten args %q; saw %v", c.CallID, want, seen) + } + } + }) +} + +// rewritingHooks is a Hooks fake whose tool.execute.before rewrites args +// per call. Every method must be safe for concurrent use: the engine now +// dispatches hooks from several goroutines at once. +type rewritingHooks struct { + before func(*plugin.ToolExecuteBeforeRequest) json.RawMessage +} + +func (h *rewritingHooks) ChatParams(_ context.Context, req *plugin.ChatParamsRequest) plugin.ChatParams { + return req.Params +} +func (h *rewritingHooks) ChatMessage(_ context.Context, req *plugin.ChatMessageRequest) message.Message { + return req.Message +} +func (h *rewritingHooks) SystemTransform(context.Context, *plugin.SystemTransformRequest) []string { + return nil +} +func (h *rewritingHooks) ShellEnv(context.Context, *plugin.ShellEnvRequest) map[string]string { + return nil +} +func (h *rewritingHooks) ToolExecuteBefore(_ context.Context, req *plugin.ToolExecuteBeforeRequest) (json.RawMessage, string) { + return h.before(req), "" +} +func (h *rewritingHooks) ToolExecuteAfter(_ context.Context, req *plugin.ToolExecuteAfterRequest) message.Parts { + return req.Output +} +func (h *rewritingHooks) ExecuteTool(_ context.Context, req *plugin.ToolExecuteRequest) (*plugin.ToolExecuteResponse, error) { + return nil, fmt.Errorf("plugin: no plugin provides tool %q", req.Tool) +} +func (h *rewritingHooks) Emit([]plugin.Event) {} +func (h *rewritingHooks) Plugins() []plugin.Info { return nil } +func (h *rewritingHooks) Tools() []plugin.ToolDef { return nil } + +// TestBatchTaskSpawnRunsBesideReads is the approved addendum's explicit +// case: batching a task spawn with ordinary reads must work. A task spawn +// is asynchronous and cheap — it hands the child to the SessionManager and +// returns — so it stays in the parallel class rather than being a Serial +// barrier. The batch is [task spawn, read, task spawn]; the two spawns and +// the read must overlap, and the results must still join in call order. +func TestBatchTaskSpawnRunsBesideReads(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "r.txt"), []byte("hello"), 0o644); err != nil { + t.Fatal(err) + } + + // Two spawned children stream at the same time, so this test needs a + // provider fake that is safe for concurrent use. scriptedProvider is + // not: it appends to a slice and bumps a counter with no lock. + prov := &lockedProvider{name: "test", text: "child done"} + mgr := NewSessionManager(context.Background(), 0, 0) + s := mgr.NewRoot(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + WorkDir: dir, + ToolConcurrency: 8, + }) + + // The read rendezvouses with both spawns: it does not return until + // both task calls are inside their own tool. An executor that ran the + // batch one call at a time could never satisfy that, so this proves + // real overlap between a task spawn and a read. + spawnIn := make(chan struct{}, 2) + readGo := make(chan struct{}) + realTask := s.tools[taskToolName] + wrapped := realTask + wrapped.Run = func(ctx context.Context, sess *Session, args json.RawMessage) (message.Parts, error) { + spawnIn <- struct{}{} + return realTask.Run(ctx, sess, args) + } + s.tools[taskToolName] = wrapped + realRead := s.tools["read_file"] + gated := realRead + gated.Run = func(ctx context.Context, sess *Session, args json.RawMessage) (message.Parts, error) { + <-readGo + return realRead.Run(ctx, sess, args) + } + s.tools["read_file"] = gated + + spawnArgs := `{"action":"spawn","agent":"general-purpose","prompt":"do a thing"}` + calls := []*message.ToolCall{ + {CallID: "tc_spawn1", Name: taskToolName, Arguments: json.RawMessage(spawnArgs)}, + {CallID: "tc_read", Name: "read_file", Arguments: json.RawMessage(`{"path":"r.txt"}`)}, + {CallID: "tc_spawn2", Name: taskToolName, Arguments: json.RawMessage(spawnArgs)}, + } + + var results message.Parts + done := make(chan struct{}) + go func() { + defer close(done) + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + <-spawnIn + <-spawnIn + close(readGo) + <-done + + wantOrder(t, results, calls...) + for i, p := range results { + if tr := p.(*message.ToolResult); tr.IsError { + t.Errorf("results[%d] (%s) failed: %s", i, tr.CallID, tr.Content.Text()) + } + } + if got := results[1].(*message.ToolResult).Content.Text(); !strings.Contains(got, "hello") { + t.Errorf("read result = %q, want it to contain the file body", got) + } +} + +// lockedProvider is a concurrency-safe provider fake: every Stream returns +// the same one-message turn. scriptedProvider cannot be used where two +// sessions stream at once — it mutates its own fields with no lock. +type lockedProvider struct { + name string + text string + + mu sync.Mutex + calls int +} + +func (p *lockedProvider) Name() string { return p.name } + +func (p *lockedProvider) Stream(context.Context, *provider.Request) (provider.Stream, error) { + p.mu.Lock() + p.calls++ + p.mu.Unlock() + return &scriptedStream{events: asstTurn(provider.StopEndTurn, &message.Text{Text: p.text})}, nil +} + +// TestBatchKeyWaiterDoesNotHoldAPoolSlot is the head-of-line finding from +// the cross-model review. A call waiting for its same-key predecessor must +// not occupy a pool slot while it waits, or an unrelated later call is +// refused admission behind a goroutine that is doing nothing. +// +// The batch is [k1-a, k1-b, free], with a cap of two. k1-a stays inside +// its tool; k1-b can only wait. "free" shares no key with either, so it +// must still be admitted. The test blocks until "free" runs, which an +// executor that admits on the submitting goroutine can never satisfy — +// its two slots are held by k1-a and the waiting k1-b. +func TestBatchKeyWaiterDoesNotHoldAPoolSlot(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + freeRan := make(chan struct{}) + holdA := make(chan struct{}) + aIn := make(chan struct{}) + + keyed := func(name, key string) Tool { + return Tool{ + Def: provider.ToolDef{Name: name, Description: "d", InputSchema: json.RawMessage(`{}`)}, + Key: func(*Session, json.RawMessage) string { return key }, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + switch in.Call { + case "a": + close(aIn) + <-holdA + case "free": + close(freeRan) + } + return message.Parts{&message.Text{Text: in.Call}}, nil + }, + } + } + + s := NewSession(Config{ToolConcurrency: 2}) + s.tools["k1"] = keyed("k1", "shared") + s.tools["free"] = keyed("free", "") + + calls := []*message.ToolCall{ + batchCall("k1", "a"), batchCall("k1", "b"), batchCall("free", "free"), + } + var results message.Parts + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + }() + + <-aIn + // The unrelated call must get in while k1-a still holds the key + // and k1-b is still waiting for it. + <-freeRan + close(holdA) + synctest.Wait() + + wantOrder(t, results, calls...) + }) +} + +// TestBatchPanicInToolYieldsOneErrorResult proves the one-result-per-call +// guarantee survives a panicking tool. A panic in a worker goroutine +// cannot be recovered by the join, so without the guard it takes the +// process down and leaves the assistant message's tool_use blocks +// unanswered forever. The panicking call must instead return one error +// result, and its siblings must still return their own. +func TestBatchPanicInToolYieldsOneErrorResult(t *testing.T) { + for _, tc := range []struct { + name string + concurrency int + }{ + {"parallel", 8}, + {"sequential", 1}, + } { + t.Run(tc.name, func(t *testing.T) { + s := NewSession(Config{ToolConcurrency: tc.concurrency}) + s.tools["ok"] = batchTool("ok", nil) + s.tools["panics"] = Tool{ + Def: provider.ToolDef{Name: "panics", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(context.Context, *Session, json.RawMessage) (message.Parts, error) { + panic("tool exploded") + }, + } + calls := []*message.ToolCall{ + batchCall("ok", "c0"), + {CallID: "tc_panic", Name: "panics", Arguments: json.RawMessage(`{}`)}, + batchCall("ok", "c2"), + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + + wantOrder(t, results, calls...) + for _, p := range results { + tr := p.(*message.ToolResult) + wantErr := tr.CallID == "tc_panic" + if tr.IsError != wantErr { + t.Errorf("%s IsError = %v, want %v", tr.CallID, tr.IsError, wantErr) + } + if wantErr && !strings.Contains(tr.Content.Text(), toolCallPanicText) { + t.Errorf("%s = %q, want it to name the panic", tr.CallID, tr.Content.Text()) + } + } + }) + } +} + +// TestBatchPanickingKeyStillSerializes proves a Key that panics cannot take +// the batch down with no results at all. Key runs on the submitting +// goroutine, before any result slot is filled, so its fallback must be a +// real key — never "", which would silently drop the exclusion the tool +// asked for. +func TestBatchPanickingKeyStillSerializes(t *testing.T) { + s := NewSession(Config{ToolConcurrency: 8}) + s.tools["bad"] = Tool{ + Def: provider.ToolDef{Name: "bad", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Key: func(*Session, json.RawMessage) string { panic("key exploded") }, + Run: func(context.Context, *Session, json.RawMessage) (message.Parts, error) { + return message.Parts{&message.Text{Text: "ran"}}, nil + }, + } + if got := s.toolKey("bad", json.RawMessage(`{}`)); got == "" { + t.Fatal("a panicking Key fell back to no key: the tool's exclusion would be dropped") + } + calls := []*message.ToolCall{ + {CallID: "tc1", Name: "bad", Arguments: json.RawMessage(`{}`)}, + {CallID: "tc2", Name: "bad", Arguments: json.RawMessage(`{}`)}, + } + wantOrder(t, s.runToolCalls(context.Background(), asstWith(calls...)), calls...) +} + +// TestFilePathKeyIsAbsoluteUnderRelativeWorkDir is the third cross-model +// finding. resolvePath joins a relative argument onto Config.WorkDir, and +// WorkDir itself may be relative, so cleaning alone left one file with two +// keys — and a write could then race an edit on it. +func TestFilePathKeyIsAbsoluteUnderRelativeWorkDir(t *testing.T) { + cwd, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + s := NewSession(Config{WorkDir: "."}) + args := func(path string) json.RawMessage { + return json.RawMessage(fmt.Sprintf(`{"path":%q}`, path)) + } + rel := s.toolKey("edit_file", args("a.txt")) + abs := s.toolKey("write_file", args(filepath.Join(cwd, "a.txt"))) + if rel != abs { + t.Errorf("relative key %q != absolute key %q: one file must take one key", rel, abs) + } + if !strings.HasPrefix(rel, filePathKeyPrefix+"/") { + t.Errorf("key %q is not absolute", rel) + } +} + +// TestFilePathKeySeesThroughSymlinks proves a valid filesystem alias +// cannot bypass same-file exclusion. Two calls naming a file directly and +// through a symlink must take ONE key, or a write_file could race an +// edit_file against the same bytes. +// +// The second case is the one a lexical fix cannot reach: a file that does +// not exist yet, inside a symlinked directory — a routine write_file +// target. +func TestFilePathKeySeesThroughSymlinks(t *testing.T) { + dir := t.TempDir() + real := filepath.Join(dir, "real.txt") + if err := os.WriteFile(real, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + link := filepath.Join(dir, "link.txt") + if err := os.Symlink(real, link); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + realDir := filepath.Join(dir, "d") + if err := os.Mkdir(realDir, 0o755); err != nil { + t.Fatal(err) + } + linkDir := filepath.Join(dir, "dlink") + if err := os.Symlink(realDir, linkDir); err != nil { + t.Fatal(err) + } + + s := NewSession(Config{WorkDir: dir}) + key := func(path string) string { + return s.toolKey("edit_file", json.RawMessage(fmt.Sprintf(`{"path":%q}`, path))) + } + if got, want := key("link.txt"), key("real.txt"); got != want { + t.Errorf("symlink key %q != real key %q: a write could race an edit on one file", got, want) + } + // Not-yet-created file inside a symlinked directory. + if got, want := key("dlink/new.txt"), key("d/new.txt"); got != want { + t.Errorf("key through a symlinked dir %q != %q", got, want) + } +} + +// TestTaskToolKeysPerTargetDescendant pins the task tool's resource key. +// Its verbs name a TARGET descendant, and two calls in one batch naming +// one descendant reordered by completion order change the outcome: a +// cancel(X) then send(X) executed as send-then-cancel delivers a message +// to a still-running child and then kills both. A spawn names no target +// and stays unkeyed, so spawns run in parallel as the design requires. +func TestTaskToolKeysPerTargetDescendant(t *testing.T) { + mgr := NewSessionManager(context.Background(), 0, 0) + s := mgr.NewRoot(Config{}) + key := func(args string) string { + return s.toolKey(taskToolName, json.RawMessage(args)) + } + + cancelX := key(`{"action":"cancel","session_id":"ses_x"}`) + sendX := key(`{"action":"send","session_id":"ses_x","prompt":"hi"}`) + if cancelX == "" { + t.Fatal("a task verb naming a descendant has no key: cancel and send on one child could reorder") + } + if cancelX != sendX { + t.Errorf("cancel key %q != send key %q: two verbs on ONE descendant must share a key", cancelX, sendX) + } + if other := key(`{"action":"cancel","session_id":"ses_y"}`); other == cancelX { + t.Error("two different descendants share a key: they must run concurrently") + } + for _, spawn := range []string{ + `{"action":"spawn","agent":"general-purpose","prompt":"go"}`, + `{"agent":"general-purpose","prompt":"go"}`, + } { + if got := key(spawn); got != "" { + t.Errorf("spawn %s took key %q, want none: spawns must stay parallel", spawn, got) + } + } + if got := key(`{`); got != "" { + t.Errorf("a malformed task call took key %q, want none", got) + } +} + +// TestPanicKeepsToolEventsBalanced proves a recovered panic leaves no +// dangling tool.start on the live event stream. runToolCall emits +// EventToolStart before anything can panic, so a subscriber that pairs +// start and end by call id — the session monitor's reducer, an ACP tool +// node, a plugin audit trail — would otherwise wait forever for an end +// that never comes. +// +// The second case is the one a naive fix gets wrong: emitToolExecuteEnd +// fires BEFORE the after-hook chain, so a panic inside ToolExecuteAfter +// must not emit a duplicate tool.execute.end. +func TestPanicKeepsToolEventsBalanced(t *testing.T) { + count := func(evs []Event, typ, callID string) int { + n := 0 + for _, ev := range evs { + if ev.Type == typ && ev.ToolCall != nil && ev.ToolCall.CallID == callID { + n++ + } + } + return n + } + + t.Run("tool panics", func(t *testing.T) { + var mu sync.Mutex + var evs []Event + s := NewSession(Config{OnEvent: func(ev Event) { + mu.Lock() + evs = append(evs, ev) + mu.Unlock() + }}) + s.tools["panics"] = Tool{ + Def: provider.ToolDef{Name: "panics", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(context.Context, *Session, json.RawMessage) (message.Parts, error) { + panic("boom") + }, + } + call := &message.ToolCall{CallID: "tc_p", Name: "panics", Arguments: json.RawMessage(`{}`)} + wantOrder(t, s.runToolCalls(context.Background(), asstWith(call)), call) + + mu.Lock() + defer mu.Unlock() + if got := count(evs, EventToolStart, "tc_p"); got != 1 { + t.Errorf("EventToolStart count = %d, want 1", got) + } + if got := count(evs, EventToolEnd, "tc_p"); got != 1 { + t.Errorf("EventToolEnd count = %d, want exactly 1 (a dangling start never closes)", got) + } + for _, ev := range evs { + if ev.Type == EventToolEnd && !ev.IsError { + t.Error("the panic's EventToolEnd is not marked as an error") + } + } + }) + + // A panic in the BEFORE chain owes no tool.execute.end at all: the + // start never fired, because emitToolExecuteStart runs after that + // chain. Emitting one would be a phantom end for a call that never + // executed — the inverse of the dangling start above — and would + // break emitToolExecuteStart's own rule that a denied call fires + // neither event. + t.Run("before-hook panics, no phantom execute-end", func(t *testing.T) { + hooks := &panickingAfterHooks{beforePanics: true} + s := NewSession(Config{Hooks: hooks}) + s.tools["ok"] = batchTool("ok", nil) + call := batchCall("ok", "c0") + wantOrder(t, s.runToolCalls(context.Background(), asstWith(call)), call) + + hooks.mu.Lock() + defer hooks.mu.Unlock() + starts, ends := 0, 0 + for _, e := range hooks.emitted { + switch e { + case plugin.EventToolExecuteStart: + starts++ + case plugin.EventToolExecuteEnd: + ends++ + } + } + if starts != 0 { + t.Errorf("tool.execute.start emitted %d times, want 0: the call never executed", starts) + } + if ends != 0 { + t.Errorf("tool.execute.end emitted %d times, want 0: a phantom end with no start", ends) + } + }) + + t.Run("after-hook panics, no duplicate execute-end", func(t *testing.T) { + hooks := &panickingAfterHooks{} + s := NewSession(Config{Hooks: hooks}) + s.tools["ok"] = batchTool("ok", nil) + call := batchCall("ok", "c0") + wantOrder(t, s.runToolCalls(context.Background(), asstWith(call)), call) + + hooks.mu.Lock() + defer hooks.mu.Unlock() + ends := 0 + for _, e := range hooks.emitted { + if e == plugin.EventToolExecuteEnd { + ends++ + } + } + if ends != 1 { + t.Errorf("tool.execute.end emitted %d times, want exactly 1", ends) + } + }) +} + +// panickingAfterHooks panics in ToolExecuteAfter and records every plugin +// event type the engine emits, so a duplicate tool.execute.end is visible. +type panickingAfterHooks struct { + beforePanics bool + + mu sync.Mutex + emitted []string +} + +func (h *panickingAfterHooks) ChatParams(_ context.Context, req *plugin.ChatParamsRequest) plugin.ChatParams { + return req.Params +} +func (h *panickingAfterHooks) ChatMessage(_ context.Context, req *plugin.ChatMessageRequest) message.Message { + return req.Message +} +func (h *panickingAfterHooks) SystemTransform(context.Context, *plugin.SystemTransformRequest) []string { + return nil +} +func (h *panickingAfterHooks) ShellEnv(context.Context, *plugin.ShellEnvRequest) map[string]string { + return nil +} +func (h *panickingAfterHooks) ToolExecuteBefore(context.Context, *plugin.ToolExecuteBeforeRequest) (json.RawMessage, string) { + if h.beforePanics { + panic("before-hook exploded") + } + return nil, "" +} +func (h *panickingAfterHooks) ToolExecuteAfter(_ context.Context, req *plugin.ToolExecuteAfterRequest) message.Parts { + if h.beforePanics { + return req.Output + } + panic("after-hook exploded") +} +func (h *panickingAfterHooks) ExecuteTool(_ context.Context, req *plugin.ToolExecuteRequest) (*plugin.ToolExecuteResponse, error) { + return nil, fmt.Errorf("plugin: no plugin provides tool %q", req.Tool) +} +func (h *panickingAfterHooks) Emit(events []plugin.Event) { + h.mu.Lock() + for _, e := range events { + h.emitted = append(h.emitted, e.Type) + } + h.mu.Unlock() +} +func (h *panickingAfterHooks) Plugins() []plugin.Info { return nil } +func (h *panickingAfterHooks) Tools() []plugin.ToolDef { return nil } From 3b6e8ccd81d3dfd62eee3507855be0f2ad05269a Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 20:30:35 -0400 Subject: [PATCH 09/95] provider/openai,config: allow a second native Responses provider (#197) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(provider/openai,config): allow a second native Responses provider A deployment can need two endpoints that both speak the OpenAI Responses API: one gateway behind the built-in "openai" family, and a second vendor serving the identical wire format at its own host. Config had no way to express that. config.validateProviders accepted an empty Provider.Type only on the native keys "anthropic" and "openai", so a Responses adapter could never be built under any other providers-map key, and the only other accepted type, "openai-compat", builds a chat-completions client that speaks a different wire. A second obstacle sat one layer down. provider/openai.Client built its request URL as base+"/v1/responses", with the path hardcoded. A vendor is free to serve a Responses-compatible endpoint under a path of its own, and no value of BaseURL can reach such a path when the adapter always appends OpenAI's. This change adds three parts. Client.ResponsesPath sets the path appended to BaseURL, defaulting to "/v1/responses" when empty, so every existing caller sends the same bytes as before. Provider.Type accepts "openai" (config.TypeOpenAI) under any providers-map key, requiring base_url for the same reason "openai-compat" does: an arbitrary endpoint under a caller-chosen key has no sensible built-in default. Provider.ResponsesPath ("responses_path") carries the path through config, and is valid only on an entry that builds the Responses adapter. Two design points are worth stating. First, responses_path is REJECTED, not ignored, on any other entry. That follows the rule cache_ttl and no_prompt_cache_key already set, and it matches on the adapter an entry builds rather than on its map key alone, because the key "openai" with type "openai-compat" builds an openaicompat client that would never read the value. Second, openai.Client gained a Family field, mirroring provider/openaicompat's. cmd/harness's registry keys adapters by the providers-map name, so routing alone did not require it, but Name() is also the ProviderData tag this adapter reads and writes. A Responses reasoning item is opaque, usually encrypted, endpoint-scoped state that history replays verbatim on every later request. Had both clients kept the shared "openai" tag, the canonical family match would have succeeded across two endpoints that share no key, and a session switching between them would replay one endpoint's ciphertext to the other. A per-client family makes that a cross-family drop instead, which costs one turn of reasoning continuity and nothing else. An empty Family still resolves to the package constant, so the built-in entry is untouched. Deliberate non-changes: the bare "openai" key with an empty type keeps its built-in base URL and its package family, the openrouter defaults are untouched, and transcodeRequest keeps its old signature, delegating to a new transcodeRequestFamily. Tests were written first and each was verified red against the exact mechanism it names. One of them, TestUnknownTypeErrorListsOpenAI, passed on its first red run: the valid-type error messages already contain the word "openai" while describing the native keys, so a substring check for the type name alone proved nothing. It now matches the rendered valid-types list. * provider/openai: normalize the join between BaseURL and ResponsesPath ResponsesPath is caller-supplied configuration, so a path written without its leading slash is a typo an operator will eventually make. Concatenated onto a base with no trailing slash it produced a URL aimed at a different host: "https://api.example.test" + "backend/responses" parses as the host "api.example.testbackend", which fails with "invalid port ... after host" against a test server and would be a silent request to somebody else's name against a real one. responsesURL applies each field's default and joins the two halves with exactly one slash, absorbing a missing leading slash and a trailing slash on the base alike. The join stays string-level rather than url.JoinPath so the default path remains byte-identical to the string this adapter has always sent; trimming every leading slash before re-adding one also rules out a "//..." path, which a URL parser reads as an authority. * test(provider/openai,config): use neutral example values in fixtures The tests for the configurable responses path and the per-client family named one particular deployment's provider and endpoint path. Nothing in the adapter is specific to it: the field exists because more than one endpoint speaks this wire, and a fixture that names a single consumer reads as if the engine knows about it. Values only; coverage is unchanged. The path cases become "/alt/responses" and its no-leading-slash twin, the family cases become "secondary", and the base-carries-a-path row keeps testing exactly what it did with a neutral prefix. * fix(cmd/harness): keep the default key env for an openai-typed entry A type:"openai" entry keyed "openai" replaces the built-in client for that key. registerOpenAIProviders read a key only from an explicit api_key_env, so adding a type to an existing entry silently unauthenticated every request it made: the built-in entry reads OPENAI_API_KEY, the replacement read nothing. An entry that names no api_key_env is asking for the adapter's default key source. Both paths now read one constant, so they cannot drift to different defaults. An explicit api_key_env still wins, and an unset named variable resolves empty rather than falling back, so naming a variable remains the stricter choice for an endpoint that must not receive the default key. The doc comment on registration order claimed validation had "already rejected every entry that could produce two adapters for one key". It has not: type:"openai" is valid under any map key. The real guarantee comes from the map — one key, one entry, hence one type — and registration order is a genuine precedence rule that lets an explicit entry replace a built-in one. The comment now says that. --------- Co-authored-by: andybons --- AGENTS.md | 39 ++++ README.md | 30 ++- cmd/harness/main.go | 92 +++++++- cmd/harness/openai_type_test.go | 278 +++++++++++++++++++++++++ config/config.go | 95 +++++++-- config/openai_type_test.go | 164 +++++++++++++++ provider/openai/openai.go | 101 +++++++-- provider/openai/responses_path_test.go | 177 ++++++++++++++++ provider/openai/transcode.go | 24 ++- 9 files changed, 962 insertions(+), 38 deletions(-) create mode 100644 cmd/harness/openai_type_test.go create mode 100644 config/openai_type_test.go create mode 100644 provider/openai/responses_path_test.go diff --git a/AGENTS.md b/AGENTS.md index 83be60b4..7f0c70ff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1496,6 +1496,45 @@ by construction: one build, one live probe, or one subagent runs longer than the window, and a user reads an answer before sending the next turn. The commit that introduced this default carries the measured evidence. +### A second native Responses provider + +`provider/openai` speaks the OpenAI Responses API. Other vendors speak the +same wire at their own host, under their own request path. Two config +fields let a deployment reach one without new adapter code. + +`Provider.Type` accepts `"openai"` (`config.TypeOpenAI`) under ANY providers +map key. The key becomes the provider family, routed by the first segment of +a `provider/model` ref, exactly like an `"openai-compat"` entry. +`cmd/harness`'s `registerOpenAIProviders` builds one `openai.Client` per such +entry. `base_url` is required: an arbitrary endpoint under a caller-chosen +key has no sensible built-in default, the same rule `"openai-compat"` +follows. The bare `openai` key with an empty type keeps its own built-in +default and is unchanged. + +`Provider.ResponsesPath` (`responses_path`) sets `Client.ResponsesPath`, the +path appended to the base URL. Empty means `/v1/responses`, the path the +Responses API documents and the only path this adapter could reach before. +The field is valid ONLY on an entry that builds this adapter — the native +`openai` key with an empty type, or any key with type `"openai"`. +`config.validateResponsesPath` rejects it elsewhere, matching on IDENTITY +(`buildsResponsesAdapter`) rather than on the map key alone, for the reason +`validateCacheTTL` already documents: the key `openai` with type +`"openai-compat"` builds an openaicompat client that would never read the +value. + +`openai.Client.Family` overrides the family key `Name()` reports AND the +`ProviderData` tag the transcoder reads and the stream writes. Empty means +the package `Family` constant, so every existing caller is unchanged; +`registerOpenAIProviders` sets it to the providers map key. The tag matters +beyond routing. A Responses reasoning item is opaque, usually ENCRYPTED, and +scoped to the endpoint that minted it, and history replays it verbatim on +every later request. One shared `"openai"` tag across two endpoints would +make the canonical family match succeed between them, so a session that +swapped models would replay one endpoint's ciphertext to the other. A +per-client family makes that a cross-family DROP instead — the canonical +crossing rule — which costs one turn of reasoning continuity and nothing +else. + ### Lazy MCP tools (deferred schemas) An MCP server's tools reach the model as full JSON Schemas in the tools diff --git a/README.md b/README.md index e9fa5def..cb40d0d9 100644 --- a/README.md +++ b/README.md @@ -56,5 +56,31 @@ and `api_key_env` above, so `"model": "openrouter/anthropic/claude-sonnet-5"` works as soon as `OPENROUTER_API_KEY` is set. Any `openrouter` entry in config — even a partial one — overrides the built-in default entirely. -An unrecognized `type`, or an `openai-compat` entry missing `base_url`, fails -config loading loudly rather than silently registering nothing. +An endpoint that speaks the OpenAI **Responses** API rather than +chat-completions uses `type: "openai"`, which builds the same native adapter +the built-in `openai` family uses. It works under any map key, so a second +Responses endpoint can sit beside the built-in one, and `responses_path` +points it at an endpoint that does not serve `/v1/responses`: + +```json +{ + "providers": { + "vendor": { + "type": "openai", + "base_url": "https://api.vendor.example", + "api_key_env": "VENDOR_API_KEY", + "responses_path": "/backend/responses" + } + } +} +``` + +`"model": "vendor/some-model"` then routes there, passing `some-model` +through as the model id. `responses_path` defaults to `/v1/responses` and is +also accepted on the built-in `openai` entry; it is rejected on any other +kind of entry, since no other adapter reads it. + +An unrecognized `type`, an `openai-compat` or `openai` entry missing +`base_url`, or a `responses_path` on an entry that builds neither Responses +adapter, fails config loading loudly rather than silently registering +nothing. diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 628951f5..ffd10f62 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -854,9 +854,10 @@ func loadConfig() (*config.Config, error) { // values. Auth is read here but validated only on first send. Adding // another built-in provider family is a two-line change: resolve its config // with providerAuth and add one entry to the returned map. Any config -// providers entry with type "openai-compat" needs no code at all — see -// registerOpenAICompatProviders — and OpenRouter itself needs no config -// entry either, see ensureDefaultOpenRouter. +// providers entry with type "openai-compat" or "openai" needs no code at +// all — see registerOpenAICompatProviders and registerOpenAIProviders — +// and OpenRouter itself needs no config entry either, see +// ensureDefaultOpenRouter. // // registry does not assume cfg came from config.LoadProject (the load path // that guarantees nativeDefaultProviders fields are filled in — see @@ -866,21 +867,102 @@ func loadConfig() (*config.Config, error) { // {"openrouter": {"api_key_env": "..."}} entry identically to one that went // through the full config-loading choke point, rather than silently // registering no adapter for it at all. +// defaultOpenAIKeyEnv is the environment variable every provider/openai +// client reads when its entry names none of its own. One constant, so the +// built-in entry (providerAuth, above) and a configured type:"openai" entry +// (registerOpenAIProviders) cannot drift to different defaults. +const defaultOpenAIKeyEnv = "OPENAI_API_KEY" + func registry(cfg *config.Config) provider.Registry { if cfg != nil { config.EnsureProviderDefaults(cfg.Providers) } akey, abase := providerAuth(cfg, anthropic.Family, "ANTHROPIC_API_KEY") - okey, obase := providerAuth(cfg, openai.Family, "OPENAI_API_KEY") + okey, obase := providerAuth(cfg, openai.Family, defaultOpenAIKeyEnv) reg := provider.Registry{ anthropic.Family: &anthropic.Client{APIKey: akey, BaseURL: abase, CacheTTL: anthropicCacheTTL(cfg)}, - openai.Family: &openai.Client{APIKey: okey, BaseURL: obase}, + // The built-in openai entry deliberately leaves Family empty: it IS + // the package default, and naming it here would only invite the two + // to drift. Its ResponsesPath comes from the native entry, if any. + openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg)}, } registerOpenAICompatProviders(reg, cfg) + registerOpenAIProviders(reg, cfg) ensureDefaultOpenRouter(reg, cfg) return reg } +// registerOpenAIProviders builds a native provider/openai (Responses API) +// client for every config.Providers entry of config.TypeOpenAI, keyed by +// its providers map name — that name is what routes "name/model" refs to +// it, exactly like an openai-compat entry, and it is also the client's own +// Family, so the entry's opaque reasoning attachments are tagged and +// replayed under the key that identifies its endpoint rather than under the +// shared package constant. +// +// Registration order IS a precedence rule, not a formality, and the earlier +// version of this comment claimed otherwise. config.validateProviders does +// NOT reject an entry that collides with a built-in key: type:"openai" is +// valid under ANY map key, including "openai" and "anthropic". What it +// rejects is an unknown type, and an empty type on a key that is neither +// native nor native-default. +// +// So the guarantee is narrower and comes from the map, not from validation: +// a providers map key has exactly one entry, hence exactly one type, so +// registerOpenAICompatProviders and this function can never both claim the +// same key. Where an entry names a built-in key, it deliberately REPLACES +// the built-in adapter, and running last is what makes the explicit entry +// win. Its API key falls back to the same environment variable the built-in +// entry reads, so replacing the built-in this way never silently +// unauthenticates it. +func registerOpenAIProviders(reg provider.Registry, cfg *config.Config) { + if cfg == nil { + return + } + for name, p := range cfg.Providers { + if p.Type != config.TypeOpenAI { + continue + } + // An entry that names no api_key_env is asking for this adapter's + // DEFAULT key source, not for no key: the built-in "openai" entry + // has always read defaultOpenAIKeyEnv (providerAuth), and an entry + // keyed "openai" replaces that client outright. Without the same + // fallback, adding a type to an existing entry would silently + // unauthenticate every request it makes. A deployment that must + // keep its OpenAI key away from a third-party endpoint names its + // own api_key_env, which wins here — and an unset named variable + // resolves empty rather than falling back, so naming a variable is + // always the stricter choice. + keyEnv := p.APIKeyEnv + if keyEnv == "" { + keyEnv = defaultOpenAIKeyEnv + } + apiKey := os.Getenv(keyEnv) + reg[name] = &openai.Client{ + Family: name, + APIKey: apiKey, + BaseURL: p.BaseURL, + ResponsesPath: p.ResponsesPath, + } + } +} + +// nativeResponsesPath reads the request path configured on the NATIVE +// "openai" entry (map key "openai", no type — the shape +// config.validateProviders permits responses_path on). Empty leaves the +// adapter's own /v1/responses default in place. A keyed type:"openai" entry +// carries its own path and is wired by registerOpenAIProviders instead. +func nativeResponsesPath(cfg *config.Config) string { + if cfg == nil { + return "" + } + p := cfg.Providers[openai.Family] + if p.Type != "" { + return "" + } + return p.ResponsesPath +} + // registerOpenAICompatProviders builds a provider/openaicompat client for // every config.Providers entry of config.TypeOpenAICompat, keyed by its // providers map name — that name is what routes "name/model" refs to it, diff --git a/cmd/harness/openai_type_test.go b/cmd/harness/openai_type_test.go new file mode 100644 index 00000000..5b05b826 --- /dev/null +++ b/cmd/harness/openai_type_test.go @@ -0,0 +1,278 @@ +package main + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/majorcontext/harness/config" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" + "github.com/majorcontext/harness/provider/openai" +) + +// TestRegistryTypeOpenAIBuildsKeyedNativeClient: a `type: "openai"` entry +// registers the native Responses adapter under its PROVIDERS-MAP KEY, not +// under the package family constant. The key is what routes a +// "/" ref, so keying it any other way would make the entry +// unreachable — or, worse, silently replace the built-in "openai" entry. +func TestRegistryTypeOpenAIBuildsKeyedNativeClient(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: "https://gateway.example", + APIKeyEnv: "SECONDARY_API_KEY", + ResponsesPath: "/alt/responses", + }, + }}) + + c, ok := reg["secondary"].(*openai.Client) + if !ok { + t.Fatalf("secondary provider is %T, want *openai.Client", reg["secondary"]) + } + if c.Family != "secondary" { + t.Errorf("Family = %q, want the providers-map key %q", c.Family, "secondary") + } + if c.BaseURL != "https://gateway.example" { + t.Errorf("BaseURL = %q", c.BaseURL) + } + if c.APIKey != "sk-secondary" { + t.Errorf("APIKey = %q, want sk-secondary", c.APIKey) + } + if c.ResponsesPath != "/alt/responses" { + t.Errorf("ResponsesPath = %q, want /alt/responses", c.ResponsesPath) + } + + // The built-in native entry must be untouched: a keyed entry ADDS a + // provider, it never rebinds the "openai" key. + builtin, ok := reg[openai.Family].(*openai.Client) + if !ok { + t.Fatalf("openai provider is %T, want *openai.Client", reg[openai.Family]) + } + if builtin.APIKey != "sk-builtin" { + t.Errorf("built-in openai APIKey = %q, want sk-builtin", builtin.APIKey) + } + if builtin.BaseURL != "" || builtin.ResponsesPath != "" { + t.Errorf("built-in openai client = %+v, want the untouched defaults", builtin) + } +} + +// TestRegistryTypeOpenAIRoutesRefToConfiguredEndpoint drives the production +// path end to end: a "/" ref resolves through the registry and +// POSTs to the configured base URL AND request path, passing the model id +// through unchanged — exactly as an openai-compat entry does. +func TestRegistryTypeOpenAIRoutesRefToConfiguredEndpoint(t *testing.T) { + var gotPath, gotAuth, gotModel string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + gotAuth = r.Header.Get("Authorization") + var body struct { + Model string `json:"model"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + t.Errorf("decoding request body: %v", err) + } + gotModel = body.Model + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_1\"}}\n\n") //nolint:errcheck + })) + defer srv.Close() + + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: srv.URL, + APIKeyEnv: "SECONDARY_API_KEY", + ResponsesPath: "/backend/responses", + }, + }}) + + ref, err := message.ParseModelRef("secondary/gpt-5") + if err != nil { + t.Fatalf("ParseModelRef: %v", err) + } + p, err := reg.For(ref) + if err != nil { + t.Fatalf("reg.For: %v", err) + } + stream, err := p.Stream(context.Background(), &provider.Request{ + Model: ref, + Messages: []message.Message{{ID: "msg_1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hello"}}}}, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer stream.Close() + for { + if _, err := stream.Next(); err == io.EOF { + break + } else if err != nil { + t.Fatalf("Next: %v", err) + } + } + + if gotPath != "/backend/responses" { + t.Errorf("path = %q, want /backend/responses", gotPath) + } + if gotAuth != "Bearer sk-secondary" { + t.Errorf("Authorization = %q, want Bearer sk-secondary", gotAuth) + } + if gotModel != "gpt-5" { + t.Errorf("model = %q, want the ref's model id passed through unchanged", gotModel) + } +} + +// TestRegistryTypeOpenAIDefaultResponsesPath: an entry that names no +// responses_path keeps the adapter's own /v1/responses default, so the new +// type is usable for an ordinary Responses endpoint with two config keys. +func TestRegistryTypeOpenAIDefaultResponsesPath(t *testing.T) { + var gotPath string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_1\"}}\n\n") //nolint:errcheck + })) + defer srv.Close() + + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": {Type: config.TypeOpenAI, BaseURL: srv.URL, APIKeyEnv: "SECONDARY_API_KEY"}, + }}) + ref, err := message.ParseModelRef("secondary/gpt-5") + if err != nil { + t.Fatalf("ParseModelRef: %v", err) + } + p, err := reg.For(ref) + if err != nil { + t.Fatalf("reg.For: %v", err) + } + stream, err := p.Stream(context.Background(), &provider.Request{ + Model: ref, + Messages: []message.Message{{ID: "msg_1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hello"}}}}, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer stream.Close() + for { + if _, err := stream.Next(); err == io.EOF { + break + } else if err != nil { + t.Fatalf("Next: %v", err) + } + } + if gotPath != "/v1/responses" { + t.Errorf("path = %q, want the default /v1/responses", gotPath) + } +} + +// TestRegistryTypeOpenAIUnderNativeKey covers the one shape where the two +// wiring paths address the same key: an entry keyed "openai" that also names +// type "openai". It is legal config, and it must apply WHOLE — base URL, +// path, and key together — rather than half-landing because the built-in +// entry supplied one field and the keyed entry another. +func TestRegistryTypeOpenAIUnderNativeKey(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "openai": { + Type: config.TypeOpenAI, + BaseURL: "https://gateway.example", + APIKeyEnv: "SECONDARY_API_KEY", + ResponsesPath: "/alt/responses", + }, + }}) + c, ok := reg[openai.Family].(*openai.Client) + if !ok { + t.Fatalf("openai provider is %T, want *openai.Client", reg[openai.Family]) + } + if c.APIKey != "sk-secondary" || c.BaseURL != "https://gateway.example" || c.ResponsesPath != "/alt/responses" { + t.Errorf("entry applied only in part: %+v", c) + } + // Family here equals the package constant, so this entry's reasoning + // attachments stay tagged exactly as the built-in entry's would be. + if got := c.Name(); got != openai.Family { + t.Errorf("Name() = %q, want %q", got, openai.Family) + } +} + +// TestRegistryNativeOpenAIHonorsResponsesPath: the bare "openai" key builds +// the same adapter, so its responses_path must reach the client too. +func TestRegistryNativeOpenAIHonorsResponsesPath(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "openai": {BaseURL: "http://proxy", ResponsesPath: "/alt/responses"}, + }}) + c := reg[openai.Family].(*openai.Client) + if c.ResponsesPath != "/alt/responses" { + t.Errorf("ResponsesPath = %q, want /alt/responses", c.ResponsesPath) + } + if c.Family != "" && c.Family != openai.Family { + t.Errorf("Family = %q, want the package default for the built-in entry", c.Family) + } +} + +// TestRegistryTypeOpenAIFallsBackToDefaultKeyEnv: an entry that names no +// api_key_env is asking for the adapter's default key source, not for no +// key at all. The bare "openai" entry has always read OPENAI_API_KEY that +// way (providerAuth), and a type:"openai" entry keyed "openai" REPLACES +// that built-in client — so without the same fallback, adding a type to an +// existing entry would silently unauthenticate every request it makes. +// +// An entry that must NOT receive that key names its own api_key_env; the +// case immediately below pins that. +func TestRegistryTypeOpenAIFallsBackToDefaultKeyEnv(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "openai": {Type: config.TypeOpenAI, BaseURL: "https://gateway.example"}, + }}) + c, ok := reg[openai.Family].(*openai.Client) + if !ok { + t.Fatalf("openai provider is %T, want *openai.Client", reg[openai.Family]) + } + if c.APIKey != "sk-builtin" { + t.Errorf("APIKey = %q, want the OPENAI_API_KEY fallback %q", c.APIKey, "sk-builtin") + } +} + +// TestRegistryTypeOpenAIKeyedEntryFallsBackToDefaultKeyEnv holds the same +// rule for a non-native key, so the fallback is a property of the adapter +// rather than of one map key. +func TestRegistryTypeOpenAIKeyedEntryFallsBackToDefaultKeyEnv(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": {Type: config.TypeOpenAI, BaseURL: "https://gateway.example"}, + }}) + c, ok := reg["secondary"].(*openai.Client) + if !ok { + t.Fatalf("secondary provider is %T, want *openai.Client", reg["secondary"]) + } + if c.APIKey != "sk-builtin" { + t.Errorf("APIKey = %q, want the OPENAI_API_KEY fallback %q", c.APIKey, "sk-builtin") + } +} + +// TestRegistryTypeOpenAIExplicitKeyEnvWins is the other half: an explicit +// api_key_env is how a deployment keeps its real OpenAI key away from a +// third-party endpoint. It must win over the fallback, and an unset named +// variable must resolve empty rather than silently borrowing OPENAI_API_KEY. +func TestRegistryTypeOpenAIExplicitKeyEnvWins(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": {Type: config.TypeOpenAI, BaseURL: "https://gateway.example", APIKeyEnv: "SECONDARY_API_KEY"}, + "tertiary": {Type: config.TypeOpenAI, BaseURL: "https://other.example", APIKeyEnv: "UNSET_API_KEY"}, + }}) + if c := reg["secondary"].(*openai.Client); c.APIKey != "sk-secondary" { + t.Errorf("secondary APIKey = %q, want sk-secondary", c.APIKey) + } + if c := reg["tertiary"].(*openai.Client); c.APIKey != "" { + t.Errorf("tertiary APIKey = %q, want empty: an entry naming an unset variable must not borrow the default key", c.APIKey) + } +} diff --git a/config/config.go b/config/config.go index 56bb6458..91b06855 100644 --- a/config/config.go +++ b/config/config.go @@ -352,10 +352,22 @@ type PluginSpec struct { // TypeOpenAICompat selects the generic OpenAI-compatible chat-completions // adapter (provider/openaicompat) for a Provider config entry — the wire -// format spoken by OpenRouter, Ollama, vLLM, and similar deployments. It is -// the only non-empty Provider.Type value Load accepts today. +// format spoken by OpenRouter, Ollama, vLLM, and similar deployments. See +// TypeOpenAI for the other non-empty Provider.Type value Load accepts. const TypeOpenAICompat = "openai-compat" +// TypeOpenAI selects the native OpenAI Responses API adapter +// (provider/openai) for a Provider config entry under ANY providers map +// key. It exists because the bare "openai" key (empty Type — see +// nativeProviderKeys) can name only ONE Responses endpoint, and a +// deployment may need a second: another vendor speaking the same wire, at +// its own base URL and its own request path, while "openai" keeps pointing +// somewhere else. Like TypeOpenAICompat, the map key becomes the provider +// family — routed by the first segment of a "provider/model" ref — and +// BaseURL is required, since an arbitrary endpoint has no sensible +// built-in default. +const TypeOpenAI = "openai" + // nativeProviderKeys are the only providers map keys allowed an empty Type // with no further defaulting: the built-in adapters cmd/harness's registry // wires directly by name (provider/anthropic.Family and @@ -450,17 +462,20 @@ type Provider struct { // (see nativeProviderKeys). TypeOpenAICompat ("openai-compat") builds a // generic provider/openaicompat client instead: the providers map key // becomes the new provider family, routed by the first segment of a - // "provider/model" ref exactly like any built-in family. Any other - // value, or an empty value on any other key, fails Load loudly — a - // typo'd or missing type must not silently produce no adapter at - // startup. + // "provider/model" ref exactly like any built-in family. TypeOpenAI + // ("openai") does the same for the native provider/openai Responses + // adapter, so a second Responses endpoint can be configured beside the + // bare "openai" key. Any other value, or an empty value on any other + // key, fails Load loudly — a typo'd or missing type must not silently + // produce no adapter at startup. Type string `json:"type,omitempty"` // APIKeyEnv names the environment variable to read the API key from. APIKeyEnv string `json:"api_key_env,omitempty"` // BaseURL overrides the provider's default API base URL when non-empty. - // Required when Type is TypeOpenAICompat — there is no built-in default - // base URL for an arbitrary compat entry (the one exception, the - // built-in "openrouter" entry, is supplied by cmd/harness, not here). + // Required when Type is TypeOpenAICompat or TypeOpenAI — there is no + // built-in default base URL for an arbitrary entry under a caller-chosen + // key (the one exception, the built-in "openrouter" entry, is supplied + // by cmd/harness, not here). BaseURL string `json:"base_url,omitempty"` // Family overrides the ProviderData tag / wire-quirk key the // openaicompat adapter uses (some deployments need family-specific @@ -490,6 +505,21 @@ type Provider struct { // same layer as its base_url. Use a *bool here only if a real // two-layer override of this one field appears. NoPromptCacheKey bool `json:"no_prompt_cache_key,omitempty"` + // ResponsesPath overrides the request path the native OpenAI Responses + // adapter POSTs to, appended to BaseURL. Empty (the default) uses + // "/v1/responses", the path the OpenAI Responses API documents and the + // only path this adapter could reach before. It exists because a + // Responses-API-compatible endpoint need not live at that path: a + // vendor may serve the identical wire format under a path of its own, + // and appending "/v1/responses" to its base URL reaches nothing. + // + // Valid ONLY on an entry that builds the Responses adapter — the + // native "openai" key with an empty Type, or a TypeOpenAI entry under + // any key. No other adapter reads it, so validateProviders rejects it + // elsewhere rather than ignoring it, the same rule NoPromptCacheKey and + // CacheTTL follow. Merge semantics are additive like every other + // Provider field (see NoPromptCacheKey's doc comment). + ResponsesPath string `json:"responses_path,omitempty"` // CacheTTL selects the Anthropic prompt-cache breakpoint lifetime: // "5m" (the Anthropic API default) or "1h" (the extended TTL, beta // extended-cache-ttl-2025-04-11). Empty (the default) leaves the @@ -573,16 +603,16 @@ func validateProviders(providers map[string]Provider) error { switch p.Type { case "": if !nativeProviderKeys[name] { - return fmt.Errorf("providers.%s: type is required (empty type is only valid for the built-in %q/%q entries); valid types: \"\" (native anthropic/openai override), %q", name, "anthropic", "openai", TypeOpenAICompat) + return fmt.Errorf("providers.%s: type is required (empty type is only valid for the built-in %q/%q entries); valid types: \"\" (native anthropic/openai override), %q, %q", name, "anthropic", "openai", TypeOpenAICompat, TypeOpenAI) } // Legacy/native provider entry (anthropic or openai); no // further validation here. - case TypeOpenAICompat: + case TypeOpenAICompat, TypeOpenAI: if p.BaseURL == "" { - return fmt.Errorf("providers.%s: base_url is required for type %q", name, TypeOpenAICompat) + return fmt.Errorf("providers.%s: base_url is required for type %q", name, p.Type) } default: - return fmt.Errorf("providers.%s: unknown type %q", name, p.Type) + return fmt.Errorf("providers.%s: unknown type %q (valid types: \"\" (native anthropic/openai override), %q, %q)", name, p.Type, TypeOpenAICompat, TypeOpenAI) } if err := validateCacheTTL(name, p); err != nil { return err @@ -590,6 +620,42 @@ func validateProviders(providers map[string]Provider) error { if p.NoPromptCacheKey && p.Type != TypeOpenAICompat { return fmt.Errorf("providers.%s: no_prompt_cache_key is only valid on a %q entry (only the openaicompat adapter reads it)", name, TypeOpenAICompat) } + if err := validateResponsesPath(name, p); err != nil { + return err + } + } + return nil +} + +// buildsResponsesAdapter reports whether a providers entry builds the +// native OpenAI Responses adapter (provider/openai) — the one adapter that +// reads ResponsesPath. +// +// Like validateCacheTTL, it matches on IDENTITY rather than on the map key +// alone: the key "openai" with type "openai-compat" builds an openaicompat +// client, and cmd/harness's registerOpenAICompatProviders overwrites the +// native client registered under that same key, so no Responses adapter +// would ever read the value. Two shapes qualify: the native "openai" key +// with no type, and a TypeOpenAI entry under any key. +func buildsResponsesAdapter(name string, p Provider) bool { + if p.Type == TypeOpenAI { + return true + } + return p.Type == "" && name == "openai" +} + +// validateResponsesPath fails loudly on a responses_path set on any entry +// that does not build the Responses adapter. Only that adapter reads the +// field, so a value elsewhere would vanish into a client that never looks +// at it — the same silent-misconfiguration class validateCacheTTL and the +// no_prompt_cache_key check refuse to allow. Empty is always valid: it +// means "adapter default" (/v1/responses). +func validateResponsesPath(name string, p Provider) error { + if p.ResponsesPath == "" { + return nil + } + if !buildsResponsesAdapter(name, p) { + return fmt.Errorf("providers.%s: responses_path is only valid on an entry that builds the OpenAI Responses adapter (map key %q with no type, or any key with type %q); no other adapter reads it", name, "openai", TypeOpenAI) } return nil } @@ -996,6 +1062,9 @@ func merge(base, over *Config) *Config { if v.CacheTTL != "" { ex.CacheTTL = v.CacheTTL } + if v.ResponsesPath != "" { + ex.ResponsesPath = v.ResponsesPath + } if v.NoPromptCacheKey { ex.NoPromptCacheKey = true } diff --git a/config/openai_type_test.go b/config/openai_type_test.go new file mode 100644 index 00000000..9ad8bb3f --- /dev/null +++ b/config/openai_type_test.go @@ -0,0 +1,164 @@ +package config + +import ( + "path/filepath" + "strings" + "testing" +) + +// TestLoadProviderTypeOpenAI: an entry typed "openai" under an arbitrary +// providers-map key is accepted and round-trips its fields. This is what +// lets a deployment run a SECOND native Responses-API provider beside the +// bare "openai" key — pointing at a different endpoint, with its own +// request path — instead of being limited to the one built-in key. +func TestLoadProviderTypeOpenAI(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{ + "providers": { + "secondary": { + "type": "openai", + "base_url": "https://gateway.example", + "api_key_env": "SECONDARY_API_KEY", + "responses_path": "/alt/responses" + } + } + }`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + pr, ok := c.Providers["secondary"] + if !ok { + t.Fatal("providers.secondary missing") + } + if pr.Type != TypeOpenAI { + t.Errorf("Type = %q, want %q", pr.Type, TypeOpenAI) + } + if pr.BaseURL != "https://gateway.example" { + t.Errorf("BaseURL = %q", pr.BaseURL) + } + if pr.ResponsesPath != "/alt/responses" { + t.Errorf("ResponsesPath = %q", pr.ResponsesPath) + } + if _, err := mergeAndValidate(c, &Config{}); err != nil { + t.Errorf("mergeAndValidate: %v", err) + } +} + +// TestLoadProviderTypeOpenAIMissingBaseURLFails: an arbitrary key has no +// sensible built-in endpoint to fall back on, so base_url is required — +// the same rule openai-compat already enforces, for the same reason. The +// bare "openai" key keeps its built-in default and is covered separately +// (TestLoadProviderEmptyTypeOnNativeKeysOK). +func TestLoadProviderTypeOpenAIMissingBaseURLFails(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "secondary": {Type: TypeOpenAI}, + }} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on missing base_url for type openai") + } + if !strings.Contains(err.Error(), "base_url") { + t.Errorf("error %q does not mention base_url", err) + } + if !strings.Contains(err.Error(), "secondary") { + t.Errorf("error %q does not name the offending key", err) + } +} + +// TestResponsesPathOnNativeOpenAIKeyOK: the bare "openai" key (empty type) +// builds the very same Responses adapter, so it may carry responses_path +// too — the field is gated on the ADAPTER an entry builds, not on its type +// string. +func TestResponsesPathOnNativeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "openai": {ResponsesPath: "/alt/responses"}, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if merged.Providers["openai"].ResponsesPath != "/alt/responses" { + t.Errorf("ResponsesPath = %q", merged.Providers["openai"].ResponsesPath) + } +} + +// TestResponsesPathOnWrongAdapterFails: only the Responses adapter reads +// responses_path. Set anywhere else it would vanish silently into a client +// that never looks at it, so it is rejected loudly instead — exactly the +// rule no_prompt_cache_key and cache_ttl already follow. +func TestResponsesPathOnWrongAdapterFails(t *testing.T) { + for _, tc := range []struct { + name string + key string + p Provider + }{ + {"openai-compat", "mycompat", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", ResponsesPath: "/alt"}}, + {"native anthropic", "anthropic", Provider{ResponsesPath: "/alt"}}, + } { + t.Run(tc.name, func(t *testing.T) { + c := &Config{Providers: map[string]Provider{tc.key: tc.p}} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatalf("mergeAndValidate accepted responses_path on a %s entry", tc.name) + } + if !strings.Contains(err.Error(), "responses_path") { + t.Errorf("error %q does not name the offending field", err) + } + if !strings.Contains(err.Error(), tc.key) { + t.Errorf("error %q does not name the offending key", err) + } + }) + } +} + +// TestUnknownTypeErrorListsOpenAI: the unknown-type and empty-type errors +// are a user's only map of what is valid. A new accepted type that never +// reaches those messages is undiscoverable. +// +// Both assertions match the rendered valid-types LIST, not the bare word +// "openai": every one of these messages already says "anthropic"/"openai" +// while describing the native keys, so a substring check for the type name +// alone passes even against code that never learned the type — a vacuous +// guard this test was caught being on its first red-verify run. +func TestUnknownTypeErrorListsOpenAI(t *testing.T) { + wantList := `"openai-compat", "openai"` + + c := &Config{Providers: map[string]Provider{ + "mystery": {Type: "carrier-pigeon", BaseURL: "http://x"}, + }} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on an unknown type") + } + if !strings.Contains(err.Error(), wantList) { + t.Errorf("unknown-type error %q does not list the valid types %s", err, wantList) + } + + c = &Config{Providers: map[string]Provider{"mycompat": {BaseURL: "http://x"}}} + _, err = mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on an empty type for an unknown key") + } + if !strings.Contains(err.Error(), wantList) { + t.Errorf("empty-type error %q does not list the valid types %s", err, wantList) + } +} + +// TestMergeProviderResponsesPath: responses_path layers like every other +// Provider field — a project layer may set it over a user layer. +func TestMergeProviderResponsesPath(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "secondary": {Type: TypeOpenAI, BaseURL: "https://gateway.example"}, + }} + over := &Config{Providers: map[string]Provider{ + "secondary": {ResponsesPath: "/alt/responses"}, + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if got := merged.Providers["secondary"].ResponsesPath; got != "/alt/responses" { + t.Errorf("ResponsesPath = %q, want the project layer's value", got) + } +} diff --git a/provider/openai/openai.go b/provider/openai/openai.go index 59ec50c1..9d8ef599 100644 --- a/provider/openai/openai.go +++ b/provider/openai/openai.go @@ -9,6 +9,7 @@ import ( "fmt" "io" "net/http" + "strings" "time" "github.com/majorcontext/harness/message" @@ -17,21 +18,66 @@ import ( const defaultBaseURL = "https://api.openai.com" +// defaultResponsesPath is the request path the OpenAI Responses API +// documents, and the only path this adapter could reach before +// Client.ResponsesPath existed. An empty ResponsesPath resolves to it, so +// every pre-existing caller's wire is unchanged. +const defaultResponsesPath = "/v1/responses" + // Client is a provider.Provider for the OpenAI Responses API. The zero value // plus APIKey is usable; nothing touches the network until Stream. type Client struct { - APIKey string - BaseURL string // defaults to https://api.openai.com + APIKey string + BaseURL string // defaults to https://api.openai.com + // ResponsesPath is the request path appended to BaseURL, defaulting to + // defaultResponsesPath. It is configurable because the Responses wire + // format is spoken by endpoints that do not serve it at OpenAI's own + // path: a vendor may expose an equivalent endpoint under a path of its + // own, which "/v1/responses" cannot reach no matter how BaseURL + // is written. + ResponsesPath string + // Family overrides the provider family key this client reports from + // Name() and uses as its ProviderData tag. Empty (the default) means + // the package Family constant, so every existing caller is unchanged. + // + // It is configurable for the same reason provider/openaicompat's is: + // more than one distinct endpoint can speak this wire, and two of them + // may be configured at once under different providers-map keys. The tag + // matters beyond routing. Reasoning items on this API are opaque, + // typically ENCRYPTED, endpoint-scoped state replayed verbatim on every + // later request. Tagging both clients "openai" would make the canonical + // format's family match succeed across two endpoints that do not share + // a key, so a session that switched between them would replay one + // endpoint's ciphertext to the other. Per-client families make that a + // cross-family drop instead — the canonical crossing rule, which costs + // a turn of reasoning continuity and nothing else. + Family string HTTPClient *http.Client // defaults to http.DefaultClient } -func (c *Client) Name() string { return Family } +// familyOrDefault resolves a configured family override to the family key +// actually used on the wire and in ProviderData: an empty override means +// the package Family constant. It is the single place that default lives, +// shared by Client (which configures it) and stream (which is also built +// directly, without a Client, by the fuzz harness). +func familyOrDefault(family string) string { + if family != "" { + return family + } + return Family +} + +// family resolves the provider family key for this client: the configured +// override, or the package constant when unset. +func (c *Client) family() string { return familyOrDefault(c.Family) } + +func (c *Client) Name() string { return c.family() } func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.Stream, error) { if c.APIKey == "" { return nil, fmt.Errorf("openai: no API key configured (set OPENAI_API_KEY)") } - wire, err := transcodeRequest(req) + wire, err := transcodeRequestFamily(req, c.family()) if err != nil { return nil, err } @@ -40,11 +86,7 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St return nil, err } - base := c.BaseURL - if base == "" { - base = defaultBaseURL - } - httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/v1/responses", bytes.NewReader(body)) + httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, responsesURL(c.BaseURL, c.ResponsesPath), bytes.NewReader(body)) if err != nil { return nil, err } @@ -65,12 +107,39 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St return nil, apiError(resp) } return &stream{ - body: resp.Body, - r: bufio.NewReader(resp.Body), - model: req.Model, + body: resp.Body, + r: bufio.NewReader(resp.Body), + model: req.Model, + family: c.family(), }, nil } +// responsesURL joins a base URL and a request path, applying each field's +// default and normalizing the separator between them to exactly one slash. +// +// Both halves are caller-supplied configuration now, so both of the obvious +// typos have to be absorbed. A path missing its leading slash is the +// dangerous one: "https://host" + "backend/responses" is not a wrong path, +// it is a request aimed at the host "hostbackend" — a different server +// entirely, and one an attacker could conceivably register. A trailing +// slash on the base is the harmless mirror image, normalized here for the +// same reason. +// +// The join is deliberately string-level rather than url.JoinPath: JoinPath +// re-encodes path segments, and this adapter must keep the default path +// byte-identical to the string it has always sent. Trimming every leading +// slash before re-adding one also rules out a "//..." path, which a URL +// parser reads as the start of an authority rather than as a path. +func responsesURL(base, path string) string { + if base == "" { + base = defaultBaseURL + } + if path == "" { + path = defaultResponsesPath + } + return strings.TrimRight(base, "/") + "/" + strings.TrimLeft(path, "/") +} + func apiError(resp *http.Response) error { raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) var body struct { @@ -146,6 +215,12 @@ type stream struct { body io.Closer r *bufio.Reader model message.ModelRef + // family is the client's resolved provider family: the ProviderData tag + // this stream writes reasoning attachments under. Empty means the + // package Family constant (see familyOrDefault), which keeps a stream + // built directly in a test — the fuzz harness does exactly that — + // behaving as it always has. + family string respID string items []*assembledItem @@ -496,7 +571,7 @@ func (s *stream) assemble() *message.Message { } msg.Parts = append(msg.Parts, &message.Reasoning{ Text: it.text.String(), - ProviderData: message.ProviderData{Family: it.raw}, + ProviderData: message.ProviderData{familyOrDefault(s.family): it.raw}, }) case "function_call": msg.Parts = append(msg.Parts, it.toolCall()) diff --git a/provider/openai/responses_path_test.go b/provider/openai/responses_path_test.go new file mode 100644 index 00000000..d2fcb094 --- /dev/null +++ b/provider/openai/responses_path_test.go @@ -0,0 +1,177 @@ +package openai + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// recordPath streams one trivial request through a client and reports the +// request path the server actually saw. The handler answers with a minimal +// completed-response SSE body so Stream returns without error; the test only +// cares about the URL the client chose. +func recordPath(t *testing.T, c *Client) string { + t.Helper() + var got string + handler := func(w http.ResponseWriter, r *http.Request) { + got = r.URL.Path + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, sse("response.completed", `{"type":"response.completed","response":{"id":"resp_1"}}`)) //nolint:errcheck + } + srv := httptest.NewServer(http.HandlerFunc(handler)) + t.Cleanup(srv.Close) + c.BaseURL = srv.URL + if c.APIKey == "" { + c.APIKey = "test-key" + } + s, err := c.Stream(context.Background(), &provider.Request{ + Model: message.ModelRef{Provider: Family, Model: "gpt-5"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + collect(t, s) + return got +} + +// TestResponsesPathDefault pins the pre-existing wire path for every caller +// that never sets ResponsesPath: the empty value must keep POSTing to +// /v1/responses, byte-identical to the hardcoded path it replaces. +// This is the "changed nothing for anyone" half of the ResponsesPath field. +func TestResponsesPathDefault(t *testing.T) { + if got := recordPath(t, &Client{}); got != "/v1/responses" { + t.Errorf("path = %q, want %q", got, "/v1/responses") + } +} + +// TestResponsesPathCustom is the reason the field exists: a Responses-API +// compatible endpoint need not live at /v1/responses. A deployment routing +// one model family at a vendor endpoint with its own path must be able to +// say so without the adapter appending its own suffix. +func TestResponsesPathCustom(t *testing.T) { + c := &Client{ResponsesPath: "/alt/responses"} + if got := recordPath(t, c); got != "/alt/responses" { + t.Errorf("path = %q, want %q", got, "/alt/responses") + } +} + +// TestClientFamilyDefaultsToPackageFamily proves the Family seam is +// backward compatible: a Client that names no family still reports the +// package constant, so every existing caller's ModelRef.Provider and +// ProviderData tag are unchanged. +func TestClientFamilyDefaultsToPackageFamily(t *testing.T) { + c := &Client{} + if got := c.Name(); got != Family { + t.Errorf("Name() = %q, want %q", got, Family) + } +} + +// TestClientFamilyOverrideTagsReasoning is the seam a SECOND native +// Responses provider needs. Two such providers can be configured at once, +// pointing at different endpoints, and a session may swap between them. +// Encrypted reasoning items are endpoint-scoped opaque state, so each +// client must tag (and later replay) them under ITS OWN family — the +// canonical cross-provider drop rule — rather than under a shared package +// constant that would replay one endpoint's items to the other. +func TestClientFamilyOverrideTagsReasoning(t *testing.T) { + c := &Client{Family: "secondary", APIKey: "test-key"} + if got := c.Name(); got != "secondary" { + t.Errorf("Name() = %q, want %q", got, "secondary") + } + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, streamFixture) //nolint:errcheck + })) + t.Cleanup(srv.Close) + c.BaseURL = srv.URL + + s, err := c.Stream(context.Background(), &provider.Request{ + Model: message.ModelRef{Provider: "secondary", Model: "gpt-5"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + + var done *provider.Event + for _, ev := range collect(t, s) { + if ev.Type == provider.EventDone { + done = &ev + } + } + if done == nil || done.Message == nil { + t.Fatal("no done event with an assembled message") + } + var reasoning *message.Reasoning + for _, p := range done.Message.Parts { + if r, ok := p.(*message.Reasoning); ok { + reasoning = r + } + } + if reasoning == nil { + t.Fatal("assembled message has no Reasoning part") + } + if _, ok := reasoning.ProviderData.Get("secondary"); !ok { + t.Errorf("ProviderData keys = %v, want the client's own family %q", keysOf(reasoning.ProviderData), "secondary") + } + if _, ok := reasoning.ProviderData.Get(Family); ok { + t.Errorf("ProviderData is tagged %q; a keyed client must not tag under the package constant", Family) + } +} + +func keysOf(pd message.ProviderData) []string { + out := make([]string, 0, len(pd)) + for k := range pd { + out = append(out, k) + } + return out +} + +// TestResponsesPathNoLeadingSlash is the malformed-URL case. ResponsesPath +// is caller-supplied configuration, so "backend/responses" is a typo an +// operator will eventually write. Concatenated naively onto a base with no +// trailing slash it yields "https://hostbackend/responses" — a request to a +// DIFFERENT HOST, not merely a wrong path, which is a far worse failure +// than the typo deserves. The adapter absorbs it. +func TestResponsesPathNoLeadingSlash(t *testing.T) { + c := &Client{ResponsesPath: "alt/responses"} + if got := recordPath(t, c); got != "/alt/responses" { + t.Errorf("path = %q, want %q", got, "/alt/responses") + } +} + +// TestResponsesURLNormalization pins the joining rule itself, including the +// mirror-image typo (a trailing slash on the base) that would otherwise +// produce a doubled separator. +func TestResponsesURLNormalization(t *testing.T) { + tests := []struct { + name string + base string + path string + want string + }{ + {name: "default path", base: "https://api.example.test", want: "https://api.example.test/v1/responses"}, + {name: "leading slash", base: "https://api.example.test", path: "/backend/responses", want: "https://api.example.test/backend/responses"}, + {name: "no leading slash", base: "https://api.example.test", path: "backend/responses", want: "https://api.example.test/backend/responses"}, + {name: "trailing slash on base", base: "https://api.example.test/", path: "/backend/responses", want: "https://api.example.test/backend/responses"}, + {name: "both typos at once", base: "https://api.example.test/", path: "backend/responses", want: "https://api.example.test/backend/responses"}, + {name: "base carries a path segment", base: "https://api.example.test/alt-api/v2", path: "/responses", want: "https://api.example.test/alt-api/v2/responses"}, + {name: "empty base uses the default", base: "", path: "/responses", want: defaultBaseURL + "/responses"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := responsesURL(tt.base, tt.path); got != tt.want { + t.Errorf("responsesURL(%q, %q) = %q, want %q", tt.base, tt.path, got, tt.want) + } + }) + } +} diff --git a/provider/openai/transcode.go b/provider/openai/transcode.go index 54764dc0..a50c084b 100644 --- a/provider/openai/transcode.go +++ b/provider/openai/transcode.go @@ -149,8 +149,21 @@ func wireCallID(id string) string { return message.ProviderCallID("call_", id, 64) } -// transcodeRequest maps a canonical request to the OpenAI Responses API. +// transcodeRequest maps a canonical request to the OpenAI Responses API +// under the package Family constant — the default for a client that +// configures no family of its own. func transcodeRequest(req *provider.Request) (*apiRequest, error) { + return transcodeRequestFamily(req, Family) +} + +// transcodeRequestFamily is transcodeRequest with the ProviderData tag made +// explicit. family is the calling client's resolved family key: the tag +// stored reasoning attachments are replayed FROM, so a second Responses +// endpoint configured under its own providers-map key reads back only the +// items it produced itself, and drops the other endpoint's (the canonical +// cross-family rule — those items are opaque, usually encrypted, and +// endpoint-scoped). +func transcodeRequestFamily(req *provider.Request, family string) (*apiRequest, error) { out := &apiRequest{ Model: req.Model.Model, Instructions: strings.Join(req.System, "\n\n"), @@ -227,7 +240,7 @@ func transcodeRequest(req *provider.Request) (*apiRequest, error) { messages := imageclamp.Clamp(message.NormalizeForWire(req.Messages), imageLimits) for i := range messages { m := &messages[i] - items, err := transcodeMessage(m, stripReasoning) + items, err := transcodeMessage(m, stripReasoning, family) if err != nil { return nil, fmt.Errorf("openai: message %s: %w", m.ID, err) } @@ -245,8 +258,9 @@ func transcodeRequest(req *provider.Request) (*apiRequest, error) { // stripReasoning reports whether stored reasoning items must be dropped (see // the Reasoning case). It is true ONLY for an explicit EffortOff, never for // the default EffortUnset — an unset session replays stored reasoning items, -// which gpt-5 stateless multi-turn tool use requires. -func transcodeMessage(m *message.Message, stripReasoning bool) ([]json.RawMessage, error) { +// which gpt-5 stateless multi-turn tool use requires. family is the calling +// client's ProviderData tag (see transcodeRequestFamily). +func transcodeMessage(m *message.Message, stripReasoning bool, family string) ([]json.RawMessage, error) { role := "user" if m.Role == message.RoleAssistant { role = "assistant" @@ -362,7 +376,7 @@ func transcodeMessage(m *message.Message, stripReasoning bool) ([]json.RawMessag // from the intact history. continue } - raw, ok := v.ProviderData.Get(Family) + raw, ok := v.ProviderData.Get(family) if !ok { // Another provider's reasoning, or a present-but-empty // entry (see message.ProviderData.Get — this is the From fd10d2724058be12837ff0a5388e78e3a73b2159 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 26 Aug 2026 21:34:55 -0400 Subject: [PATCH 10/95] feat(engine,server): answer session reads from a metadata index (#195) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(engine,server): answer session reads from a metadata index Opening a box console is slow in proportion to the box's history. The deepest single cause sits here. GET /session/{id} called engine.LoadSession for any session this process does not hold live. That call replays the whole journal. It decodes every message body, rebuilds the history, and repairs it. The handler then read a dozen scalar fields and dropped the rest. Measured against production on 2026-08-26: about 7 s per read on the fleet's longest session. GET /session paid that once per non-live session, about 2 s in total. Every control-plane proxy route calls the list first. One goal read therefore cost a list plus a load, in series, uncached. See meetneptune/boxes docs/design/console-read-path.md, workstream 1. Each session log now has a sidecar .index.json. It holds one engine.SessionIndex. The index carries every wire Session field with a durable source: timestamps, model, effort, workdir, parent session, task lineage, message count, usage, durable goal state, queue depth, and compaction counters. GET /session and GET /session/{id} render a non-live session from it. The index is a fold of the journal. Three rules keep it honest. It folds records, not memory. Session.writeRecord folds each record it appends. ensureLog folds the header records it writes directly, which is the one write that does not pass writeRecord. The distinction is load-bearing. EnqueuePromptDurable writes its record before it mutates the queue. A fold of memory at that instant would disagree with the log it claims to summarize. It is a cache, never an authority. ReadSessionIndex serves a stored index only when three checks pass: a checksum over the stored bytes, the journal's byte length, and the journal's modification time. Anything else refolds from byte 0. There is no repair path, so no repair path can be wrong. It reports what a full load reports. Messages and LastActivityAt run through message.ResolveOrphanToolCalls. That repair reads roles and tool-call ids, so the fold runs it over a skeleton of exactly those fields. A crash between a tool call and its result therefore counts the same on both paths. DurableMessages counts the records alone, for a reader that must map a message to a byte offset. The fold is slim. It decodes ids, roles, timestamps, and tool-call ids, never message bodies. On a 723 KB journal a refold costs 2.8 ms. A full LoadSession costs 13.0 ms. A current index costs 55 us. Write-through adds about 27 us per record, which is one marshal plus one rewrite. A turn writes a handful of records. Some journals cannot be answered by a fold. A legacy header records no workdir. A crash can tear away the initial model record. LoadSession answers both from the loading Config. A fold has no Config. SessionIndex.Complete reports the gap, and Server.coldSessionJSON then uses the load path. That predicate covers the whole fallback surface only because the loading Config must stay generic, which Options.LoadSession now states. A cold read also re-checks residency at the end. That narrows the window where a session goes live between the residency check and the index read. Four alternatives were rejected. A fold of live Session state needs no records, but it puts a second source of truth beside the journal, and the EnqueuePromptDurable window proves the two can disagree. A rebuild through LoadSession is correct, but every restart-era read then pays the cost this change removes. An incremental catch-up saves milliseconds on a path that is already milliseconds, and it needs state the sidecar does not carry. Publication by rename is atomic, but it creates a directory entry per write, and a write that races a directory removal resurrects an entry; the checksum answers the same hazard and also catches a mixed read of two same-length indexes. Three folds have real state machines. Each now has one implementation, shared rather than copied: applyCompactRecord (compact.go), applyGoalRecord (store.go), and promptQueueFold (queue.go). LoadSession's behavior is unchanged. The rules moved verbatim. One durability fix rides along, because the index made the window legible. A record write that fails after putting bytes on disk leaves a torn final line. ensureLog repairs such a line, but its fast path returns while a handle is open. The next append therefore landed on the torn line with no separator. The two lines became one unparseable line, which scanLog hard-fails as soon as a later record makes it non-final. A retry of a failed EnqueuePromptDurable is exactly that shape. writeRecord now closes the handle on a failed write, so the next persist repairs the tail. A descriptor fix rides along for the same reason. A Session holds two handles for its whole life, its journal and now its sidecar, and a server keeps one Session per session it has touched. Session.ReleaseFiles drops both, and the server calls it when it evicts a session from residency — the point it has already decided the session is idle and reloadable. The session stays usable: the next persist reopens both through ensureLog. The call runs after s.mu is released, because s.mu is a leaf lock with respect to a session's own mutex. Semantic changes, stated precisely. engine.SessionInfo.Messages, and so GET /session/status, now counts messages after compaction folds. The previous header scan counted every message record ever written, and over-reported a compacted session. server.Options.Plugins is new and takes a session id. Plugins are process configuration, not durable session state, and a cold read has no Session to ask. Reads of a live session are unchanged in every respect. A running turn is still rendered from its live object, so status never falsely reports idle. Verification: go test -race ./... clean. Every guard is red-verified against the mechanism its name claims. The oracle test fails on its compaction case when the fold appends a summary instead of splicing, and on its orphan case when the repair is skipped. The currency and resume tests fail without the write-through flush and without LoadSession's fold seeding. The refold tests fail without the length check, without the modification-time check, and without the checksum. The unusable-sidecar cases fail without the version and id checks; each mangled sidecar also carries a wrong message count, so serving it is detectable. The completeness test fails when every journal is called complete. The oversized-header test fails when a too-long first line is read as a verdict. The handle test fails when a failed write keeps its handle. The eviction test counts this process's open descriptors across an eviction and fails when the release is a no-op. The cold-path server tests fail when the handlers revert to the LoadSession lookup; they corrupt the journal to unreadable bytes of the same length and modification time first, so a handler that still replays it cannot pass. * perf(engine): keep the index write path constant per record Review round 5 on the metadata index found the write path had the cost profile inverted. Session.writeRecord flushes the sidecar after every record, and indexFold.snapshot ran message.ResolveOrphanToolCalls over the whole message skeleton each time. That is O(n) per record, so O(n squared) over a session's life. An index that makes reads cheap and writes quadratic is a net loss on exactly the long sessions it exists for. The repair reads one pair of messages at a time — an assistant message and whatever follows it. Appending a message can therefore change the decision for exactly two messages: the new one, which has no follower yet, and the one before it, whose follower just changed. Every earlier pair is untouched. indexFold.appendMessage updates both and keeps a running count. It still CALLS the real repair rather than restating its rule, over a two-message window, and attributes insertions positionally: the repair preserves order and inserts directly after the message that earned the insertion, so anything before the follower in the result belongs to the message under test. Restating "an assistant message with an unmatched tool call gets one synthetic result unless a tool message follows" would be a second copy of a rule this repository has already been burned by copying. Compaction is the one operation that rewrites the skeleton's middle, so it recounts from scratch: O(n) per compact record, not per record. Measured on a skeleton of 8000 messages, snapshot fell from 881 us to 198 ns. It is now flat across 100, 1000, and 8000 messages. Three smaller findings from the same round: GET /session held an index for every session it listed, then passed the id to coldSessionJSON, which read the index again. It now passes the index it holds, so an incomplete session costs one stat and one sidecar read instead of two. Session.indexFile's comment described a lifetime the same PR had already changed: it named collection as the release point, after ReleaseFiles and releaseEvicted existed. writeSessionIndex now documents why two uncoordinated writers to one sidecar are safe — each writes a complete index of the prefix it folded, with that prefix's staleness key, so the loser leaves a file that is current or visibly stale, never a blend, and a stale sidecar costs one refold. Verification: two new guards, both red-verified. One compares bytes allocated per snapshot at 100 and at 4000 messages, and fails at 13 KB against 516 KB when the repair runs over the whole skeleton again. The other checks the maintained count against a full recount after every append, over six tool-call shapes, and fails when the incremental update drops a term. The existing oracle test still pins every index field against a full LoadSession. Two existing tests started a turn and returned without waiting for it. Their background writes raced t.TempDir's cleanup, which fails with "directory not empty" once a session creates a second file. Both now wait on a production seam — GET /wait, and SessionManager.Changed — never a sleep. * fix(engine): find a repair window's follower by marker, not by id Review of the previous commit found the incremental repair count wrong for a journal that repeats a message id, or carries none. repairsAt runs the real repair over a two-message window and attributes insertions positionally: everything before the FOLLOWER in the result belongs to the message under test. It located the follower by its id. Two adjacent records with the same id — or with no id, which a malformed record produces — made that search land on the window's FIRST message instead, returning -1. The running count then drifted below zero, and SessionIndex.Messages fell below the durable message count for the rest of the session. Both shapes are reachable. A provider-derived message id is hashed from the message's own text (message.ProviderCallID), so two identical replies carry one id. An absent id is what a truncated or hand-repaired record leaves. The follower now carries a marker: a Text part appended to the copied window. The repair reads roles, tool calls, and tool results, so the marker changes no decision, and the repair never moves a part between messages, so exactly one message in the result carries it. The window is a throwaway copy, so the marker never reaches a caller. Verification: four shapes added to the count-versus-recount table — duplicate ids on adjacent orphans, absent ids, an id wearing the repair's own synthetic shape, and duplicate ids across a matched call — plus an assertion that the count is never negative. Against the id-based lookup the table fails with "repair count is negative (-1)" and "count 0, but the repair inserts 2". * fix(engine,server): recover a broken fold, and skip indexes for live sessions Two findings from the latest review round on the metadata index. A failed record write marks the session's fold broken, because the fold no longer knows what the journal holds. That flag was set-only. One transient write failure therefore disabled the index for the whole life of the session object: every later read of that session refolded the entire journal, which is the cost the index exists to remove. A failed write already forces a reopen — writeRecord drops the handle so ensureLog runs its tail repair. That reopen now also re-seeds the fold from the journal as the repair left it, so the cost is one slim fold per failure rather than one per read. A fold that fails again leaves the flag set, exactly as before. GET /session read an index for every session file, then discarded it for any session it renders from a live object. Reading them is work thrown away, and worse: a stale sidecar for a LIVE session was refolded and written back by the listing while that session's own writer held it. The listing now resolves ids first (engine.ListSessionIDs, one directory scan, no journal and no sidecar read), and reads an index only for an id nothing live holds. Verification: both guards red-verified. The recovery test injects a write failure at the OS level through the production persist path, then checks the next turn clears the flag and leaves a CURRENT sidecar — read against a journal corrupted at an unchanged staleness key, so only a current sidecar can answer. The listing test removes a live session's sidecar and fails with "the listing refolded and wrote a sidecar for a live session" when the id scan is replaced by an index scan. * docs(engine): say why a broken fold is re-seeded, never just cleared A review round asked for the reasoning behind the broken-fold flag to be written down, because the obvious "simplification" of the recovery path reintroduces a silent wrong-index bug. A failed Write can land a record's bytes and not its trailing newline. ensureLog's tail repair then takes its case-2 branch: the tail parses, so the record is terminated and KEPT. The fold never saw that record. Clearing the flag at that point would resume flushing a sidecar short by one message, while logSize claimed the whole file — a stale index that reads as current, which no reader would ever refold. Folding the file again is what makes the fold agree with the bytes on disk, whichever branch the repair took. The recovery already re-seeds. This states the distinction where the code is, and pins it with a test. Verification: the new test writes a complete record with no trailing newline, marks the fold broken, and drives a reopen. Replacing the re-seed with a bare `broken = false` fails it with "Messages = 4, want 5: two turns of two, plus the record the repair kept". * fix(engine,server): answer session existence with one stat Two findings from the latest review round on the metadata index. sessionOnDisk backs the existence check on three hot paths — abort, end, and wait — and it walked the whole session directory. Since ListSessions became a projection of the index, that walk stats and reads a sidecar for every session in the directory, and refolds and writes back any that is stale, including live ones. That is the exact hazard GET /session was reworked to avoid, paid per abort. engine.SessionExists answers the question directly: one stat, no read. It also answers PRESENCE rather than readability. The listing skips a journal it cannot fold, so an existence check built on it reported "no such session" for a session whose bytes are damaged — and an abort or a wait against a real, damaged session deserves better than a 404. GET /session rendered an index-backed session directly, while GET /session/{id} rendered the same case through coldSessionJSON, which re-checks residency after reading the index. The listing therefore skipped that re-check, so the two endpoints could disagree about whether a session is running: claimForPrompt can make a session live between the residency check and the index read, and the listing would report the running session as idle. Both paths now go through coldSessionJSON. The window itself is not reachable from a test, so the guarantee is structural — one shared path rather than two kept in step by hand. Verification: the existence test corrupts a session's journal beyond folding, drops its sidecar, and requires POST /abort to answer 204 and to write no sidecar. Against the directory walk it fails with "POST abort on an existing but unreadable session = 404". A second test pins that the listing and the single-session read report the same status for a running session. * feat(engine,server): page message reads from the journal tail (#196) * feat(engine,server): page message reads from the journal tail GET /session/{id}/message answered with the entire message history, always. On the fleet's longest production session that is 1.4 MB and 5.8 s, measured on 2026-08-26, paid on every console open. The renderer then froze under about 500 messages arriving at once. A console shows the tail and pages older messages in on scroll. It needs a bounded read. The existing max_bytes truncation is not one: it truncates after the whole history is already materialized, so it saves wire bytes and no load time. See meetneptune/boxes docs/design/console-read-path.md, workstream 2. The endpoint now takes before_seq and limit. A request that names neither is unchanged, byte for byte: the bare array of the whole history every existing caller already parses. A request that names either gets a MessagePage envelope: messages, first_seq, last_seq, total, has_more. A client that pages needs the page's position. A client that does not must not have to learn a new shape. A message's seq is its 1-based ordinal in the DURABLE message sequence: message records in log order, with each compact record's fold applied. engine.SessionIndex.DurableMessages counts that sequence, so the newest message's seq equals it and a client can size a scrollbar from one response. That definition is the load-bearing decision. An ordinal over durable records can be counted backwards from the end of a file. An ordinal over a materialized history cannot be known without materializing it, which is the cost this change removes. A page therefore reports a total that can be LOWER than the messages field of GET /session. That field counts what a full load produces, including the synthetic tool results message.ResolveOrphanToolCalls derives. A derived message has no record. It has no byte offset, so it can carry no seq, and no page carries it. Numbering derived messages instead would let a page a client already holds renumber under it. Two paths serve a page, both numbered by the same index. The tail walk reads blocks backwards from the index's LogSize and numbers message records down from the total, so it touches only the tail. It gives up the moment it meets a compact record. The messages a fold KEPT sit in the log between the folded range and the compact record itself; an earlier revision tried to undo that in reverse, and its first compaction test caught the page short by the whole kept tail. Rather than grow a second implementation of a fold, the fallback reuses the forward one: engine's own indexFold, and so the same applyCompactRecord LoadSession uses. It learns which ids occupy the requested seqs, then reads back just those records. That costs one slim pass, and it runs only for a page that reaches into compacted history. The fold path reads exactly the journal prefix its index summarized, and nothing past it. A compaction landing during the read would otherwise renumber the page while the response still reported the old total: one answer describing two instants of the journal. The reverse scanner works in offsets, never in an accumulating buffer. Each block read searches backwards for a newline, and the line it delimits is then read once at its exact length. An earlier revision prepended each block to a carry buffer, which copies the whole partial line per block: one 20 MB record spans hundreds of blocks and copied gigabytes before decoding one message. Journals carry records that large. The scan is bounded by SessionIndex.LogSize rather than the file's current size, so a turn appending records cannot renumber a page under its reader. A journal that SHRANK under the read — ensureLog repairing a torn tail — invalidates the numbering instead. Two checks report that, covering different windows: one before the scan, and one that re-stats after a scan fails on bytes that are gone. Either way the read takes a fresh index and retries once. A page carries durable messages verbatim. It never runs message.ResolveOrphanToolCalls. That repair keeps a provider request valid, this endpoint builds no request, and fabricating a tool failure in a read view has production history: see Server.lookup's doc comment, where a healthy child's in-flight tool call rendered as failed for as long as it kept running. Compaction renumbers. A fold replaces N messages with one summary, so every later seq shifts down. A client paging across a compaction can see one page overlap another. Message ids are stable and are the way to de-duplicate. An opaque id cursor was rejected for the opposite reason: it cannot answer "how far along is this page" without a second call, which is what a scrollbar needs. A page is read from the durable records even for a resident session. One seq definition then covers both cases. A resident history can carry messages the log does not — load-time repairs, recovery's memory-only closers — and numbering those would give one message two seqs depending on residency. The fallback keeps that contract rather than bending it. Resident history is paged for exactly one case: a live session with no durable journal at all, whose history IS its durable sequence. A journal that exists but cannot be read is a 500, for a live session too. Paging memory there would hand a repair message a sequence number, and a client that paged again after the journal became readable would see its pages renumbered. A 404 would be a different lie: it sends an operator looking for an id that is on disk in front of them. Verification: go test -race ./... clean. The oracle is independent: it derives the durable sequence from the journal's bytes in the test itself, never through LoadSession or indexFold, because those share applyCompactRecord with the code under test. A separate check pins the oracle against LoadSession's own history for the shapes where the two must agree. The page oracle drives every (before_seq, limit) pair against a 12-message session. A second test walks a compacted session one seq at a time across the compaction boundary, so the tail path and the fold path are proved to agree message for message. Red-verified, each against the mechanism its name claims. The compaction tests fail when the tail walk does not give up at a compact record. The derived-repair test fails when the page is numbered against SessionIndex.Messages. The tail-only test overwrites everything except the final 8 KiB of an 80-message journal and still serves the newest page, and fails when the fold path is forced. The scanner boundary table fails when the bound is not clamped to the file size. The truncating-repair test fails without the index's length check. The compatibility test fails when the envelope is served unconditionally. The classification tests fail when an unreadable journal reports 404, when a live session's resident history is paged for an unreadable journal, and when a repeated parameter is silently accepted. The stale-bound tests drive readMessagePageWithIndex and pageError directly, because the window they cover opens after the public call has already taken its index. The same seam proves the fold path's bound: a compact record appended after the index was taken must not reach the page, and the test fails when the read is unbounded. * perf(engine,server): decode only the records a message page carries Review on the paginated message read found the fold path still paying the cost the endpoint exists to remove. foldedPage serves a page that reaches into compacted history. It ran two scans: a slim fold, to learn which message ids occupy the requested seqs, and then a second scanLog over the same journal that decoded EVERY line into a full record to find them. That second scan decoded every message body in the file. A page of ten messages from a session with two thousand paid for two thousand. There is now one pass with two decode depths. scanLogRaw hands each raw line to the fold, which decodes it through the slim indexRecord — ids, roles, timestamps, tool-call ids, never a body — and keeps that line, by the id it contributes to the sequence, as a subslice of the data already in memory. Only the handful of lines the page carries is then decoded in full. scanLogRaw is scanLog with the decode removed, and scanLog is now written in terms of it, so the corruption discipline both readers depend on has one implementation: a corrupt or truncated FINAL line ends iteration silently, corruption anywhere else is an error. A sentinel carries that first case across the split. The OpenAPI spec now documents the 500 the page routes can return, and says why an unreadable journal is not a 404: a session whose journal exists is not "no such session", and reporting it as one sends an operator looking for an id that is on disk in front of them. The 404 description names the same distinction. Verification: a new guard, red-verified. It rewrites a message record that a compaction folded away to carry a part of an unknown type. That record is valid JSON, so the slim fold walks past it, but a full decode fails — unmarshalPart rejects an unknown part type. A page that excludes the record therefore proves it was never fully decoded. Against the two-scan shape the test fails with "unknown part type \"from_a_newer_binary\"", from a record no page asked for. * fix(engine,server): absorb only the scanner's own end-of-scan sentinel Review of the previous commit found a real defect in the scanLog split it introduced, plus three smaller points. scanLogRaw ended a scan cleanly whenever a callback's error satisfied errors.Is(err, errTruncatedFinalRecord). The sentinel means one thing — the decoder hit a corrupt FINAL line, which every reader ignores as a crash mid-write. A callback that wrapped it into a genuine failure, say "cannot update index: %w", had that failure read as a torn record and reported as a clean scan. The comparison is now by identity, so the signal stays with the one decoder that raises it. foldedPage's line map holds one entry per message RECORD in the folded prefix, not one per message in the resulting sequence: a record a compaction folded away keeps its entry. The comment claimed otherwise. The behavior is deliberate — each entry is an id string and a slice header beside a prefix already held in memory in full — and now says so. An id appearing on two records now keeps the FIRST rather than the last. Engine-minted ids are unique. The one production source of a repeat is a provider-derived id hashed from the message's own text, where both records carry the same content anyway. A journal that repeats an id with different content is damaged, and first-wins at least matches spliceCompact's own first-match rule. TestFoldedPageDecodesOnlyThePage now asserts its own premise: the probe record must fail a full decode. The guard rests on canonical decoding rejecting an unknown part type, and if that ever becomes forward-compatible the test fails loudly there instead of quietly becoming a test that passes from birth. The OpenAPI 500 description said "a corrupt record" where only a corrupt NON-FINAL record produces it. Verification: a new table test drives scanLogRaw with the bare sentinel, a wrapped sentinel, and an unrelated error. It fails against errors.Is with "scanLogRaw swallowed cannot update index: engine: truncated final record". * fix(server,engine): number the fallback page over durable messages only Review triage on the paginated message read: one correctness point and four small ones. The resident fallback is the one place a page is numbered from memory rather than from records. It numbered sess.History() directly. A resident history can carry a message that message.ResolveOrphanToolCalls derived, which has no record, so no byte offset and no sequence number — and giving one a seq would make a fallback page contradict a journal page for the same session. The fallback now filters those out. The filter is reachable only for a session with no journal, whose history carries no derived message today, so this makes the two paths agree by construction rather than by an argument about which shapes can reach the fallback. The OpenAPI spec documents the 500 that fallback can return, and says why an unreadable journal is not a 404. That landed with the previous commit; this one adds nothing there. The scanLog double-pass thread is settled by the previous commit: foldedPage runs ONE scanLogRaw pass, folds each line through the slim indexRecord, keeps the raw line by the id it contributes, and fully decodes only the records the page carries. The second pass, which decoded every message body in the journal, is gone. Three small ones. A case table declared a `want` field and then decided by test name, so the field was dead — it now drives the assertion. handleMessages parsed r.URL.Query() up to four times per request, once per parameter test and once per read; it parses once and passes the values down. scanLogBackward re-stat'd a file its caller had just stat'd; it takes the size instead. Verification: durableOnly has a direct test that fails with "returned 4 messages, want 3" when the filter is a passthrough. A second test pages the same live session from its journal and then with the journal removed, and requires the two pages to agree on total, on both seqs, and on every message id. * perf(engine,server): decode only the page in the tail path too Three findings from the latest review round on the paginated message read. tailPage full-decoded every record it walked past. Paging back to an older page therefore decoded one message body per record NEWER than the page — the same O(n) cost the fold path was rewritten to remove, in the other code path. It also made a page fail on a record it never asked for: a message a newer binary wrote, carrying a part this build cannot decode, failed the whole read even when the page lay entirely before it. The walk now decodes a slim head — a type, and a message's id — and decodes in full only a record INSIDE the requested range. Both page paths now have the same rule: decode what the page carries, walk past the rest. The window arithmetic — the limit clamp and the hi/lo range — lived twice, once in the engine's page read and once in the server's resident fallback. Two copies would give one session two different paginations depending on which path answered it. engine.MessagePageWindow is now the one implementation, and both call it. The `limit` contract disagreed with itself. The published schema names a maximum, which a generated client or a gateway enforces, while the server silently clamped a larger value and answered a smaller page. The HTTP boundary now rejects an oversized limit with 400. The engine API still clamps, for a direct caller that has no schema to honor, and both say so. Verification: three guards, all red-verified. The tail-path guard rewrites the NEWEST record to carry a part of an unknown type — valid JSON, so a slim decode passes it, while a full decode fails — and asks for the oldest page; against the full decode it fails with "journal holds 1 of those messages". The limit guard fails with a 200 and a full page when the rejection is removed. The window guard pages one session four ways from its journal and again with the journal removed, requiring identical windows. --------- Co-authored-by: andybons * fix(engine,server): list from the journals, accelerate with the index Four findings from the latest review round, two of them correctness. ListSessions had become a projection of ListSessionIndexes, which made the index the source of truth about EXISTENCE. A session whose fold breaks — a damaged compact record, say — has no index, so the listing dropped it. A list that omits a real session lies to every caller that asks what is here, including the existence checks behind abort, end, and wait. The journals are what exist. ListSessions enumerates them, answers each from its index when the index can answer, and falls back to a direct scan of the journal when it cannot. Only a file that is not a session log at all is skipped. ReadSessionInfo is the single-session form of the same path, so GET /session/status and GET /session cannot disagree about which sessions are there — before this, status skipped a session the listing kept. A read must not mutate. Listing a directory refolded and wrote back a sidecar for every session whose index was stale, including sessions this process holds live, racing their own writers. readSessionIndexAt now takes a cache flag: a single-session read still memoizes, because that is one fold and it makes the next read of that session a stat and a small read; a listing never writes. GET /session rendered an index-backed session directly, while GET /session/{id} rendered the same case through coldSessionJSON, which re-checks residency after reading the index. The listing therefore skipped that re-check, so the two endpoints could disagree about whether a session is running. Both paths now go through coldSessionJSON. The prompt-queue fold's nextID comment claimed the counter advances for every record "folded or skipped". It advances only for a record the fold accepts, which is correct — a duplicate id was already cleared by the record that folded first, and an id at or below zero can never reach a counter that starts at 1 — but the comment invited a maintainer to hoist the advance above the guard that rejects those records. The fallback scan reports what the index would have reported, not what the pre-index scan did: it counts a compact record's usage, which LoadSession and the index both count. LastInputTokens still moves for message records only. Verification: four guards, all red-verified. A listing must keep a session whose fold breaks ("a session whose fold breaks was dropped from the listing"); a listing must write no sidecar ("ListSessions wrote a sidecar; a listing must not repair one"); status and list must report the same sessions ("status is missing session ..."); and the fallback must count a compact record's usage. A fifth pins that the listing and the single-session read agree on a running session's status. Known and unchanged from before this branch: `harness run -c` picks the newest LISTED session and fails if it cannot load, so a damaged newest session breaks it. That is main's behavior today — verified directly against main, where the same journal is listed and the same load fails — and it is a pre-existing wart this change restores rather than introduces. * fix(engine): count only the usage a load counts, and pin the filename id A regression hunt over the previous commit found two defects in its new fallback scan, and one asymmetry worth stating rather than fixing. The fallback accumulated usage from ANY record carrying a usage field. LoadSession reads exactly two — a message record and a compact record — so a stray usage field, which a future build could write on a goal record, inflated a listing past what the authoritative load reports. The scan now reads the same two, and LastInputTokens still moves for message records only, so a summarization call is never reported as a session's last request size. The fallback returned the id from the session's own header. The index path pins the FILENAME, as LoadSession does, so one file could report two different ids depending on which path answered it. The fallback pins the same way. The asymmetry, stated rather than changed: a journal whose fold breaks and whose load fails is omitted from GET /session and reported by GET /session/status. A listing entry names a model, a workdir, and lineage, and a load that fails supplies none of them; GET /session/{id} 404s for the same session, so omitting it keeps the two reads consistent. Status promises only counts, which the scan still gives. This is main's behavior, verified directly against main, where the same journal is absent from GET /session and present in GET /session/status — the index did not introduce it. A test now names it, so a later reader finds it stated. Verification: both fixes red-verified. The stray-usage guard fails with "usage = 511, want 11" against the old accumulation, and the id guard fails reporting the header's id for a file named after another session. * perf(engine): a tail page reads the span it walks, once A review round found the tail page path reading far more than it needs. Walking back to an older page passes every record after it. Two things made that expensive. Each line was read WHOLE before the walk classified it, so a 20 MB image blob or tool result newer than the page was pulled into memory to learn a type string and dropped. And the newline search read a fresh 64 KiB block per LINE, so a journal of small records was read several times over: a boundary 200 bytes away cost a block. A line now arrives as a PREFIX plus the means to read the rest. The walk classifies from the prefix — the type is the first field of every record this package writes — and reads a body only for a record the page carries. The block search keeps one block in memory and moves backwards through it, serving every line whose search and prefix fall inside it, so the span between the page and the end of the file is read about once. Reading that span at all is not avoidable: the line boundaries are in it, and there is no way to count records without finding them. Reading it twice was. Two shapes need the whole line regardless, and both still get it. The newest line is the one a crash can leave torn, and a torn line's prefix still parses a type, so it is read whole and required to parse — exactly as the fold that numbered the page did. A prefix that ends mid-key is inconclusive rather than corrupt, so the line is read whole before any verdict. scanLogBackward now takes an io.ReaderAt rather than an *os.File, so a test can count what a page read actually reads. That is the only way to prove this: decoding less means nothing if the bytes are read anyway. Verification: red-verified twice against one guard. A journal with one 164 KB record and six small ones, paged for its two OLDEST messages, reads 165 KB — the span, once. Reading each line whole to classify it takes 329 KB, and refilling a block per line takes 301 KB. The same test checks the page that DOES carry the big record still reads it in full, so the number is a bounded read rather than a scan that skipped the work. * docs(engine): say why a page read memoizes its index and a listing does not A review thread asked whether the page path's sidecar write-back races a live session's own writer. It can, and the overlap is bounded and benign: each writer writes a complete index of the prefix it folded, carrying that prefix's staleness key, and the checksum covers those bytes, so a reader sees a file that is current or visibly stale, never a blend. The loser costs one refold. Removing the write-back would cost more than it saves on this path: this is one session, and every page request for a session whose sidecar is stale would refold the whole journal again. A LISTING is the opposite case — N sessions, per call, including live ones — and those paths already pass cache=false. readMessagePage now carries that reasoning, so the difference between the two is stated where a maintainer meets it. * fix(engine): keep the record-type fast path an optimization, not a format rule The tail page's prefix scan reads ONE key to classify a record, which answers only for a record whose first field is the type. That is every record this package writes today, because record's Type field is declared first — but a record with another field order is not corrupt, and a page that failed on one would turn a fast path into a format requirement that nothing states or enforces. The scan now falls back to a decode that finds the type wherever it sits, after reading the line whole. The prefix stays an optimization. Verification: a journal whose message record puts "message" before "type" pages correctly, and fails with "corrupt record" without the fallback. --------- Co-authored-by: andybons --- AGENTS.md | 125 ++++ cmd/harness/main.go | 6 + cmd/harness/plugins.go | 16 + engine/compact.go | 18 + engine/engine.go | 26 + engine/index.go | 934 +++++++++++++++++++++++++ engine/index_test.go | 1243 ++++++++++++++++++++++++++++++++++ engine/messagepage.go | 690 +++++++++++++++++++ engine/messagepage_test.go | 1117 ++++++++++++++++++++++++++++++ engine/queue.go | 109 +++ engine/store.go | 507 ++++++++++---- engine/store_failure_test.go | 179 +++++ engine/store_test.go | 29 + server/cold_read_test.go | 602 ++++++++++++++++ server/handlers.go | 499 ++++++++++++-- server/id_test.go | 7 + server/journal.go | 16 +- server/message_page_test.go | 441 ++++++++++++ server/openapi.yaml | 150 +++- server/server.go | 22 + server/session_tree_test.go | 11 + 21 files changed, 6522 insertions(+), 225 deletions(-) create mode 100644 engine/index.go create mode 100644 engine/index_test.go create mode 100644 engine/messagepage.go create mode 100644 engine/messagepage_test.go create mode 100644 server/cold_read_test.go create mode 100644 server/message_page_test.go diff --git a/AGENTS.md b/AGENTS.md index 7f0c70ff..cc3dddaa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -940,6 +940,131 @@ The goal loop is a **plan-artifact-free, gate-free** control loop: it is mode, and no permission gate. It does not violate the no-plan-mode decision below. +### Session metadata index + +`GET /session` and `GET /session/{id}` do not replay a session journal. Each +session log has a sidecar `.index.json` holding one +`engine.SessionIndex`. The index carries every wire `Session` field with a +durable source: timestamps, model, effort, workdir, parent session, task +lineage, message count, usage, durable goal state, queue depth, and +compaction counters. + +Before the index, a read of a non-live session called `LoadSession`. That +call decodes every message body and rebuilds the whole history. The handler +then reported a dozen scalars and dropped the rest. The list endpoint paid +that cost once per non-live session (meetneptune/boxes +`docs/design/console-read-path.md`, workstream 1). + +The index is a fold of the journal (`engine/index.go`). Three rules keep it +honest. + +**It folds records, not memory.** `Session.writeRecord` folds every record +it appends. `ensureLog` folds the header records it writes directly, which +is the one write that does not pass `writeRecord`. Memory is never the +source: `EnqueuePromptDurable` writes its record before it mutates the +queue, so a fold of memory at that instant would disagree with the log it +claims to summarize. + +**It is a cache, never an authority.** `ReadSessionIndex` serves a stored +index only when three checks pass: a checksum over the stored bytes, the +journal's byte length, and the journal's modification time. Anything else +refolds from byte 0. There is no repair path, so no repair path can be +wrong. The checksum catches a torn read — both writers replace the sidecar +in place, and a reader in another process can otherwise mix old bytes with +new ones that parse. Length and modification time are a staleness key, not a +proof: they rest on the journal's own contract of one writer and append-only +writes. + +**It reports what a full load reports.** `Messages` and `LastActivityAt` run +through `message.ResolveOrphanToolCalls`, over a skeleton of ids, roles, and +tool-call ids. A crash between a tool call and its result therefore counts +the same on both paths. `DurableMessages` counts the records alone, for a +reader that must map a message to a byte offset; paginated message reads are +numbered against it. + +Some journals cannot be answered by a fold at all. A legacy header records +no workdir, and a crash can tear away the initial model record. `LoadSession` +answers those from the loading `Config`; a fold has no `Config`. +`SessionIndex.Complete` reports the difference, and +`Server.coldSessionJSON` uses the authoritative load path for such a +session. + +Three folds have real state machines. Each has one implementation, shared by +`LoadSession` and the index: `applyCompactRecord` (`compact.go`), +`applyGoalRecord` (`store.go`), and `promptQueueFold` (`queue.go`). +`engine/index_test.go`'s oracle test pins every index field against a full +`LoadSession`. + +A refold is far cheaper than a full `LoadSession`, and a current index is +cheaper again. Write-through costs one marshal and one rewrite per record. +The sidecar never gets an `fsync`: losing it in a crash costs one refold. + +`server.Options.Plugins` supplies a session's plugins to a cold read. +Plugins are process configuration, not durable session state, and a cold +read has no `Session` to ask. + +### Paginated message reads + +`GET /session/{id}/message?before_seq=N&limit=K` answers one bounded page of +a session's messages, read from the journal's tail. The unparameterized call +is unchanged, byte for byte: no `before_seq` and no `limit` still returns the +bare array of the whole history every existing caller expects. A request that +names either parameter gets a `MessagePage` envelope instead — `messages`, +`first_seq`, `last_seq`, `total`, `has_more` — because a client that pages +needs the page's position and a client that does not must not have to learn a +new shape. A console loads the tail and pages older messages in on scroll +(meetneptune/boxes `docs/design/console-read-path.md`, workstream 2 and +directive 1). Before this, every console open transferred the whole +transcript. + +**Seq is an ordinal over the DURABLE message sequence**: message records in +log order, with each compact record's fold applied (the folded range replaced +by that record's summary). `SessionIndex.DurableMessages` counts that same +sequence, so the newest message's seq equals it. `SessionIndex.Messages` can +be larger — it also counts the synthetic tool results +`message.ResolveOrphanToolCalls` derives — and a derived message has no +record, so it has no seq and no page carries it. This definition is what +makes a bounded read possible: an ordinal over durable records can be counted +backwards from the end of a file, while an ordinal over a materialized +history cannot be known without materializing it. + +`engine.ReadMessagePage` (`engine/messagepage.go`) serves a page two ways, +and both are numbered by the same index: + +- The **tail walk** reads `revChunkBytes` blocks backwards from + `SessionIndex.LogSize` and numbers message records down from the total. It + touches only the tail however long the journal is. It gives up the moment + it meets a compact record, because the messages a fold KEPT sit in the log + between the folded range and the compact record itself — undoing that + backwards would be a second, subtly different implementation of a fold. +- The **fold path** then reuses `indexFold` — the same forward fold, so the + same `applyCompactRecord` — to learn which ids occupy the requested seqs, + and reads back just those records. It costs one slim pass (ids and roles, + never message bodies), which is still three orders of magnitude below + materializing the history. + +The scan is bounded by `SessionIndex.LogSize`, never the file's current size, +so a turn appending records while a page is read cannot renumber that page +under it. + +Two properties are deliberate. A page carries durable messages **verbatim**: +it never runs `message.ResolveOrphanToolCalls`. That repair exists to keep a +provider REQUEST valid, this endpoint builds no request, and fabricating a +tool failure in a read view has real production history (see `Server.lookup`'s +doc comment: a healthy child's in-flight tool call rendered as failed in the +console for as long as it kept running). And compaction **renumbers** — a fold +replaces N messages with one summary, so every later seq shifts down by N-1. +A client paging across a compaction can see one page overlap another; message +ids are stable and are the way to de-duplicate. + +A page is read from the durable records even when the session is resident, so +one seq definition covers both cases: a resident history can carry messages +the log does not (load-time repairs, recovery's memory-only closers), and +numbering those would give one message two different seqs depending on +residency. A session with no journal at all — created through the API and +never prompted — falls back to its resident history, which for such a session +IS the durable sequence. + ### Prompt queue `POST /session/{id}/prompt_async` against a session already busy (another diff --git a/cmd/harness/main.go b/cmd/harness/main.go index ffd10f62..0d54af2d 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -1545,6 +1545,12 @@ func serveCmd(args []string) error { // (handleSpawnChild, handleSessionSend, buildSession's lineage) // consult). SessionManager: sessMgr, + // Plugins: the same host mkCfg wires into every session's + // engine.Config.Hooks. A read of a session this process does not + // hold live is answered from its metadata index and has no + // Session to ask, so the process's plugin list is supplied here + // directly — see server.Options.Plugins. + Plugins: pluginInfoFn(pluginHost), // MonitorPage: every `harness serve` box offers its own same-origin // monitor at GET /monitor — no CORS/-cors-origin dance, no // separately hosted copy required (see AGENTS.md's "Session diff --git a/cmd/harness/plugins.go b/cmd/harness/plugins.go index 1303c039..0e960fb9 100644 --- a/cmd/harness/plugins.go +++ b/cmd/harness/plugins.go @@ -415,6 +415,22 @@ func pluginHooks(host *plugin.Host) engine.Hooks { return host } +// pluginInfoFn adapts a possibly-nil *plugin.Host to +// server.Options.Plugins: the list of configured plugins a read of a +// NON-LIVE session reports (see that field's doc comment — such a read is +// answered from the session's metadata index, which has no Session to ask). +// A nil host reports no plugins, the same answer a session with no hooks +// gives through engine.Session.Plugins. +func pluginInfoFn(host *plugin.Host) func(string) []plugin.Info { + if host == nil { + return nil + } + // harness serve wires ONE host for every session it runs, so the id is + // not consulted. The parameter exists for an embedder whose own + // NewSession/LoadSession wrappers vary hooks per session. + return func(string) []plugin.Info { return host.Plugins() } +} + // pluginCmd dispatches `harness plugin `. func pluginCmd(args []string) error { if len(args) == 0 { diff --git a/engine/compact.go b/engine/compact.go index d5dfe6be..7e69dd59 100644 --- a/engine/compact.go +++ b/engine/compact.go @@ -721,6 +721,24 @@ func healCompactFoldEnd(history []message.Message, firstID string, turnsFolded i return history[foldEnd].ID, nil } +// applyCompactRecord folds one journaled compact record into history: the +// LastID heal above, then spliceCompact. It is the ONE implementation of +// "what a compact record does to a history", shared by LoadSession's replay +// (store.go) and the metadata index's own fold (index.go), so the two can +// never disagree about how many messages a fold removed. +// +// A FOUND lastID keeps the pre-heal behavior exactly: the heal never runs +// for it. A failed heal falls through unchanged, so spliceCompact returns +// its usual loud error rather than a silent best-effort guess. +func applyCompactRecord(history []message.Message, firstID, lastID string, turnsFolded int, summary message.Message) ([]message.Message, error) { + if _, found := indexOfMessageID(history, lastID); !found { + if healed, err := healCompactFoldEnd(history, firstID, turnsFolded); err == nil { + lastID = healed + } + } + return spliceCompact(history, firstID, lastID, summary) +} + // bytesPerTokenEstimate is the standard ~4-bytes-per-token heuristic used by // estimatePromptTokensFromHistory below when a provider's own usage // accounting is unavailable. diff --git a/engine/engine.go b/engine/engine.go index c122b086..adc47b6b 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -983,6 +983,32 @@ type Session struct { logStarted bool // the log file exists on disk lastPersistErr error + // index is the running fold of every record this session has written + // or replayed, and logSize the journal length that fold covers (see + // index.go). Together they are what flushIndexLocked writes to the + // session's sidecar after every record, so a reader answers GET + // /session for a non-resident session without replaying the journal. + // + // The fold reads RECORDS, never these in-memory fields, because a + // record does not always reach disk at the same instant memory changes + // — EnqueuePromptDurable writes its record BEFORE it mutates the queue, + // deliberately. logSize counts only bytes this session wrote (or found + // at ensureLog time), so a second writer on the same log can only ever + // make it too SMALL, which reads as stale and refolds. + index indexFold + logSize int64 + // indexFile is the sidecar's handle, opened beside the log in + // ensureLog and rewritten in place from then on. A session holds two + // descriptors, its journal and this one, and ReleaseFiles (store.go) + // drops both: the server calls it when it evicts a session from + // residency. Either handle reopens on the next persist, through + // ensureLog, so releasing them never ends a session. + indexFile *os.File + // lastIndexErr holds the most recent sidecar write failure. It is + // deliberately NOT lastPersistErr: the index is a cache, and its loss + // must never be reported to a caller as a durability failure. + lastIndexErr error + // Project-instruction segment, loaded once on the first Prompt (see // instructions.go). instrLoaded gates the one-time disk read; instrSeg is // the cached system-prompt segment (empty when none); instrErr records a diff --git a/engine/index.go b/engine/index.go new file mode 100644 index 00000000..c675e172 --- /dev/null +++ b/engine/index.go @@ -0,0 +1,934 @@ +// Session metadata index: a per-session summary a reader can answer +// GET /session and GET /session/{id} from, without replaying the journal. +// +// The problem it solves. A session's durable state lives in one append-only +// JSONL journal (store.go). Before this file, the only way to read a +// non-resident session's model, usage, or message count was LoadSession — +// a full replay that decodes every message body, builds the whole history +// slice, and runs message.ResolveOrphanToolCalls over it. A read endpoint +// then used four fields of the result and dropped the rest. On a long +// production session (1.4 MB, ~500 messages) that replay cost about 7 s +// per read, and the list endpoint paid it once per non-resident session. +// +// The shape. SessionIndex is a memoized FOLD of the journal, keyed by the +// journal's byte length. Two rules make it safe: +// +// 1. The index is derived from RECORDS, never from live Session memory. +// Every record reaches disk through Session.writeRecord (store.go), so +// applyIndexRecord folds exactly what the log holds. This matters for +// EnqueuePromptDurable (queue.go), which deliberately writes its +// record BEFORE it mutates memory: a fold of live memory taken at that +// instant would disagree with the log it claims to summarize. +// +// 2. The index is a CACHE, never an authority. ReadSessionIndex trusts a +// stored index only when it still describes the journal on disk: same +// byte length, same modification time, and a checksum that covers the +// stored bytes. Anything else — a missing index, a torn one, an older +// format, a shorter journal, a journal a second writer grew — is +// refolded from byte 0. No repair path exists, so no repair path can be +// wrong. +// +// Byte length plus modification time is a staleness key, not a proof. +// It rests on the journal's own contract: one writer, append only. +// Nothing in this package rewrites a journal in place. The one repair +// that touches existing bytes, ensureLog's torn-tail repair, always +// changes the length. An external rewrite that preserved both length +// and modification time would defeat the key, and is outside that +// contract. +// +// The fold is deliberately SLIM: it decodes message ids, roles, and +// timestamps, never message bodies (indexRecord below). A full refold of +// the 1.4 MB session above costs milliseconds, not seconds, so even the +// cold path — a journal written by an older binary, or the first read after +// a crash — is cheap. See engine/index_test.go's oracle test, which pins +// every field against the value a full LoadSession produces. +// +// The fold counts messages twice, on purpose. Messages is what a full +// LoadSession reports: the durable messages PLUS the synthetic tool results +// message.ResolveOrphanToolCalls adds for a tool call whose result never +// reached the log. The fold gets that count by running that exact function +// over a skeleton of the history — ids, roles, and tool-call ids, no +// bodies — so the index can never disagree with the repair about how many +// messages a reader sees. DurableMessages counts only the records +// themselves. A reader that must map a message to a byte offset needs the +// second number, because a repair message has no record to map to; that is +// what paginated message reads are numbered against. +package engine + +import ( + "bufio" + "bytes" + "encoding/json" + "errors" + "fmt" + "hash/crc32" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// sessionIndexVersion is the sidecar format version. ReadSessionIndex +// refolds — never guesses — when a stored index carries any other value, so +// a field added here needs no migration: bump this and every stale sidecar +// is rebuilt on its next read. +const sessionIndexVersion = 1 + +// sessionIndexSuffix is appended to a session id to name its sidecar. It +// deliberately does NOT end in ".jsonl", so ListSessionIndexes' own scan for +// session journals can never pick a sidecar up as a session. +const sessionIndexSuffix = ".index.json" + +// SessionIndex summarizes one persisted session: everything GET /session +// and GET /session/{id} report about it that has a durable source. +// +// Fields with no durable source are absent by construction, not omitted by +// accident: a session's live/idle status, its in-process goal presentation +// (server/journal.go's goalTracker), and its last turn outcome all belong +// to the process that ran the turn, not to the log. +type SessionIndex struct { + Version int `json:"version"` + ID string `json:"id"` + + CreatedAt time.Time `json:"created_at"` + // LastActivityAt is the CreatedAt of the newest durable message, or + // CreatedAt when the session has none — the same fallback + // Session.LastActivityAt applies for a log whose message records + // predate the timestamp field. + LastActivityAt time.Time `json:"last_activity_at"` + + Model message.ModelRef `json:"model,omitzero"` + Effort message.Effort `json:"effort,omitempty"` + + WorkDir string `json:"workdir,omitempty"` + ParentSession string `json:"parent_session,omitempty"` + + // TaskParentID, TaskAgentType, TaskDepth, and SpawnedChildIDs are the + // durable half of a session's subagent lineage — the same fields + // lineageJSONFor's cold branch (server/handlers.go) reads off a loaded + // Session. The live half (status, result, fail reason) has no durable + // source and is not here. + TaskParentID string `json:"task_parent_id,omitempty"` + TaskAgentType string `json:"task_agent_type,omitempty"` + TaskDepth int `json:"task_depth,omitempty"` + SpawnedChildIDs []string `json:"spawned_child_ids,omitempty"` + + // Messages is the message count a full LoadSession reports: durable + // messages after compaction splices, plus the synthetic tool results + // message.ResolveOrphanToolCalls adds. DurableMessages counts the + // records alone. The two differ only for a journal that carries a tool + // call whose result never reached the log — see this file's header + // comment. + Messages int `json:"messages"` + DurableMessages int `json:"durable_messages"` + Usage provider.Usage `json:"usage,omitzero"` + LastInputTokens int `json:"last_input_tokens,omitempty"` + + // GoalActive and GoalCondition are the durable goal state LoadSession + // restores (store.go's recGoalSet fold): the condition of a goal set + // without a later goal.achieved or goal.cleared. The run counters + // deliberately do not survive, live or here. + GoalActive bool `json:"goal_active,omitempty"` + GoalCondition string `json:"goal_condition,omitempty"` + + // Queued is the durable prompt-queue depth (queue.go): prompts + // enqueued and not yet dequeued. + Queued int `json:"queued,omitempty"` + + CompactionCount int `json:"compaction_count,omitempty"` + LastCompactedAt time.Time `json:"last_compacted_at,omitzero"` + + // Complete reports whether the journal recorded a model and a workdir. + // Both are fields LoadSession will otherwise take from the loading + // Config: a legacy header carries no workdir, and a crash can tear away + // the initial model record a fresh log writes beside its header. A fold + // has no Config, so it reports the gap instead of an empty value, and + // the caller uses the authoritative load path for that session (see + // server.Server.coldSessionJSON). + // + // Model and workdir are the WHOLE fallback surface for a reader, and + // that rests on a contract the reader owes this package: the Config it + // loads a session with must not name state that belongs to one specific + // session. LoadSession applies the same "absent means keep the Config + // value" rule to ParentSession, TaskParentID, TaskAgentType, and + // TaskDepth. Absence is the NORMAL case for all four — most sessions + // have no parent and no task lineage — so treating an absent one as + // incomplete would send every read back to the load path and delete the + // point of this index. A Config that carries a generic model and + // workdir, as harness serve's own loader does (loadSessionFn, + // cmd/harness/main.go), therefore diverges nowhere. See + // server.Options.LoadSession for the same rule stated to the embedder + // who supplies that Config. + Complete bool `json:"complete"` + + // LogSize and LogModTime are the journal length and modification time + // this index folds, and together they are the staleness key: an index + // is current when, and only when, both still match the journal on disk. + // Journals are append-only, so a longer journal means records this fold + // never saw, and a shorter one means a torn-tail repair (ensureLog) + // rewrote history under it. Both refold. See this file's header comment + // for what the key rests on and what it does not prove. + LogSize int64 `json:"log_size"` + LogModTime time.Time `json:"log_mod_time,omitzero"` +} + +// indexMessage is the slim decode of a record's message body: identity, +// role, timestamp, and the call id of every tool call and tool result. +// Nothing else is decoded — a Text part's text and a Blob part's bytes are +// walked by encoding/json and dropped, never turned into message.Part +// values, which is what keeps a refold milliseconds rather than seconds. +// +// The call ids are here for one reason: message.ResolveOrphanToolCalls +// decides its repair from roles and call ids alone, so a skeleton carrying +// them lets the fold ask that exact function how many messages a full load +// would produce (see skeleton and indexFold.snapshot). +type indexMessage struct { + ID string `json:"id"` + Role message.Role `json:"role"` + CreatedAt time.Time `json:"created_at"` + Parts []indexPart `json:"parts"` +} + +// indexPart is the slim decode of one part: its kind and, for the two kinds +// that pair a call with its result, the call id. +type indexPart struct { + Type message.PartType `json:"type"` + CallID string `json:"call_id"` +} + +// skeleton builds the message.Message the fold keeps for one record: the +// identity fields, plus a ToolCall or ToolResult part for each call id, and +// nothing else. It is never served to a reader. It exists so the fold can +// run the real repair and the real compaction splice over it. +func (m indexMessage) skeleton() message.Message { + out := message.Message{ID: m.ID, Role: m.Role, CreatedAt: m.CreatedAt} + for _, p := range m.Parts { + switch p.Type { + case message.PartToolCall: + out.Parts = append(out.Parts, &message.ToolCall{CallID: p.CallID}) + case message.PartToolResult: + out.Parts = append(out.Parts, &message.ToolResult{CallID: p.CallID}) + } + } + return out +} + +// indexCompact is the slim decode of a compact record's payload (see +// compactRecord). Summary carries only the summary message's identity — +// the fold splices by id, never by content. +type indexCompact struct { + FirstID string `json:"first_id"` + LastID string `json:"last_id"` + TurnsFolded int `json:"turns_folded"` + Summary indexMessage `json:"summary"` +} + +// indexRecord is one journal line, decoded to just the fields the fold +// reads. It mirrors record (store.go) field for field where they overlap, +// so a record type the fold ignores costs a type-string compare and +// nothing else. +type indexRecord struct { + Type string `json:"type"` + ID string `json:"id,omitempty"` + CreatedAt time.Time `json:"created_at,omitzero"` + WorkDir string `json:"workdir,omitempty"` + ParentSession string `json:"parent_session,omitempty"` + TaskParentID string `json:"task_parent_id,omitempty"` + TaskAgentType string `json:"task_agent_type,omitempty"` + TaskDepth int `json:"task_depth,omitempty"` + Model message.ModelRef `json:"model,omitzero"` + Effort message.Effort `json:"effort,omitempty"` + Message *indexMessage `json:"message,omitempty"` + Usage *provider.Usage `json:"usage,omitempty"` + Goal *goalRecord `json:"goal,omitempty"` + Prompt *promptRecord `json:"prompt,omitempty"` + TaskSpawn *taskSpawnRecord `json:"task_spawn,omitempty"` + Compact *indexCompact `json:"compact,omitempty"` +} + +// indexRecordOf projects a full record (the shape the write path and +// LoadSession already hold) onto the fold's input. The two paths therefore +// share ONE fold: a record folded as it is written and the same record +// folded from disk take the identical branch. +func indexRecordOf(rec record) indexRecord { + out := indexRecord{ + Type: rec.Type, + ID: rec.ID, + CreatedAt: rec.CreatedAt, + WorkDir: rec.WorkDir, + ParentSession: rec.ParentSession, + TaskParentID: rec.TaskParentID, + TaskAgentType: rec.TaskAgentType, + TaskDepth: rec.TaskDepth, + Model: rec.Model, + Effort: rec.Effort, + Usage: rec.Usage, + Goal: rec.Goal, + Prompt: rec.Prompt, + TaskSpawn: rec.TaskSpawn, + } + if rec.Message != nil { + out.Message = indexMessageOf(*rec.Message) + } + if rec.Compact != nil { + out.Compact = &indexCompact{ + FirstID: rec.Compact.FirstID, + LastID: rec.Compact.LastID, + TurnsFolded: rec.Compact.TurnsFolded, + Summary: *indexMessageOf(rec.Compact.Summary), + } + } + return out +} + +// indexMessageOf projects a full message onto the fold's slim shape — the +// write path's counterpart to the slim JSON decode a refold performs. It +// keeps the call ids for the same reason indexMessage carries them. +func indexMessageOf(m message.Message) *indexMessage { + out := &indexMessage{ID: m.ID, Role: m.Role, CreatedAt: m.CreatedAt} + for _, p := range m.Parts { + switch v := p.(type) { + case *message.ToolCall: + out.Parts = append(out.Parts, indexPart{Type: message.PartToolCall, CallID: v.CallID}) + case *message.ToolResult: + out.Parts = append(out.Parts, indexPart{Type: message.PartToolResult, CallID: v.CallID}) + } + } + return out +} + +// indexFold accumulates a SessionIndex from journal records in log order. +// +// It keeps a message SKELETON — one message.Message per durable message, +// carrying id, role, and timestamp and no parts — rather than a plain +// counter, because compaction folds a RANGE of messages named by their +// first and last id. Holding the skeleton lets the fold call the same +// spliceCompact and healCompactFoldEnd that LoadSession calls (compact.go), +// so the two can never disagree about what a compact record does to a +// history. +type indexFold struct { + ix SessionIndex + messages []message.Message + // repairs is how many messages message.ResolveOrphanToolCalls would + // insert into the skeleton, maintained as records arrive so snapshot + // stays constant time. See appendMessage. + repairs int + queue promptQueueFold + // header is set by the session header record. A journal whose first + // record is not a header is not a session log (events.jsonl is the one + // in-tree example), and snapshot refuses it. + header bool + // broken records a fold that hit a record it could not apply — today, + // only a compact record whose range is absent from the skeleton. Such + // a fold is never written to disk and never served: the caller falls + // back to the authority, LoadSession. + broken bool +} + +// applyIndexRecord folds one record. It mirrors LoadSession's own switch +// (store.go) case for case, and shares its helpers for the three folds with +// real state machines behind them: compaction (applyCompactRecord), +// goals (applyGoalRecord), and the prompt queue (promptQueueFold). +func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { + switch rec.Type { + case recSession: + f.header = true + f.ix.ID = rec.ID + f.ix.CreatedAt = rec.CreatedAt + f.ix.WorkDir = rec.WorkDir + f.ix.ParentSession = rec.ParentSession + f.ix.TaskParentID = rec.TaskParentID + f.ix.TaskAgentType = rec.TaskAgentType + f.ix.TaskDepth = rec.TaskDepth + f.ix.Effort = rec.Effort + case recMessage: + if rec.Message == nil { + if isLast { + return nil + } + return errors.New("message record without message") + } + f.appendMessage(rec.Message.skeleton()) + if rec.Usage != nil { + f.addUsage(*rec.Usage) + f.ix.LastInputTokens = rec.Usage.InputTokens + } + case recModel: + f.ix.Model = rec.Model + case recEffort: + f.ix.Effort = rec.Effort + case recGoalSet, recGoalUpdated, recGoalAchieved, recGoalCleared: + f.ix.GoalActive, f.ix.GoalCondition = applyGoalRecord(f.ix.GoalActive, f.ix.GoalCondition, rec.Type, rec.Goal) + case recPromptQueued: + if rec.Prompt != nil { + f.queue.queued(*rec.Prompt) + } + case recPromptDequeued: + if rec.Prompt != nil { + f.queue.dequeued(*rec.Prompt) + } + case recTaskSpawned: + if rec.TaskSpawn != nil && rec.TaskSpawn.ChildID != "" { + f.ix.SpawnedChildIDs = append(f.ix.SpawnedChildIDs, rec.TaskSpawn.ChildID) + } + case recCompact: + if rec.Compact == nil { + return errors.New("compact record without payload") + } + spliced, err := applyCompactRecord(f.messages, rec.Compact.FirstID, rec.Compact.LastID, rec.Compact.TurnsFolded, rec.Compact.Summary.skeleton()) + if err != nil { + // The same corruption LoadSession fails on. Mark the fold + // broken rather than returning: a listing must still report + // every OTHER session, and this one's reader falls back to + // LoadSession, which produces the authoritative error. + f.broken = true + return nil + } + f.messages = spliced + f.recountRepairs() + f.ix.CompactionCount++ + f.ix.LastCompactedAt = rec.CreatedAt + if rec.Usage != nil { + // Cumulative usage only — never LastInputTokens. See + // record.Usage's doc comment (store.go): a reloaded session + // must not report the small summarization call as its last + // request size. + f.addUsage(*rec.Usage) + } + } + return nil +} + +// applyIndexRecordBestEffort is the live write path's entry point: it folds +// a record and treats any failure as "this fold can no longer be trusted" +// rather than as an error to propagate. Nothing on the write path may fail +// a session because a cache could not be updated — see writeRecord. +func (f *indexFold) applyIndexRecordBestEffort(rec indexRecord, isLast bool) { + if f.broken { + return + } + if err := f.applyIndexRecord(rec, isLast); err != nil { + f.broken = true + } +} + +func (f *indexFold) addUsage(u provider.Usage) { + f.ix.Usage.InputTokens += u.InputTokens + f.ix.Usage.OutputTokens += u.OutputTokens + f.ix.Usage.CacheReadTokens += u.CacheReadTokens + f.ix.Usage.CacheWriteTokens += u.CacheWriteTokens +} + +// repairsAt returns how many messages message.ResolveOrphanToolCalls +// inserts on account of the skeleton message at index i. +// +// It CALLS that function, over a two-message window, rather than restating +// its rule. The repair reads exactly one pair at a time — an assistant +// message and whatever follows it (message/message.go) — so a window of +// that pair decides for message i exactly as the whole slice does. Reusing +// the real function is what keeps this in step with it; restating "an +// assistant message with an unmatched tool call gets one synthetic result +// unless a tool message follows" would be a second copy of a rule this +// repository has already been burned by copying. +// +// Insertions are attributed POSITIONALLY, and the follower is found by a +// MARKER rather than by its id. The window's second message is evaluated +// too, as a message with no follower of its own, and any insertion it earns +// belongs to ITS index, not to i. The repair preserves order and inserts +// directly after the message that earned the insertion, so everything +// before the follower in the result is i's. +// +// Searching for the follower's id instead would break on a journal that +// repeats one: two adjacent records with the same id — or with none, which +// a malformed record can produce — made the search land on the FIRST +// message and report a negative count, so the running total drifted below +// zero and Messages fell below the durable count. A review found it. The +// marker is a Text part appended to the copied follower; the repair reads +// only roles and tool-call parts, so it changes no decision, and the repair +// never moves a part between messages, so exactly one message in the result +// carries it. +// +// The window is a deep copy: the repair appends parts to the messages it +// returns, and a shallow copy would share the skeleton's own Parts backing +// array. +func (f *indexFold) repairsAt(i int) int { + if i < 0 || i >= len(f.messages) { + return 0 + } + end := i + 2 + if end > len(f.messages) { + end = len(f.messages) + } + window := make([]message.Message, 0, end-i) + for _, m := range f.messages[i:end] { + cp := m + cp.Parts = make(message.Parts, len(m.Parts)) + copy(cp.Parts, m.Parts) + window = append(window, cp) + } + if len(window) == 2 { + window[1].Parts = append(window[1].Parts, &message.Text{Text: repairWindowMarker}) + } + out := message.ResolveOrphanToolCalls(window) + if len(window) == 1 { + // No follower: message i is the last, so every insertion is its + // own. + return len(out) - 1 + } + for pos := range out { + if hasRepairWindowMarker(out[pos]) { + return pos - 1 // out[0] is message i; anything between is its repair + } + } + // Unreachable: the repair never drops a message and never moves a part + // between messages. Attribute nothing rather than guess. + return 0 +} + +// repairWindowMarker tags the follower inside a repair window (see +// repairsAt). It is a Text part's text, so the repair — which reads roles, +// tool calls, and tool results — cannot see it, and it never reaches a +// caller: the window is a throwaway copy. +const repairWindowMarker = "\x00harness/engine: repair-window follower" + +func hasRepairWindowMarker(m message.Message) bool { + for _, p := range m.Parts { + if t, ok := p.(*message.Text); ok && t.Text == repairWindowMarker { + return true + } + } + return false +} + +// appendMessage adds one durable message to the skeleton and keeps the +// running repair count in step. +// +// Appending at index n can change the repair decision for exactly two +// messages: n-1, whose follower changes from nothing to this message, and n +// itself, which now has no follower. Every earlier pair is untouched, +// because the repair never looks further than one message ahead. That is +// what makes the count maintainable in constant time. +// +// Constant time is the point. snapshot runs after EVERY record a session +// writes, and it used to re-run the repair over the whole skeleton — O(n) +// per record, so O(n^2) over a session's life. An index that makes reads +// cheap and writes quadratic is a net loss on exactly the long sessions it +// exists for. A review caught it. +func (f *indexFold) appendMessage(m message.Message) { + last := len(f.messages) - 1 + f.repairs -= f.repairsAt(last) + f.messages = append(f.messages, m) + f.repairs += f.repairsAt(last) + f.repairsAt(last+1) +} + +// recountRepairs recomputes the repair count from scratch. Compaction is +// the one operation that rewrites the skeleton's middle — a fold replaces a +// whole range with one summary — so pairs far from the tail change and an +// incremental update cannot see them. It is O(n), and it runs once per +// compact record rather than once per record. +func (f *indexFold) recountRepairs() { + f.repairs = 0 + for i := range f.messages { + f.repairs += f.repairsAt(i) + } +} + +// snapshot renders the fold as an index covering a journal of logSize bytes +// last modified at modTime. ok is false for a fold that never saw a session +// header, or one a record broke — neither is a summary anything may serve +// or store. +// +// It is constant time. Messages and LastActivityAt describe the history a +// full LoadSession produces, repair included, but the repair count is +// maintained as records arrive (see appendMessage) rather than recomputed +// here. +func (f *indexFold) snapshot(logSize int64, modTime time.Time) (SessionIndex, bool) { + if !f.header || f.broken { + return SessionIndex{}, false + } + ix := f.ix + ix.Version = sessionIndexVersion + ix.DurableMessages = len(f.messages) + ix.Messages = len(f.messages) + f.repairs + ix.Queued = len(f.queue.queue) + ix.LogSize = logSize + ix.LogModTime = modTime + // Complete: the journal recorded every field a reader needs, so no + // Config fallback is involved. See SessionIndex.Complete. + ix.Complete = !ix.Model.IsZero() && ix.WorkDir != "" + + ix.LastActivityAt = ix.CreatedAt + if n := len(f.messages); n > 0 && f.repairsAt(n-1) == 0 { + // Same fallback as Session.LastActivityAt, over the same input. A + // message record written before the timestamp field existed + // replays as zero, and so does a synthetic repair message — and a + // repair on the LAST message puts exactly such a message at the + // end of the repaired history, which is why a non-zero repair + // count there keeps the session's own CreatedAt. + if t := f.messages[n-1].CreatedAt; !t.IsZero() { + ix.LastActivityAt = t + } + } + if len(ix.SpawnedChildIDs) > 0 { + ix.SpawnedChildIDs = append([]string(nil), ix.SpawnedChildIDs...) + } + return ix, true +} + +// sessionIndexPath names a session's sidecar index file. +func sessionIndexPath(dir, id string) string { + return filepath.Join(dir, id+sessionIndexSuffix) +} + +// foldJournalBytes folds an entire journal's bytes into a fresh fold. It +// applies scanLog's corruption discipline unchanged: a corrupt final line +// is a crash mid-write and ends the fold silently, corruption anywhere else +// is an error. +func foldJournalBytes(data []byte) (indexFold, error) { + var f indexFold + err := scanLog(data, func(rec indexRecord, line int, isLast bool) error { + if err := f.applyIndexRecord(rec, isLast); err != nil { + return fmt.Errorf("%w at line %d", err, line) + } + return nil + }) + if err != nil { + return indexFold{}, err + } + return f, nil +} + +// foldSessionJournal folds an entire journal's bytes into an index — the +// refold path, the one ReadSessionIndex takes whenever a stored index is +// missing or stale. +func foldSessionJournal(data []byte, modTime time.Time) (SessionIndex, error) { + f, err := foldJournalBytes(data) + if err != nil { + return SessionIndex{}, err + } + ix, ok := f.snapshot(int64(len(data)), modTime) + if !ok { + return SessionIndex{}, errNotSessionJournal + } + return ix, nil +} + +// ReadSessionIndex returns id's summary index, reading the journal only +// when the stored sidecar does not already cover it. +// +// The fast path is one stat plus one small read: a sidecar whose LogSize +// equals the journal's current size is returned as it stands, and the +// journal is never opened. Every other case — no sidecar, an unreadable or +// torn one, a different format version, a journal that grew since (records +// this process did not write, or a write whose sidecar flush failed), or a +// journal that SHRANK (ensureLog's torn-tail repair) — refolds from byte 0 +// and writes the result back, so the next read takes the fast path. +// +// The write-back is best effort. A read-only session directory, or a +// racing writer, costs a refold on the next read and nothing else. +func ReadSessionIndex(dir, id string) (SessionIndex, error) { + if dir == "" { + return SessionIndex{}, errors.New("engine: ReadSessionIndex requires a session dir") + } + // The id reaches the filesystem through sessionPath below, so it is + // validated here rather than trusted — the same defense in depth + // LoadSession applies, for the callers (the CLI's resume flags) that + // never pass through an HTTP boundary. + if !ValidSessionID(id) { + return SessionIndex{}, fmt.Errorf("%w: %q", ErrInvalidSessionID, id) + } + return readSessionIndexAt(dir, id, true) +} + +// errNotSessionJournal marks a .jsonl file in the session directory that is +// not a session log at all — the server's own event journal (events.jsonl) +// is the in-tree example. It is a skip signal for ListSessionIndexes, never +// a failure. +var errNotSessionJournal = errors.New("engine: not a session journal") + +// readSessionIndexAt is ReadSessionIndex without the id validation, for +// ListSessionIndexes, whose ids come from directory entries (which cannot +// contain a path separator) rather than from a caller. +// +// cache selects whether a REFOLD writes its result back. A single-session +// read memoizes: one session, one fold, and the next read of it is a stat +// and a small read. A LISTING does not. It would refold and rewrite a +// sidecar for every session in the directory, including sessions this +// process holds live — racing their own writers — and a read that mutates +// N files as a side effect of listing them is the wrong shape whatever the +// race. The write path repairs the sidecar; the list path only reads it. +func readSessionIndexAt(dir, id string, cache bool) (SessionIndex, error) { + path := sessionPath(dir, id) + fi, err := os.Stat(path) + if err != nil { + return SessionIndex{}, err + } + if ix, ok := readStoredIndex(dir, id, fi.Size(), fi.ModTime()); ok { + return ix, nil + } + // Refold. Check the first record before reading the whole file: a + // directory holds one session journal per session AND the server's + // event journal, which can be megabytes and never yields a sidecar, so + // a listing must not read it end to end every time. + if err := checkSessionJournalHead(path); err != nil { + return SessionIndex{}, err + } + data, err := os.ReadFile(path) + if err != nil { + return SessionIndex{}, err + } + // Re-stat AFTER the read, and key the index on that: a journal the + // writer grew between the stat above and this read would otherwise be + // summarized under a modification time older than its own bytes. The + // worst case now is a key that looks stale on the next read, which + // costs one refold. + modTime := fi.ModTime() + if after, err := os.Stat(path); err == nil { + modTime = after.ModTime() + } + ix, err := foldSessionJournal(data, modTime) + if err != nil { + return SessionIndex{}, fmt.Errorf("engine: session %s: %w", id, err) + } + // The FILENAME names the session, not the header record inside it. + // LoadSession pins the same way (it assigns s.ID = id before replaying), + // so a journal copied to a new name reports the new name on both paths. + // Without this, a header that disagrees with its filename would make + // GET /session/{id} answer with a different id than it was asked about, + // and every later read would refold, since readStoredIndex requires the + // stored id to match. + ix.ID = id + if cache { + writeSessionIndex(dir, id, ix) + } + return ix, nil +} + +// journalHeadPeekBytes bounds checkSessionJournalHead's read. It is far +// larger than an ordinary session header and far smaller than a journal, so +// the peek stays O(1) against the file it exists to avoid reading. +const journalHeadPeekBytes = 64 << 10 + +// checkSessionJournalHead reports whether a file's FIRST record is a +// session header, reading only that first line. It answers the same +// question the fold answers (see indexFold.header), at O(1) cost instead of +// the file's whole length. A first line too long to peek at is not a +// verdict: it returns nil, and the caller reads the file. +func checkSessionJournalHead(path string) error { + f, err := os.Open(path) + if err != nil { + return err + } + defer f.Close() + line, err := bufio.NewReaderSize(f, journalHeadPeekBytes).ReadSlice('\n') + if errors.Is(err, bufio.ErrBufferFull) { + // A first line longer than the peek buffer. A session header CAN be + // long: ensureLog writes task_tool_names into it, and a large + // restricted tool set has no fixed bound. Report nothing rather + // than a verdict — the caller reads the whole file and the fold + // decides. Skipping here instead would hide a loadable session from + // GET /session/{id} and from every listing. + return nil + } + if err != nil && len(line) == 0 { + return errNotSessionJournal + } + var head struct { + Type string `json:"type"` + } + if json.Unmarshal(bytes.TrimSpace(line), &head) != nil || head.Type != recSession { + return errNotSessionJournal + } + return nil +} + +// sessionIndexFile is the sidecar's on-disk shape: the index bytes, plus a +// checksum over exactly those bytes. +// +// The checksum is what makes a torn read detectable. Both writers replace +// the sidecar in place, so a reader in another process can, in principle, +// read part of the old file and part of the new one. Such a mix can parse +// as JSON — the fields are the same and in the same order — and can even +// carry a staleness key that matches the journal. A checksum over the exact +// bytes it covers turns that into a miss, and a miss refolds. +// +// CRC-32 detects that corruption; it does not prove its absence. A mix that +// happens to collide passes. The residual is a cache read, under a +// single-writer contract, of a file this package alone writes — not a +// trust boundary — so a wider hash would buy nothing an operator can use. +type sessionIndexFile struct { + CRC32 uint32 `json:"crc32"` + Index json.RawMessage `json:"index"` +} + +// readStoredIndex loads the sidecar and reports whether it is usable AND +// still describes a journal of exactly logSize bytes last modified at +// modTime. Every failure — absent, unreadable, malformed, checksum +// mismatch, wrong version, wrong length, wrong time — returns false, which +// means "refold": the sidecar is a cache with no repair path. +func readStoredIndex(dir, id string, logSize int64, modTime time.Time) (SessionIndex, bool) { + data, err := os.ReadFile(sessionIndexPath(dir, id)) + if err != nil { + return SessionIndex{}, false + } + var file sessionIndexFile + if err := json.Unmarshal(data, &file); err != nil { + return SessionIndex{}, false + } + if crc32.ChecksumIEEE(file.Index) != file.CRC32 { + return SessionIndex{}, false + } + var ix SessionIndex + if err := json.Unmarshal(file.Index, &ix); err != nil { + return SessionIndex{}, false + } + if ix.Version != sessionIndexVersion || ix.ID != id { + return SessionIndex{}, false + } + if ix.LogSize != logSize || !ix.LogModTime.Equal(modTime) { + return SessionIndex{}, false + } + return ix, true +} + +// marshalSessionIndex renders an index as its sidecar bytes, checksum and +// all. +func marshalSessionIndex(ix SessionIndex) ([]byte, error) { + inner, err := json.Marshal(ix) + if err != nil { + return nil, err + } + return json.Marshal(sessionIndexFile{CRC32: crc32.ChecksumIEEE(inner), Index: inner}) +} + +// writeSessionIndex replaces id's sidecar, from a caller that holds no open +// handle on it (ReadSessionIndex's write-back). +// +// It never fsyncs. The index is a cache: losing an unsynced sidecar in a +// crash costs one refold, and fsync is exactly the call some FUSE/9p +// transports deadlock on (see Config.SessionSync). +func writeSessionIndex(dir, id string, ix SessionIndex) error { + b, err := marshalSessionIndex(ix) + if err != nil { + return err + } + return os.WriteFile(sessionIndexPath(dir, id), b, 0o644) +} + +// writeIndexTo rewrites an already-open sidecar in place: truncate to zero, +// then write from offset 0. Session.flushIndexLocked takes this path, once +// per journal record, through a handle opened beside the journal itself +// (ensureLog). +// +// In place, not a temporary file and a rename. A rename publishes +// atomically, but it creates a directory entry per write, and a write that +// races a directory removal resurrects an entry the remover already passed. +// The handle here survives an unlinked file exactly like the journal handle +// beside it. A reader that catches a rewrite mid-flight is answered by the +// sidecar's checksum instead (see sessionIndexFile), which is a stronger +// guard than atomic publication alone: it also catches a mixed read of two +// same-length indexes. +func writeIndexTo(f *os.File, ix SessionIndex) error { + b, err := marshalSessionIndex(ix) + if err != nil { + return err + } + if err := f.Truncate(0); err != nil { + return err + } + _, err = f.WriteAt(b, 0) + return err +} + +// SessionExists reports whether dir holds a journal for id: one stat, no +// read of any kind. +// +// It is the existence check for a hot path — an abort, an end, a wait — and +// it answers presence, not readability. A journal that exists but cannot be +// folded is still a session that exists, and a caller asking "is there a +// session here" must not be told no because its bytes are damaged. An +// invalid id is false without touching the filesystem. +func SessionExists(dir, id string) bool { + if dir == "" || !ValidSessionID(id) { + return false + } + fi, err := os.Stat(sessionPath(dir, id)) + return err == nil && !fi.IsDir() +} + +// ListSessionIDs returns the id of every session journal in dir, unsorted, +// reading no journal and no sidecar — one directory scan. +// +// It exists for a caller that resolves its own per-session state before +// deciding what to read. GET /session renders a session this process holds +// live from that live object, so reading an index for it would be work +// thrown away; worse, a stale sidecar for a live session would be refolded +// and written back by the listing while the session's own writer holds it. +// The ids come from file names, so a name that is not a valid session id is +// skipped here rather than reaching the filesystem again. +func ListSessionIDs(dir string) ([]string, error) { + entries, err := os.ReadDir(dir) + if errors.Is(err, fs.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, err + } + var out []string + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".jsonl") { + continue + } + id := strings.TrimSuffix(e.Name(), ".jsonl") + if !ValidSessionID(id) { + continue // events.jsonl, or any file that is not a session log + } + out = append(out, id) + } + return out, nil +} + +// ListSessionIndexes returns the index of every persisted session in dir +// whose index can be read, sorted by creation time. A missing directory +// yields an empty list, not an error, and a file that is unreadable, +// corrupt, or not a session journal at all (events.jsonl, the server's own +// event journal, lives in the same directory) is skipped rather than +// failing the whole listing. +// +// It never writes. A refold here answers this call and is then dropped, so +// listing a directory cannot rewrite the sidecar of a session another +// goroutine or process is writing. +// +// A caller that must not MISS a session cannot use this alone: a journal +// whose fold breaks has no index and is skipped here. ListSessions pairs it +// with a direct scan for exactly that reason. +func ListSessionIndexes(dir string) ([]SessionIndex, error) { + entries, err := os.ReadDir(dir) + if errors.Is(err, fs.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, err + } + var out []SessionIndex + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".jsonl") { + continue + } + // The id comes from the directory entry, so it names this exact + // file; sessionIndexSuffix keeps a sidecar from ever matching the + // ".jsonl" test above. + ix, err := readSessionIndexAt(dir, strings.TrimSuffix(e.Name(), ".jsonl"), false) + if err != nil { + continue + } + out = append(out, ix) + } + sort.Slice(out, func(i, j int) bool { return out[i].CreatedAt.Before(out[j].CreatedAt) }) + return out, nil +} diff --git a/engine/index_test.go b/engine/index_test.go new file mode 100644 index 00000000..88609b6a --- /dev/null +++ b/engine/index_test.go @@ -0,0 +1,1243 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "hash/crc32" + "os" + "path/filepath" + "runtime" + "testing" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// indexOracle is what a full LoadSession reports for the fields +// SessionIndex claims to answer without one. The oracle is the AUTHORITY +// the index replaces — GET /session builds its body from exactly these +// accessors (server/handlers.go's buildSession) — so an index that +// disagrees with it is wrong by definition. +type indexOracle struct { + ID string + CreatedAt time.Time + LastActivityAt time.Time + Model message.ModelRef + Effort message.Effort + WorkDir string + ParentSession string + TaskParentID string + TaskAgentType string + TaskDepth int + SpawnedChildIDs []string + Messages int + Usage provider.Usage + LastInputTokens int + GoalActive bool + GoalCondition string + Queued int + CompactionCount int + LastCompactedAt time.Time +} + +// oracleOf reads the authority: a session loaded from its journal. +func oracleOf(t *testing.T, sess *Session) indexOracle { + t.Helper() + o := indexOracle{ + ID: sess.ID, + CreatedAt: sess.CreatedAt(), + LastActivityAt: sess.LastActivityAt(), + Model: sess.Model(), + Effort: sess.Effort(), + WorkDir: sess.WorkDir(), + ParentSession: sess.ParentSession(), + TaskParentID: sess.TaskParentID(), + TaskAgentType: sess.TaskAgentType(), + TaskDepth: sess.TaskDepth(), + SpawnedChildIDs: sess.SpawnedChildIDs(), + Messages: len(sess.History()), + Usage: sess.Usage(), + Queued: len(sess.QueuedPrompts()), + CompactionCount: sess.CompactionCount(), + LastCompactedAt: sess.LastCompactedAt(), + } + o.GoalCondition, o.GoalActive = sess.ActiveGoal() + if last, ok := sess.LastUsage(); ok { + o.LastInputTokens = last.InputTokens + } + return o +} + +// oracleOfIndex projects a SessionIndex onto the same shape. +func oracleOfIndex(ix SessionIndex) indexOracle { + return indexOracle{ + ID: ix.ID, + CreatedAt: ix.CreatedAt, + LastActivityAt: ix.LastActivityAt, + Model: ix.Model, + Effort: ix.Effort, + WorkDir: ix.WorkDir, + ParentSession: ix.ParentSession, + TaskParentID: ix.TaskParentID, + TaskAgentType: ix.TaskAgentType, + TaskDepth: ix.TaskDepth, + SpawnedChildIDs: ix.SpawnedChildIDs, + Messages: ix.Messages, + Usage: ix.Usage, + LastInputTokens: ix.LastInputTokens, + GoalActive: ix.GoalActive, + GoalCondition: ix.GoalCondition, + Queued: ix.Queued, + CompactionCount: ix.CompactionCount, + LastCompactedAt: ix.LastCompactedAt, + } +} + +// assertIndexMatchesLoad reads id's index and compares it, field for field, +// against a full LoadSession of the same journal. +func assertIndexMatchesLoad(t *testing.T, cfg Config, id string) SessionIndex { + t.Helper() + loaded, err := LoadSession(cfg, id) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + ix, err := ReadSessionIndex(cfg.SessionDir, id) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + want, got := oracleOf(t, loaded), oracleOfIndex(ix) + if wantJSON, gotJSON := mustJSON(t, want), mustJSON(t, got); wantJSON != gotJSON { + t.Errorf("index disagrees with LoadSession\n index = %s\n load = %s", gotJSON, wantJSON) + } + return ix +} + +// TestSessionIndexMatchesLoadSession is the oracle test: for every journal +// shape a session can reach through its own production entry points, the +// index must report exactly what a full LoadSession reports. Each case +// drives the real API (Prompt, Compact, RegisterGoal, EnqueuePrompt, +// SetModel, SetEffort), never a hand-written journal, so the fold is +// verified against the records production actually writes. +func TestSessionIndexMatchesLoadSession(t *testing.T) { + cases := []struct { + name string + // turns scripts the provider; drive runs the session. + turns [][]provider.Event + drive func(t *testing.T, s *Session) + }{ + { + name: "never prompted", + turns: [][]provider.Event{}, + drive: func(t *testing.T, s *Session) { + if err := s.Persist(); err != nil { + t.Fatal(err) + } + }, + }, + { + name: "plain turns", + turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10, OutputTokens: 5}), + compactTurn("two", provider.Usage{InputTokens: 20, OutputTokens: 7, CacheReadTokens: 3, CacheWriteTokens: 4}), + }, + drive: func(t *testing.T, s *Session) { runTurns(t, s, 2) }, + }, + { + name: "tool loop", + turns: [][]provider.Event{ + asstTurn(provider.StopToolUse, toolCall("tc1", "bash", `{"command":"echo hi"}`)), + asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), + }, + drive: func(t *testing.T, s *Session) { + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatal(err) + } + }, + }, + { + name: "model and effort switches", + turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }, + drive: func(t *testing.T, s *Session) { + runTurns(t, s, 1) + s.SetModel(message.ModelRef{Provider: "test", Model: "m2"}) + s.SetEffort(message.EffortHigh) + }, + }, + { + name: "compaction folds a prefix", + turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + compactTurn("three", provider.Usage{InputTokens: 30}), + compactSummaryTurn("SUMMARY", provider.Usage{InputTokens: 40, OutputTokens: 8}), + }, + drive: func(t *testing.T, s *Session) { + runTurns(t, s, 3) + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}); err != nil { + t.Fatal(err) + } + }, + }, + { + name: "active goal", + turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }, + drive: func(t *testing.T, s *Session) { + runTurns(t, s, 1) + if err := s.RegisterGoal("ship it"); err != nil { + t.Fatal(err) + } + if err := s.UpdateGoal("ship it twice"); err != nil { + t.Fatal(err) + } + }, + }, + { + name: "goal cleared", + turns: [][]provider.Event{}, + drive: func(t *testing.T, s *Session) { + if err := s.RegisterGoal("ship it"); err != nil { + t.Fatal(err) + } + s.ClearGoal() + }, + }, + { + name: "prompt queue", + turns: [][]provider.Event{}, + drive: func(t *testing.T, s *Session) { + if _, err := s.EnqueuePrompt("first"); err != nil { + t.Fatal(err) + } + if _, err := s.EnqueuePrompt("second"); err != nil { + t.Fatal(err) + } + if _, _, err := s.EnqueuePromptDurable("third", 1); err != nil { + t.Fatal(err) + } + if _, _, ok := s.DequeuePrompt("delivered"); !ok { + t.Fatal("DequeuePrompt: queue empty") + } + }, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: tc.turns} + cfg := persistCfg(dir, prov) + cfg.WorkDir = dir + cfg.ParentSession = "ses_00000000000000000000000000" + s := NewSession(cfg) + tc.drive(t, s) + if err := s.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + assertIndexMatchesLoad(t, cfg, s.ID) + }) + } +} + +// TestSessionIndexIsCurrentAfterEveryRecord proves the write-through half: +// after each mutation the sidecar already describes the journal, so a +// reader never has to open it. +// +// The proof is destructive on purpose. After each mutation the journal is +// overwritten with garbage of the SAME byte length: any read of the journal +// itself would now fail or fold nonsense, so an index that still reports +// the right message count can only have come from the sidecar. +func TestSessionIndexIsCurrentAfterEveryRecord(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 2) + + // 2 turns x (user + assistant) = 4 durable messages. + want := 4 + corruptJournalKeepingSize(t, dir, s.ID) + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != want { + t.Errorf("Messages = %d, want %d (index must be served from the sidecar, not the journal)", ix.Messages, want) + } + if ix.Usage.InputTokens != 30 { + t.Errorf("Usage.InputTokens = %d, want 30", ix.Usage.InputTokens) + } +} + +// corruptJournalKeepingSize replaces a session journal with unparseable +// bytes of identical length AND identical modification time. Length and +// modification time are the whole staleness key ReadSessionIndex checks, so +// this leaves a "current" sidecar in front of a journal no fold can read: +// any answer that still names the right message count can only have come +// from the sidecar. +// +// Production never produces this state — a journal is append-only and has +// one writer — which is exactly why it is a usable probe here. +func corruptJournalKeepingSize(t *testing.T, dir, id string) { + t.Helper() + path := filepath.Join(dir, id+".jsonl") + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + junk := make([]byte, len(data)) + for i := range junk { + junk[i] = 'x' + } + if err := os.WriteFile(path, junk, 0o644); err != nil { + t.Fatal(err) + } + if err := os.Chtimes(path, fi.ModTime(), fi.ModTime()); err != nil { + t.Fatal(err) + } +} + +// TestReadSessionIndexRefoldsWhenJournalGrows covers the crash window: a +// process that wrote a record but died before its sidecar flush, and any +// journal an older binary wrote with no sidecar at all. The stored index +// covers fewer bytes than the journal holds, so it must be refolded, never +// served. +func TestReadSessionIndexRefoldsWhenJournalGrows(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + // Freeze the sidecar as it stands after turn 1, then run turn 2 with + // the sidecar pinned to the older fold — the exact state a crash + // between a record write and its flush leaves behind. + frozen, err := os.ReadFile(filepath.Join(dir, s.ID+sessionIndexSuffix)) + if err != nil { + t.Fatal(err) + } + runTurns(t, s, 1) + if err := os.WriteFile(filepath.Join(dir, s.ID+sessionIndexSuffix), frozen, 0o644); err != nil { + t.Fatal(err) + } + + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 4 { + t.Errorf("Messages = %d, want 4 (a stale sidecar must refold)", ix.Messages) + } + // The refold writes back, so the next read takes the fast path. + corruptJournalKeepingSize(t, dir, s.ID) + again, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex (second): %v", err) + } + if again.Messages != 4 { + t.Errorf("second read Messages = %d, want 4 (refold must write back)", again.Messages) + } +} + +// TestReadSessionIndexRefoldsAfterJournalShrinks covers ensureLog's +// torn-tail truncation: the journal gets SHORTER than the sidecar's fold. +// A byte-length comparison alone (rather than "grew since") is what catches +// it. +func TestReadSessionIndexRefoldsAfterJournalShrinks(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 2) + + path := filepath.Join(dir, s.ID+".jsonl") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + // Drop the final record, as a truncating tail repair would. + cut := len(data) - 1 + for cut > 0 && data[cut-1] != '\n' { + cut-- + } + if err := os.WriteFile(path, data[:cut], 0o644); err != nil { + t.Fatal(err) + } + + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 3 { + t.Errorf("Messages = %d, want 3 (a shorter journal must refold)", ix.Messages) + } +} + +// TestReadSessionIndexIgnoresUnusableSidecar covers every way a stored +// index can be unusable: torn (a crash inside the rewrite), empty, a +// checksum that does not cover its bytes, a format version from another +// binary, or an id naming another session. Each must refold silently rather +// than serve or fail. +// +// The version and id cases build a VALID envelope and recompute the +// checksum over the altered bytes. Mangling the index bytes alone would +// fail the checksum first, and the test would pass without ever reaching +// the check it names. +func TestReadSessionIndexIgnoresUnusableSidecar(t *testing.T) { + sidecars := map[string]func(t *testing.T, ix SessionIndex) []byte{ + "empty": func(*testing.T, SessionIndex) []byte { return nil }, + "torn prefix": func(t *testing.T, ix SessionIndex) []byte { + b := mustMarshalIndex(t, ix) + return b[:len(b)/2] + }, + "checksum does not cover the bytes": func(t *testing.T, ix SessionIndex) []byte { + ix.Messages = 99 + inner, err := json.Marshal(ix) + if err != nil { + t.Fatal(err) + } + b, err := json.Marshal(sessionIndexFile{CRC32: crc32.ChecksumIEEE(inner) + 1, Index: inner}) + if err != nil { + t.Fatal(err) + } + return b + }, + // Each mangled sidecar also carries a WRONG message count, so + // serving it is detectable. A sidecar mangled only in the field + // under test would still report the right count, and the case + // would pass whether the check ran or not. + "old version": func(t *testing.T, ix SessionIndex) []byte { + ix.Version = sessionIndexVersion + 1 + ix.Messages = 99 + return mustMarshalIndex(t, ix) + }, + "wrong id": func(t *testing.T, ix SessionIndex) []byte { + ix.ID = "ses_0123456789abcdef" + ix.Messages = 99 + return mustMarshalIndex(t, ix) + }, + } + for name, mangle := range sidecars { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, s.ID+sessionIndexSuffix), mangle(t, ix), 0o644); err != nil { + t.Fatal(err) + } + got, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if got.Messages != 2 { + t.Errorf("Messages = %d, want 2 (an unusable sidecar must refold)", got.Messages) + } + }) + } +} + +// mustMarshalIndex renders a sidecar exactly as the production writers do, +// checksum included, so a test that alters a field still produces a file +// that reaches the check it means to exercise. +func mustMarshalIndex(t *testing.T, ix SessionIndex) []byte { + t.Helper() + b, err := marshalSessionIndex(ix) + if err != nil { + t.Fatal(err) + } + return b +} + +// TestSessionIndexSurvivesResume pins the seam a resumed session depends +// on: LoadSession seeds the fold from the journal it replays, so the first +// record the reloaded session writes flushes an index describing the WHOLE +// journal, not just that one record. +func TestSessionIndexSurvivesResume(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + // Resume in a second Session object, exactly as the serve API does for + // a session it evicted, and drive one more turn. + resumed, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatal(err) + } + if _, err := resumed.Prompt(context.Background(), "go"); err != nil { + t.Fatal(err) + } + if err := resumed.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + + corruptJournalKeepingSize(t, dir, s.ID) + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 4 { + t.Errorf("Messages = %d, want 4 (a resumed session must flush the whole fold)", ix.Messages) + } + if ix.Usage.InputTokens != 30 { + t.Errorf("Usage.InputTokens = %d, want 30", ix.Usage.InputTokens) + } +} + +// TestListSessionIndexesSkipsNonSessionFiles: the server's own event +// journal (events.jsonl) and a session's sidecar live in the same +// directory. Neither is a session, and neither may appear in a listing or +// fail it. +func TestListSessionIndexesSkipsNonSessionFiles(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + if err := os.WriteFile(filepath.Join(dir, "events.jsonl"), []byte(`{"type":"message","seq":1}`+"\n"), 0o644); err != nil { + t.Fatal(err) + } + + list, err := ListSessionIndexes(dir) + if err != nil { + t.Fatalf("ListSessionIndexes: %v", err) + } + if len(list) != 1 || list[0].ID != s.ID { + t.Fatalf("ListSessionIndexes = %+v, want exactly the one session %s", list, s.ID) + } +} + +// TestListSessionsMatchesIndex pins the projection: ListSessions now reads +// indexes, and every field SessionInfo carries must still be the value the +// index folded. +func TestListSessionsMatchesIndex(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10, OutputTokens: 5}), + compactTurn("two", provider.Usage{InputTokens: 20, OutputTokens: 6}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 2) + + infos, err := ListSessions(dir) + if err != nil { + t.Fatalf("ListSessions: %v", err) + } + if len(infos) != 1 { + t.Fatalf("ListSessions returned %d sessions, want 1", len(infos)) + } + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatal(err) + } + got, want := infos[0], SessionInfo{ + ID: ix.ID, + CreatedAt: ix.CreatedAt, + Messages: ix.Messages, + Usage: ix.Usage, + LastInputTokens: ix.LastInputTokens, + } + if mustJSON(t, got) != mustJSON(t, want) { + t.Errorf("ListSessions entry = %s, want %s", mustJSON(t, got), mustJSON(t, want)) + } +} + +// TestReadSessionIndexRejectsUnknownSession: an id with no journal is an +// error, not an empty summary — the 404 GET /session/{id} depends on. +func TestReadSessionIndexRejectsUnknownSession(t *testing.T) { + dir := t.TempDir() + if _, err := ReadSessionIndex(dir, "ses_0123456789abcdef"); err == nil { + t.Fatal("ReadSessionIndex on a missing journal = nil error, want an error") + } + if _, err := ReadSessionIndex(dir, "../escape"); err == nil { + t.Fatal("ReadSessionIndex on a path-traversal id = nil error, want an error") + } +} + +// TestSessionIndexRejectsMixedSidecarBytes is the torn-read guard. Both +// sidecar writers replace the file in place, so a reader in another process +// can read part of the old file and part of the new one. Such a mix parses: +// the fields are the same, in the same order, and can carry a staleness key +// that matches the journal. The checksum is what turns it into a clean +// miss. +// +// The test builds the exact hazard: a sidecar whose index bytes describe an +// OLD session state while its staleness key describes the CURRENT journal. +func TestSessionIndexRejectsMixedSidecarBytes(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + stale, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatal(err) + } + runTurns(t, s, 1) + current, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatal(err) + } + + // The mix: turn 1's counts under turn 2's staleness key, checksummed as + // a naive writer would leave it — over the OLD bytes. + mixed := stale + mixed.LogSize, mixed.LogModTime = current.LogSize, current.LogModTime + inner, err := json.Marshal(mixed) + if err != nil { + t.Fatal(err) + } + staleInner, err := json.Marshal(stale) + if err != nil { + t.Fatal(err) + } + file, err := json.Marshal(sessionIndexFile{CRC32: crc32.ChecksumIEEE(staleInner), Index: inner}) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, s.ID+sessionIndexSuffix), file, 0o644); err != nil { + t.Fatal(err) + } + + got, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if got.Messages != 4 { + t.Errorf("Messages = %d, want 4 (a mixed sidecar must refold, never be served)", got.Messages) + } +} + +// TestSessionIndexRefoldsWhenJournalIsRewrittenInPlace: byte length alone +// cannot tell a journal from a same-length replacement. The modification +// time is the second half of the key. +// +// The check is only as good as the filesystem's timestamp resolution, which +// is why this test asserts its own precondition first: on a filesystem that +// reports the rewrite under the SAME modification time, there is nothing +// for the key to catch and the test skips rather than fails. +func TestSessionIndexRefoldsWhenJournalIsRewrittenInPlace(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + if _, err := ReadSessionIndex(dir, s.ID); err != nil { + t.Fatal(err) + } + + // Rewrite the journal at the same length, WITHOUT restoring its + // modification time — an ordinary external rewrite, unlike + // corruptJournalKeepingSize's deliberate probe. + path := filepath.Join(dir, s.ID+".jsonl") + before, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + junk := make([]byte, len(data)) + for i := range junk { + junk[i] = 'x' + } + junk[len(junk)-1] = '\n' + if err := os.WriteFile(path, junk, 0o644); err != nil { + t.Fatal(err) + } + after, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if after.ModTime().Equal(before.ModTime()) { + t.Skip("filesystem reports the same modification time after a rewrite; the staleness key has nothing to detect here") + } + if _, err := ReadSessionIndex(dir, s.ID); err == nil { + t.Fatal("ReadSessionIndex served an index for a rewritten journal, want a refold that fails on the new bytes") + } +} + +// TestSessionIndexMatchesLoadSessionForOrphanToolCall: a journal whose last +// turn died between a tool call and its result. LoadSession repairs it with +// a synthetic tool result, which CHANGES the message count and the activity +// timestamp a reader sees. The index must report the repaired numbers, or +// GET /session and GET /session/{id}/message would disagree about how many +// messages a session has. +func TestSessionIndexMatchesLoadSessionForOrphanToolCall(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test"} + cfg := persistCfg(dir, prov) + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_u1","role":"user","parts":[{"type":"text","text":"go"}],"created_at":"2026-01-02T03:04:06Z"}} +{"type":"message","message":{"id":"msg_a1","role":"assistant","parts":[{"type":"tool_call","call_id":"tc1","name":"bash","arguments":{}}],"created_at":"2026-01-02T03:04:07Z"}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + ix := assertIndexMatchesLoad(t, cfg, id) + if ix.DurableMessages != 2 { + t.Errorf("DurableMessages = %d, want 2 (the records themselves)", ix.DurableMessages) + } + if ix.Messages != 3 { + t.Errorf("Messages = %d, want 3 (the repair adds one)", ix.Messages) + } +} + +// TestSessionIndexReportsIncompleteForALegacyJournal: a journal that never +// recorded a model (a crash tore the initial model record away, or the log +// predates it) cannot be answered from a fold — LoadSession answers those +// from the loading Config. The index must say so rather than report an +// empty model. +func TestSessionIndexReportsIncompleteForALegacyJournal(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + for name, journal := range map[string]string{ + "torn model record": `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","mod`, + "legacy header without workdir": `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z"} +{"type":"model","model":"test/m1"} +`, + } { + t.Run(name, func(t *testing.T) { + dir := filepath.Join(dir, name) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + ix, err := ReadSessionIndex(dir, id) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Complete { + t.Error("Complete = true, want false: the journal records no model or no workdir, so only a load can answer") + } + }) + } +} + +// TestSessionIndexSurvivesAnOversizedHeader: a session header is not +// bounded in size — ensureLog writes task_tool_names into it. A header +// larger than the first-line peek must not make the session invisible to a +// read or a listing. +func TestSessionIndexSurvivesAnOversizedHeader(t *testing.T) { + dir := t.TempDir() + tools := make([]string, 4000) + for i := range tools { + tools[i] = "mcp__server__tool_with_a_long_name_" + string(rune('a'+i%26)) + } + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + cfg.WorkDir = dir + cfg.TaskParentID = "ses_0123456789abcdef" + cfg.TaskAgentType = "explore" + cfg.TaskToolNames = tools + s := NewSession(cfg) + runTurns(t, s, 1) + + // Drop the sidecar, forcing the refold path — the one that peeks at the + // first line before reading the file. + if err := os.Remove(filepath.Join(dir, s.ID+sessionIndexSuffix)); err != nil { + t.Fatal(err) + } + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex on an oversized header: %v", err) + } + if ix.Messages != 2 { + t.Errorf("Messages = %d, want 2", ix.Messages) + } + list, err := ListSessionIndexes(dir) + if err != nil || len(list) != 1 { + t.Fatalf("ListSessionIndexes = %v, %v; want the one session", list, err) + } +} + +// TestSessionIndexAfterEnsureLogTailRepair drives ensureLog's OWN two +// repair branches, rather than editing a journal by hand, and checks the +// index still agrees with a full load afterwards. Branch 1 truncates a torn +// record away; branch 2 terminates a complete record whose newline never +// landed. The two move the journal's length in opposite directions, and +// logSize has to follow each. +func TestSessionIndexAfterEnsureLogTailRepair(t *testing.T) { + for _, tc := range []struct { + name string + tail string + }{ + {"torn record is truncated", `{"type":"message","message":{"id":"msg_x`}, + {"complete record is terminated", `{"type":"message","message":{"id":"msg_x","role":"user","parts":[{"type":"text","text":"hi"}]}}`}, + } { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + first := NewSession(cfg) + runTurns(t, first, 1) + + // Append an unterminated tail, as a crash mid-write leaves. + path := filepath.Join(dir, first.ID+".jsonl") + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(tc.tail); err != nil { + t.Fatal(err) + } + f.Close() + + // Resume and write one more record: ensureLog repairs the tail + // first, and the flush that follows must describe the repaired + // journal. + resumed, err := LoadSession(cfg, first.ID) + if err != nil { + t.Fatal(err) + } + if _, err := resumed.Prompt(context.Background(), "go"); err != nil { + t.Fatal(err) + } + if err := resumed.PersistErr(); err != nil { + t.Fatalf("PersistErr: %v", err) + } + assertIndexMatchesLoad(t, cfg, first.ID) + + // And the sidecar must be current, not merely correct after a + // refold: corrupt the journal at the same key and read again. + corruptJournalKeepingSize(t, dir, first.ID) + if _, err := ReadSessionIndex(dir, first.ID); err != nil { + t.Errorf("sidecar is not current after a tail repair: %v", err) + } + }) + } +} + +// TestSessionIndexPinsTheRequestedID: the filename names the session, not +// the header record inside it. LoadSession pins the same way, so a journal +// copied to a new name reports the new name on both paths. Without this, +// GET /session/{id} could answer with a different id than it was asked +// about, and every later read would refold. +func TestSessionIndexPinsTheRequestedID(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + // Copy the journal under a second id, header and all. + data, err := os.ReadFile(filepath.Join(dir, s.ID+".jsonl")) + if err != nil { + t.Fatal(err) + } + const copied = "ses_0123456789abcdef" + if err := os.WriteFile(filepath.Join(dir, copied+".jsonl"), data, 0o644); err != nil { + t.Fatal(err) + } + + ix, err := ReadSessionIndex(dir, copied) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.ID != copied { + t.Errorf("ID = %q, want %q (the filename names the session)", ix.ID, copied) + } + loaded, err := LoadSession(cfg, copied) + if err != nil { + t.Fatal(err) + } + if loaded.ID != ix.ID { + t.Errorf("LoadSession reports %q, the index reports %q", loaded.ID, ix.ID) + } + // The second read must take the stored index, not refold: an id that + // never matches would make every read pay a fold forever. + corruptJournalKeepingSize(t, dir, copied) + again, err := ReadSessionIndex(dir, copied) + if err != nil { + t.Fatalf("second ReadSessionIndex: %v", err) + } + if again.Messages != ix.Messages { + t.Errorf("second read = %d messages, want %d from the stored index", again.Messages, ix.Messages) + } +} + +// TestIndexFoldWorkIsConstantPerRecord is the cost guard for the WRITE +// path. Session.writeRecord folds and flushes after every record, so any +// work here that grows with history length is O(n^2) over a session's life +// — an index that makes reads cheap and writes quadratic is a net loss on +// exactly the long sessions it exists for. A review caught that shape. +// +// Bytes allocated, not elapsed time, are the measurement: they are +// deterministic under a fixed input, and they separate constant work from a +// re-fold cleanly. Object COUNT does not: a re-fold of a skeleton with no +// orphan returns the same slice and allocates one copy, so it looks +// constant while copying the whole history. +// +// The ratio, not an absolute, is the assertion. A skeleton forty times +// longer must not cost anything like forty times as much per record. +func TestIndexFoldWorkIsConstantPerRecord(t *testing.T) { + build := func(n int) *indexFold { + f := &indexFold{header: true} + for i := 0; i < n; i++ { + role := message.RoleUser + if i%2 == 1 { + role = message.RoleAssistant + } + f.appendMessage(message.Message{ID: fmt.Sprintf("m%d", i), Role: role}) + } + return f + } + bytesPerOp := func(f *indexFold) uint64 { + const runs = 200 + var before, after runtime.MemStats + runtime.GC() + runtime.ReadMemStats(&before) + for i := 0; i < runs; i++ { + f.snapshot(1, time.Time{}) + } + runtime.ReadMemStats(&after) + return (after.TotalAlloc - before.TotalAlloc) / runs + } + small := bytesPerOp(build(100)) + large := bytesPerOp(build(4000)) + // 40x the history. Constant work stays flat; a re-fold copies the whole + // skeleton every time. Four is generous for the first and unreachable + // for the second. + if small > 0 && large > small*4 { + t.Errorf("snapshot costs %d bytes at 100 messages and %d at 4000: work grows with history length", small, large) + } +} + +// TestIndexFoldRepairCountMatchesAFullRecount: the repair count is +// maintained as records arrive, and a full recount is the truth it must +// equal. This is the drift the incremental path can develop and the oracle +// test cannot see on its own, because a shape whose incremental and full +// answers agree by luck passes both. +func TestIndexFoldRepairCountMatchesAFullRecount(t *testing.T) { + call := func(id string) message.Parts { + return message.Parts{&message.ToolCall{CallID: id}} + } + result := func(id string) message.Parts { + return message.Parts{&message.ToolResult{CallID: id}} + } + shapes := map[string][]message.Message{ + "plain turns": { + {ID: "u1", Role: message.RoleUser}, + {ID: "a1", Role: message.RoleAssistant}, + }, + "matched tool call": { + {ID: "u1", Role: message.RoleUser}, + {ID: "a1", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: "t1", Role: message.RoleTool, Parts: result("tc1")}, + }, + "orphan at the tail": { + {ID: "u1", Role: message.RoleUser}, + {ID: "a1", Role: message.RoleAssistant, Parts: call("tc1")}, + }, + "orphan mid-history": { + {ID: "u1", Role: message.RoleUser}, + {ID: "a1", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: "u2", Role: message.RoleUser}, + {ID: "a2", Role: message.RoleAssistant, Parts: call("tc2")}, + {ID: "t2", Role: message.RoleTool, Parts: result("tc2")}, + }, + "partial results": { + {ID: "a1", Role: message.RoleAssistant, Parts: message.Parts{&message.ToolCall{CallID: "tc1"}, &message.ToolCall{CallID: "tc2"}}}, + {ID: "t1", Role: message.RoleTool, Parts: result("tc1")}, + }, + "two orphans in a row": { + {ID: "a1", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: "a2", Role: message.RoleAssistant, Parts: call("tc2")}, + {ID: "u1", Role: message.RoleUser}, + }, + } + // Shapes a journal can hold that break an id-based lookup: repeated + // ids, absent ids, and an id shaped like the repair's own synthetic + // one. A review found the first of these reporting a NEGATIVE count. + shapes["duplicate ids on adjacent orphans"] = []message.Message{ + {ID: "same", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: "same", Role: message.RoleAssistant, Parts: call("tc2")}, + } + shapes["absent ids"] = []message.Message{ + {Role: message.RoleAssistant, Parts: call("tc1")}, + {Role: message.RoleAssistant, Parts: call("tc2")}, + {Role: message.RoleUser}, + } + shapes["a follower wearing the repair's own id shape"] = []message.Message{ + {ID: "a1", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: message.SyntheticOrphanIDPrefix + "0-tc1", Role: message.RoleTool, Parts: result("tc1")}, + } + shapes["duplicate ids across a matched call"] = []message.Message{ + {ID: "dup", Role: message.RoleAssistant, Parts: call("tc1")}, + {ID: "dup", Role: message.RoleTool, Parts: result("tc1")}, + {ID: "dup", Role: message.RoleAssistant, Parts: call("tc2")}, + } + + for name, msgs := range shapes { + t.Run(name, func(t *testing.T) { + f := &indexFold{header: true} + // Check after EVERY append, not only at the end: the + // incremental update runs once per message, and a shape that + // converges at the end can still be wrong in the middle, which + // is what a reader of a live session would see. + for i := range msgs { + f.appendMessage(msgs[i]) + incremental := f.repairs + f.recountRepairs() + if incremental != f.repairs { + t.Fatalf("after %d messages: incremental count %d, full recount %d", i+1, incremental, f.repairs) + } + if f.repairs < 0 { + t.Fatalf("after %d messages: repair count is negative (%d)", i+1, f.repairs) + } + repaired := message.ResolveOrphanToolCalls(append([]message.Message(nil), f.messages...)) + if want := len(repaired) - len(f.messages); f.repairs != want { + t.Fatalf("after %d messages: count %d, but the repair inserts %d", i+1, f.repairs, want) + } + } + }) + } +} + +// TestListSessionsNeverDropsASession: the journals are what exist. The +// index is an acceleration over them, never the source of truth about +// existence — a session whose fold breaks has no index, and a listing that +// dropped it would lie to every caller that asks what is here. +// +// The probe is a journal with a compact record naming a range that is not +// in it. That folds to an error, so the index cannot answer, and the +// session must still be listed from a direct scan. +func TestListSessionsNeverDropsASession(t *testing.T) { + dir := t.TempDir() + healthy := "ses_0123456789abcdef" + broken := "ses_fedcba9876543210" + if err := os.WriteFile(filepath.Join(dir, healthy+".jsonl"), []byte( + `{"type":"session","id":"`+healthy+`","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":7}} +`), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, broken+".jsonl"), []byte( + `{"type":"session","id":"`+broken+`","created_at":"2026-01-02T03:04:06Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":9}} +{"type":"compact","compact":{"first_id":"absent_first","last_id":"absent_last","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +`), 0o644); err != nil { + t.Fatal(err) + } + + // The broken one has no index to serve. + if _, err := ReadSessionIndex(dir, broken); err == nil { + t.Fatal("test setup: the broken journal folded cleanly") + } + + infos, err := ListSessions(dir) + if err != nil { + t.Fatalf("ListSessions: %v", err) + } + got := map[string]SessionInfo{} + for _, info := range infos { + got[info.ID] = info + } + if _, ok := got[healthy]; !ok { + t.Errorf("the healthy session is missing from the listing: %+v", infos) + } + if info, ok := got[broken]; !ok { + t.Errorf("a session whose fold breaks was dropped from the listing: %+v", infos) + } else if info.Messages != 1 || info.Usage.InputTokens != 9 { + t.Errorf("the fallback scan reported %d messages and %d input tokens, want 1 and 9", info.Messages, info.Usage.InputTokens) + } +} + +// TestListingNeverWritesSidecars: a read must not mutate. Listing a +// directory used to refold and write back a sidecar for every session whose +// index was stale — including sessions another writer holds — so the list +// path repaired what only the write path should. +func TestListingNeverWritesSidecars(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + sidecar := filepath.Join(dir, s.ID+sessionIndexSuffix) + if err := os.Remove(sidecar); err != nil { + t.Fatal(err) + } + + for _, call := range []struct { + name string + run func() error + }{ + {"ListSessions", func() error { _, err := ListSessions(dir); return err }}, + {"ListSessionIndexes", func() error { _, err := ListSessionIndexes(dir); return err }}, + } { + if err := call.run(); err != nil { + t.Fatalf("%s: %v", call.name, err) + } + if _, err := os.Stat(sidecar); err == nil { + t.Fatalf("%s wrote a sidecar; a listing must not repair one", call.name) + } + } + + // A single-session read still memoizes: that is one fold, and it makes + // the next read of that session a stat and a small read. + if _, err := ReadSessionIndex(dir, s.ID); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(sidecar); err != nil { + t.Errorf("ReadSessionIndex did not write the sidecar back: %v", err) + } +} + +// TestListSessionsFallbackCountsCompactUsage: the fallback scan reports the +// numbers the index would have reported, not the numbers the pre-index scan +// did. A compact record carries the summarization call's own spend; +// LoadSession adds it to cumulative usage and so does the index, so the +// fallback does too. LastInputTokens still moves for message records only. +func TestListSessionsFallbackCountsCompactUsage(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + // A journal whose compact record names an absent range: the fold + // breaks, so the listing must take its fallback path. + journal := `{"type":"session","id":"` + id + `","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":11,"output_tokens":3}} +{"type":"compact","usage":{"input_tokens":5,"output_tokens":2},"compact":{"first_id":"absent","last_id":"absent","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + if _, err := ReadSessionIndex(dir, id); err == nil { + t.Fatal("test setup: the journal folded cleanly, so the fallback never runs") + } + + infos, err := ListSessions(dir) + if err != nil { + t.Fatalf("ListSessions: %v", err) + } + if len(infos) != 1 { + t.Fatalf("listed %d sessions, want 1", len(infos)) + } + got := infos[0] + if got.Usage.InputTokens != 16 || got.Usage.OutputTokens != 5 { + t.Errorf("usage = %d in / %d out, want 16 and 5 (the message plus the compaction)", got.Usage.InputTokens, got.Usage.OutputTokens) + } + if got.LastInputTokens != 11 { + t.Errorf("last_input_tokens = %d, want 11 (a compact record must not move it)", got.LastInputTokens) + } +} + +// TestListSessionsFallbackIgnoresStrayUsage: two record types carry usage a +// reader counts — a message record and a compact record — because those are +// the two LoadSession reads. A usage field on any other record, which a +// future build could write, must not inflate a listing past what the +// authoritative load reports. +func TestListSessionsFallbackIgnoresStrayUsage(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"` + id + `","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":11}} +{"type":"goal.set","goal":{"condition":"x"},"usage":{"input_tokens":500}} +{"type":"compact","compact":{"first_id":"absent","last_id":"absent","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + if _, err := ReadSessionIndex(dir, id); err == nil { + t.Fatal("test setup: the journal folded cleanly, so the fallback never runs") + } + infos, err := ListSessions(dir) + if err != nil { + t.Fatalf("ListSessions: %v", err) + } + if len(infos) != 1 { + t.Fatalf("listed %d sessions, want 1", len(infos)) + } + if infos[0].Usage.InputTokens != 11 { + t.Errorf("usage = %d, want 11: a goal record's usage is not a reader's to count", infos[0].Usage.InputTokens) + } +} + +// TestListSessionsPinsTheFilenameID: the filename names the session on BOTH +// answers. LoadSession pins the same way, and so does the index, so a +// journal copied to a new name reports the new name however it was read. +func TestListSessionsPinsTheFilenameID(t *testing.T) { + dir := t.TempDir() + const named = "ses_0123456789abcdef" + // The header names a DIFFERENT session, and the fold breaks, so the + // fallback answers. + journal := `{"type":"session","id":"ses_fedcba9876543210","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]}} +{"type":"compact","compact":{"first_id":"absent","last_id":"absent","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +` + if err := os.WriteFile(filepath.Join(dir, named+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + infos, err := ListSessions(dir) + if err != nil { + t.Fatalf("ListSessions: %v", err) + } + if len(infos) != 1 || infos[0].ID != named { + t.Fatalf("listing reported %+v, want the filename id %s", infos, named) + } + info, err := ReadSessionInfo(dir, named) + if err != nil { + t.Fatalf("ReadSessionInfo: %v", err) + } + if info.ID != named { + t.Errorf("ReadSessionInfo reported %q, want the filename id %q", info.ID, named) + } +} diff --git a/engine/messagepage.go b/engine/messagepage.go new file mode 100644 index 00000000..4db49657 --- /dev/null +++ b/engine/messagepage.go @@ -0,0 +1,690 @@ +// Paginated message reads: the newest K messages of a session, or the K +// before a given sequence number, read from the journal's tail instead of +// its whole length. +// +// The problem. GET /session/{id}/message answered with the ENTIRE message +// history, always. On the fleet's longest production session that is 1.4 MB +// and 5.8 s per load, every time a console opens (meetneptune/boxes +// docs/design/console-read-path.md, workstream 2). A console shows the tail +// first and pages older messages in on scroll, so it needs a bounded read. +// +// The sequence number. A message's seq is its 1-based ordinal in the +// session's DURABLE message sequence: message records in log order, with +// each compact record's fold applied (the folded range replaced by that +// record's summary). SessionIndex.DurableMessages (index.go) is the count +// of that same sequence, so the newest message's seq equals it. Note that +// SessionIndex.Messages can be LARGER: it also counts the synthetic tool +// results message.ResolveOrphanToolCalls derives for a tool call whose +// result never reached the log. Those messages have no record, so they have +// no seq and no page carries them. This definition is +// what makes a bounded read possible at all: an ordinal over the durable +// records can be counted backwards from the end, while an ordinal over a +// materialized history cannot be known without materializing it. +// +// Two consequences follow, and both are deliberate: +// +// - A page carries the durable messages VERBATIM. It never runs +// message.ResolveOrphanToolCalls, the load-time repair that synthesizes +// an is_error tool result for a tool call whose result is absent. That +// repair exists so a REQUEST is wire-valid; this endpoint builds no +// request. Running it here would also fabricate a failure at every page +// boundary that splits an assistant message from its results, and a +// fabricated tool failure in a read view is a defect with production +// history (see Server.lookup's doc comment, server/handlers.go: a +// healthy child's in-flight tool call rendered as failed in the console +// for as long as it kept running). +// - Compaction renumbers. A fold replaces N messages with one summary, so +// every seq after it shifts down by N-1. A client paging across a +// compaction can see one page overlap another; message ids, which are +// stable, are the way to de-duplicate. +package engine + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "os" + + "github.com/majorcontext/harness/message" +) + +// DefaultMessagePageLimit is the page size a caller gets when it asks for a +// page without naming one. MaxMessagePageLimit caps what it may ask for: +// the point of this API is a bounded read, so an unbounded limit is a +// contradiction, and a caller that truly wants everything still has the +// unparameterized call. +const ( + DefaultMessagePageLimit = 100 + MaxMessagePageLimit = 1000 +) + +// ErrStaleMessagePage reports that the journal changed under a page read in +// a way that invalidates the sequence numbers it was about to serve: it is +// now shorter than the index that numbered it. A caller retries with a +// fresh index; ReadMessagePage does that itself once (see readMessagePage). +var ErrStaleMessagePage = errors.New("engine: session journal changed under a message page read") + +// MessagePage is one page of a session's durable message sequence, oldest +// first — the same order the unparameterized read returns. +type MessagePage struct { + // Messages holds the page, ascending by seq. Empty when the session has + // no messages at or below the requested point. + Messages []message.Message + // FirstSeq and LastSeq are the seqs of the first and last entries of + // Messages, and 0 for an empty page. A client pages further back by + // asking again with BeforeSeq = FirstSeq. + FirstSeq int + LastSeq int + // Total is the session's whole durable message count — the seq of its + // newest message — so a client knows where the page sits without a + // second call. It can be lower than the `messages` field of GET + // /session, which also counts derived repair messages that have no + // record and so no seq. + Total int + // HasMore reports whether at least one message older than FirstSeq + // exists. It is false for a page that starts at seq 1. + HasMore bool +} + +// revChunkBytes is the backward scan's read granularity. It is comfortably +// larger than one record of ordinary size, so a page of a few dozen +// messages usually costs one or two reads, and it bounds how much of a +// journal a page read touches at all. +const revChunkBytes = 64 << 10 + +// MessagePageWindow resolves a page request against a total, returning the +// inclusive sequence range [lo, hi] the page covers and the limit actually +// applied. hi < lo means an empty page: nothing sits at or below the +// requested point. +// +// beforeSeq <= 0 means "the newest page". limit <= 0 means +// DefaultMessagePageLimit, and a limit above MaxMessagePageLimit is capped +// to it — an engine caller gets a bounded answer rather than an error. The +// HTTP boundary is stricter: it rejects an oversized limit, because its +// published schema names a maximum and a client generator enforces it. +// +// It is exported because the server computes the same window when it pages +// a resident history for a session with no journal. Two copies of this +// arithmetic drifting apart would give one session two different +// paginations depending on which path answered. +func MessagePageWindow(total, beforeSeq, limit int) (lo, hi, appliedLimit int) { + if limit <= 0 { + limit = DefaultMessagePageLimit + } + if limit > MaxMessagePageLimit { + limit = MaxMessagePageLimit + } + hi = total + if beforeSeq > 0 && beforeSeq-1 < hi { + hi = beforeSeq - 1 + } + if hi < 1 { + return 1, 0, limit // empty page + } + lo = hi - limit + 1 + if lo < 1 { + lo = 1 + } + return lo, hi, limit +} + +// ReadMessagePage returns the durable messages immediately BEFORE +// beforeSeq, at most limit of them, reading only the journal bytes it needs. +// +// beforeSeq <= 0 means "the newest page". limit <= 0 means +// DefaultMessagePageLimit, and a limit above MaxMessagePageLimit is capped +// to it rather than rejected — a caller asking for too much gets a bounded +// answer, not an error it has to handle. +// +// The scan is bounded by SessionIndex.LogSize, not by the file's current +// size, which is what keeps a page consistent with the Total it reports: a +// turn appending records while this runs cannot renumber the page under it, +// because those bytes are past the end this read agreed to look at. +func ReadMessagePage(dir, id string, beforeSeq, limit int) (MessagePage, error) { + page, err := readMessagePage(dir, id, beforeSeq, limit) + if errors.Is(err, ErrStaleMessagePage) { + // One retry, with a freshly folded index. A journal shrinks only + // when a torn tail is repaired, which happens once per crash, so a + // second stale answer is a real fault rather than a race. + page, err = readMessagePage(dir, id, beforeSeq, limit) + } + return page, err +} + +// readMessagePage takes the index through ReadSessionIndex, which memoizes +// a refold. That write is deliberate here and not the anti-pattern a +// LISTING has: this is one session, and without it every page request for +// a session whose sidecar is stale refolds the whole journal again. +// +// It can overlap the session's own writer. The overlap is bounded and +// benign: each writer writes a COMPLETE index of the prefix it folded, +// carrying that prefix's own staleness key, and the checksum covers the +// bytes (see sessionIndexFile), so a reader sees a file that is current or +// visibly stale, never a blend. The window is also small — Session. +// writeRecord's append and its flush are two steps under one lock — so a +// page read only refolds when its stat and its sidecar read straddle that +// gap. +func readMessagePage(dir, id string, beforeSeq, limit int) (MessagePage, error) { + ix, err := ReadSessionIndex(dir, id) + if err != nil { + return MessagePage{}, err + } + return readMessagePageWithIndex(dir, id, ix, beforeSeq, limit) +} + +// readMessagePageWithIndex is readMessagePage with the index already in +// hand. The split is a test seam: the window this function's own stale +// checks exist for opens between taking an index and reading the journal, +// which nothing outside can drive through the public call. +func readMessagePageWithIndex(dir, id string, ix SessionIndex, beforeSeq, limit int) (MessagePage, error) { + page := MessagePage{Total: ix.DurableMessages} + lo, hi, _ := MessagePageWindow(ix.DurableMessages, beforeSeq, limit) + if hi < lo { + return page, nil + } + + f, err := os.Open(sessionPath(dir, id)) + if err != nil { + return MessagePage{}, err + } + defer f.Close() + + // The index named a journal length; the file has to still be at least + // that long. It can be shorter: another process's ensureLog repairs a + // torn tail by truncating it. A read that started BEFORE such a repair + // would number its page against records that no longer exist, so this + // reports staleness and the caller takes a fresh index. + // + // This check and pageError below overlap on purpose. This one answers + // the cheap, common case before any scan runs. pageError answers the + // narrower window the check cannot cover: a repair that lands after it + // and before the scan reads. Removing either leaves the other reporting + // the same classification, one attempt later. + fi, err := f.Stat() + if err != nil { + return MessagePage{}, err + } + if fi.Size() < ix.LogSize { + return MessagePage{}, ErrStaleMessagePage + } + + msgs, ok, err := tailPage(f, ix.LogSize, fi.Size(), ix.DurableMessages, lo, hi) + if err != nil { + return MessagePage{}, pageError(f, id, ix, err) + } + if !ok { + // The page reaches into compacted history. Fall back to the forward + // fold, over exactly the bytes the index summarized, so the page + // and its seqs describe one instant of the journal. + // + // The read is bounded by LogSize, not by the file's current size: a + // turn appending a large record while this runs must not enlarge + // the buffer, and bytes past LogSize are not part of the sequence + // being numbered. It still holds the journal prefix in memory — + // one slim pass, the same shape LoadSession and every refold + // already use — which is why the tail walk above exists for pages + // that do not need it. + data := make([]byte, ix.LogSize) + if _, err := io.ReadFull(f, data); err != nil { + return MessagePage{}, pageError(f, id, ix, err) + } + if msgs, err = foldedPage(data, lo, hi); err != nil { + return MessagePage{}, pageError(f, id, ix, err) + } + } + page.Messages = msgs + if n := len(msgs); n > 0 { + page.FirstSeq = hi - n + 1 + page.LastSeq = hi + page.HasMore = page.FirstSeq > 1 + } + return page, nil +} + +// pageError classifies a failure from a page scan. The check before the +// scan cannot close the whole window: another process's ensureLog can +// truncate a torn tail AFTER that check and BEFORE the scan reads. The scan +// then fails on bytes that no longer exist, and reports a numbering +// mismatch or a short read rather than the truth, which is that the index +// is stale. Re-stat and say so, and ReadMessagePage's one retry answers +// from a fresh index. +func pageError(f *os.File, id string, ix SessionIndex, cause error) error { + if fi, statErr := f.Stat(); statErr == nil && fi.Size() < ix.LogSize { + return ErrStaleMessagePage + } + return fmt.Errorf("engine: session %s: %w", id, cause) +} + +// tailPage is the fast path: a page whose whole range lies in the journal's +// UNCOMPACTED tail. Every message record there is one durable message, in +// order, so the walk numbers them down from total and stops as soon as the +// page is complete — touching only the tail of the file however long the +// journal is. +// +// It gives up (ok false) the moment it meets a compact record, because a +// compact record means the records older than it are not a plain sequence: +// the fold replaced a range of them with one summary, and the messages the +// fold KEPT sit in the log between that range and the record itself. Undoing +// that in reverse is exactly the kind of second, subtly different +// implementation of a fold this repository forbids, so the general path +// below reuses the forward fold instead. +func tailPage(src io.ReaderAt, logSize, size int64, total, lo, hi int) ([]message.Message, bool, error) { + cur := total + var out []message.Message + compacted := false + + err := scanLogBackward(src, logSize, size, func(line logLine, isTail bool) (bool, error) { + // Classify from the line's PREFIX. Walking back to an older page + // passes every record after it, and reading each of those whole — + // a 20 MB image blob or tool result among them — to learn a type + // string and drop it is the cost this endpoint exists to avoid. + // + // The type is the first field of every record this package writes + // (see record), so a prefix answers it. A prefix that does not + // parse is inconclusive rather than corrupt, and the line is read + // whole before any verdict. + if isTail { + // The newest line is the one a crash can leave torn, and a + // TORN line's prefix still parses a type — the type is its + // first field. Read this one whole and require the whole thing + // to parse, exactly as the fold that numbered this page did. + // Otherwise a record the index never counted would shift every + // seq in the page. + whole, err := line.All() + if err != nil { + return false, err + } + if !json.Valid(bytes.TrimSpace(whole)) { + return true, nil + } + } + raw := line.Prefix() + head, ok := decodeRecordHead(raw) + if !ok && !line.Complete() { + // Inconclusive rather than corrupt: a prefix can end mid-key. + whole, err := line.All() + if err != nil { + return false, err + } + raw = whole + head, ok = decodeRecordHead(raw) + } + if !ok { + // The prefix scan reads ONE key, so it answers only for a + // record whose first field is the type — which is every record + // this package writes today (see record, store.go). A record + // with another field order is not corrupt, so fall back to a + // decode that finds the type wherever it sits. The fast path + // stays an optimization rather than a format requirement. + whole, err := line.All() + if err != nil { + return false, err + } + var slim struct { + Type string `json:"type"` + } + if err := json.Unmarshal(bytes.TrimSpace(whole), &slim); err != nil || slim.Type == "" { + return false, errors.New("corrupt record") + } + head = slim.Type + } + switch head { + case recCompact: + compacted = true + return false, nil + case recMessage: + if cur >= lo && cur <= hi { + whole, err := line.All() + if err != nil { + return false, err + } + var rec struct { + Message *message.Message `json:"message"` + } + if err := json.Unmarshal(whole, &rec); err != nil || rec.Message == nil { + return false, fmt.Errorf("message record at offset %d: %v", line.start, err) + } + msg := *rec.Message + // The same ingest-time repair LoadSession applies to every + // message it replays (message.Message.Normalize's doc + // comment): an empty ToolResult persisted by an older + // binary must not reach a reader unrepaired. + msg.Normalize() + out = append(out, msg) + } + // A message record with no message body would make this count + // wrong. It cannot reach here: the index that numbered this + // page folds such a record as an error, so a journal carrying + // one has no index and this path never runs for it. + cur-- + } + return cur >= lo, nil + }) + if err != nil { + return nil, false, err + } + if compacted { + return nil, false, nil + } + if len(out) != hi-lo+1 { + // The walk ran out of journal before it produced the page it was + // numbered for: the index and the records disagree. Report it + // rather than serve messages under seqs that do not describe them. + return nil, false, fmt.Errorf("message page [%d,%d]: journal holds %d of those messages", lo, hi, len(out)) + } + reverseMessages(out) + return out, true, nil +} + +// decodeRecordHead reads a record's type from the START of its bytes, +// without decoding the rest. ok is false when the bytes do not (yet) parse +// as an object whose first key is "type", which for a PREFIX means +// inconclusive — read more — and for a whole line means corrupt. +// +// Type is the first field of every record this package writes (see record, +// store.go), so this returns after one key and one value. +func decodeRecordHead(raw []byte) (recordType string, ok bool) { + dec := json.NewDecoder(bytes.NewReader(bytes.TrimSpace(raw))) + tok, err := dec.Token() + if err != nil || tok != json.Delim('{') { + return "", false + } + key, err := dec.Token() + if err != nil { + return "", false + } + name, isString := key.(string) + if !isString || name != "type" { + return "", false + } + value, err := dec.Token() + if err != nil { + return "", false + } + recordType, isString = value.(string) + return recordType, isString +} + +// foldedPage is the general path, for a journal that carries at least one +// compact record. It folds the journal exactly as LoadSession does — the +// same indexFold, so the same applyCompactRecord — to learn WHICH message +// ids occupy seqs lo..hi, then decodes just those records. +// +// One pass, two decode depths. Every line is folded through indexRecord: +// ids, roles, timestamps, and tool-call ids, never a message body. The same +// pass keeps each record's raw line, keyed by the id that record +// contributes, as a subslice of data rather than a copy. Only the handful +// of lines a page actually carries is then decoded in full. +// +// The line map holds one entry per message record in the folded prefix, not +// one per message in the resulting sequence: a record a compaction folded +// away keeps its entry. That is deliberate. Each entry is an id string and +// a slice header beside a prefix this function already holds in memory in +// full, so pruning would trade a few percent of that for a pass per +// compaction. +// +// An id that appears on two records keeps the FIRST. Engine-minted ids are +// unique, and the one production source of a repeat is a provider-derived +// id hashed from the message's own text (message.ProviderCallID), where the +// two records carry the same content anyway. A journal that repeats an id +// with DIFFERENT content is damaged, and this renders the first of them +// rather than the last. +// +// An earlier revision ran a SECOND scanLog over the journal, decoding every +// line into a full record to find the wanted ones. That decoded every +// message body in the file, which is the cost this whole endpoint exists to +// avoid — a review caught it. The raw-line map is what removes it. +func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { + var fold indexFold + // lineByID aliases data; it never copies a record. + lineByID := make(map[string][]byte) + err := scanLogRaw(data, func(line []byte, n int, isLast bool) error { + var rec indexRecord + if err := json.Unmarshal(line, &rec); err != nil { + if isLast { + return errTruncatedFinalRecord + } + return fmt.Errorf("corrupt record at line %d: %v", n, err) + } + if err := fold.applyIndexRecord(rec, isLast); err != nil { + return fmt.Errorf("%w at line %d", err, n) + } + var contributes string + switch { + case rec.Type == recMessage && rec.Message != nil: + contributes = rec.Message.ID + case rec.Type == recCompact && rec.Compact != nil: + // A compact record contributes its summary to the sequence, + // and the summary lives inside that record's own line. + contributes = rec.Compact.Summary.ID + } + if contributes != "" { + if _, seen := lineByID[contributes]; !seen { + lineByID[contributes] = line + } + } + return nil + }) + if err != nil { + return nil, err + } + if fold.broken { + return nil, errors.New("message page: journal fold is not usable") + } + if hi > len(fold.messages) { + return nil, fmt.Errorf("message page [%d,%d]: journal folds to %d messages", lo, hi, len(fold.messages)) + } + out := make([]message.Message, hi-lo+1) + for seq := lo; seq <= hi; seq++ { + id := fold.messages[seq-1].ID + line, ok := lineByID[id] + if !ok { + return nil, fmt.Errorf("message page [%d,%d]: no record for message %q at seq %d", lo, hi, id, seq) + } + var rec record + if err := json.Unmarshal(line, &rec); err != nil { + return nil, fmt.Errorf("message page [%d,%d]: message %q: %v", lo, hi, id, err) + } + var msg *message.Message + switch { + case rec.Type == recMessage: + msg = rec.Message + case rec.Type == recCompact && rec.Compact != nil: + msg = &rec.Compact.Summary + } + if msg == nil { + return nil, fmt.Errorf("message page [%d,%d]: record for message %q carries no message", lo, hi, id) + } + m := *msg + // The same ingest-time repair LoadSession applies to every message + // it replays (message.Message.Normalize's doc comment). + m.Normalize() + out[seq-lo] = m + } + return out, nil +} + +// reverseMessages flips a newest-first slice into the oldest-first order +// every message response uses. +func reverseMessages(msgs []message.Message) { + for i, j := 0, len(msgs)-1; i < j; i, j = i+1, j-1 { + msgs[i], msgs[j] = msgs[j], msgs[i] + } +} + +// logLine is one line the backward scan found, handed to a callback as a +// PREFIX plus the means to read the rest. +// +// A page reader classifies most of the records it walks and keeps almost +// none of them: walking back to an older page passes every record after it. +// Reading each of those whole cost the bytes of a 20 MB image blob or tool +// result to learn a type string and drop it. The prefix answers that +// question, and All reads the body only for a record the page carries. +type logLine struct { + src io.ReaderAt + start int64 + length int64 + prefix []byte +} + +// Prefix returns the line's first bytes — the whole line when it is short. +func (l logLine) Prefix() []byte { return l.prefix } + +// Complete reports whether Prefix already holds the whole line. +func (l logLine) Complete() bool { return int64(len(l.prefix)) == l.length } + +// All reads the whole line. +func (l logLine) All() ([]byte, error) { + if l.Complete() { + return l.prefix, nil + } + buf := make([]byte, l.length) + if _, err := l.src.ReadAt(buf, l.start); err != nil { + return nil, err + } + return buf, nil +} + +// logLinePeekBytes bounds the prefix read. It is far larger than a record +// head — a type, and a message's identity fields — and far smaller than the +// large records this exists to avoid reading. +const logLinePeekBytes = 4 << 10 + +// scanLogBackward calls fn for each non-empty line in [0, end) of f, newest +// first, stopping when fn returns false or an error, or at byte 0. isTail is +// true only for the file's final line, which is the one a crash can leave +// torn — the same rule scanLog applies reading forward. +// +// size is the file's current length, which the caller has already stat'd; +// end is clamped to it, so a caller that names a bound from a stale index +// reads the file's real bytes rather than an EOF. A caller whose SEQUENCE +// NUMBERS depend on that bound must compare the two itself; ReadMessagePage +// does. +// +// It works in offsets, never in an accumulating buffer. Each block read +// searches backwards for a newline, and the line it delimits is then read +// once — bounded to logLinePeekBytes unless the callback asks for the rest. +// An earlier revision prepended each block to a carry buffer, which copies +// the whole partial line per block: one 20 MB record spans hundreds of +// blocks and copied gigabytes before decoding one message. +// src is an io.ReaderAt rather than an *os.File so a test can measure what +// a page read actually reads: the whole point of the prefix is that a deep +// page does not pull the bytes of the records it walks past, and only a +// counting reader can prove that. +func scanLogBackward(src io.ReaderAt, end, size int64, fn func(line logLine, isTail bool) (bool, error)) error { + if end > size { + end = size + } + if end <= 0 { + return nil + } + // One block stays in memory, and the walk moves backwards through it. + // Every line whose newline search and prefix fall inside it is served + // without touching the file again. A block per LINE — which an earlier + // revision did — reads a 64 KiB block to find a boundary 200 bytes + // away, so a journal of small records was read several times over. + win := newBackwardWindow(src) + lineEnd := end + first := true + for lineEnd > 0 { + lineStart, err := win.newlineBefore(lineEnd) + if err != nil { + return err + } + if lineStart >= lineEnd { + // An empty line: the terminator of the line before it. Step + // over it without consuming the one torn-line allowance. + lineEnd = lineStart - 1 + continue + } + length := lineEnd - lineStart + peek := length + if peek > logLinePeekBytes { + peek = logLinePeekBytes + } + prefix, err := win.at(lineStart, peek) + if err != nil { + return err + } + line := logLine{src: src, start: lineStart, length: length, prefix: prefix} + lineEnd = lineStart - 1 // step over the newline itself + if len(bytes.TrimSpace(prefix)) > 0 { + cont, err := fn(line, first) + first = false + if err != nil || !cont { + return err + } + } + } + return nil +} + +// backwardWindow holds one block of a file and serves reads that fall +// inside it, refilling backwards as a scan moves towards byte 0. It exists +// so a backward scan reads each byte of the span it walks about once. +type backwardWindow struct { + src io.ReaderAt + buf []byte + start int64 // file offset of buf[0] +} + +func newBackwardWindow(src io.ReaderAt) *backwardWindow { + return &backwardWindow{src: src, buf: nil, start: -1} +} + +// covers reports whether [off, off+n) lies inside the block in memory. +func (w *backwardWindow) covers(off, n int64) bool { + return w.start >= 0 && off >= w.start && off+n <= w.start+int64(len(w.buf)) +} + +// fillEndingAt loads the block of up to revChunkBytes that ENDS at end. +func (w *backwardWindow) fillEndingAt(end int64) error { + n := int64(revChunkBytes) + if n > end { + n = end + } + start := end - n + buf := make([]byte, n) + if _, err := w.src.ReadAt(buf, start); err != nil { + return err + } + w.buf, w.start = buf, start + return nil +} + +// newlineBefore returns the offset just past the closest newline before +// end, or 0 when the span back to byte 0 holds none. +func (w *backwardWindow) newlineBefore(end int64) (int64, error) { + searchEnd := end + for searchEnd > 0 { + if !w.covers(searchEnd-1, 1) { + if err := w.fillEndingAt(searchEnd); err != nil { + return 0, err + } + } + // Search only the part of the block below searchEnd. + hi := searchEnd - w.start + if i := bytes.LastIndexByte(w.buf[:hi], '\n'); i >= 0 { + return w.start + int64(i) + 1, nil + } + searchEnd = w.start + } + return 0, nil +} + +// at returns n bytes at off, from the block when it covers them and from +// the file otherwise. The result is a copy: the block is refilled as the +// scan moves on. +func (w *backwardWindow) at(off, n int64) ([]byte, error) { + out := make([]byte, n) + if w.covers(off, n) { + copy(out, w.buf[off-w.start:off-w.start+n]) + return out, nil + } + if _, err := w.src.ReadAt(out, off); err != nil { + return nil, err + } + return out, nil +} diff --git a/engine/messagepage_test.go b/engine/messagepage_test.go new file mode 100644 index 00000000..575197b6 --- /dev/null +++ b/engine/messagepage_test.go @@ -0,0 +1,1117 @@ +package engine + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// pagedSession drives n plain turns against a fresh session in dir and +// returns it. Each turn appends exactly two durable messages: the user +// prompt and the assistant reply. +func pagedSession(t *testing.T, dir string, n int) *Session { + t.Helper() + turns := make([][]provider.Event, 0, n) + for i := 0; i < n; i++ { + turns = append(turns, compactTurn("reply", provider.Usage{InputTokens: 1})) + } + prov := &scriptedProvider{name: "test", turns: turns} + s := NewSession(persistCfg(dir, prov)) + runTurns(t, s, n) + if err := s.PersistErr(); err != nil { + t.Fatalf("PersistErr: %v", err) + } + return s +} + +// wholeSequence is the ORACLE: the durable message sequence, derived from +// the journal's bytes by this test alone. +// +// It calls no engine function. The rule it applies is the contract the +// endpoint publishes, restated here in full: read the records in order; +// each message record appends its id; each compact record removes the ids +// from first_id through last_id and puts its summary id in their place. +// Deriving it from LoadSession instead would share applyCompactRecord with +// the implementation under test, and a fold defect would then agree with +// itself (AGENTS.md's oracle rule). +func wholeSequence(t *testing.T, dir, id string) []string { + t.Helper() + data, err := os.ReadFile(filepath.Join(dir, id+".jsonl")) + if err != nil { + t.Fatal(err) + } + var ids []string + lines := strings.Split(strings.TrimRight(string(data), "\n"), "\n") + for i, line := range lines { + if strings.TrimSpace(line) == "" { + continue + } + var rec struct { + Type string `json:"type"` + Message *struct { + ID string `json:"id"` + } `json:"message"` + Compact *struct { + FirstID string `json:"first_id"` + LastID string `json:"last_id"` + Summary struct { + ID string `json:"id"` + } `json:"summary"` + } `json:"compact"` + } + if err := json.Unmarshal([]byte(line), &rec); err != nil { + if i == len(lines)-1 { + continue // a torn final line, which no reader counts + } + t.Fatalf("oracle: line %d: %v", i+1, err) + } + switch { + case rec.Type == "message" && rec.Message != nil: + ids = append(ids, rec.Message.ID) + case rec.Type == "compact" && rec.Compact != nil: + first, last := -1, -1 + for j, existing := range ids { + if first == -1 && existing == rec.Compact.FirstID { + first = j + } + if first != -1 && existing == rec.Compact.LastID { + last = j + break + } + } + if first == -1 || last == -1 { + t.Fatalf("oracle: compact range [%s, %s] not in the sequence", rec.Compact.FirstID, rec.Compact.LastID) + } + folded := append([]string{}, ids[:first]...) + folded = append(folded, rec.Compact.Summary.ID) + ids = append(folded, ids[last+1:]...) + } + } + return ids +} + +// assertOracleAgreesWithLoad cross-checks the oracle against the production +// authority for the shapes where the two must agree: a full LoadSession's +// history, minus the repair messages it derives, is the durable sequence. +// The oracle stays independent; this is the integration check that keeps it +// honest. +func assertOracleAgreesWithLoad(t *testing.T, cfg Config, dir, id string) { + t.Helper() + loaded, err := LoadSession(cfg, id) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + var durable []string + for _, m := range loaded.History() { + if message.IsSyntheticOrphanID(m.ID) { + continue + } + durable = append(durable, m.ID) + } + if !sameIDs(durable, wholeSequence(t, dir, id)) { + t.Fatalf("oracle disagrees with LoadSession: load = %v, oracle = %v", durable, wholeSequence(t, dir, id)) + } +} + +func idsOf(msgs []message.Message) []string { + out := make([]string, 0, len(msgs)) + for _, m := range msgs { + out = append(out, m.ID) + } + return out +} + +func sameIDs(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +// TestReadMessagePageMatchesFullHistory is the oracle test: every page, at +// every offset and size, must be exactly the corresponding window of the +// session's whole durable message sequence — in the same order, with the +// same ids, under the seqs the page reports. +func TestReadMessagePageMatchesFullHistory(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: nil} + cfg := persistCfg(dir, prov) + sess := pagedSession(t, dir, 6) // 12 durable messages + assertOracleAgreesWithLoad(t, cfg, dir, sess.ID) + want := wholeSequence(t, dir, sess.ID) + if len(want) != 12 { + t.Fatalf("test setup: %d durable messages, want 12", len(want)) + } + + for _, limit := range []int{1, 2, 5, 12, 50} { + for beforeSeq := 0; beforeSeq <= len(want)+1; beforeSeq++ { + page, err := ReadMessagePage(dir, sess.ID, beforeSeq, limit) + if err != nil { + t.Fatalf("ReadMessagePage(before=%d, limit=%d): %v", beforeSeq, limit, err) + } + hi := len(want) + if beforeSeq > 0 && beforeSeq-1 < hi { + hi = beforeSeq - 1 + } + lo := hi - limit + 1 + if lo < 1 { + lo = 1 + } + if hi < 1 { + if len(page.Messages) != 0 { + t.Errorf("before=%d limit=%d: got %d messages, want an empty page", beforeSeq, limit, len(page.Messages)) + } + continue + } + if !sameIDs(idsOf(page.Messages), want[lo-1:hi]) { + t.Errorf("before=%d limit=%d: page ids = %v, want %v", beforeSeq, limit, idsOf(page.Messages), want[lo-1:hi]) + } + if page.FirstSeq != lo || page.LastSeq != hi { + t.Errorf("before=%d limit=%d: seqs = [%d,%d], want [%d,%d]", beforeSeq, limit, page.FirstSeq, page.LastSeq, lo, hi) + } + if page.Total != len(want) { + t.Errorf("before=%d limit=%d: total = %d, want %d", beforeSeq, limit, page.Total, len(want)) + } + if page.HasMore != (lo > 1) { + t.Errorf("before=%d limit=%d: has_more = %v, want %v", beforeSeq, limit, page.HasMore, lo > 1) + } + } + } +} + +// TestReadMessagePageWalksBackToTheStart drives the whole sequence one page +// at a time, exactly as a console scrolling up does, and asserts the pages +// reassemble into the full history with no gap and no repeat. +func TestReadMessagePageWalksBackToTheStart(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: nil} + cfg := persistCfg(dir, prov) + sess := pagedSession(t, dir, 5) // 10 durable messages + assertOracleAgreesWithLoad(t, cfg, dir, sess.ID) + want := wholeSequence(t, dir, sess.ID) + + var got []message.Message + before := 0 + for { + page, err := ReadMessagePage(dir, sess.ID, before, 3) + if err != nil { + t.Fatalf("ReadMessagePage(before=%d): %v", before, err) + } + got = append(page.Messages, got...) + if !page.HasMore { + break + } + before = page.FirstSeq + } + if !sameIDs(idsOf(got), want) { + t.Errorf("paged walk = %v\nwant %v", idsOf(got), want) + } +} + +// TestReadMessagePageUndoesCompaction: a compact record replaces its folded +// range with one summary message. A page numbered over the durable sequence +// must see the summary and never the folded messages, however far back it +// reaches — the reverse of the splice LoadSession applies forward. +func TestReadMessagePageUndoesCompaction(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + compactTurn("three", provider.Usage{InputTokens: 10}), + compactTurn("four", provider.Usage{InputTokens: 10}), + compactSummaryTurn("SUMMARY ONE", provider.Usage{InputTokens: 5}), + compactSummaryTurn("SUMMARY TWO", provider.Usage{InputTokens: 5}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 4) + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 2}); err != nil { + t.Fatalf("first Compact: %v", err) + } + // A second compaction folds the FIRST summary into the second one — the + // nested case, where a compact record met while skipping is itself the + // message the outer fold started at. + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}); err != nil { + t.Fatalf("second Compact: %v", err) + } + + assertOracleAgreesWithLoad(t, cfg, dir, s.ID) + want := wholeSequence(t, dir, s.ID) + page, err := ReadMessagePage(dir, s.ID, 0, 100) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if !sameIDs(idsOf(page.Messages), want) { + t.Errorf("page ids = %v\nwant %v", idsOf(page.Messages), want) + } + if page.Total != len(want) { + t.Errorf("total = %d, want %d", page.Total, len(want)) + } +} + +// TestReadMessagePageReadsOnlyTheTail is the cost claim: a bounded page +// must not depend on the bytes at the start of the journal. Overwriting +// everything except the tail with unreadable bytes leaves a page of the +// newest messages answerable; a reader that walked the whole log could not +// answer it. +// +// The index is written first and left alone, so the seq numbering the page +// reports still comes from the intact fold. +func TestReadMessagePageReadsOnlyTheTail(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 40) // 80 durable messages, comfortably > one chunk + if _, err := ReadSessionIndex(dir, sess.ID); err != nil { + t.Fatal(err) + } + + path := filepath.Join(dir, sess.ID+".jsonl") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + // Keep the final 8 KiB intact (aligned to a record boundary); make the + // rest unreadable. + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + cut := len(data) - 8192 + if cut <= 0 { + t.Fatalf("test setup: journal is only %d bytes", len(data)) + } + for data[cut-1] != '\n' { + cut++ + } + for i := 0; i < cut-1; i++ { + if data[i] != '\n' { + data[i] = 'x' + } + } + if err := os.WriteFile(path, data, 0o644); err != nil { + t.Fatal(err) + } + // Length and modification time are the index's staleness key. Leaving + // both alone is what keeps the intact fold in front of a journal whose + // head is now unreadable. + if err := os.Chtimes(path, fi.ModTime(), fi.ModTime()); err != nil { + t.Fatal(err) + } + + page, err := ReadMessagePage(dir, sess.ID, 0, 2) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if len(page.Messages) != 2 { + t.Fatalf("got %d messages, want 2", len(page.Messages)) + } + if page.LastSeq != 80 || page.FirstSeq != 79 { + t.Errorf("seqs = [%d,%d], want [79,80]", page.FirstSeq, page.LastSeq) + } + if !page.HasMore { + t.Error("has_more = false, want true") + } +} + +// TestReadMessagePageToleratesTornTail: a crash mid-write leaves an +// unparseable final line. The forward reader (scanLog) ignores exactly that +// one line, and so must the backward one — otherwise the newest page of a +// crashed session, the page an operator most wants, is the one that fails. +func TestReadMessagePageToleratesTornTail(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 3) + path := filepath.Join(dir, sess.ID+".jsonl") + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_torn"`); err != nil { + t.Fatal(err) + } + f.Close() + + page, err := ReadMessagePage(dir, sess.ID, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != 6 { + t.Errorf("total = %d, want 6 (the torn record must not count)", page.Total) + } + for _, m := range page.Messages { + if m.ID == "msg_torn" { + t.Error("page carries the torn record") + } + } +} + +// TestReadMessagePageEmptySession: a session with a journal but no messages +// pages to an empty result, not an error. +func TestReadMessagePageEmptySession(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test"} + s := NewSession(persistCfg(dir, prov)) + if err := s.Persist(); err != nil { + t.Fatal(err) + } + page, err := ReadMessagePage(dir, s.ID, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if len(page.Messages) != 0 || page.Total != 0 || page.HasMore { + t.Errorf("page = %+v, want an empty page", page) + } +} + +// TestReadMessagePageCapsLimit: the ENGINE API bounds a read rather than +// erroring, for a caller with no schema to honor. The HTTP boundary is +// stricter — see TestMessagePageRejectsAnOversizedLimit. +func TestReadMessagePageCapsLimit(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 2) + page, err := ReadMessagePage(dir, sess.ID, 0, MaxMessagePageLimit*10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if len(page.Messages) != 4 { + t.Errorf("got %d messages, want all 4", len(page.Messages)) + } +} + +// TestReadMessagePageTailAndFoldPathsAgree: a compacted session is served +// two different ways depending on how far back the page reaches — the +// backward tail walk while the page stays in the uncompacted tail, the +// forward fold once it crosses a compact record. The two must produce the +// same messages under the same seqs, or a console would see a page change +// shape as it scrolled. +func TestReadMessagePageTailAndFoldPathsAgree(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + compactTurn("three", provider.Usage{InputTokens: 10}), + compactTurn("four", provider.Usage{InputTokens: 10}), + compactSummaryTurn("SUMMARY", provider.Usage{InputTokens: 5}), + compactTurn("five", provider.Usage{InputTokens: 10}), + compactTurn("six", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 4) + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}); err != nil { + t.Fatalf("Compact: %v", err) + } + runTurns(t, s, 2) + + assertOracleAgreesWithLoad(t, cfg, dir, s.ID) + want := wholeSequence(t, dir, s.ID) + whole, err := ReadMessagePage(dir, s.ID, 0, 100) + if err != nil { + t.Fatalf("whole page: %v", err) + } + if !sameIDs(idsOf(whole.Messages), want) { + t.Fatalf("whole page ids = %v, want %v", idsOf(whole.Messages), want) + } + // Page one message at a time across the compaction boundary: each page + // must equal the same window of the whole sequence. + for seq := 1; seq <= whole.Total; seq++ { + page, err := ReadMessagePage(dir, s.ID, seq+1, 1) + if err != nil { + t.Fatalf("page at seq %d: %v", seq, err) + } + if len(page.Messages) != 1 || page.Messages[0].ID != want[seq-1] { + t.Errorf("page at seq %d = %v, want %s", seq, idsOf(page.Messages), want[seq-1]) + } + if page.FirstSeq != seq || page.LastSeq != seq { + t.Errorf("page at seq %d reports seqs [%d,%d]", seq, page.FirstSeq, page.LastSeq) + } + } +} + +// TestReadMessagePageSkipsDerivedRepairMessages: a journal whose last turn +// died between a tool call and its result. A full load repairs it with a +// synthetic tool result, and GET /session counts that message. A page must +// NOT: the synthetic message has no record, so it has no seq, and inventing +// one would renumber every page a client already holds. +func TestReadMessagePageSkipsDerivedRepairMessages(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_u1","role":"user","parts":[{"type":"text","text":"go"}],"created_at":"2026-01-02T03:04:06Z"}} +{"type":"message","message":{"id":"msg_a1","role":"assistant","parts":[{"type":"tool_call","call_id":"tc1","name":"bash","arguments":{}}],"created_at":"2026-01-02T03:04:07Z"}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + page, err := ReadMessagePage(dir, id, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != 2 { + t.Errorf("total = %d, want 2 (the two records, not the derived repair)", page.Total) + } + if len(page.Messages) != 2 { + t.Fatalf("got %d messages, want 2", len(page.Messages)) + } + for _, m := range page.Messages { + if message.IsSyntheticOrphanID(m.ID) { + t.Errorf("page carries a derived repair message %q", m.ID) + } + } + // The index still reports the repaired count, which is what GET + // /session reports; the two numbers are deliberately different. + ix, err := ReadSessionIndex(dir, id) + if err != nil { + t.Fatal(err) + } + if ix.Messages != 3 || ix.DurableMessages != 2 { + t.Errorf("index reports messages=%d durable=%d, want 3 and 2", ix.Messages, ix.DurableMessages) + } +} + +// TestScanLogBackwardBoundaries drives the reverse scanner directly, over +// the shapes a journal actually reaches: a record larger than one read +// block, a record spanning three blocks, a file with no final newline, +// blank lines, and an empty file. The scanner's contract is "every +// non-empty line, newest first, and only the file's final line may be +// torn"; each case checks the lines it yields, in order. +func TestScanLogBackwardBoundaries(t *testing.T) { + big := strings.Repeat("A", revChunkBytes+7) // one block plus change + huge := strings.Repeat("B", revChunkBytes*2+11) // spans three blocks + cases := []struct { + name string + content string + want []string + }{ + {"empty file", "", nil}, + {"single line with newline", "one\n", []string{"one"}}, + {"single line without newline", "one", []string{"one"}}, + {"two lines", "one\ntwo\n", []string{"two", "one"}}, + {"blank lines between records", "one\n\n\ntwo\n", []string{"two", "one"}}, + {"trailing blank lines", "one\ntwo\n\n\n", []string{"two", "one"}}, + {"record larger than one block", "one\n" + big + "\ntwo\n", []string{"two", big, "one"}}, + {"record spanning three blocks", "one\n" + huge + "\n", []string{huge, "one"}}, + {"two oversized records", big + "\n" + huge + "\n", []string{huge, big}}, + // A bound past the end of the file: a caller holding a stale index + // names one. The scan reads the file's real bytes rather than + // failing with an EOF from ReadAt. + {"bound past the end", "one\ntwo\n", []string{"two", "one"}}, + {"CRLF terminators", "one\r\ntwo\r\n", []string{"two", "one"}}, + {"line exactly one block", strings.Repeat("C", revChunkBytes) + "\n", []string{strings.Repeat("C", revChunkBytes)}}, + {"terminator on a block boundary", strings.Repeat("D", revChunkBytes-1) + "\n" + "tail\n", []string{"tail", strings.Repeat("D", revChunkBytes-1)}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "log") + if err := os.WriteFile(path, []byte(tc.content), 0o644); err != nil { + t.Fatal(err) + } + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + end := int64(len(tc.content)) + if tc.name == "bound past the end" { + end += 4096 + } + var got []string + fi, err := f.Stat() + if err != nil { + t.Fatal(err) + } + if err := scanLogBackward(f, end, fi.Size(), func(line logLine, _ bool) (bool, error) { + whole, err := line.All() + if err != nil { + return false, err + } + got = append(got, string(bytes.TrimSpace(whole))) + return true, nil + }); err != nil { + t.Fatalf("scanLogBackward: %v", err) + } + if !sameIDs(got, tc.want) { + if len(got) != len(tc.want) { + t.Fatalf("got %d lines, want %d", len(got), len(tc.want)) + } + for i := range got { + if got[i] != tc.want[i] { + t.Fatalf("line %d differs: got %d bytes, want %d bytes", i, len(got[i]), len(tc.want[i])) + } + } + } + }) + } +} + +// TestScanLogBackwardMarksOnlyTheFinalLineAsTail: the torn-line allowance +// belongs to the file's last record and to nothing else, matching scanLog's +// forward rule. A trailing newline, or trailing blank lines, must not spend +// it on a complete record. +func TestScanLogBackwardMarksOnlyTheFinalLineAsTail(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "log") + if err := os.WriteFile(path, []byte("one\ntwo\nthree\n\n"), 0o644); err != nil { + t.Fatal(err) + } + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + var tails []string + fi, err := f.Stat() + if err != nil { + t.Fatal(err) + } + if err := scanLogBackward(f, 15, fi.Size(), func(line logLine, isTail bool) (bool, error) { + if isTail { + whole, err := line.All() + if err != nil { + return false, err + } + tails = append(tails, string(bytes.TrimSpace(whole))) + } + return true, nil + }); err != nil { + t.Fatal(err) + } + if len(tails) != 1 || tails[0] != "three" { + t.Errorf("lines marked as the tail = %v, want exactly [three]", tails) + } +} + +// TestReadMessagePageOversizedRecord is the page-level counterpart: a +// journal carrying one message far larger than a read block must page +// correctly, and the page must carry that message whole. +func TestReadMessagePageOversizedRecord(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn(strings.Repeat("x", revChunkBytes*2+64), provider.Usage{InputTokens: 10}), + compactTurn("small", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 2) + assertOracleAgreesWithLoad(t, cfg, dir, s.ID) + + page, err := ReadMessagePage(dir, s.ID, 0, 4) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if !sameIDs(idsOf(page.Messages), wholeSequence(t, dir, s.ID)) { + t.Fatalf("page ids = %v, want the whole sequence", idsOf(page.Messages)) + } + var found bool + for _, m := range page.Messages { + for _, p := range m.Parts { + if txt, ok := p.(*message.Text); ok && len(txt.Text) > revChunkBytes { + found = true + } + } + } + if !found { + t.Error("the oversized message did not survive the page read whole") + } +} + +// TestReadMessagePageAfterATruncatingRepair: a crash leaves a torn record, +// a later ensureLog truncates it away, and the journal is now SHORTER than +// the index that already folded it. The next page read must notice and +// refold, rather than number a page against a record that no longer exists. +func TestReadMessagePageAfterATruncatingRepair(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 3) + path := filepath.Join(dir, sess.ID+".jsonl") + + // Append a torn record, let the index fold cover it, then truncate it + // away exactly as ensureLog's repair does. + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_torn`); err != nil { + t.Fatal(err) + } + f.Close() + before, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if _, err := ReadSessionIndex(dir, sess.ID); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + cut := bytes.LastIndexByte(data, '\n') + 1 + if err := os.WriteFile(path, data[:cut], 0o644); err != nil { + t.Fatal(err) + } + // Keep the modification time: without this the index refolds on the + // stat alone, and the stale-bound path under test never runs. + if err := os.Chtimes(path, before.ModTime(), before.ModTime()); err != nil { + t.Fatal(err) + } + + page, err := ReadMessagePage(dir, sess.ID, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage after a truncating repair: %v", err) + } + if page.Total != 6 || len(page.Messages) != 6 { + t.Errorf("page = total %d, %d messages; want 6 and 6", page.Total, len(page.Messages)) + } +} + +// TestReadMessagePageStaleIndexBound drives the window the public call +// cannot: a page read that has ALREADY taken an index when another process +// repairs a torn tail and shortens the journal. The bytes the page was +// numbered against are gone, so the read must report staleness rather than +// serve a page whose sequence numbers describe records that no longer +// exist, or fail with a bare EOF from its scan. +// +// The seam is readMessagePageWithIndex, which takes the index the caller +// already holds. ReadMessagePage takes a fresh index and retries once, so +// the public call answers correctly in the same situation. +func TestReadMessagePageStaleIndexBound(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 3) + stale, err := ReadSessionIndex(dir, sess.ID) + if err != nil { + t.Fatal(err) + } + + // The repair: drop the final record, as ensureLog's truncating branch + // does. The index in hand still names the longer journal. + path := filepath.Join(dir, sess.ID+".jsonl") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + cut := bytes.LastIndexByte(data[:len(data)-1], '\n') + 1 + if err := os.WriteFile(path, data[:cut], 0o644); err != nil { + t.Fatal(err) + } + + if _, err := readMessagePageWithIndex(dir, sess.ID, stale, 0, 10); !errors.Is(err, ErrStaleMessagePage) { + t.Errorf("readMessagePageWithIndex with a stale bound = %v, want ErrStaleMessagePage", err) + } + // The public call refolds and answers: five durable messages remain. + page, err := ReadMessagePage(dir, sess.ID, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != 5 || len(page.Messages) != 5 { + t.Errorf("page = total %d, %d messages; want 5 and 5", page.Total, len(page.Messages)) + } +} + +// TestPageErrorClassifiesATruncationDuringTheScan covers the narrower race +// the pre-scan size check cannot: the truncation lands AFTER that check and +// the scan itself fails on bytes that are gone. The failure must still be +// reported as staleness, not as a numbering fault the caller cannot act on. +func TestPageErrorClassifiesATruncationDuringTheScan(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 2) + ix, err := ReadSessionIndex(dir, sess.ID) + if err != nil { + t.Fatal(err) + } + path := filepath.Join(dir, sess.ID+".jsonl") + if err := os.Truncate(path, ix.LogSize-10); err != nil { + t.Fatal(err) + } + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + if err := pageError(f, sess.ID, ix, errors.New("short read")); !errors.Is(err, ErrStaleMessagePage) { + t.Errorf("pageError on a shortened journal = %v, want ErrStaleMessagePage", err) + } + if err := pageError(f, sess.ID, SessionIndex{LogSize: 1}, errors.New("real fault")); errors.Is(err, ErrStaleMessagePage) { + t.Error("pageError reported staleness for a journal that did not shrink") + } +} + +// TestFoldedPageReadsOnlyTheIndexedPrefix: the fold path reads exactly the +// journal prefix its index summarized, and nothing past it. +// +// The record appended below is a COMPACT record, not just a large one, +// because that is the shape that proves the bound: a compaction folds +// earlier messages away and renumbers every seq after it. A page that read +// past its own index would answer with those new numbers while reporting +// the old total — one response describing two instants of the journal. +func TestFoldedPageReadsOnlyTheIndexedPrefix(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + compactTurn("three", provider.Usage{InputTokens: 10}), + compactSummaryTurn("SUMMARY", provider.Usage{InputTokens: 5}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 3) + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 2}); err != nil { + t.Fatalf("Compact: %v", err) + } + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatal(err) + } + want := wholeSequence(t, dir, s.ID) + + // A second compaction lands AFTER the index was taken, exactly as a + // concurrent turn's own compaction would. It folds the sequence the + // index numbered, so a read that saw it would renumber the page. + history := s.History() + if len(history) < 3 { + t.Fatalf("test setup: history is %d messages", len(history)) + } + record := map[string]any{ + "type": "compact", + "compact": map[string]any{ + "first_id": history[0].ID, + "last_id": history[1].ID, + "turns_folded": 1, + "summary": map[string]any{"id": "cmpsum_appended", "role": "user"}, + }, + } + line, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + f, err := os.OpenFile(filepath.Join(dir, s.ID+".jsonl"), os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.Write(append(line, '\n')); err != nil { + t.Fatal(err) + } + f.Close() + + // The page is served from the index already in hand — the seam that + // makes "one instant of the journal" checkable. + page, err := readMessagePageWithIndex(dir, s.ID, ix, 0, 100) + if err != nil { + t.Fatalf("readMessagePageWithIndex: %v", err) + } + if !sameIDs(idsOf(page.Messages), want) { + t.Errorf("page ids = %v, want %v (the appended compaction is outside the read)", idsOf(page.Messages), want) + } + if page.Total != len(want) { + t.Errorf("total = %d, want %d", page.Total, len(want)) + } +} + +// TestFoldedPageDecodesOnlyThePage is the cost guard for the fold path. A +// page that reaches into compacted history folds every line through the +// slim shape, but it must decode IN FULL only the records the page carries. +// An earlier revision ran a second scan that decoded every line into a full +// record to find the wanted ones, which decoded every message body in the +// journal — the cost this endpoint exists to remove. A review caught it. +// +// The probe is a record carrying a part of an unknown type. It is valid +// JSON, and the slim fold walks past it: indexPart keeps only tool-call and +// tool-result parts. A FULL decode of that same line fails, because +// unmarshalPart rejects an unknown part type (message/message.go). So a +// page that excludes that message proves the record was never fully +// decoded, and a page that includes it fails loudly. +func TestFoldedPageDecodesOnlyThePage(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + compactTurn("three", provider.Usage{InputTokens: 10}), + compactSummaryTurn("SUMMARY", provider.Usage{InputTokens: 5}), + compactTurn("four", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 3) + if _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}); err != nil { + t.Fatalf("Compact: %v", err) + } + runTurns(t, s, 1) + + // Rewrite the OLDEST message record — folded away by the compaction, so + // no page below asks for it — to carry an unknown part type. + path := filepath.Join(dir, s.ID+".jsonl") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(strings.TrimRight(string(data), "\n"), "\n") + var poisonedID string + poisonedLine := -1 + for i, line := range lines { + var probe struct { + Type string `json:"type"` + Message *struct { + ID string `json:"id"` + } `json:"message"` + } + if json.Unmarshal([]byte(line), &probe) != nil || probe.Type != recMessage || probe.Message == nil { + continue + } + lines[i] = `{"type":"message","message":{"id":"` + probe.Message.ID + + `","role":"user","parts":[{"type":"from_a_newer_binary","text":"x"}]}}` + poisonedID, poisonedLine = probe.Message.ID, i + break + } + if poisonedID == "" { + t.Fatal("test setup: no message record to rewrite") + } + if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + t.Fatal(err) + } + + // Assert the probe's premise directly: the rewritten line must be + // foldable and NOT fully decodable. If canonical message decoding ever + // accepts an unknown part type, this fails loudly here rather than + // quietly turning the guard below into a test that passes from birth. + var full record + if err := json.Unmarshal([]byte(lines[poisonedLine]), &full); err == nil { + t.Fatal("the probe record decodes in full, so excluding it proves nothing") + } + + // The slim fold must still accept the journal: this probe is only + // meaningful while the record is foldable and merely undecodable. + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("the slim fold rejected the record, so this probe proves nothing: %v", err) + } + if ix.DurableMessages < 3 { + t.Fatalf("test setup: %d durable messages", ix.DurableMessages) + } + + // A page over the compacted history: it crosses the compact record, so + // it takes the fold path, and it must not decode the rewritten record. + page, err := ReadMessagePage(dir, s.ID, 0, ix.DurableMessages) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + for _, m := range page.Messages { + if m.ID == poisonedID { + t.Fatalf("the page carried the folded-away record %q", poisonedID) + } + } + if !sameIDs(idsOf(page.Messages), wholeSequence(t, dir, s.ID)) { + t.Errorf("page ids = %v, want the whole durable sequence", idsOf(page.Messages)) + } +} + +// TestTailPageDecodesOnlyThePage is the fold path's cost guard, applied to +// the tail path. Walking back to page K passes every record newer than it, +// and it must not decode those in full: that is one message body per record +// after the page, the same O(n) the fold path was rewritten to remove. It +// is also a failure surface — a record a newer binary wrote would fail a +// page that never asked for it. +// +// Same probe as the fold path's guard: a record carrying a part of an +// unknown type is valid JSON, so a slim decode walks past it, while a full +// decode fails. +func TestTailPageDecodesOnlyThePage(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 4) // 8 durable messages, no compaction + + path := filepath.Join(dir, sess.ID+".jsonl") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(strings.TrimRight(string(data), "\n"), "\n") + // Rewrite the NEWEST message record: the tail walk passes it on its way + // back to an older page. + rewritten := -1 + for i := len(lines) - 1; i >= 0; i-- { + var probe struct { + Type string `json:"type"` + Message *struct { + ID string `json:"id"` + } `json:"message"` + } + if json.Unmarshal([]byte(lines[i]), &probe) != nil || probe.Type != recMessage || probe.Message == nil { + continue + } + lines[i] = `{"type":"message","message":{"id":"` + probe.Message.ID + + `","role":"assistant","parts":[{"type":"from_a_newer_binary","text":"x"}]}}` + rewritten = i + break + } + if rewritten < 0 { + t.Fatal("test setup: no message record to rewrite") + } + var full record + if err := json.Unmarshal([]byte(lines[rewritten]), &full); err == nil { + t.Fatal("the probe record decodes in full, so passing it proves nothing") + } + if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + t.Fatal(err) + } + os.Remove(filepath.Join(dir, sess.ID+sessionIndexSuffix)) + + // A page of the OLDEST messages: the walk passes the rewritten record. + page, err := ReadMessagePage(dir, sess.ID, 3, 2) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.FirstSeq != 1 || page.LastSeq != 2 || len(page.Messages) != 2 { + t.Fatalf("page = [%d,%d] with %d messages, want [1,2] with 2", page.FirstSeq, page.LastSeq, len(page.Messages)) + } +} + +// pageByteCounter records how many bytes a scan actually pulls from a +// file. It is the only way to prove the claim the prefix exists for: a page +// deep in history must not read the bodies of the records it walks past. +type pageByteCounter struct { + f *os.File + bytes int64 +} + +func (c *pageByteCounter) ReadAt(p []byte, off int64) (int, error) { + n, err := c.f.ReadAt(p, off) + c.bytes += int64(n) + return n, err +} + +// TestTailPageDoesNotReadRecordsItOnlyClassifies is the byte-level cost +// guard for the tail path. Walking back to an older page passes every +// record after it, and pulling each of those whole — a 20 MB image blob or +// tool result among them — to learn a type string and drop it is the cost +// this endpoint exists to avoid. Decoding less is not enough if the bytes +// are read anyway. +func TestTailPageDoesNotReadRecordsItOnlyClassifies(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 3) // six small durable messages + path := filepath.Join(dir, sess.ID+".jsonl") + + // One record far larger than the peek window, then a small one after + // it so the big record is never the tail (a torn tail is read whole by + // design). + const bigBytes = logLinePeekBytes * 40 + beforeBig, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_big","role":"user","parts":[{"type":"text","text":"` + + strings.Repeat("Z", bigBytes) + "\"}]}}\n"); err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"model","model":"test/m2"}` + "\n"); err != nil { + t.Fatal(err) + } + f.Close() + + ix, err := ReadSessionIndex(dir, sess.ID) + if err != nil { + t.Fatal(err) + } + if ix.DurableMessages != 7 { + t.Fatalf("index counts %d durable messages, want 7", ix.DurableMessages) + } + + read := func(lo, hi int) (int64, []message.Message) { + t.Helper() + jf, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer jf.Close() + fi, err := jf.Stat() + if err != nil { + t.Fatal(err) + } + counter := &pageByteCounter{f: jf} + msgs, ok, err := tailPage(counter, ix.LogSize, fi.Size(), ix.DurableMessages, lo, hi) + if err != nil || !ok { + t.Fatalf("tailPage([%d,%d]) = ok %v, err %v", lo, hi, ok, err) + } + return counter.bytes, msgs + } + + // A page of the two OLDEST messages walks past the big record. It must + // not pull its body: the whole journal is bigger than the big record, + // and a walk that read every record whole would exceed it. + deepBytes, deep := read(1, 2) + if len(deep) != 2 { + t.Fatalf("deep page returned %d messages, want 2", len(deep)) + } + for _, m := range deep { + if m.ID == "msg_big" { + t.Fatal("the deep page carried the record it should only have classified") + } + } + // A backward walk must read the span between the page and the end of + // the file: that is where the line boundaries are, and there is no way + // to count records without finding them. What it must NOT do is read + // that span and then pull the big record's body a SECOND time to + // classify it. So the bound is the span, with room to spare, not twice + // it. + final, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + span := final.Size() - beforeBig.Size() + if deepBytes > span*3/2 { + t.Errorf("a page walking past the big record read %d bytes for a %d-byte span: the record's body is being read to classify it", deepBytes, span) + } + + // The page that DOES carry it reads it whole, which is what makes the + // number above meaningful rather than a scan that skipped everything. + newestBytes, newest := read(7, 7) + if len(newest) != 1 || newest[0].ID != "msg_big" { + t.Fatalf("newest page = %v, want the big record", idsOf(newest)) + } + if newestBytes < bigBytes { + t.Errorf("the page carrying the big record read only %d bytes, want at least %d", newestBytes, bigBytes) + } +} + +// TestTailPageHandlesAnUnusualFieldOrder: the prefix scan reads one key, so +// it answers only for a record whose FIRST field is the type — which is +// every record this package writes today. A record with another field +// order is not corrupt, and a page that failed on one would make a fast +// path into a format requirement. +func TestTailPageHandlesAnUnusualFieldOrder(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + // The third record puts "message" before "type". + journal := `{"type":"session","id":"` + id + `","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"first"}]},"type":"message"} +{"type":"message","message":{"id":"msg_2","role":"assistant","parts":[{"type":"text","text":"second"}]}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + page, err := ReadMessagePage(dir, id, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if !sameIDs(idsOf(page.Messages), []string{"msg_1", "msg_2"}) { + t.Errorf("page ids = %v, want both messages", idsOf(page.Messages)) + } + if page.Total != 2 { + t.Errorf("total = %d, want 2", page.Total) + } +} diff --git a/engine/queue.go b/engine/queue.go index e1c80f30..f301c74b 100644 --- a/engine/queue.go +++ b/engine/queue.go @@ -37,6 +37,115 @@ type QueuedPrompt struct { Seq int64 } +// promptQueueFold replays prompt.queued/prompt.dequeued records into the +// exact set of undelivered prompts, in the order live memory held them. It +// is the ONE implementation of that fold, shared by LoadSession (store.go) +// and the session metadata index (index.go), which needs the same depth +// without paying for a full replay. +// +// nextID and seq mirror Session.promptQueueNextID and Session.enqueueSeq: a +// caller seeds them with the session's current values and reads them back +// after the fold. +type promptQueueFold struct { + queue []QueuedPrompt + nextID int64 + seq int64 +} + +// queued folds one prompt.queued record. +// +// A record carrying Seq (durable enqueue) folds last-writer-wins against +// any already-folded entry with the SAME Seq: a failed fsync can leave a +// torn record on disk whose write reported failure, followed by its +// successful retry under a fresh ID — live memory only ever held the +// retry's entry, so replay must converge to that one too (a later +// prompt.dequeued references the retry's ID — this holds under +// EnqueuePromptDurable's caller contract that the same seq is retried +// before any higher seq is accepted). Seq also advances the enqueueSeq +// high-water mark, which is what makes duplicate detection survive a +// process restart. +// +// The fold REMOVES the old same-Seq entry from its slot and APPENDS the new +// one at the tail, rather than replacing it in place: a plain EnqueuePrompt +// can land BETWEEN the torn write and its retry (log order id1/seq5 torn, +// id2/seq0 plain, id3/seq5 retry). Live memory only ever appended id2 then +// id3, in that order — an in-place replacement at id1's old slot would +// instead fold to [id3, id2], reordering delivery relative to what actually +// happened live. Remove+append reconstructs live append order faithfully +// (the retry always carries the highest ID seen so far, so this can never +// misorder against a later, genuinely-newer plain entry); the common case +// with no interposed record degenerates to the exact same single-entry +// result as an in-place replacement. +// +// Malformed-record guards (found by FuzzLoadSessionReplay): the live path +// can never write a queued record with ID <= 0 (promptQueueNextID starts at +// 1) nor two records with the same ID (IDs are burned, never reused — see +// EnqueuePromptDurable), so either shape in a journal is corruption, not +// history. Folding them anyway would violate the queue's ID-uniqueness +// invariant (two ID-0 entries from two `{"prompt":{}}` lines) and a later +// dequeue-by-ID would remove an arbitrary one. Skip the record; same +// defensive posture as message.ResolveOrphanToolCalls at this layer. +// +// nextID advances past every ID this fold ACCEPTS, which is what keeps a +// resumed session's counter collision-free: IDs are burned on failed +// durable writes, so a counter must clear every ID that ever reached the +// log. A SKIPPED record does not advance it, and does not need to. A +// duplicate ID was already cleared by the record that folded first, and an +// ID at or below zero can never reach a counter that starts at 1. Do not +// "fix" this by hoisting the advance above the validity guard: that would +// let a malformed record's ID move the counter, which is exactly what the +// guard rejects it for. +func (f *promptQueueFold) queued(p promptRecord) { + q := QueuedPrompt{ID: p.ID, Text: p.Text, Seq: p.Seq} + valid := q.ID > 0 + for _, existing := range f.queue { + if existing.ID == q.ID { + valid = false + break + } + } + if !valid { + return + } + if q.Seq > 0 { + for i, existing := range f.queue { + if existing.Seq == q.Seq { + f.queue = append(f.queue[:i], f.queue[i+1:]...) + break + } + } + if q.Seq > f.seq { + f.seq = q.Seq + } + } + f.queue = append(f.queue, q) + if p.ID >= f.nextID { + f.nextID = p.ID + 1 + } +} + +// dequeued folds one prompt.dequeued record: it removes the matching queued +// entry by ID, not by position (see promptRecord's doc comment), so the +// folded queue ends up exactly the undelivered set however many other +// records separate a queued record from its own dequeued record. +// +// This fold reads FORWARD only: a dequeued record for an ID not folded yet +// is a no-op, and a queued record arriving after it re-appends the item. +// Every writer therefore owes this fold one ordering guarantee — a queued +// record reaches disk before its own dequeued record. The two writers that +// defer a prompt-queue write out from under the tree-wide m.mu keep it by +// parking the record on the session, not in their own closure; see +// queueRecordDeferredLocked for the resurrection defect a closure-held +// record caused. +func (f *promptQueueFold) dequeued(p promptRecord) { + for i, existing := range f.queue { + if existing.ID == p.ID { + f.queue = append(f.queue[:i], f.queue[i+1:]...) + return + } + } +} + // ErrEmptyPromptText is returned for a prompt whose text is empty or // whitespace-only. One shared sentinel, not a fresh errors.New per call // site — a review finding: SessionManager.SendToDescendant validates the diff --git a/engine/store.go b/engine/store.go index 9c229c39..714c5d14 100644 --- a/engine/store.go +++ b/engine/store.go @@ -299,6 +299,36 @@ type record struct { MCPTools []string `json:"mcp_tools,omitempty"` } +// applyGoalRecord folds one goal.* record into the durable goal state a +// resumed session restores: an active goal is one set without a later +// goal.achieved or goal.cleared, and per Claude Code semantics the run +// counters reset, so nothing else carries over. It is the ONE +// implementation of that rule, shared by LoadSession's replay and the +// session metadata index's own fold (index.go). +// +// Every other goal.* record type (goal.eval, goal.stalled, +// goal.eval_failed, goal.parked) is per-turn trace with no resume state, +// and returns the state unchanged — in particular a park never clears the +// goal, live or on replay. +func applyGoalRecord(active bool, condition string, recType string, g *goalRecord) (bool, string) { + switch recType { + case recGoalSet: + active = true + if g != nil { + condition = g.Condition + } + case recGoalUpdated: + // Only meaningful while active (see UpdateGoal): rewrites the + // restored condition in place, same as the live path. + if active && g != nil { + condition = g.Condition + } + case recGoalAchieved, recGoalCleared: + active, condition = false, "" + } + return active, condition +} + // toolResultRecord carries the durable payload of a toolresult.retained // record (see toolresult.go's writeRetainedToolResult). It is a POINTER // record: Handle names the sidecar file holding the actual bytes, and @@ -461,10 +491,10 @@ type SessionInfo struct { CreatedAt time.Time Messages int // Usage is cumulative token usage summed from every message record's - // optional Usage (see record.Usage, persistMessage), computed by the - // same cheap header-only scan that counts Messages — no full - // LoadSession/message.Message replay required (issue #62 layer 2: - // GET /session/status needs this without paying for a full session + // optional Usage (see record.Usage, persistMessage). It comes from the + // session's metadata index (index.go), like every other field here — + // no full LoadSession/message.Message replay required (issue #62 layer + // 2: GET /session/status needs this without paying for a full session // load per entry). Usage provider.Usage // LastInputTokens is the input-token count of the most recent message @@ -472,6 +502,14 @@ type SessionInfo struct { LastInputTokens int } +// addUsage accumulates one record's usage into a listing summary. +func (info *SessionInfo) addUsage(u provider.Usage) { + info.Usage.InputTokens += u.InputTokens + info.Usage.OutputTokens += u.OutputTokens + info.Usage.CacheReadTokens += u.CacheReadTokens + info.Usage.CacheWriteTokens += u.CacheWriteTokens +} + func sessionPath(dir, id string) string { return filepath.Join(dir, id+".jsonl") } @@ -969,6 +1007,14 @@ func (s *Session) ensureLog() error { s.logFile = nil return err } + // The header records bypass writeRecord (they go out in ONE Write, + // see above), so fold them here — the metadata index must see + // every record the journal holds, starting with the header that + // names the session at all. + size += int64(buf.Len()) + for _, rec := range headerRecs { + s.index.applyIndexRecordBestEffort(indexRecordOf(rec), false) + } // A file fsync (as EnqueuePromptDurable does before attesting // durability — see queue.go) commits the file's *contents* but not // its directory entry: POSIX leaves the entry itself up to the @@ -1007,18 +1053,151 @@ func (s *Session) ensureLog() error { } } s.logStarted = true + // A fold marked broken by a failed record write is RE-SEEDED here, from + // the journal as the repair above left it. Without this, one transient + // write failure disabled the index for the rest of the session object's + // life: every later read of that session refolded the whole journal, + // which is the cost the index exists to remove. A review caught it. + // + // Re-seed, never merely clear the flag. That distinction is the whole + // correctness argument, and a maintainer who "simplifies" this to + // `s.index.broken = false` reintroduces a silent wrong-index bug. A + // failed Write can land the record's bytes and not its trailing + // newline. The tail repair above then takes its case-2 branch: the tail + // parses, so it terminates the record and KEEPS it. The fold never saw + // that record. Clearing the flag would resume flushing a sidecar that + // is short by one message while claiming, through logSize, to cover the + // whole file — a stale index that reads as current. Folding the file + // again is what makes the fold agree with the bytes on disk, whichever + // branch the repair took. + // + // This runs on a reopen, which a failed write forces (see writeRecord), + // so it costs one slim fold per failure rather than per record. A fold + // that fails again leaves broken set, exactly as before. + if s.index.broken { + if data, rerr := os.ReadFile(sessionPath(s.cfg.SessionDir, s.ID)); rerr == nil { + if reseeded, ferr := foldJournalBytes(data); ferr == nil { + s.index = reseeded + } + } + } + // The sidecar index handle is opened beside the log, once, and rewritten + // in place from then on (see writeIndexTo). A failure to open it is + // never a session failure: the index is a cache, and a reader that + // cannot find one refolds the journal. + if s.indexFile == nil { + if idxf, err := os.OpenFile(sessionIndexPath(s.cfg.SessionDir, s.ID), os.O_CREATE|os.O_WRONLY, 0o644); err == nil { + s.indexFile = idxf + } else { + s.lastIndexErr = err + } + } + // size is the journal length after any tail repair above, which is + // exactly the bytes the index fold covers: a repair that TRUNCATED a + // torn tail dropped a record scanLog never folded either, and a repair + // that only terminated a complete record added one byte to a record + // the fold already holds. + s.logSize = size + // Flush now, not only after the first record: a session that is + // created and persisted but never prompted (Session.Persist, which the + // serve API calls so an evicted session can be reloaded) must still + // answer GET /session/{id} from its index. + s.flushIndexLocked() return nil } +// ReleaseFiles closes the session's log and sidecar-index handles and drops +// them. The session stays fully usable: the next persist call re-enters +// ensureLog, which reopens both, repairs a torn tail if one is there, and +// continues appending. Nothing in memory changes, so a caller can release a +// session it may still use. +// +// It exists because a Session holds two descriptors for its whole life, and +// a server keeps one Session per session it has touched. A long-lived box +// with many subagent sessions accumulates them. The server calls this when +// it evicts a session from residency (evictResidentLocked), which is the +// point it has already decided the session is idle and can be reloaded from +// disk. +// +// Errors are dropped on purpose: a close failure on a handle being +// discarded tells a caller nothing it can act on, and the next ensureLog +// reopens from the path regardless. +func (s *Session) ReleaseFiles() { + s.mu.Lock() + defer s.mu.Unlock() + if s.logFile != nil { + s.logFile.Close() + s.logFile = nil + } + if s.indexFile != nil { + s.indexFile.Close() + s.indexFile = nil + } +} + // writeRecord marshals one record and appends it as a line. Caller holds // s.mu and has called ensureLog. +// +// It is also the session's single record choke point, so it is where the +// metadata index folds (see index.go): every durable record, from every +// persist path, passes here exactly once. A failed write folds nothing and +// advances no byte counter — it marks the fold broken instead, so this +// session stops writing a sidecar that could claim to summarize a record it +// never saw. The next reader refolds the journal from byte 0. func (s *Session) writeRecord(rec record) error { b, err := json.Marshal(rec) if err != nil { return err } - _, err = s.logFile.Write(append(b, '\n')) - return err + n, err := s.logFile.Write(append(b, '\n')) + if err != nil { + // The file may have grown by a partial line. Two things follow. + // + // The fold and logSize both stay put, and the fold is marked + // broken, so this session never again writes a sidecar that could + // claim to summarize a record it did not see. Readers refold. + // + // The handle is closed, so the next persist call re-enters + // ensureLog instead of returning at its fast path. That is what + // runs the torn-tail repair over the partial line. Without it the + // next append concatenates onto that line with no separator, and + // the pair becomes a hard load error as soon as any later record + // makes it non-final — a retry of a failed EnqueuePromptDurable + // could poison the whole session log. + s.index.broken = true + s.logFile.Close() + s.logFile = nil + return err + } + s.logSize += int64(n) + s.index.applyIndexRecordBestEffort(indexRecordOf(rec), false) + s.flushIndexLocked() + return nil +} + +// flushIndexLocked writes the session's sidecar metadata index for the +// journal as it now stands (see index.go). Best effort by design: the index +// is a memoized fold, and a reader that finds it missing, torn, or stale +// refolds the journal instead. Caller holds s.mu. +func (s *Session) flushIndexLocked() { + if s.indexFile == nil || s.logFile == nil { + return + } + // The journal's modification time is half the staleness key (see + // SessionIndex.LogModTime), and it must be read AFTER the record write + // this flush follows. One fstat on a handle already open. + fi, err := s.logFile.Stat() + if err != nil { + s.lastIndexErr = err + return + } + ix, ok := s.index.snapshot(s.logSize, fi.ModTime()) + if !ok { + return + } + if err := writeIndexTo(s.indexFile, ix); err != nil { + s.lastIndexErr = err + } } // taskToolNamesPtr converts a Config.TaskToolNames value into the pointer @@ -1077,7 +1256,20 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.ID = id s.logStarted = true + // The prompt queue folds through promptQueueFold (queue.go), seeded + // from this fresh session's own counters and written back after the + // scan — the same fold the metadata index uses, so the two can never + // drift on the torn-write and ID-burn rules it holds. + qf := promptQueueFold{nextID: s.promptQueueNextID, seq: s.enqueueSeq} + err = scanLog(data, func(rec record, line int, isLast bool) error { + // Seed the session's metadata-index fold from the same records + // (index.go). A resumed session keeps writing that index through + // on every later record, so it must start from the state this + // journal already holds — a fold that began empty here would + // flush a summary claiming to cover the whole journal while + // describing one record of it. + s.index.applyIndexRecordBestEffort(indexRecordOf(rec), isLast) switch rec.Type { case recSession: s.createdAt = rec.CreatedAt @@ -1249,23 +1441,11 @@ func LoadSession(cfg Config, id string) (*Session, error) { } s.mcpSelected[name] = true } - case recGoalSet: - // An active goal is one set without a later achieved/cleared. The - // condition is restored; per Claude Code semantics the run counters - // reset, so nothing else carries over. - s.goalActive = true - if rec.Goal != nil { - s.goalCondition = rec.Goal.Condition - } - case recGoalUpdated: - // Only meaningful while active (see UpdateGoal): rewrites the - // restored condition in place, same as the live path. - if s.goalActive && rec.Goal != nil { - s.goalCondition = rec.Goal.Condition - } - case recGoalAchieved, recGoalCleared: - s.goalActive = false - s.goalCondition = "" + case recGoalSet, recGoalUpdated, recGoalAchieved, recGoalCleared: + // applyGoalRecord holds the rule (an active goal is one set + // without a later achieved/cleared; the run counters reset) — + // shared with the metadata index's own fold (index.go). + s.goalActive, s.goalCondition = applyGoalRecord(s.goalActive, s.goalCondition, rec.Type, rec.Goal) case recGoalEval, recGoalStalled, recGoalEvalFailed, recGoalParked: // Per-turn evaluation/stall/eval-failure/park trace; no resume // state (counters reset). None of these ever change goalActive @@ -1277,96 +1457,17 @@ func LoadSession(cfg Config, id string) (*Session, error) { // set s.goalActive=true and the condition) — a park never // clears the goal, live or on replay. case recPromptQueued: - // Append to the folded queue and advance the next-ID counter past - // whatever this record used (IDs are burned on failed durable - // writes — see EnqueuePromptDurable — so advancing past every ID - // seen, folded or not, is what keeps a resumed session's counter - // collision-free). - // - // A record carrying Seq (durable enqueue) folds last-writer-wins - // against any already-folded entry with the SAME Seq: a failed - // fsync can leave a torn record on disk whose write reported - // failure, followed by its successful retry under a fresh ID — - // live memory only ever held the retry's entry, so replay must - // converge to that one too (a later prompt.dequeued references - // the retry's ID — this holds under EnqueuePromptDurable's caller - // contract that the same seq is retried before any higher seq is - // accepted, see queue.go). Seq also advances the enqueueSeq - // high-water mark, which is what makes duplicate detection - // survive a process restart. - // - // The fold REMOVES the old same-Seq entry from its slot and - // APPENDS the new one at the tail, rather than replacing it - // in place: a plain EnqueuePrompt can land BETWEEN the torn - // write and its retry (log order id1/seq5 torn, id2/seq0 - // plain, id3/seq5 retry). Live memory only ever appended - // id2 then id3, in that order — an in-place replacement at - // id1's old slot would instead fold to [id3, id2], reordering - // delivery relative to what actually happened live. Remove+ - // append reconstructs live append order faithfully (the retry - // always carries the highest ID seen so far, so this can never - // misorder against a later, genuinely-newer plain entry); the - // common case with no interposed record degenerates to the - // exact same single-entry result as an in-place replacement. + // Both prompt-queue cases fold through promptQueueFold (queue.go), + // which owns the torn-write last-writer-wins rule, the malformed- + // record guards, and the ID-burn counter advance. See its own doc + // comments; the rules moved there verbatim so the metadata index + // can share them. if rec.Prompt != nil { - q := QueuedPrompt{ID: rec.Prompt.ID, Text: rec.Prompt.Text, Seq: rec.Prompt.Seq} - // Malformed-record guards (found by FuzzLoadSessionReplay): - // the live path can never write a queued record with ID <= 0 - // (promptQueueNextID starts at 1) nor two records with the - // same ID (IDs are burned, never reused — see - // EnqueuePromptDurable), so either shape in a journal is - // corruption, not history. Folding them anyway would violate - // the queue's ID-uniqueness invariant (two ID-0 entries from - // two `{"prompt":{}}` lines) and a later dequeue-by-ID would - // remove an arbitrary one. Skip the record; same defensive - // posture as message.ResolveOrphanToolCalls at this layer. - valid := q.ID > 0 - for _, p := range s.promptQueue { - if p.ID == q.ID { - valid = false - break - } - } - if valid { - if q.Seq > 0 { - for i, p := range s.promptQueue { - if p.Seq == q.Seq { - s.promptQueue = append(s.promptQueue[:i], s.promptQueue[i+1:]...) - break - } - } - if q.Seq > s.enqueueSeq { - s.enqueueSeq = q.Seq - } - } - s.promptQueue = append(s.promptQueue, q) - if rec.Prompt.ID >= s.promptQueueNextID { - s.promptQueueNextID = rec.Prompt.ID + 1 - } - } + qf.queued(*rec.Prompt) } case recPromptDequeued: - // Remove the matching queued entry (by ID, not position — see - // promptRecord's doc comment) so the folded queue ends up exactly - // the undelivered set, in ID order, however many other records - // separate a queued record from its own dequeued record. - // - // This fold reads FORWARD only: a dequeued record for an ID - // not folded yet is a no-op, and a queued record arriving - // after it re-appends the item. Every writer therefore owes - // this fold one ordering guarantee — a queued record reaches - // disk before its own dequeued record. The two writers that - // defer a prompt-queue write out from under the tree-wide - // m.mu keep it by parking the record on the session, not in - // their own closure; see queueRecordDeferredLocked (queue.go) - // for the resurrection defect a closure-held record caused. if rec.Prompt != nil { - for i, p := range s.promptQueue { - if p.ID == rec.Prompt.ID { - s.promptQueue = append(s.promptQueue[:i], s.promptQueue[i+1:]...) - break - } - } + qf.dequeued(*rec.Prompt) } case recTaskSpawned: // Folded into s.spawnedChildIDs — see recTaskSpawned's own doc @@ -1517,18 +1618,15 @@ func LoadSession(cfg Config, id string) (*Session, error) { // error. Not a regression: main hard-fails this load every time, // and a session that loads with a slightly-wrong fold beats a // session that never loads again. - lastID := rec.Compact.LastID - if _, found := indexOfMessageID(s.history, lastID); !found { - healed, herr := healCompactFoldEnd(s.history, rec.Compact.FirstID, rec.Compact.TurnsFolded) - if herr == nil { - lastID = healed - } - // A failed heal falls through unchanged: spliceCompact below - // will look for the original (unhealed) LastID, fail to find - // it exactly as before, and return its usual loud, explicit - // error — never a silent best-effort guess. - } - spliced, err := spliceCompact(s.history, rec.Compact.FirstID, lastID, rec.Compact.Summary) + // applyCompactRecord (compact.go) runs the heal and then + // spliceCompact. It is shared with the metadata index's own + // fold (index.go), so both agree on how many messages a + // compact record removes. A failed heal falls through + // unchanged: spliceCompact looks for the original (unhealed) + // LastID, fails to find it exactly as before, and returns its + // usual loud, explicit error — never a silent best-effort + // guess. + spliced, err := applyCompactRecord(s.history, rec.Compact.FirstID, rec.Compact.LastID, rec.Compact.TurnsFolded, rec.Compact.Summary) if err != nil { return fmt.Errorf("%w at line %d", err, line) } @@ -1553,6 +1651,7 @@ func LoadSession(cfg Config, id string) (*Session, error) { if err != nil { return nil, fmt.Errorf("engine: session %s: %w", id, err) } + s.promptQueue, s.promptQueueNextID, s.enqueueSeq = qf.queue, qf.nextID, qf.seq // A log from an older binary or an external writer can carry an // assistant tool_call whose turn died before a result was recorded. // Repair at ingest so every downstream consumer sees a protocol-valid @@ -1673,6 +1772,42 @@ func advanceToolResultNextIDFromHistory(s *Session) { // a corrupt or truncated final line (crash mid-write) ends iteration // silently; corruption anywhere else is an error. func scanLog[T any](data []byte, fn func(rec T, line int, isLast bool) error) error { + return scanLogRaw(data, func(raw []byte, line int, isLast bool) error { + var rec T + if err := json.Unmarshal(raw, &rec); err != nil { + if isLast { + return errTruncatedFinalRecord // crash mid-write, ignore + } + return fmt.Errorf("corrupt record at line %d: %v", line, err) + } + return fn(rec, line, isLast) + }) +} + +// errTruncatedFinalRecord ends a scan at a corrupt FINAL line, which is +// scanLog's documented tolerance for a crash mid-write. scanLogRaw absorbs +// it, so a caller sees the same clean end scanLog has always returned. +// Every OTHER error propagates. +// +// scanLogRaw compares it by IDENTITY, never with errors.Is. A callback that +// wrapped this sentinel into a genuine failure — "cannot update index: %w" +// — would otherwise have that failure read as a torn final record and +// reported as a clean scan. Identity keeps the signal to the one decoder +// that raises it. +var errTruncatedFinalRecord = errors.New("engine: truncated final record") + +// scanLogRaw is scanLog without the decode: it hands fn each non-empty line +// as raw bytes, aliasing data rather than copying it, and owns the same +// corruption discipline (a corrupt or truncated FINAL line ends iteration +// silently; corruption anywhere else is the caller's error to report). +// +// It exists for a reader that must decide, per line, HOW MUCH of it to +// decode. foldedPage (messagepage.go) folds every line through a slim shape +// and then fully decodes only the handful of records a page actually +// carries. Routed through scanLog instead, that reader would decode every +// message body in the journal — the cost the paginated read exists to +// avoid. +func scanLogRaw(data []byte, fn func(raw []byte, line int, isLast bool) error) error { lines := bytes.Split(data, []byte("\n")) last := len(lines) - 1 for last >= 0 && len(bytes.TrimSpace(lines[last])) == 0 { @@ -1683,14 +1818,10 @@ func scanLog[T any](data []byte, fn func(rec T, line int, isLast bool) error) er if len(line) == 0 { continue } - var rec T - if err := json.Unmarshal(line, &rec); err != nil { - if i == last { - return nil // truncated final line: crash mid-write, ignore + if err := fn(line, i+1, i == last); err != nil { + if err == errTruncatedFinalRecord { //nolint:errorlint // identity on purpose; see the sentinel's doc comment + return nil } - return fmt.Errorf("corrupt record at line %d: %v", i+1, err) - } - if err := fn(rec, i+1, i == last); err != nil { return err } } @@ -1698,8 +1829,25 @@ func scanLog[T any](data []byte, fn func(rec T, line int, isLast bool) error) er } // ListSessions lists persisted sessions in dir, sorted by creation time. A -// missing directory yields an empty list, not an error. Only headers and -// record types are decoded, never message bodies. +// missing directory yields an empty list, not an error. +// +// The session JOURNALS are what exist. The metadata index (index.go) is an +// acceleration over them, never the source of truth about existence: a +// session whose sidecar is missing, stale, or unusable — or whose fold +// breaks on a damaged compact record — is still a session, and a listing +// that dropped it would lie to every caller that asks "what is here". So +// each journal is answered by its index when the index can answer, and by +// a direct scan of the journal when it cannot. Only a file that is not a +// session log at all is skipped. +// +// The index path never writes. Listing a directory must not rewrite the +// sidecar of a session another writer holds; the write path repairs it. +// +// One semantic follows from the two answers. Messages counts messages +// after compaction folds on the index path, which is what a full load +// reports. The fallback scan cannot fold — a broken fold is why it ran — +// so for that session it counts message records instead, the number the +// previous header-only scan always reported. func ListSessions(dir string) ([]SessionInfo, error) { entries, err := os.ReadDir(dir) if errors.Is(err, fs.ErrNotExist) { @@ -1713,9 +1861,11 @@ func ListSessions(dir string) ([]SessionInfo, error) { if e.IsDir() || !strings.HasSuffix(e.Name(), ".jsonl") { continue } - info, err := readSessionInfo(filepath.Join(dir, e.Name())) + // The file NAME, not a validated id: a listing has always reported + // any *.jsonl carrying a session header, whatever it is called. + info, err := sessionInfoAt(dir, strings.TrimSuffix(e.Name(), ".jsonl")) if err != nil { - continue // unreadable or corrupt header: not listable + continue // unreadable, or not a session log: not listable } infos = append(infos, info) } @@ -1723,24 +1873,82 @@ func ListSessions(dir string) ([]SessionInfo, error) { return infos, nil } +// ReadSessionInfo returns one session's listing summary: its index when +// that index can answer, and a direct scan of its journal when it cannot. +// +// It is the single-session form of ListSessions, for a caller that already +// knows which sessions it wants — GET /session/status walks ids it resolved +// itself. Sharing this one path is what keeps that endpoint and a listing +// from disagreeing about which sessions exist: a session whose fold breaks +// must appear in both, or in neither. +// +// Like a listing, it never writes: a refold here answers this call and is +// dropped. The write path repairs a sidecar. +func ReadSessionInfo(dir, id string) (SessionInfo, error) { + if dir == "" { + return SessionInfo{}, errors.New("engine: ReadSessionInfo requires a session dir") + } + if !ValidSessionID(id) { + return SessionInfo{}, fmt.Errorf("%w: %q", ErrInvalidSessionID, id) + } + return sessionInfoAt(dir, id) +} + +// sessionInfoAt is ReadSessionInfo without the id validation, for +// ListSessions, whose ids are directory entries rather than caller input. +func sessionInfoAt(dir, id string) (SessionInfo, error) { + if ix, err := readSessionIndexAt(dir, id, false); err == nil { + return SessionInfo{ + ID: ix.ID, + CreatedAt: ix.CreatedAt, + Messages: ix.Messages, + Usage: ix.Usage, + LastInputTokens: ix.LastInputTokens, + }, nil + } + // No usable index. Read the journal itself rather than report nothing. + info, err := readSessionInfo(sessionPath(dir, id)) + if err != nil { + return SessionInfo{}, err + } + // The FILENAME names the session, on both paths. LoadSession pins the + // same way, and so does the index (see readSessionIndexAt), so a + // journal copied to a new name reports the new name however it was + // read. Without this, a listing could report one id from its index and + // another from its fallback for the same file. + info.ID = id + return info, nil +} + +// readSessionInfo scans one journal for the fields a listing needs. It is +// ListSessions' fallback for a journal the index cannot answer for, and it +// decodes only record heads — never message bodies — so it stays cheap on a +// large session. +// +// It does not fold compaction: a compact record's own payload is skipped +// like any other unknown field, so Messages counts message RECORDS. That is +// the number this scan has always reported, and it runs only where the fold +// that would have corrected it is the thing that failed. +// +// It DOES count a compact record's usage, which the pre-index version of +// this scan did not. A compact record carries the summarization call's own +// spend, LoadSession adds it to cumulative usage, and so does the index. A +// fallback should report the number its fast path would have reported, so +// this one follows the index rather than its own history. LastInputTokens +// still moves for message records only — the same rule LoadSession applies, +// so a reload never reports the small summarization call as the session's +// last request size. func readSessionInfo(path string) (SessionInfo, error) { data, err := os.ReadFile(path) if err != nil { return SessionInfo{}, err } - - // headRecord decodes only the fields listings need — never message - // bodies, which keeps ListSessions cheap on large sessions. Usage is a - // small flat sub-object (sibling to the message body, never nested - // inside it — see record.Usage), so decoding it here costs nothing - // like a full message.Message unmarshal would. type headRecord struct { Type string `json:"type"` ID string `json:"id"` CreatedAt time.Time `json:"created_at"` Usage *provider.Usage `json:"usage,omitempty"` } - var info SessionInfo first := true err = scanLog(data, func(rec headRecord, line int, isLast bool) error { @@ -1751,15 +1959,26 @@ func readSessionInfo(path string) (SessionInfo, error) { info.ID = rec.ID info.CreatedAt = rec.CreatedAt first = false - } else if rec.Type == recMessage { + return nil + } + // Two record types carry usage a reader counts, and only two: a + // message record and a compact record. LoadSession reads exactly + // those, so a stray usage field on any other record — a goal + // record written by a future build, say — must not inflate a + // listing that the authoritative load would not. + switch rec.Type { + case recMessage: info.Messages++ if rec.Usage != nil { - info.Usage.InputTokens += rec.Usage.InputTokens - info.Usage.OutputTokens += rec.Usage.OutputTokens - info.Usage.CacheReadTokens += rec.Usage.CacheReadTokens - info.Usage.CacheWriteTokens += rec.Usage.CacheWriteTokens + info.addUsage(*rec.Usage) info.LastInputTokens = rec.Usage.InputTokens } + case recCompact: + if rec.Usage != nil { + // Cumulative only. LastInputTokens must not move for a + // summarization call — see record.Usage's doc comment. + info.addUsage(*rec.Usage) + } } return nil }) diff --git a/engine/store_failure_test.go b/engine/store_failure_test.go index eea9b652..5868b215 100644 --- a/engine/store_failure_test.go +++ b/engine/store_failure_test.go @@ -3,7 +3,10 @@ package engine import ( "os" "path/filepath" + "strings" "testing" + + "github.com/majorcontext/harness/provider" ) // unwritableSessionDir returns a SessionDir path guaranteed to make @@ -73,3 +76,179 @@ func TestRegisterGoalSurvivesUnwritableSessionDir(t *testing.T) { t.Fatal("PersistErr() = nil after RegisterGoal against an unwritable SessionDir, want the write failure reported") } } + +// TestFailedRecordWriteDropsTheLogHandle covers the window a partial write +// opens. When Write fails after putting bytes on disk, the journal's last +// line is torn. ensureLog knows how to repair that, but it only runs when +// the session has no open handle: its fast path returns immediately while +// one exists. A session that kept its handle would append the NEXT record +// directly onto the torn line with no separator. The two lines become one +// unparseable line, and scanLog hard-fails the whole session as soon as any +// later record makes it non-final — so one failed write could poison a log +// permanently. The retry of a failed EnqueuePromptDurable is exactly that +// shape. +// +// The failure is injected at the OS level, by closing the session's own +// file descriptor: the next Write returns an error, through the production +// persist path, with no stub in the way. +func TestFailedRecordWriteDropsTheLogHandle(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + // The bytes a partial write leaves behind: a torn final line. + path := sessionPath(dir, s.ID) + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_torn`); err != nil { + t.Fatal(err) + } + f.Close() + + // Make the session's own next write fail. + s.mu.Lock() + s.logFile.Close() + s.mu.Unlock() + + if err := s.RegisterGoal("a goal whose record cannot be written"); err != nil { + t.Fatalf("RegisterGoal: %v", err) + } + if s.PersistErr() == nil { + t.Fatal("test setup: the record write did not fail") + } + s.mu.Lock() + handle := s.logFile + s.mu.Unlock() + if handle != nil { + t.Error("a failed record write kept the log handle; the next write appends onto the torn line instead of repairing it") + } + + // The next record must reopen, repair, and append cleanly. + s.ClearGoal() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "msg_torn") { + t.Error("the torn line survived; ensureLog's tail repair did not run") + } + if _, err := LoadSession(cfg, s.ID); err != nil { + t.Errorf("journal is no longer loadable after a failed write: %v", err) + } +} + +// TestIndexRecoversAfterAFailedWrite: a failed record write marks the +// session's fold broken, because the fold no longer knows what the journal +// holds. It must not stay broken for the life of the session object — every +// later read would refold the whole journal, which is the cost the index +// exists to remove. The reopen a failed write forces (see writeRecord) +// re-seeds the fold from the repaired journal. +func TestIndexRecoversAfterAFailedWrite(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) + + // Make the next write fail, at the OS level, through the production + // persist path. + s.mu.Lock() + s.logFile.Close() + s.mu.Unlock() + if err := s.RegisterGoal("a goal whose record cannot be written"); err != nil { + t.Fatalf("RegisterGoal: %v", err) + } + if s.PersistErr() == nil { + t.Fatal("test setup: the record write did not fail") + } + s.mu.Lock() + broken := s.index.broken + s.mu.Unlock() + if !broken { + t.Fatal("test setup: the failed write did not mark the fold broken") + } + + // The next turn reopens the log, and the fold must come back with it. + // PersistErr is deliberately not checked: it is sticky, so it still + // reports the failure this test injected. + runTurns(t, s, 1) + s.mu.Lock() + broken = s.index.broken + s.mu.Unlock() + if broken { + t.Error("the fold is still broken after a reopen; the session writes no index for the rest of its life") + } + + // And the sidecar it writes must be current: corrupt the journal at an + // unchanged staleness key, so only a current sidecar can answer. + corruptJournalKeepingSize(t, dir, s.ID) + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 4 { + t.Errorf("Messages = %d, want 4 (the re-seeded fold must cover the whole journal)", ix.Messages) + } +} + +// TestIndexRecoveryCountsARecordTheFoldNeverSaw is the sharp edge of the +// recovery above. A failed Write can land a record's bytes and not its +// trailing newline. ensureLog's tail repair then takes its case-2 branch: +// the tail parses, so the record is terminated and KEPT. The fold never saw +// it. +// +// Re-seeding from the file is what makes the fold agree with those bytes. +// Merely clearing the broken flag would resume flushing a sidecar short by +// one message, while logSize claimed the whole file — a stale index that +// reads as current, which no reader would ever refold. +func TestIndexRecoveryCountsARecordTheFoldNeverSaw(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 20}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + runTurns(t, s, 1) // two durable messages + + // The exact shape of a write that landed its bytes and not its + // newline, with the session's fold left broken by that failure. + path := sessionPath(dir, s.ID) + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_landed","role":"user","parts":[{"type":"text","text":"hi"}]}}`); err != nil { + t.Fatal(err) + } + f.Close() + s.mu.Lock() + s.index.broken = true + s.logFile.Close() + s.logFile = nil + s.mu.Unlock() + + // The next turn reopens, repairs (case 2: the record is kept), and + // re-seeds the fold. + runTurns(t, s, 1) // two more durable messages + + // Only a CURRENT sidecar can answer once the journal is unreadable at + // an unchanged staleness key. + corruptJournalKeepingSize(t, dir, s.ID) + ix, err := ReadSessionIndex(dir, s.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 5 { + t.Errorf("Messages = %d, want 5: two turns of two, plus the record the repair kept", ix.Messages) + } +} diff --git a/engine/store_test.go b/engine/store_test.go index 578547e5..541a1411 100644 --- a/engine/store_test.go +++ b/engine/store_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "fmt" "io/fs" "os" "path/filepath" @@ -788,3 +789,31 @@ func TestLoadSessionRepairsOrphanedToolCalls(t *testing.T) { t.Errorf("synthetic result text = %q, want %q", got, message.SyntheticOrphanResultText) } } + +// TestScanLogRawAbsorbsOnlyItsOwnSentinel: scanLogRaw ends a scan cleanly +// at errTruncatedFinalRecord, which its decoder raises for a corrupt final +// line. It must not do that for a callback's own failure that merely wraps +// the sentinel — "cannot update index: %w" is a real error, and reporting +// it as a clean scan would drop it silently. +func TestScanLogRawAbsorbsOnlyItsOwnSentinel(t *testing.T) { + data := []byte("{\"type\":\"session\"}\n{\"type\":\"model\"}\n") + wrapped := fmt.Errorf("cannot update index: %w", errTruncatedFinalRecord) + unrelated := errors.New("disk on fire") + cases := map[string]struct { + give error + want error // nil means the scan must end cleanly + }{ + "the sentinel itself ends the scan": {give: errTruncatedFinalRecord, want: nil}, + "a wrapped sentinel propagates": {give: wrapped, want: wrapped}, + "an unrelated error propagates": {give: unrelated, want: unrelated}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + give := tc.give + got := scanLogRaw(data, func([]byte, int, bool) error { return give }) + if got != tc.want { //nolint:errorlint // identity: the callback returns these exact values + t.Errorf("scanLogRaw = %v, want %v", got, tc.want) + } + }) + } +} diff --git a/server/cold_read_test.go b/server/cold_read_test.go new file mode 100644 index 00000000..cd1811c1 --- /dev/null +++ b/server/cold_read_test.go @@ -0,0 +1,602 @@ +package server + +import ( + "context" + "encoding/json" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" + "github.com/majorcontext/harness/provider" +) + +// coldSession writes a real session log into dir through the engine's own +// production path (NewSession + Prompt), without the server ever seeing it. +// The result is exactly the state GET /session/{id} answers cold: a journal +// on disk, no residency entry, no SessionManager node. +func coldSession(t *testing.T, dir string, cfgMutate func(*engine.Config)) *engine.Session { + t.Helper() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("cold reply")}} + cfg := engine.Config{ + Providers: provider.Registry{prov.name: prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + SessionDir: dir, + WorkDir: dir, + } + if cfgMutate != nil { + cfgMutate(&cfg) + } + sess := engine.NewSession(cfg) + if _, err := sess.Prompt(context.Background(), "hello"); err != nil { + t.Fatalf("Prompt: %v", err) + } + if err := sess.PersistErr(); err != nil { + t.Fatalf("PersistErr: %v", err) + } + return sess +} + +// breakJournal overwrites a session journal with unreadable bytes of the +// same length and modification time — the index's whole staleness key, +// left untouched. A handler that still replays the journal therefore fails +// visibly, which is what makes this a proof and not a hope. Production +// never produces this state: a journal is append-only with one writer. +func breakJournal(t *testing.T, dir, id string) { + t.Helper() + path := filepath.Join(dir, id+".jsonl") + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + defer func() { + // Restore the modification time: it is the other half of the + // index's staleness key, and this probe must change neither half. + if err := os.Chtimes(path, fi.ModTime(), fi.ModTime()); err != nil { + t.Fatal(err) + } + }() + junk := make([]byte, len(data)) + for i := range junk { + junk[i] = 'x' + } + if err := os.WriteFile(path, junk, 0o644); err != nil { + t.Fatal(err) + } +} + +func decodeSession(t *testing.T, data []byte) sessionJSON { + t.Helper() + var out sessionJSON + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("decode session: %v (%s)", err, data) + } + return out +} + +// TestGetSessionColdAnswersFromIndex is the workstream's headline claim: +// GET /session/{id} for a session this process does not hold live never +// replays the journal. +func TestGetSessionColdAnswersFromIndex(t *testing.T) { + dir := t.TempDir() + sess := coldSession(t, dir, nil) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + breakJournal(t, dir, sess.ID) + + resp, data := h.do("GET", "/session/"+sess.ID, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", sess.ID, resp.StatusCode, data) + } + got := decodeSession(t, data) + if got.ID != sess.ID { + t.Errorf("id = %q, want %q", got.ID, sess.ID) + } + if got.Messages != 2 { + t.Errorf("messages = %d, want 2 (user + assistant)", got.Messages) + } + if got.Model != sess.Model() { + t.Errorf("model = %v, want %v", got.Model, sess.Model()) + } + if got.WorkDir != dir { + t.Errorf("workdir = %q, want %q", got.WorkDir, dir) + } + if got.Status != "idle" || got.State != "idle" { + t.Errorf("status/state = %q/%q, want idle/idle", got.Status, got.State) + } + if got.LastActivityAt.IsZero() { + t.Error("last_activity_at is zero, want the newest message's timestamp") + } + if got.Plugins == nil { + t.Error("plugins = null, want an array") + } +} + +// TestListSessionsColdAnswersFromIndex is the same claim for the list +// endpoint, which used to pay one full replay per non-resident session. +func TestListSessionsColdAnswersFromIndex(t *testing.T) { + dir := t.TempDir() + first := coldSession(t, dir, nil) + second := coldSession(t, dir, nil) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + breakJournal(t, dir, first.ID) + breakJournal(t, dir, second.ID) + + resp, data := h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatalf("decode list: %v (%s)", err, data) + } + if len(list) != 2 { + t.Fatalf("listed %d sessions, want 2: %s", len(list), data) + } + for _, entry := range list { + if entry.Messages != 2 { + t.Errorf("session %s: messages = %d, want 2", entry.ID, entry.Messages) + } + } +} + +// TestGetSessionColdReportsLineage: a task child's lineage is durable, and +// the cold read must still report it — the same durable-only block a +// disk-loaded session reported before, now sourced from the index. +func TestGetSessionColdReportsLineage(t *testing.T) { + dir := t.TempDir() + child := coldSession(t, dir, func(cfg *engine.Config) { + cfg.TaskParentID = "ses_0123456789abcdef" + cfg.TaskAgentType = "explore" + cfg.TaskDepth = 2 + }) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + breakJournal(t, dir, child.ID) + + resp, data := h.do("GET", "/session/"+child.ID, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", child.ID, resp.StatusCode, data) + } + got := decodeSession(t, data) + if got.Lineage == nil { + t.Fatalf("lineage absent for a task child: %s", data) + } + if got.Lineage.ParentID != "ses_0123456789abcdef" { + t.Errorf("lineage.parent_id = %q, want ses_0123456789abcdef", got.Lineage.ParentID) + } + if got.Lineage.AgentType != "explore" { + t.Errorf("lineage.agent_type = %q, want explore", got.Lineage.AgentType) + } + if got.Lineage.Depth != 2 { + t.Errorf("lineage.depth = %d, want 2", got.Lineage.Depth) + } +} + +// TestGetSessionColdReportsConfiguredPlugins: plugins are process +// configuration, not durable session state, so the cold path reads them +// from Options.Plugins. Without that seam a cold read would silently +// report no plugins for a process that has them. +func TestGetSessionColdReportsConfiguredPlugins(t *testing.T) { + dir := t.TempDir() + sess := coldSession(t, dir, nil) + want := []plugin.Info{{Name: "guard", Tools: []string{"scan"}}} + srv := newServer(t, dir, &scriptedProvider{name: "test"}, 0, func(o *Options) { + o.Plugins = func(string) []plugin.Info { return want } + }) + ts := httptest.NewServer(srv) + t.Cleanup(ts.Close) + h := &harness{t: t, dir: dir, token: "secret-run-token", srv: srv, ts: ts} + breakJournal(t, dir, sess.ID) + + resp, data := h.do("GET", "/session/"+sess.ID, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", sess.ID, resp.StatusCode, data) + } + got := decodeSession(t, data) + if len(got.Plugins) != 1 || got.Plugins[0].Name != "guard" { + t.Errorf("plugins = %+v, want the one configured plugin", got.Plugins) + } +} + +// TestGetSessionUnknownIDIsNotFound: an id with no journal must still 404, +// not report an empty summary. +func TestGetSessionUnknownIDIsNotFound(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, _ := h.do("GET", "/session/ses_0123456789abcdef", nil) + if resp.StatusCode != 404 { + t.Fatalf("GET unknown session = %d, want 404", resp.StatusCode) + } +} + +// TestGetSessionPrefersLiveObjectOverIndex: a session this process is +// actively running must be rendered from its live object. The index is a +// summary of the JOURNAL, which cannot know a turn is in flight, so a cold +// read of a running session would report "idle" — the exact false-idle +// answer an orchestrator acts on. +func TestGetSessionPrefersLiveObjectOverIndex(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{"parts": []map[string]string{{"type": "text", "text": "go"}}}) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + <-prov.started + defer prov.releaseAll() + + resp, data = h.do("GET", "/session/"+id, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", id, resp.StatusCode, data) + } + got := decodeSession(t, data) + if got.Status != "busy" { + t.Errorf("status = %q, want busy (a live session must not be read from its index)", got.Status) + } +} + +// TestGetSessionColdFallsBackForLegacyJournal: a journal that never +// recorded a model cannot be answered from a fold — engine.LoadSession +// answers that from the loading Config, and the index says so through +// SessionIndex.Complete. The handler must then use the load path and report +// the same model it always did, not an empty one. OpenAPI makes `model` +// required, so an empty value is not a smaller answer, it is an invalid +// one. +func TestGetSessionColdFallsBackForLegacyJournal(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + // A crash between the header write and the model record beside it: the + // header is complete, the model record is torn away. + journal := `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/tmp"} +{"type":"model","mod` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session/"+id, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", id, resp.StatusCode, data) + } + got := decodeSession(t, data) + if got.Model.IsZero() { + t.Errorf("model = %v, want the configured default the load path restores", got.Model) + } + + // The same session must still appear in a listing, with the same model. + resp, data = h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatal(err) + } + if len(list) != 1 || list[0].Model.IsZero() { + t.Fatalf("listing = %s, want one entry carrying a model", data) + } +} + +// TestColdPluginsAreAskedPerSession pins the seam's shape: an embedder may +// wire different hooks per session through its own NewSession/LoadSession +// wrappers, so the cold read asks for THAT session's plugins, not the +// process's. +func TestColdPluginsAreAskedPerSession(t *testing.T) { + dir := t.TempDir() + first := coldSession(t, dir, nil) + second := coldSession(t, dir, nil) + srv := newServer(t, dir, &scriptedProvider{name: "test"}, 0, func(o *Options) { + o.Plugins = func(sessionID string) []plugin.Info { + return []plugin.Info{{Name: "for-" + sessionID}} + } + }) + ts := httptest.NewServer(srv) + t.Cleanup(ts.Close) + h := &harness{t: t, dir: dir, token: "secret-run-token", srv: srv, ts: ts} + + for _, id := range []string{first.ID, second.ID} { + resp, data := h.do("GET", "/session/"+id, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", id, resp.StatusCode, data) + } + got := decodeSession(t, data) + if len(got.Plugins) != 1 || got.Plugins[0].Name != "for-"+id { + t.Errorf("session %s: plugins = %+v, want the per-session answer", id, got.Plugins) + } + } +} + +// TestEvictionReleasesSessionFileHandles: a Session holds two descriptors +// for its whole life — its journal and its sidecar index — and a server +// keeps one Session per session it has touched. A long-lived box with many +// subagent sessions accumulates them. Eviction is the point the server has +// already decided a session is idle and reloadable, so it is the point that +// releases them. +// +// The session must stay usable: the next persist reopens both handles, and +// the index it then writes must still describe the whole journal. +func TestEvictionReleasesSessionFileHandles(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("one"), asstTurn("two"), asstTurn("three")}} + h := newHarnessOpts(t, dir, prov, 1) // MaxResident 1: the next create evicts + first := h.createSession("") + resp, data := h.do("POST", "/session/"+first+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(first) + + sess := h.srv.residentSession(first) + if sess == nil { + t.Fatal("test setup: the first session is not resident") + } + openBefore := openDescriptors(t) + + // A second session evicts the first. + h.createSession("") + if h.srv.residentSession(first) != nil { + t.Fatal("test setup: the first session was not evicted") + } + if openDescriptors(t) > openBefore { + t.Errorf("descriptor count rose from %d to %d across an eviction", openBefore, openDescriptors(t)) + } + + // The evicted session object stays usable: a further append reopens its + // handles, and the index still describes the whole journal. + if _, err := sess.Prompt(context.Background(), "again"); err != nil { + t.Fatalf("Prompt on an evicted session: %v", err) + } + if err := sess.PersistErr(); err != nil { + t.Fatalf("PersistErr after reopen: %v", err) + } + ix, err := engine.ReadSessionIndex(dir, first) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.Messages != 4 { + t.Errorf("index reports %d messages, want 4 after the reopened append", ix.Messages) + } +} + +// openDescriptors counts this process's open file descriptors. It is a +// Linux-only reading of /proc/self/fd; on any other system the test that +// uses it still checks the reopen behavior, and skips the count. +func openDescriptors(t *testing.T) int { + t.Helper() + entries, err := os.ReadDir("/proc/self/fd") + if err != nil { + t.Skip("no /proc/self/fd; cannot count descriptors on this system") + } + return len(entries) +} + +// TestListDoesNotReadIndexesForLiveSessions: GET /session renders a live +// session from its live object, so reading that session's index is work +// thrown away — and a stale sidecar would be refolded and written back by +// the listing while the session's own writer holds it. +// +// The probe removes a live session's sidecar and leaves its journal intact. +// A listing that reads indexes for every file would refold this one and +// write the sidecar back. A listing that resolves residency first never +// touches it, so the sidecar stays absent. +func TestListDoesNotReadIndexesForLiveSessions(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("one")}} + h := newHarnessDir(t, dir, prov) + id := h.createSession("") + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + if err := os.Remove(filepath.Join(dir, id+".index.json")); err != nil { + t.Fatal(err) + } + + resp, data = h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatal(err) + } + if len(list) != 1 || list[0].ID != id { + t.Fatalf("listing = %s, want the one live session %s", data, id) + } + if list[0].Messages != 2 { + t.Errorf("messages = %d, want 2 from the live object", list[0].Messages) + } + // The listing must not have written a sidecar for it either. + if _, err := os.Stat(filepath.Join(dir, id+".index.json")); err == nil { + t.Error("the listing refolded and wrote a sidecar for a live session") + } +} + +// TestListAndGetAgreeOnLiveness: GET /session and GET /session/{id} must +// not disagree about whether a session is running. +// +// Both cold paths now render through coldSessionJSON, which re-checks +// residency after reading the index. That window — a session going live +// between the residency check and the index read — is not reachable +// deterministically from a test, so the guarantee is structural: one +// shared path, rather than two that must be kept in step. This test pins +// the observable half, that the two endpoints agree. +func TestListAndGetAgreeOnLiveness(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("") + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + <-prov.started + defer prov.releaseAll() + + resp, data = h.do("GET", "/session/"+id, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", id, resp.StatusCode, data) + } + fromGet := decodeSession(t, data) + + resp, data = h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatal(err) + } + if len(list) != 1 { + t.Fatalf("listed %d sessions, want 1", len(list)) + } + if list[0].Status != fromGet.Status || list[0].State != fromGet.State { + t.Errorf("listing says %q/%q, GET says %q/%q", list[0].Status, list[0].State, fromGet.Status, fromGet.State) + } + if fromGet.Status != "busy" { + t.Errorf("status = %q, want busy", fromGet.Status) + } +} + +// TestSessionExistenceCheckIsOneStat: abort, end, and wait ask only whether +// a session's journal is there. That check must not walk the directory, +// fold every journal in it, or write sidecars back — and it must answer YES +// for a journal that exists but cannot be read, because a damaged session +// is still a session that exists. +func TestSessionExistenceCheckIsOneStat(t *testing.T) { + dir := t.TempDir() + sess := coldSession(t, dir, nil) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + // Corrupt the journal so nothing can fold it, and drop its sidecar. + if err := os.Remove(filepath.Join(dir, sess.ID+".index.json")); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, sess.ID+".jsonl"), []byte("{\"type\":\"session\"}\n{not json\n{}\n"), 0o644); err != nil { + t.Fatal(err) + } + + // POST /abort answers from the existence check alone. + resp, data := h.do("POST", "/session/"+sess.ID+"/abort", nil) + if resp.StatusCode != 204 { + t.Errorf("POST abort on an existing but unreadable session = %d: %s; want 204", resp.StatusCode, data) + } + // And no sidecar was written for it by that check. + if _, err := os.Stat(filepath.Join(dir, sess.ID+".index.json")); err == nil { + t.Error("the existence check refolded and wrote a sidecar") + } +} + +// TestStatusAndListAgreeOnWhichSessionsExist: GET /session/status and GET +// /session must not disagree about which sessions are there. Both resolve +// ids first and then take the same index-then-scan path, so a session whose +// fold breaks — no usable index, a readable journal — appears in both. +func TestStatusAndListAgreeOnWhichSessionsExist(t *testing.T) { + dir := t.TempDir() + healthy := coldSession(t, dir, nil) + const broken = "ses_fedcba9876543210" + // A compact record naming an absent range: the fold fails, the journal + // reads fine. + journal := `{"type":"session","id":"` + broken + `","created_at":"2026-01-02T03:04:06Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":9}} +{"type":"compact","compact":{"first_id":"absent","last_id":"absent","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +` + if err := os.WriteFile(filepath.Join(dir, broken+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session/status", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/status = %d: %s", resp.StatusCode, data) + } + var status map[string]struct { + Usage usageJSON `json:"usage"` + } + if err := json.Unmarshal(data, &status); err != nil { + t.Fatal(err) + } + for _, id := range []string{healthy.ID, broken} { + if _, ok := status[id]; !ok { + t.Errorf("status is missing session %s: %s", id, data) + } + } + if got := status[broken].Usage.InputTokens; got != 9 { + t.Errorf("status usage for the fold-broken session = %d, want 9 from its journal", got) + } +} + +// TestListOmitsWhatItCannotRenderWhileStatusReportsIt pins a deliberate +// asymmetry, so a later reader finds it stated rather than discovers it. +// +// A journal whose fold breaks and whose load fails cannot be rendered as a +// listing entry: that entry names a model, a workdir, and lineage, and none +// of those survive a load that fails. GET /session/{id} 404s for the same +// session, so omitting it keeps the listing and the single-session read +// consistent. GET /session/status promises only counts, which a direct +// journal scan still supplies, so it reports the session. +// +// This is main's behavior, not something the metadata index introduced — +// verified directly against main, where the same journal is absent from +// GET /session and present in GET /session/status. +func TestListOmitsWhatItCannotRenderWhileStatusReportsIt(t *testing.T) { + dir := t.TempDir() + healthy := coldSession(t, dir, nil) + const broken = "ses_fedcba9876543210" + journal := `{"type":"session","id":"` + broken + `","created_at":"2026-01-02T03:04:06Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]},"usage":{"input_tokens":9}} +{"type":"compact","compact":{"first_id":"absent","last_id":"absent","turns_folded":1,"summary":{"id":"cmpsum_x","role":"user"}}} +` + if err := os.WriteFile(filepath.Join(dir, broken+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatal(err) + } + if len(list) != 1 || list[0].ID != healthy.ID { + t.Fatalf("listing = %s, want only the renderable session %s", data, healthy.ID) + } + + // The single-session read agrees with the listing. + resp, _ = h.do("GET", "/session/"+broken, nil) + if resp.StatusCode != 404 { + t.Errorf("GET /session/%s = %d, want 404: the listing omits what this cannot render", broken, resp.StatusCode) + } + + // Status reports it, from the journal scan. + resp, data = h.do("GET", "/session/status", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/status = %d: %s", resp.StatusCode, data) + } + var status map[string]struct { + Usage usageJSON `json:"usage"` + } + if err := json.Unmarshal(data, &status); err != nil { + t.Fatal(err) + } + if got, ok := status[broken]; !ok || got.Usage.InputTokens != 9 { + t.Errorf("status for the fold-broken session = %+v (present=%v), want its journal's 9 input tokens", got, ok) + } +} diff --git a/server/handlers.go b/server/handlers.go index 56eebe50..f13d31b2 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -6,13 +6,16 @@ import ( "errors" "fmt" "io" + "io/fs" "net/http" + "net/url" "os" "path/filepath" "regexp" "runtime/debug" "runtime/pprof" "sort" + "strconv" "strings" "time" @@ -250,10 +253,10 @@ func usageJSONForSession(sess *engine.Session) usageJSON { return out } -// usageJSONForInfo builds usageJSON from a cheap engine.SessionInfo (no full -// session load) — used by handleStatus's non-resident branch, where paying -// for a full LoadSession per listed session would defeat the point of a -// lightweight status endpoint. +// usageJSONForInfo builds usageJSON from a cheap engine.SessionInfo — the +// non-resident branch of GET /session/status, where paying for a full +// LoadSession per listed session would defeat the point of a lightweight +// status endpoint. func usageJSONForInfo(info engine.SessionInfo) usageJSON { return usageJSON{ InputTokens: info.Usage.InputTokens, @@ -838,8 +841,9 @@ func (s *Server) handleCreate(w http.ResponseWriter, r *http.Request) { s.timedCreatePhase(sess.ID, "register", func() error { //nolint:errcheck // never errors; see timedCreatePhase's uniform shape s.mu.Lock() s.sessions[sess.ID] = &sessionState{sess: sess, lastUsed: time.Now(), shareWorkdir: body.ShareWorkdir, isolation: isolation, worktree: wt} - s.evictResidentLocked() + evicted := s.evictResidentLocked() s.mu.Unlock() + releaseEvicted(evicted) return nil }) @@ -981,43 +985,124 @@ func (s *Server) handleList(w http.ResponseWriter, _ *http.Request) { out = append(out, s.buildSession(liveFromResident(m.sess.ID, m.sess, m.running).withManager(s.sessMgr))) seen[m.sess.ID] = true } - infos, err := engine.ListSessions(s.opts.SessionDir) + // Ids first, indexes second. A session this loop renders from a live + // object needs no index at all, and reading one for it is work thrown + // away — and a stale sidecar would be refolded and written back here + // while that session's own writer holds it. + ids, err := engine.ListSessionIDs(s.opts.SessionDir) if err != nil { writeErr(w, http.StatusInternalServerError, "cannot list sessions") return } - for _, info := range infos { - if seen[info.ID] { + for _, id := range ids { + if seen[id] { continue } - // Server.lookup, not a bare LoadSession: an id on disk can still - // be live in this process — a Spawn-driven child is never a - // residency key, so it lands in this branch, and its status and - // lineage must come from SessionManager's own node. lookup reads - // the log only when nothing live holds the id (a live review - // finding: the old unconditional load re-read and then discarded - // such a child's whole log on every listing). - lv, ok := s.lookup(info.ID) - if !ok { + // resolveLive first, index second. An id on disk can still be live + // in this process — a Spawn-driven child is never a residency key, + // so it lands in this branch, and its status and lineage must come + // from SessionManager's own node. Only an id nothing live holds is + // rendered from its index, which is the case this listing used to + // pay a full LoadSession for, per session, on every call. + lv := s.resolveLive(id) + if lv.session() != nil { + out = append(out, s.buildSession(lv)) continue } - out = append(out, s.buildSession(lv)) + ix, ixErr := engine.ReadSessionIndex(s.opts.SessionDir, id) + // A session neither the index nor a load can render is omitted + // here, while GET /session/status still reports its usage from a + // direct journal scan. That asymmetry predates this index — see + // TestListOmitsWhatItCannotRenderWhileStatusReportsIt — and it is + // what the two endpoints promise: a listing entry names a session's + // model, workdir, and lineage, which a journal that will not load + // cannot supply, and GET /session/{id} 404s for the same session. + // Status promises only counts, which a scan can still give. + // + // coldSessionJSON for BOTH cases, index-backed and load-backed. It + // renders from the index when that index can answer, falls back to + // the authoritative load path when it cannot (see + // SessionIndex.Complete), and re-checks residency at the end. The + // re-check is why this listing does not build from the index + // directly: claimForPrompt can make a session live between the + // resolveLive above and this call, and GET /session/{id} closes + // that window the same way. The two must not disagree about a + // session's liveness. A session neither path can render is skipped, + // exactly as this listing always has. + if body, ok := s.coldSessionJSON(id, ix, ixErr == nil && ix.Complete); ok { + out = append(out, body) + } } sort.Slice(out, func(i, j int) bool { return out[i].CreatedAt.Before(out[j].CreatedAt) }) writeJSON(w, http.StatusOK, out) } +// handleGet answers GET /session/{id}. +// +// A session any live source in this process holds — this server's own +// residency map or SessionManager's tree — is rendered from that live +// object, unchanged. Anything else is rendered from its metadata index +// (engine.ReadSessionIndex): one small sidecar read, no journal replay. +// That cold path used to call engine.LoadSession, decode every message +// body, rebuild the whole history, and then throw all of it away to report +// a dozen scalar fields — about 7 s per read on the fleet's longest session +// (see docs/design/console-read-path.md in meetneptune/boxes). func (s *Server) handleGet(w http.ResponseWriter, r *http.Request) { id, ok := s.sessionIDOrNotFound(w, r) if !ok { return } - lv, ok := s.lookup(id) - if !ok { - writeErr(w, http.StatusNotFound, "no such session") + lv := s.resolveLive(id) + if lv.session() != nil { + writeJSON(w, http.StatusOK, s.buildSession(lv)) + return + } + ix, err := engine.ReadSessionIndex(s.opts.SessionDir, id) + if body, ok := s.coldSessionJSON(id, ix, err == nil && ix.Complete); ok { + writeJSON(w, http.StatusOK, body) return } - writeJSON(w, http.StatusOK, s.buildSession(lv)) + writeErr(w, http.StatusNotFound, "no such session") +} + +// coldSessionJSON renders a session no live source holds, from the index +// the caller already has, and falls back to a full load when that index +// cannot answer. +// +// The caller passes the index it holds rather than an id to re-read: GET +// /session has one per session already, and re-reading would cost a second +// stat and sidecar read per session in a listing. usable is false when the +// caller has no index at all, or holds one that is not Complete. +// +// The fallback is not a safety net, it is a correctness rule. A journal +// does not always record every field a reader needs: a legacy header +// carries no workdir, and a crash can tear away the initial model record a +// fresh log writes beside its header. engine.LoadSession answers those from +// the loading Config; a fold has no Config, and reports +// SessionIndex.Complete false rather than an empty model. Then the +// authoritative path answers, exactly as it did before this index existed. +// +// It also re-checks residency at the end. resolveLive ran before the index +// read, so a concurrent claimForPrompt can make the session live in that +// gap, and reporting "idle" for a session this process is now running is +// the false-idle answer an orchestrator acts on. The re-check does not +// close the window — nothing can, without holding a lock across a disk read +// — but it narrows it to the width of one map lookup. +func (s *Server) coldSessionJSON(id string, ix engine.SessionIndex, usable bool) (sessionJSON, bool) { + var body sessionJSON + if usable { + body = s.buildSessionFromIndex(ix) + } else { + sess, err := s.opts.LoadSession(id) + if err != nil { + return sessionJSON{}, false + } + body = s.buildSession(liveSession{id: id}.withLoaded(sess)) + } + if lv := s.resolveLive(id); lv.session() != nil { + body = s.buildSession(lv) + } + return body, true } // messagePlaceholder substitutes for a resident message that fails to @@ -1050,12 +1135,25 @@ func (s *Server) handleMessages(w http.ResponseWriter, r *http.Request) { if !ok { return } + query := r.URL.Query() + if query.Has("before_seq") || query.Has("limit") { + s.handleMessagePage(w, query, id) + return + } sess, ok := s.lookupSession(id) if !ok { writeErr(w, http.StatusNotFound, "no such session") return } msgs := sess.History() + writeJSON(w, http.StatusOK, marshalMessages(msgs)) +} + +// marshalMessages renders messages for the wire, one at a time, replacing +// any that fails to marshal with a messagePlaceholder — see handleMessages' +// own doc comment for the production incident that rule exists for. It +// always returns a non-nil slice, so an empty history serializes as []. +func marshalMessages(msgs []message.Message) []json.RawMessage { out := make([]json.RawMessage, 0, len(msgs)) for i := range msgs { m := &msgs[i] @@ -1076,7 +1174,177 @@ func (s *Server) handleMessages(w http.ResponseWriter, r *http.Request) { } out = append(out, raw) } - writeJSON(w, http.StatusOK, out) + return out +} + +// messagePageJSON is the openapi MessagePage shape: one bounded page of a +// session's durable message sequence, plus where that page sits. +// +// The envelope is deliberately a DIFFERENT shape from the unparameterized +// response's bare array. A client that pages needs the page's position, and +// a client that does not page must keep working byte for byte — so the +// array response stays exactly as it was, and only a request that names +// before_seq or limit gets this. +type messagePageJSON struct { + Messages []json.RawMessage `json:"messages"` + // FirstSeq and LastSeq bound the page: a client fetches the next older + // page with before_seq=first_seq. Both are 0 for an empty page. + FirstSeq int `json:"first_seq"` + LastSeq int `json:"last_seq"` + // Total is the session's whole durable message count (see + // engine.SessionIndex.DurableMessages), so a client can size a + // scrollbar without a second call. It can be lower than the `messages` + // field of GET /session, which also counts the repair messages a + // replay derives; those have no record, so they have no seq. + Total int `json:"total"` + // HasMore reports whether older messages exist before FirstSeq. + HasMore bool `json:"has_more"` +} + +// handleMessagePage answers GET /session/{id}/message?before_seq=N&limit=K: +// the K durable messages immediately before seq N, newest page by default. +// +// It reads the journal's TAIL through engine.ReadMessagePage, never the +// whole log, and it does so whether or not the session is resident. Reading +// the durable records even for a live session is what keeps one seq +// definition for both cases: a resident history can carry messages the log +// does not (message.ResolveOrphanToolCalls repairs applied at load, +// recovery's memory-only closers), and numbering those would give the same +// message two different seqs depending on residency. +// +// A session with no journal at all — created and never persisted — has no +// durable sequence to page. It falls back to the resident history, which +// for such a session is exactly the durable sequence it would have had. +func (s *Server) handleMessagePage(w http.ResponseWriter, query url.Values, id string) { + beforeSeq, ok := intParam(w, query, "before_seq") + if !ok { + return + } + limit, ok := intParam(w, query, "limit") + if !ok { + return + } + // Reject, do not clamp. The published schema names a maximum, and a + // generated client or a gateway enforces it, so silently answering a + // larger request with a smaller page would make the server disagree + // with its own spec. engine.MessagePageWindow still clamps for a + // direct engine caller, which has no schema to honor. + if limit > engine.MaxMessagePageLimit { + writeErr(w, http.StatusBadRequest, fmt.Sprintf("limit must be at most %d", engine.MaxMessagePageLimit)) + return + } + page, err := engine.ReadMessagePage(s.opts.SessionDir, id, beforeSeq, limit) + if err != nil { + s.messagePageFallback(w, id, beforeSeq, limit, err) + return + } + writeJSON(w, http.StatusOK, messagePageJSON{ + Messages: marshalMessages(page.Messages), + FirstSeq: page.FirstSeq, + LastSeq: page.LastSeq, + Total: page.Total, + HasMore: page.HasMore, + }) +} + +// messagePageFallback answers a page request that engine.ReadMessagePage +// could not, and it classifies the reason rather than giving one answer for +// all of them. +// +// It pages resident history for exactly ONE case: a live session with no +// durable journal at all. Such a session has no records, so its resident +// history IS its durable sequence, and numbering it invents nothing. Every +// other case keeps the durable contract instead of bending it. A resident +// history can carry messages the log does not — message.ResolveOrphanToolCalls +// repairs applied at load, recovery's memory-only closers — so paging it for +// a session whose journal merely could not be READ would hand those messages +// sequence numbers. A client that then paged again, after the journal +// became readable, would see its pages renumbered. +// +// So: a missing journal with no live session is a 404, the same answer the +// unparameterized read gives. A journal that exists but cannot be read, or +// keeps changing under the read, is a 500 — for a live session too. A +// session whose journal exists is not "no such session", and reporting it +// as one sends an operator looking for an id that is on disk in front of +// them. +func (s *Server) messagePageFallback(w http.ResponseWriter, id string, beforeSeq, limit int, cause error) { + // A session dir the process never configured has no durable sequence + // for ANY session, so resident history is the only answer there is. + noJournal := errors.Is(cause, fs.ErrNotExist) || s.opts.SessionDir == "" + if !noJournal { + writeErr(w, http.StatusInternalServerError, "cannot read session messages") + return + } + sess := s.liveSessionObject(id) + if sess == nil { + writeErr(w, http.StatusNotFound, "no such session") + return + } + // Number the same sequence the journal path numbers: durable messages + // only. A session with no journal has no repair applied to its history + // today — the repair runs at load, and this session was never loaded — + // but filtering makes the two paths agree by CONSTRUCTION rather than + // by an argument about which shapes can reach here. + msgs := durableOnly(sess.History()) + total := len(msgs) + // The same window arithmetic the journal path uses, from the same + // helper: two copies would give one session two different paginations + // depending on which path answered it. + lo, hi, _ := engine.MessagePageWindow(total, beforeSeq, limit) + if hi < lo { + writeJSON(w, http.StatusOK, messagePageJSON{Messages: []json.RawMessage{}, Total: total}) + return + } + writeJSON(w, http.StatusOK, messagePageJSON{ + Messages: marshalMessages(msgs[lo-1 : hi]), + FirstSeq: lo, + LastSeq: hi, + Total: total, + HasMore: lo > 1, + }) +} + +// durableOnly drops the messages a load-time repair derives +// (message.ResolveOrphanToolCalls) from a resident history. Such a message +// has no record in the journal, so it has no byte offset and no sequence +// number — see engine/messagepage.go's package comment. A page must never +// give one a seq, whichever path produced the page. +func durableOnly(msgs []message.Message) []message.Message { + out := make([]message.Message, 0, len(msgs)) + for _, m := range msgs { + if message.IsSyntheticOrphanID(m.ID) { + continue + } + out = append(out, m) + } + return out +} + +// intParam reads a non-negative integer query parameter, writing 400 and +// returning ok=false when it is present but not one. An absent parameter is +// 0, which every caller reads as "unset". It takes the already-parsed +// query: url.URL.Query() re-parses the raw string on every call, and a page +// request asks for two parameters after testing for two more. +// +// A repeated parameter is a 400, not a silent choice of the first value: +// "?limit=2&limit=nonsense" names two different intentions, and answering +// one of them hides a client bug. An explicitly empty value ("?limit=") is +// a 400 for the same reason: it is present, and it is not an integer. +func intParam(w http.ResponseWriter, query url.Values, name string) (int, bool) { + values := query[name] + if len(values) == 0 { + return 0, true + } + if len(values) > 1 { + writeErr(w, http.StatusBadRequest, name+" must be given at most once") + return 0, false + } + v, err := strconv.Atoi(values[0]) + if err != nil || v < 0 { + writeErr(w, http.StatusBadRequest, name+" must be a non-negative integer") + return 0, false + } + return v, true } // requestJSON is the openapi Request shape: the latest fully-assembled model @@ -1160,19 +1428,31 @@ func (s *Server) handleStatus(w http.ResponseWriter, _ *http.Request) { Usage: usageJSONForSession(m.sess), } } - infos, err := engine.ListSessions(s.opts.SessionDir) + // Ids first, indexes second — the same rule GET /session follows. A + // session already answered from memory above needs no index, and + // reading one for it would refold and rewrite the sidecar of a session + // this process holds live, racing that session's own writer. + ids, err := engine.ListSessionIDs(s.opts.SessionDir) if err != nil { writeErr(w, http.StatusInternalServerError, "cannot list sessions") return } - for _, info := range infos { - if _, ok := result[info.ID]; !ok { - result[info.ID] = entry{ - Type: "idle", - State: s.compositeStateFor(info.ID, false), - LastTurn: s.lastTurnFor(info.ID), - Usage: usageJSONForInfo(info), - } + for _, id := range ids { + if _, ok := result[id]; ok { + continue + } + // The same index-then-scan path a listing takes, so this endpoint + // and GET /session never disagree about which sessions exist: a + // session whose fold breaks appears in both, from its journal. + info, err := engine.ReadSessionInfo(s.opts.SessionDir, id) + if err != nil { + continue // unreadable or not a session journal: not listable + } + result[id] = entry{ + Type: "idle", + State: s.compositeStateFor(id, false), + LastTurn: s.lastTurnFor(id), + Usage: usageJSONForInfo(info), } } writeJSON(w, http.StatusOK, result) @@ -1708,8 +1988,9 @@ func (s *Server) releasePromptClaim(st *sessionState) { st.cancel = nil st.goalLoop = false st.lastUsed = time.Now() - s.evictResidentLocked() + evicted := s.evictResidentLocked() s.mu.Unlock() + releaseEvicted(evicted) s.wg.Done() } @@ -1758,10 +2039,11 @@ func (s *Server) freeRunSlotAndEmitIdle(id string, st *sessionState) { st.cancel = nil st.goalLoop = false st.lastUsed = time.Now() - s.evictResidentLocked() + evicted := s.evictResidentLocked() s.emitDurableLocked(&Event{Type: evtSessionStatus, SessionID: id, Status: "idle"}) s.queueDrainPending[id] = true s.mu.Unlock() + releaseEvicted(evicted) } // dispatchQueueHead dequeues the session's queue head (reason "delivered") @@ -2497,14 +2779,16 @@ func (s *Server) handleGoalDelete(w http.ResponseWriter, r *http.Request) { return } s.mu.Lock() + var evicted []*engine.Session if ex := s.sessions[id]; ex != nil { st = ex // a resident appeared while we loaded; use the winner } else { st = &sessionState{sess: sess, lastUsed: time.Now()} s.sessions[id] = st - s.evictResidentLocked() + evicted = s.evictResidentLocked() } s.mu.Unlock() + releaseEvicted(evicted) } s.mu.Lock() var cancel context.CancelFunc @@ -2590,14 +2874,16 @@ func (s *Server) handleSetModel(w http.ResponseWriter, r *http.Request) { return } s.mu.Lock() + var evicted []*engine.Session if ex := s.sessions[id]; ex != nil { st = ex // a resident appeared while we loaded; use the winner } else { st = &sessionState{sess: sess, lastUsed: time.Now()} s.sessions[id] = st - s.evictResidentLocked() + evicted = s.evictResidentLocked() } s.mu.Unlock() + releaseEvicted(evicted) } if body.Model.IsZero() { @@ -2664,14 +2950,16 @@ func (s *Server) handleSetThinking(w http.ResponseWriter, r *http.Request) { return } s.mu.Lock() + var evicted []*engine.Session if ex := s.sessions[id]; ex != nil { st = ex } else { st = &sessionState{sess: sess, lastUsed: time.Now()} s.sessions[id] = st - s.evictResidentLocked() + evicted = s.evictResidentLocked() } s.mu.Unlock() + releaseEvicted(evicted) } effort, err := message.ParseEffort(body.Effort) @@ -2700,10 +2988,16 @@ func (s *Server) handleSetThinking(w http.ResponseWriter, r *http.Request) { // sessMgr-pinned one, not a fresh LoadSession reread from disk — eviction // here narrows only this server's own bookkeeping, never a guarantee that // the next access is served cold. Caller holds s.mu. -func (s *Server) evictResidentLocked() { +// It returns the evicted sessions for the caller to release, and never +// releases them itself: releaseEvicted takes each session's OWN mutex, and +// s.mu is a leaf lock with respect to that (see syncMessages' lock-ordering +// note, journal.go). Taking a session mutex under s.mu would close exactly +// the cycle that rule forbids — the engine holds a session's mutex while +// emitting events into Publish, which takes s.mu. +func (s *Server) evictResidentLocked() (evicted []*engine.Session) { excess := len(s.sessions) - s.opts.MaxResident if excess <= 0 { - return + return nil } type cand struct { id string @@ -2718,6 +3012,9 @@ func (s *Server) evictResidentLocked() { } sort.Slice(cands, func(i, j int) bool { return cands[i].last.Before(cands[j].last) }) for i := 0; i < excess && i < len(cands); i++ { + if st := s.sessions[cands[i].id]; st != nil { + evicted = append(evicted, st.sess) + } delete(s.sessions, cands[i].id) // Release the request snapshot (it holds a full copy of the // assembled system segments). lastReqHash survives deliberately: @@ -2725,6 +3022,22 @@ func (s *Server) evictResidentLocked() { // session is later reloaded. delete(s.lastRequest, cands[i].id) } + return evicted +} + +// releaseEvicted closes the file descriptors of sessions eviction just +// dropped: a session's journal handle and its sidecar-index handle. Call it +// only after s.mu is released — see evictResidentLocked's own doc comment +// for why. +// +// The sessions stay usable. Eviction has already decided each one is idle +// and reloadable, and the next persist call reopens both handles through +// ensureLog. Without this, a process holds two descriptors for every +// session it has ever touched. +func releaseEvicted(evicted []*engine.Session) { + for _, sess := range evicted { + sess.ReleaseFiles() + } } // handleAbort interrupts a session's in-flight prompt. Unknown session (not @@ -2897,14 +3210,16 @@ func (s *Server) handleQueueDelete(w http.ResponseWriter, r *http.Request) { s.queueDeleteRace() } s.mu.Lock() + var evicted []*engine.Session if ex := s.sessions[id]; ex != nil { st = ex // a resident appeared while we loaded; use the winner } else { st = &sessionState{sess: sess, lastUsed: time.Now()} s.sessions[id] = st - s.evictResidentLocked() + evicted = s.evictResidentLocked() } s.mu.Unlock() + releaseEvicted(evicted) } st.sess.DequeueAllPrompts("cleared") w.WriteHeader(http.StatusNoContent) @@ -3080,19 +3395,7 @@ func (s *Server) handleCompact(w http.ResponseWriter, r *http.Request) { // sessionOnDisk reports whether a session log for id exists in the session // directory, without loading the session. func (s *Server) sessionOnDisk(id string) bool { - if s.opts.SessionDir == "" { - return false - } - infos, err := engine.ListSessions(s.opts.SessionDir) - if err != nil { - return false - } - for _, info := range infos { - if info.ID == id { - return true - } - } - return false + return engine.SessionExists(s.opts.SessionDir, id) } // lookup resolves a session for read endpoints and returns its whole @@ -3279,8 +3582,9 @@ func (s *Server) claimForPrompt(id string) (st *sessionState, ctx context.Contex s.wg.Add(1) // A cold load grew the resident set; cap it now. st is running, so // evictResidentLocked will not evict the session we just claimed. - s.evictResidentLocked() + evicted := s.evictResidentLocked() s.mu.Unlock() + releaseEvicted(evicted) return st, ctx, fromSeq, 0, "" } @@ -3475,6 +3779,75 @@ func (s *Server) buildSession(lv liveSession) sessionJSON { } } +// buildSessionFromIndex renders the wire Session for an id NO live source +// in this process holds, from its durable metadata index alone. +// +// It is buildSession's cold twin, and the split is along one line: what has +// a durable source and what does not. The index answers everything the +// session log records — created/activity timestamps, model, effort, +// workdir, message count, usage, durable goal state, queue depth, +// compaction, lineage. The three fields that are process-local answer +// exactly as they do in buildSession, from the same server-side maps: +// journal seq, the goal presentation (goalTracker, which survives a +// restart through the event journal, not the session log), and last_turn. +// +// Status is always "idle" here, by construction: a session running a turn +// in this process is, by definition, live in it, so it never reaches this +// path. That matches what the old cold path reported — a disk-loaded +// session contributed no status either (see liveSession.status). +// +// Plugins come from Options.Plugins, because plugins are process +// configuration rather than durable session state, and an index has no +// Session to ask. +func (s *Server) buildSessionFromIndex(ix engine.SessionIndex) sessionJSON { + s.mu.Lock() + seq := s.sessionSeqLocked(ix.ID) + goal := goalJSONFrom(s.goalState[ix.ID]) + lastTurn := s.lastTurnJSONLocked(ix.ID) + s.mu.Unlock() + return sessionJSON{ + ID: ix.ID, + CreatedAt: ix.CreatedAt, + Model: ix.Model, + Effort: ix.Effort, + Status: "idle", + State: compositeState(false, goal != nil && goal.Active, forcesIdlePause(goal)), + Messages: ix.Messages, + Seq: seq, + Goal: goal, + WorkDir: ix.WorkDir, + LastTurn: lastTurn, + Usage: usageJSON{ + InputTokens: ix.Usage.InputTokens, + OutputTokens: ix.Usage.OutputTokens, + CacheReadTokens: ix.Usage.CacheReadTokens, + CacheWriteTokens: ix.Usage.CacheWriteTokens, + Messages: ix.Messages, + LastInputTokens: ix.LastInputTokens, + }, + LastActivityAt: ix.LastActivityAt, + ParentSession: ix.ParentSession, + CompactionCount: ix.CompactionCount, + LastCompactedAt: ix.LastCompactedAt, + Plugins: s.pluginInfo(ix.ID), + Queued: ix.Queued, + Lineage: coldLineageJSON(ix.TaskParentID, ix.TaskAgentType, ix.TaskDepth, ix.SpawnedChildIDs), + } +} + +// pluginInfo reports a session's configured plugins for a cold read (see +// Options.Plugins). It never returns nil: the wire contract for +// Session.plugins is an array, empty when nothing is configured. +func (s *Server) pluginInfo(sessionID string) []plugin.Info { + if s.opts.Plugins == nil { + return []plugin.Info{} + } + if infos := s.opts.Plugins(sessionID); infos != nil { + return infos + } + return []plugin.Info{} +} + // lineageJSONFor returns lv.id's subagent-sessions lineage. It reads only // the caller's already-taken liveSession snapshot (see that type's own doc // comment) — never sessMgr again — so the lineage block always describes @@ -3614,11 +3987,23 @@ func lineageJSONFor(lv liveSession) *lineageJSON { // reliable positive signal regardless of log vintage (a durably // recorded child is a durably recorded child), so that case is // reported normally, unmodified. + return coldLineageJSON(parentID, sess.TaskAgentType(), sess.TaskDepth(), sess.SpawnedChildIDs()) +} + +// coldLineageJSON builds the durable-only lineage block from the four +// fields a session log carries, whether they were read off a loaded +// Session (lineageJSONFor's cold branch) or off its metadata index +// (buildSessionFromIndex). Both callers describe a session no live source +// in this process holds, so both omit the live-only fields identically. +func coldLineageJSON(parentID, agentType string, depth int, children []string) *lineageJSON { + if parentID == "" { + return nil + } return &lineageJSON{ ParentID: parentID, - Depth: sess.TaskDepth(), - AgentType: sess.TaskAgentType(), - Children: sess.SpawnedChildIDs(), + Depth: depth, + AgentType: agentType, + Children: children, } } diff --git a/server/id_test.go b/server/id_test.go index 033270e9..4db6d73f 100644 --- a/server/id_test.go +++ b/server/id_test.go @@ -130,4 +130,11 @@ func TestLegacySessionIDOverHTTP(t *testing.T) { if resp.StatusCode != 202 { t.Fatalf("prompt_async on legacy session status = %d: %s", resp.StatusCode, data) } + // prompt_async returns before the turn runs. Wait for it: the turn + // writes durable records — its journal appends, and the session's + // sidecar index — and a test that returns first leaves those writes + // racing t.TempDir's own cleanup, which then fails with "directory not + // empty". GET /wait is the production seam for this (AGENTS.md's + // in-process-state rule), not a sleep. + h.waitIdle(legacyID) } diff --git a/server/journal.go b/server/journal.go index f315565b..b4518925 100644 --- a/server/journal.go +++ b/server/journal.go @@ -992,23 +992,27 @@ func (s *Server) reconcile() error { } s.jf = f - infos, err := engine.ListSessions(s.opts.SessionDir) + // Ids only. This pass loads every session in full anyway, so reading + // each session's metadata index first would be pure waste — and it + // would refold and rewrite a stale sidecar for every session in the + // directory before the load that supersedes it. + ids, err := engine.ListSessionIDs(s.opts.SessionDir) if err != nil { return err } - for _, info := range infos { - sess, err := s.opts.LoadSession(info.ID) + for _, id := range ids { + sess, err := s.opts.LoadSession(id) if err != nil { continue // unreadable log: skip, do not fail the whole boot } history := sess.History() for i := range history { m := history[i] - if s.isSeenLocked(info.ID, m.ID) { + if s.isSeenLocked(id, m.ID) { continue } - s.markSeenLocked(info.ID, m.ID) - s.emitDurableLocked(&Event{Type: evtMessage, SessionID: info.ID, Message: &m}) + s.markSeenLocked(id, m.ID) + s.emitDurableLocked(&Event{Type: evtMessage, SessionID: id, Message: &m}) } } return nil diff --git a/server/message_page_test.go b/server/message_page_test.go new file mode 100644 index 00000000..f8622725 --- /dev/null +++ b/server/message_page_test.go @@ -0,0 +1,441 @@ +package server + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// pageResponse mirrors messagePageJSON for a test that decodes it as a +// client would, without reaching into the server's own type for anything +// but the field names. +type pageResponse struct { + Messages []message.Message `json:"messages"` + FirstSeq int `json:"first_seq"` + LastSeq int `json:"last_seq"` + Total int `json:"total"` + HasMore bool `json:"has_more"` +} + +// coldMessages writes a session with n turns into dir through the engine's +// own path, so the server sees it exactly as it sees any session it did not +// create: a journal on disk, nothing resident. +func coldMessages(t *testing.T, dir string, n int) *engine.Session { + t.Helper() + turns := make([][]provider.Event, 0, n) + for i := 0; i < n; i++ { + turns = append(turns, asstTurn(fmt.Sprintf("reply %d", i))) + } + prov := &scriptedProvider{name: "test", turns: turns} + sess := engine.NewSession(engine.Config{ + Providers: provider.Registry{prov.name: prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + SessionDir: dir, + WorkDir: dir, + }) + for i := 0; i < n; i++ { + if _, err := sess.Prompt(context.Background(), fmt.Sprintf("ask %d", i)); err != nil { + t.Fatalf("Prompt %d: %v", i, err) + } + } + if err := sess.PersistErr(); err != nil { + t.Fatalf("PersistErr: %v", err) + } + return sess +} + +func getPage(t *testing.T, h *harness, id, query string) pageResponse { + t.Helper() + resp, data := h.do("GET", "/session/"+id+"/message"+query, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET message%s = %d: %s", query, resp.StatusCode, data) + } + var page pageResponse + if err := json.Unmarshal(data, &page); err != nil { + t.Fatalf("decode page: %v (%s)", err, data) + } + return page +} + +// TestMessagesUnparameterizedIsUnchanged pins the compatibility promise: a +// caller that asks for no page still gets the bare array of the WHOLE +// history it has always got, not an envelope. +func TestMessagesUnparameterizedIsUnchanged(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 3) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session/"+sess.ID+"/message", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET message = %d: %s", resp.StatusCode, data) + } + var msgs []message.Message + if err := json.Unmarshal(data, &msgs); err != nil { + t.Fatalf("unparameterized response must stay a bare array: %v (%s)", err, data) + } + if len(msgs) != 6 { + t.Errorf("got %d messages, want 6", len(msgs)) + } +} + +// TestMessagePageServesBoundedTail is the endpoint's reason to exist: a +// console asks for the newest K and gets K, plus where they sit. +func TestMessagePageServesBoundedTail(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 5) // 10 messages + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + page := getPage(t, h, sess.ID, "?limit=4") + if len(page.Messages) != 4 { + t.Fatalf("got %d messages, want 4", len(page.Messages)) + } + if page.Total != 10 { + t.Errorf("total = %d, want 10", page.Total) + } + if page.FirstSeq != 7 || page.LastSeq != 10 { + t.Errorf("seqs = [%d,%d], want [7,10]", page.FirstSeq, page.LastSeq) + } + if !page.HasMore { + t.Error("has_more = false, want true") + } +} + +// TestMessagePageScrollsBackToTheStart drives the console's own loop: fetch +// the tail, then page older with before_seq, until has_more is false. The +// pages must reassemble into the full transcript exactly once. +func TestMessagePageScrollsBackToTheStart(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 4) // 8 messages + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session/"+sess.ID+"/message", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET message = %d: %s", resp.StatusCode, data) + } + var whole []message.Message + if err := json.Unmarshal(data, &whole); err != nil { + t.Fatal(err) + } + + var got []message.Message + query := "?limit=3" + for { + page := getPage(t, h, sess.ID, query) + got = append(append([]message.Message{}, page.Messages...), got...) + if !page.HasMore { + break + } + query = fmt.Sprintf("?limit=3&before_seq=%d", page.FirstSeq) + } + if len(got) != len(whole) { + t.Fatalf("paged walk returned %d messages, want %d", len(got), len(whole)) + } + for i := range got { + if got[i].ID != whole[i].ID { + t.Fatalf("paged walk differs at %d: %q, want %q", i, got[i].ID, whole[i].ID) + } + } +} + +// TestMessagePageServesRunningSession: a page must be answerable while the +// session is mid-turn. It reads the durable records, so it neither waits +// for the turn nor reports the turn's unfinished state. +func TestMessagePageServesRunningSession(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + <-prov.started + defer prov.releaseAll() + + page := getPage(t, h, id, "?limit=10") + if page.Total != 1 || len(page.Messages) != 1 { + t.Fatalf("page = %+v, want the one durable user message", page) + } + if page.Messages[0].Role != message.RoleUser { + t.Errorf("role = %q, want user", page.Messages[0].Role) + } +} + +// TestMessagePageOnNeverPersistedSession: a session created through the API +// and never prompted has no messages at all. The page endpoint answers an +// empty page, never a 404 or a 500. +func TestMessagePageOnNeverPersistedSession(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("") + + page := getPage(t, h, id, "?limit=5") + if len(page.Messages) != 0 || page.Total != 0 || page.HasMore { + t.Errorf("page = %+v, want an empty page", page) + } +} + +// TestMessagePageRejectsBadParameters: a malformed page request is a client +// error, not a silently-defaulted one — a caller that sends limit=abc has a +// bug and must be told. +func TestMessagePageRejectsBadParameters(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 1) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + for _, query := range []string{"?limit=abc", "?limit=-1", "?before_seq=nope", "?before_seq=-3", "?limit=", "?before_seq="} { + resp, data := h.do("GET", "/session/"+sess.ID+"/message"+query, nil) + if resp.StatusCode != 400 { + t.Errorf("GET message%s = %d, want 400: %s", query, resp.StatusCode, data) + } + } +} + +// TestMessagePageUnknownSessionIsNotFound: an id with no journal and no +// live session is a 404, exactly as the unparameterized read reports it. +func TestMessagePageUnknownSessionIsNotFound(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, _ := h.do("GET", "/session/ses_0123456789abcdef/message?limit=5", nil) + if resp.StatusCode != 404 { + t.Fatalf("GET unknown session page = %d, want 404", resp.StatusCode) + } +} + +// TestMessagePageBeforeSeqOne: paging older than the oldest message is an +// empty page with has_more false — the loop terminator a client relies on. +func TestMessagePageBeforeSeqOne(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 2) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + page := getPage(t, h, sess.ID, "?before_seq=1&limit=5") + if len(page.Messages) != 0 { + t.Errorf("got %d messages, want 0", len(page.Messages)) + } + if page.HasMore { + t.Error("has_more = true, want false") + } + if page.Total != 4 { + t.Errorf("total = %d, want 4", page.Total) + } +} + +// TestMessagePageDistinguishesMissingFromUnreadable: a session whose +// journal cannot be read is not "no such session". Reporting 404 for it +// sends an operator looking for a session id that is on disk in front of +// them. +func TestMessagePageDistinguishesMissingFromUnreadable(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + // A journal whose SECOND record is corrupt: scanLog's tolerance covers + // only a corrupt final line, so this file exists and cannot be folded. + journal := `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"message","message":{"id":"msg_1","ro +{"type":"message","message":{"id":"msg_2","role":"user","parts":[{"type":"text","text":"hi"}]}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", "/session/"+id+"/message?limit=5", nil) + if resp.StatusCode != 500 { + t.Errorf("unreadable journal = %d: %s; want 500", resp.StatusCode, data) + } + + resp, data = h.do("GET", "/session/ses_00000000000000000000000000/message?limit=5", nil) + if resp.StatusCode != 404 { + t.Errorf("absent journal = %d: %s; want 404", resp.StatusCode, data) + } +} + +// TestMessagePageRejectsRepeatedParameters: "?limit=2&limit=nonsense" names +// two intentions. Answering the first hides a client bug. +func TestMessagePageRejectsRepeatedParameters(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 1) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + for _, query := range []string{"?limit=2&limit=3", "?before_seq=1&before_seq=2", "?limit=2&limit=nonsense"} { + resp, data := h.do("GET", "/session/"+sess.ID+"/message"+query, nil) + if resp.StatusCode != 400 { + t.Errorf("GET message%s = %d, want 400: %s", query, resp.StatusCode, data) + } + } +} + +// TestMessagePageKeepsTheDurableContractWhenAJournalIsUnreadable: a live +// session's resident history can carry messages the log does not — a repair +// applied at load, or recovery's memory-only closer. Paging it would give +// those messages sequence numbers, and a client that paged again after the +// journal became readable would see its pages renumbered. An unreadable +// journal is therefore a 500, even for a session this process holds live. +func TestMessagePageKeepsTheDurableContractWhenAJournalIsUnreadable(t *testing.T) { + dir := t.TempDir() + h := newHarnessDir(t, dir, &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("hi")}}) + id := h.createSession("") + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + // Corrupt a NON-final record, which no reader tolerates, while the + // session stays resident. + path := filepath.Join(dir, id+".jsonl") + raw, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(string(raw), "\n") + if len(lines) < 4 { + t.Fatalf("test setup: journal has %d lines", len(lines)) + } + lines[2] = `{"type":"message","message":{"id":"broken` + if err := os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o644); err != nil { + t.Fatal(err) + } + + resp, data = h.do("GET", "/session/"+id+"/message?limit=5", nil) + if resp.StatusCode != 500 { + t.Errorf("unreadable journal for a LIVE session = %d: %s; want 500, never resident history under durable seqs", resp.StatusCode, data) + } +} + +// TestMessagePageFallbackNumbersTheDurableSequence: the resident fallback +// is the one place a page is numbered from memory rather than from records, +// and it must number the SAME sequence a journal page would. This drives a +// live session whose journal is gone — the only case the fallback serves — +// and checks the page carries the durable seqs, not a memory-only count. +func TestMessagePageFallbackNumbersTheDurableSequence(t *testing.T) { + dir := t.TempDir() + h := newHarnessDir(t, dir, &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("one")}}) + id := h.createSession("") + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + // A page from the journal, then the same page with the journal gone. + fromJournal := getPage(t, h, id, "?limit=10") + if err := os.Remove(filepath.Join(dir, id+".jsonl")); err != nil { + t.Fatal(err) + } + if err := os.Remove(filepath.Join(dir, id+".index.json")); err != nil { + t.Fatal(err) + } + fromMemory := getPage(t, h, id, "?limit=10") + + if fromMemory.Total != fromJournal.Total { + t.Errorf("total = %d from memory, %d from the journal", fromMemory.Total, fromJournal.Total) + } + if fromMemory.FirstSeq != fromJournal.FirstSeq || fromMemory.LastSeq != fromJournal.LastSeq { + t.Errorf("seqs = [%d,%d] from memory, [%d,%d] from the journal", + fromMemory.FirstSeq, fromMemory.LastSeq, fromJournal.FirstSeq, fromJournal.LastSeq) + } + if len(fromMemory.Messages) != len(fromJournal.Messages) { + t.Fatalf("%d messages from memory, %d from the journal", len(fromMemory.Messages), len(fromJournal.Messages)) + } + for i := range fromMemory.Messages { + if fromMemory.Messages[i].ID != fromJournal.Messages[i].ID { + t.Errorf("message %d: %q from memory, %q from the journal", i, fromMemory.Messages[i].ID, fromJournal.Messages[i].ID) + } + } +} + +// TestDurableOnlyDropsDerivedRepairMessages pins the filter the fallback +// leans on. message.ResolveOrphanToolCalls derives a tool result for a call +// whose result never reached the log; that message has no record, so it has +// no byte offset and no sequence number. A page must never give it one. +func TestDurableOnlyDropsDerivedRepairMessages(t *testing.T) { + history := []message.Message{ + {ID: "msg_u1", Role: message.RoleUser}, + {ID: "msg_a1", Role: message.RoleAssistant}, + {ID: message.SyntheticOrphanIDPrefix + "1-tc1", Role: message.RoleTool}, + {ID: "msg_u2", Role: message.RoleUser}, + } + got := durableOnly(history) + want := []string{"msg_u1", "msg_a1", "msg_u2"} + if len(got) != len(want) { + t.Fatalf("durableOnly returned %d messages, want %d", len(got), len(want)) + } + for i := range got { + if got[i].ID != want[i] { + t.Errorf("message %d = %q, want %q", i, got[i].ID, want[i]) + } + } +} + +// TestMessagePageRejectsAnOversizedLimit: the published schema names a +// maximum for `limit`, and a generated client or a gateway enforces it. +// Answering a larger request with a smaller page would make the server +// disagree with its own spec, so the boundary rejects rather than clamps. +// The engine API still clamps, for a caller with no schema to honor. +func TestMessagePageRejectsAnOversizedLimit(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 2) + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + resp, data := h.do("GET", fmt.Sprintf("/session/%s/message?limit=%d", sess.ID, engine.MaxMessagePageLimit+1), nil) + if resp.StatusCode != 400 { + t.Errorf("limit above the maximum = %d: %s; want 400", resp.StatusCode, data) + } + // The maximum itself is accepted. + resp, data = h.do("GET", fmt.Sprintf("/session/%s/message?limit=%d", sess.ID, engine.MaxMessagePageLimit), nil) + if resp.StatusCode != 200 { + t.Errorf("limit at the maximum = %d: %s; want 200", resp.StatusCode, data) + } +} + +// TestMessagePageWindowIsSharedWithTheJournalPath: the resident fallback +// and the journal path must paginate identically. Both call +// engine.MessagePageWindow, and this pins the observable half — the same +// request against the same session yields the same window either way. +func TestMessagePageWindowIsSharedWithTheJournalPath(t *testing.T) { + dir := t.TempDir() + h := newHarnessDir(t, dir, &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("a"), asstTurn("b"), asstTurn("c")}}) + id := h.createSession("") + for i := 0; i < 3; i++ { + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": fmt.Sprintf("go %d", i)}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + } + + queries := []string{"?limit=2", "?limit=2&before_seq=5", "?limit=100", "?before_seq=1&limit=2"} + fromJournal := make([]pageResponse, 0, len(queries)) + for _, q := range queries { + fromJournal = append(fromJournal, getPage(t, h, id, q)) + } + if err := os.Remove(filepath.Join(dir, id+".jsonl")); err != nil { + t.Fatal(err) + } + if err := os.Remove(filepath.Join(dir, id+".index.json")); err != nil { + t.Fatal(err) + } + for i, q := range queries { + got := getPage(t, h, id, q) + want := fromJournal[i] + if got.FirstSeq != want.FirstSeq || got.LastSeq != want.LastSeq || got.Total != want.Total || got.HasMore != want.HasMore { + t.Errorf("%s: memory page [%d,%d] total=%d more=%v, journal page [%d,%d] total=%d more=%v", + q, got.FirstSeq, got.LastSeq, got.Total, got.HasMore, want.FirstSeq, want.LastSeq, want.Total, want.HasMore) + } + } +} diff --git a/server/openapi.yaml b/server/openapi.yaml index 670de5eb..4479d194 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -126,7 +126,13 @@ components: $ref: "#/components/schemas/SessionState" messages: type: integer - description: Count of messages in the session log. + description: | + Count of messages in the session's history. For a session this + process holds live it is the resident history length. For any + other session it is the same count taken from the session's + metadata index: message records with compaction folds applied, + plus the synthetic tool results a replay derives for a tool call + whose result never reached the log. seq: type: integer description: Sequence number of the latest durable record. @@ -609,6 +615,47 @@ components: type: string description: The encoding/json error raised while marshaling this message. + MessagePage: + type: object + description: > + One bounded page of a session's durable message sequence, oldest + first — the response shape of GET /session/{id}/message when the + request names `before_seq` or `limit`. + + A message's sequence number is its 1-based ordinal in the DURABLE + sequence: message records in log order, with each compaction's fold + applied (the folded range replaced by that compaction's summary). + Compaction renumbers, so a client paging across one can see two + pages overlap; message ids are stable and are the way to + de-duplicate. + required: [messages, first_seq, last_seq, total, has_more] + properties: + messages: + type: array + description: The page, oldest first. Empty when nothing sits at or below the requested point. + items: + oneOf: + - $ref: "#/components/schemas/Message" + - $ref: "#/components/schemas/MessagePlaceholder" + first_seq: + type: integer + description: Sequence number of the first message in the page; 0 for an empty page. Pass it back as `before_seq` for the next older page. + last_seq: + type: integer + description: Sequence number of the last message in the page; 0 for an empty page. + total: + type: integer + description: > + The session's whole durable message count — the sequence number + of its newest message. It can be lower than Session.messages, + which also counts the synthetic tool results a replay derives + for a tool call whose result never reached the log: a derived + message has no record, so it has no sequence number and no page + carries it. + has_more: + type: boolean + description: Whether at least one message older than `first_seq` exists. + Part: type: object required: [type] @@ -1674,6 +1721,11 @@ components: content: application/json: schema: { $ref: "#/components/schemas/Error" } + BadRequest: + description: A malformed request parameter or body. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } ProcessNotFound: description: No declared process with that name. content: @@ -1840,33 +1892,95 @@ paths: /session/{id}/message: get: operationId: getMessages - summary: Full canonical message history (bootstrap for renderers). + summary: Canonical message history — whole, or one bounded page. description: > - Marshals each resident message independently: a single message that - fails to marshal (e.g. a Reasoning part carrying an invalid - provider-native attachment that ingest-time normalization does not - catch) is replaced in the response by a MessagePlaceholder rather - than failing the whole request — this endpoint always returns 200 - with as much of the transcript as is actually renderable, never a - 500 for one bad message. + Two response shapes, chosen by the request. WITHOUT `before_seq` and + `limit` the response is the bare array of the whole history it has + always been, unchanged for every existing caller. WITH either + parameter the response is a MessagePage envelope: one bounded page + of the session's durable message sequence, newest page by default, + read from the journal's tail rather than its whole length. + + Either way, each message is marshaled independently: a single + message that fails to marshal (e.g. a Reasoning part carrying an + invalid provider-native attachment that ingest-time normalization + does not catch) is replaced in the response by a MessagePlaceholder + rather than failing the whole request — this endpoint always returns + 200 with as much of the transcript as is actually renderable, never + a 500 for one bad message. + + A page reports messages VERBATIM from the durable log. Unlike the + unparameterized read, it never adds the load-time repair that + synthesizes an is_error tool result for a tool call whose result + never reached the log: that repair exists to keep a provider REQUEST + valid, and fabricating a tool failure in a read view has caused a + console to render a healthy in-flight call as failed. parameters: - $ref: "#/components/parameters/sessionID" + - name: before_seq + in: query + required: false + schema: { type: integer, minimum: 0 } + description: > + Return the messages immediately BEFORE this sequence number. + Omitted (or 0) means the newest page. A client pages backwards by + passing the previous response's `first_seq`. + - name: limit + in: query + required: false + schema: { type: integer, minimum: 0, default: 100, maximum: 1000 } + description: > + Maximum messages in the page. 0 or omitted means 100. A value + above the maximum is REJECTED with 400, not clamped, so a + generated client that enforces the schema and this server agree + on what a request means. responses: "200": description: OK content: application/json: schema: - type: array - items: - description: > - A Message, or a MessagePlaceholder in place of any - message that failed to marshal. - oneOf: - - $ref: "#/components/schemas/Message" - - $ref: "#/components/schemas/MessagePlaceholder" + oneOf: + - type: array + description: > + The whole history, for a request that names no page. + items: + description: > + A Message, or a MessagePlaceholder in place of any + message that failed to marshal. + oneOf: + - $ref: "#/components/schemas/Message" + - $ref: "#/components/schemas/MessagePlaceholder" + - $ref: "#/components/schemas/MessagePage" + "400": { $ref: "#/components/responses/BadRequest" } "401": { $ref: "#/components/responses/Unauthorized" } - "404": { $ref: "#/components/responses/NotFound" } + "404": + description: > + No session with that id: no journal on disk, and nothing live in + this process. A session whose journal EXISTS but cannot be read + is a 500, not a 404 — see below. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "500": + description: > + A page request whose journal exists but cannot be read: a + corrupt NON-FINAL record, an I/O error, or a journal that keeps + changing under the read. Error text "cannot read session + messages". A corrupt FINAL record is a crash mid-write, which + every reader ignores, and does not produce this. + + A live session is answered from its resident history for ONE + case only — a session that has no durable journal at all — so a + live session with an unreadable journal is a 500 as well. Paging + resident history there would give a sequence number to a message + that has no record, and a client that paged again after the + journal became readable would see its pages renumbered. Only the + page routes (`before_seq`/`limit`) can answer 500; a request + naming no page is unchanged. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } /session/{id}/journal: get: diff --git a/server/server.go b/server/server.go index 91c3cd64..031939dd 100644 --- a/server/server.go +++ b/server/server.go @@ -28,6 +28,7 @@ import ( "github.com/majorcontext/harness/engine" "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" ) // Workdir isolation modes for POST /session's workdir_isolation field (see @@ -98,7 +99,28 @@ type Options struct { // way. It returns an error when no log with that ID exists. The // session's workdir is restored from its log header (see // engine.LoadSession), not passed in here. + // + // The Config it builds must be GENERIC: a model, a workdir, and process + // wiring, never state that belongs to one specific session. + // engine.LoadSession keeps the Config's value for any header field a + // journal does not carry, and a cold read answers from that session's + // metadata index instead, which has no Config at all (see + // engine.SessionIndex.Complete). A wrapper that injected, say, a parent + // session id per session would make those two reads disagree. LoadSession func(id string) (*engine.Session, error) + // Plugins reports the plugin.Info list a session carries + // (engine.Session.Plugins, which reads the plugin.Host out of that + // session's own Config). GET /session and GET /session/{id} answer a + // session that is NOT live in this process from its metadata index + // (engine.SessionIndex), and an index has no Session to ask — plugins + // are process configuration, not durable session state. + // + // It takes the session id because an embedder may wire a different host + // per session through its own NewSession/LoadSession wrappers, even + // though harness serve wires one host for the whole process. nil + // reports no plugins, which is also what a process with no plugin host + // configured reports. + Plugins func(sessionID string) []plugin.Info // WorkspaceRoots bounds the directories POST /session may accept as an // explicit workdir: the request value must clean-resolve (absolute, // cleaned) to one of these roots or a descendant of one. Empty means the diff --git a/server/session_tree_test.go b/server/session_tree_test.go index 2a6192c9..c7a9aec4 100644 --- a/server/session_tree_test.go +++ b/server/session_tree_test.go @@ -260,6 +260,10 @@ func TestSessionCreateWithParentIDFiresOnTaskEvent(t *testing.T) { if resp.StatusCode != 201 { t.Fatalf("spawn child status %d: %s", resp.StatusCode, data) } + var child struct { + ID string `json:"id"` + } + mustUnmarshal(t, data, &child) mu.Lock() got := append([]string(nil), events...) @@ -267,6 +271,13 @@ func TestSessionCreateWithParentIDFiresOnTaskEvent(t *testing.T) { if len(got) != 1 || got[0] != "spawned" { t.Errorf("events = %v, want [spawned]", got) } + + // Spawn returns before the child's turn runs. Wait for it to settle: + // that turn creates and writes the child's durable files, and a test + // that returns first leaves those writes racing t.TempDir's cleanup, + // which then fails with "directory not empty". waitForLineageStatus + // blocks on SessionManager.Changed, the production seam, not a sleep. + waitForLineageStatus(t, h, child.ID, "done", 5*time.Second) } // TestReportTaskEventClassifiesEachSentinel proves reportTaskEvent's From 0fe0076f4edae73c9e46cb4d3ca7b34608125b1e Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 06:25:38 -0400 Subject: [PATCH 11/95] feat(engine): tell the model to batch independent tool calls (#200) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine executes one assistant message's tool calls concurrently (engine/toolexec.go, capped by Config.ToolConcurrency), but nothing ever asked the model to produce a batch. A model that emits one tool call per turn never makes a batch wider than one, so the executor's capacity went unclaimed on every such turn: measured on three independent `sleep 2` bash calls, a batched turn finishes in ~2s where one-per-turn takes ~6s. Inject one system segment saying to put independent calls in the same message, and to wait when a call's arguments depend on an earlier call's result. Both halves matter — the second is what stops a model parallelizing genuinely dependent work, which no amount of executor correctness can repair. The segment is gated on the session's resolved concurrency and is empty at 1, so a session running strictly sequentially is never told its calls run concurrently. The cap in the text is rendered from s.toolConcurrency, so the number the model reads is the number the executor enforces. It sits immediately after Config.System and before the instructions segment: it describes how the engine runs tools, not anything about the project. That shifts every later segment's index, so the segment-layout assertions across engine/ and server/ now pin it explicitly rather than asserting a layout no default-configured session has. Co-authored-by: harness --- AGENTS.md | 28 ++++++++ engine/engine.go | 7 ++ engine/engine_test.go | 2 +- engine/instructions_test.go | 60 +++++++++-------- engine/mcp_lazy_test.go | 16 ++--- engine/onrequest_test.go | 19 +++--- engine/session_info_test.go | 4 +- engine/skills_test.go | 41 +++++------ engine/toolbatching_test.go | 131 ++++++++++++++++++++++++++++++++++++ engine/toolexec.go | 28 ++++++++ server/request_test.go | 21 ++++-- 11 files changed, 286 insertions(+), 71 deletions(-) create mode 100644 engine/toolbatching_test.go diff --git a/AGENTS.md b/AGENTS.md index cc3dddaa..0e205512 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -134,6 +134,34 @@ never written to the session log — a resumed session rediscovers them. Config `skills_dirs` (array; a non-empty project value overrides the user value entirely) and the repeatable `-skills-dir` run/serve flag drive it. +### Tool-batching guidance + +The engine executes one assistant message's tool calls concurrently +(`engine/toolexec.go`, capped by `Config.ToolConcurrency`), but a model +that emits one call per turn never produces a batch wider than one. The +executor is only as useful as the model's willingness to batch, so the +engine asks for it: `toolBatchingSegment` (`engine/toolexec.go`) injects +one system segment telling the model to put independent calls in the same +message, and to wait when a call's arguments depend on an earlier call's +result. Both halves matter — the second is what stops a model +parallelizing genuinely dependent work, which no amount of executor +correctness can repair. + +The segment sits immediately after `Config.System` and before the +instructions segment (`engine/engine.go`): it describes how this engine +runs tools, not anything about the project. It is **gated on the +session's resolved concurrency and is empty at 1** — an operator who set +`HARNESS_SEQUENTIAL_TOOLS=1`, or an embedder who set `ToolConcurrency: 1`, +must not be told calls run concurrently when for that session they do not. +The cap in the text is rendered from `s.toolConcurrency`, so the number +the model reads is the number the executor enforces. Like every other +engine-injected segment it is never written to the session log. + +Adding a base segment shifts every later segment's index, so the +segment-layout assertions across `engine/*_test.go` pin it explicitly via +`isBatchingSegment` (`engine/toolbatching_test.go`); only that file pins +the wording itself. + ### read_file image support The built-in `read_file` tool (`engine/filetools.go`) can return an image diff --git a/engine/engine.go b/engine/engine.go index adc47b6b..da9c24aa 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -2417,6 +2417,13 @@ func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message tools, mcpCatalog := s.toolDefsWithCatalog(ctx) system := append([]string(nil), s.cfg.System...) + // Tool-batching guidance sits with the base system prompt, ahead of + // project instructions: it describes how this engine executes tools, + // not anything about the project. Empty for a session that runs tools + // one at a time (see toolBatchingSegment in toolexec.go). + if seg := s.toolBatchingSegment(); seg != "" { + system = append(system, seg) + } // Project instructions sit after the base system prompt and before any // hook-contributed segments (see ensureInstructions in instructions.go). if seg := s.instructionSegment(); seg != "" { diff --git a/engine/engine_test.go b/engine/engine_test.go index 902fd49d..a72cd7e8 100644 --- a/engine/engine_test.go +++ b/engine/engine_test.go @@ -293,7 +293,7 @@ func TestHooksIntegration(t *testing.T) { } // system.transform segments appended after base. - if sys := prov.requests[0].System; len(sys) != 2 || sys[1] != "injected rules" { + if sys := prov.requests[0].System; len(sys) != 3 || sys[2] != "injected rules" { t.Errorf("system = %v", sys) } // shell.env consulted for the bash command. diff --git a/engine/instructions_test.go b/engine/instructions_test.go index 4d03b427..13c70450 100644 --- a/engine/instructions_test.go +++ b/engine/instructions_test.go @@ -228,17 +228,20 @@ func TestInstructionsInjectedIntoSystem(t *testing.T) { writeInstr(t, filepath.Join(dir, "AGENTS.md"), "project says hi") prov := instrSession(t, Config{WorkDir: dir}, 1) sys := prov.requests[0].System - if len(sys) != 2 { - t.Fatalf("system = %v, want 2 segments", sys) + if len(sys) != 3 { + t.Fatalf("system = %v, want 3 segments", sys) } if sys[0] != "base" { t.Errorf("sys[0] = %q, want base", sys[0]) } - if !strings.HasPrefix(sys[1], "Project instructions from AGENTS.md:") { - t.Errorf("sys[1] header = %q", sys[1]) + if !isBatchingSegment(sys[1]) { + t.Errorf("sys[1] = %q, want the tool-batching segment", sys[1]) + } + if !strings.HasPrefix(sys[2], "Project instructions from AGENTS.md:") { + t.Errorf("sys[2] header = %q", sys[2]) } - if !strings.Contains(sys[1], "project says hi") { - t.Errorf("sys[1] body = %q", sys[1]) + if !strings.Contains(sys[2], "project says hi") { + t.Errorf("sys[2] body = %q", sys[2]) } } @@ -247,8 +250,8 @@ func TestInstructionsDisabled(t *testing.T) { writeInstr(t, filepath.Join(dir, "AGENTS.md"), "should be ignored") prov := instrSession(t, Config{WorkDir: dir, Instructions: &InstructionsConfig{Disabled: true}}, 1) sys := prov.requests[0].System - if len(sys) != 1 || sys[0] != "base" { - t.Errorf("system = %v, want only [base] when disabled", sys) + if len(sys) != 2 || sys[0] != "base" || !isBatchingSegment(sys[1]) { + t.Errorf("system = %v, want [base, tool-batching] when instructions are disabled", sys) } } @@ -257,8 +260,8 @@ func TestInstructionsMissingNoSegment(t *testing.T) { mkdirAll(t, filepath.Join(dir, ".git")) prov := instrSession(t, Config{WorkDir: dir}, 1) sys := prov.requests[0].System - if len(sys) != 1 || sys[0] != "base" { - t.Errorf("system = %v, want only [base] when no AGENTS.md", sys) + if len(sys) != 2 || sys[0] != "base" || !isBatchingSegment(sys[1]) { + t.Errorf("system = %v, want [base, tool-batching] when there is no AGENTS.md", sys) } } @@ -269,17 +272,17 @@ func TestInstructionsPathOverride(t *testing.T) { writeInstr(t, override, "override rules") prov := instrSession(t, Config{WorkDir: dir, Instructions: &InstructionsConfig{Path: override}}, 1) sys := prov.requests[0].System - if len(sys) != 2 { - t.Fatalf("system = %v, want 2 segments", sys) + if len(sys) != 3 { + t.Fatalf("system = %v, want 3 segments", sys) } - if !strings.Contains(sys[1], "override rules") { - t.Errorf("sys[1] = %q, want override rules", sys[1]) + if !strings.Contains(sys[2], "override rules") { + t.Errorf("sys[2] = %q, want override rules", sys[2]) } - if strings.Contains(sys[1], "default file") { - t.Errorf("override ignored the discovered AGENTS.md: %q", sys[1]) + if strings.Contains(sys[2], "default file") { + t.Errorf("override ignored the discovered AGENTS.md: %q", sys[2]) } - if !strings.Contains(sys[1], override) { - t.Errorf("sys[1] should name the override path %q: %q", override, sys[1]) + if !strings.Contains(sys[2], override) { + t.Errorf("sys[2] should name the override path %q: %q", override, sys[2]) } } @@ -290,7 +293,7 @@ func TestInstructionsPathOverrideRelative(t *testing.T) { writeInstr(t, filepath.Join(dir, "custom.md"), "relative override rules") prov := instrSession(t, Config{WorkDir: dir, Instructions: &InstructionsConfig{Path: "custom.md"}}, 1) sys := prov.requests[0].System - if len(sys) != 2 || !strings.Contains(sys[1], "relative override rules") { + if len(sys) != 3 || !strings.Contains(sys[2], "relative override rules") { t.Fatalf("system = %v, want relative override injected", sys) } } @@ -320,8 +323,8 @@ func TestInstructionsLoadedOncePerSession(t *testing.T) { if len(prov.requests) != 2 { t.Fatalf("requests = %d, want 2", len(prov.requests)) } - seg0 := prov.requests[0].System[1] - seg1 := prov.requests[1].System[1] + seg0 := prov.requests[0].System[2] + seg1 := prov.requests[1].System[2] if seg0 != seg1 { t.Errorf("segment changed between prompts:\n%q\n%q", seg0, seg1) } @@ -336,17 +339,20 @@ func TestInstructionsBeforeHookSegments(t *testing.T) { hooks := &fakeHooks{segments: []string{"hook seg"}} prov := instrSession(t, Config{WorkDir: dir, Hooks: hooks}, 1) sys := prov.requests[0].System - if len(sys) != 3 { - t.Fatalf("system = %v, want [base, instructions, hook seg]", sys) + if len(sys) != 4 { + t.Fatalf("system = %v, want [base, tool-batching, instructions, hook seg]", sys) } if sys[0] != "base" { t.Errorf("sys[0] = %q, want base", sys[0]) } - if !strings.Contains(sys[1], "instr body") { - t.Errorf("sys[1] = %q, want instructions segment", sys[1]) + if !isBatchingSegment(sys[1]) { + t.Errorf("sys[1] = %q, want the tool-batching segment", sys[1]) + } + if !strings.Contains(sys[2], "instr body") { + t.Errorf("sys[2] = %q, want instructions segment", sys[2]) } - if sys[2] != "hook seg" { - t.Errorf("sys[2] = %q, want hook seg (hooks run after instructions)", sys[2]) + if sys[3] != "hook seg" { + t.Errorf("sys[3] = %q, want hook seg (hooks run after instructions)", sys[3]) } } diff --git a/engine/mcp_lazy_test.go b/engine/mcp_lazy_test.go index 453c2a2f..8182a2ed 100644 --- a/engine/mcp_lazy_test.go +++ b/engine/mcp_lazy_test.go @@ -624,17 +624,17 @@ func TestCatalogSegmentPositionInAssembledSystem(t *testing.T) { t.Fatalf("OnRequest fired %d times, want 1", len(seen)) } sys := seen[0].system - if len(sys) != 5 { - t.Fatalf("system has %d segments, want 5 (base, instructions, skills, mcp catalog, hook):\n%q", len(sys), sys) + if len(sys) != 6 { + t.Fatalf("system has %d segments, want 6 (base, tool-batching, instructions, skills, mcp catalog, hook):\n%q", len(sys), sys) } - if sys[0] != "base" || !strings.Contains(sys[1], "instr body") || !strings.Contains(sys[2], "Skill one") { - t.Fatalf("segments 0-2 are not base/instructions/skills:\n%q", sys) + if sys[0] != "base" || !isBatchingSegment(sys[1]) || !strings.Contains(sys[2], "instr body") || !strings.Contains(sys[3], "Skill one") { + t.Fatalf("segments 0-3 are not base/tool-batching/instructions/skills:\n%q", sys) } - if !strings.HasPrefix(sys[3], mcpCatalogHeader) { - t.Fatalf("segment 3 is not the MCP catalog:\n%q", sys[3]) + if !strings.HasPrefix(sys[4], mcpCatalogHeader) { + t.Fatalf("segment 4 is not the MCP catalog:\n%q", sys[4]) } - if sys[4] != "hook seg" { - t.Fatalf("segment 4 is not the hook segment:\n%q", sys[4]) + if sys[5] != "hook seg" { + t.Fatalf("segment 5 is not the hook segment:\n%q", sys[5]) } for _, name := range seen[0].tools { if isMCPToolName(name) { diff --git a/engine/onrequest_test.go b/engine/onrequest_test.go index c386b689..2846f104 100644 --- a/engine/onrequest_test.go +++ b/engine/onrequest_test.go @@ -69,20 +69,23 @@ func TestOnRequestFiresPerTurnWithAssembledSystem(t *testing.T) { } sys := seen[0].system - if len(sys) != 4 { - t.Fatalf("system = %v, want [base, instructions, skills, hook seg]", sys) + if len(sys) != 5 { + t.Fatalf("system = %v, want [base, tool-batching, instructions, skills, hook seg]", sys) } if sys[0] != "base" { t.Errorf("sys[0] = %q, want base", sys[0]) } - if !strings.Contains(sys[1], "instr body") { - t.Errorf("sys[1] = %q, want instructions", sys[1]) + if !isBatchingSegment(sys[1]) { + t.Errorf("sys[1] = %q, want the tool-batching segment", sys[1]) } - if !strings.Contains(sys[2], "one — Skill one") { - t.Errorf("sys[2] = %q, want skills", sys[2]) + if !strings.Contains(sys[2], "instr body") { + t.Errorf("sys[2] = %q, want instructions", sys[2]) } - if sys[3] != "hook seg" { - t.Errorf("sys[3] = %q, want hook seg", sys[3]) + if !strings.Contains(sys[3], "one — Skill one") { + t.Errorf("sys[3] = %q, want skills", sys[3]) + } + if sys[4] != "hook seg" { + t.Errorf("sys[4] = %q, want hook seg", sys[4]) } } diff --git a/engine/session_info_test.go b/engine/session_info_test.go index 69961acc..42507ee8 100644 --- a/engine/session_info_test.go +++ b/engine/session_info_test.go @@ -133,8 +133,8 @@ func TestSessionInfoNothingInjected(t *testing.T) { t.Errorf("plugins must serialize as [], got %q", rawSessionInfoPlugins(t, info)) } // System still carries the base segment. - if len(info.System) != 1 || info.System[0] != "base" { - t.Errorf("system = %v, want [base]", info.System) + if len(info.System) != 2 || info.System[0] != "base" || !isBatchingSegment(info.System[1]) { + t.Errorf("system = %v, want [base, tool-batching]", info.System) } // Effort was never set: report it honestly as EffortUnset ("", the // provider default), not omitted and not an invented level. diff --git a/engine/skills_test.go b/engine/skills_test.go index 662f43f8..e6c9c77e 100644 --- a/engine/skills_test.go +++ b/engine/skills_test.go @@ -30,10 +30,10 @@ func TestSkillsInjectedIntoSystem(t *testing.T) { prov := instrSession(t, Config{WorkDir: work, SkillsDirs: []string{skills}, Instructions: &InstructionsConfig{Disabled: true}}, 1) sys := prov.requests[0].System - if len(sys) != 2 { - t.Fatalf("system = %v, want [base, skills]", sys) + if len(sys) != 3 { + t.Fatalf("system = %v, want [base, tool-batching, skills]", sys) } - seg := sys[1] + seg := sys[2] // Header must instruct reading SKILL.md before use. if !strings.Contains(seg, "read_file") || !strings.Contains(strings.ToLower(seg), "skill.md") { t.Errorf("skills header must mention reading SKILL.md with read_file: %q", seg) @@ -63,20 +63,23 @@ func TestSkillsSegmentOrder(t *testing.T) { prov := instrSession(t, Config{WorkDir: work, SkillsDirs: []string{skills}, Hooks: hooks}, 1) sys := prov.requests[0].System - if len(sys) != 4 { - t.Fatalf("system = %v, want [base, instructions, skills, hook seg]", sys) + if len(sys) != 5 { + t.Fatalf("system = %v, want [base, tool-batching, instructions, skills, hook seg]", sys) } if sys[0] != "base" { t.Errorf("sys[0] = %q, want base", sys[0]) } - if !strings.Contains(sys[1], "instr body") { - t.Errorf("sys[1] = %q, want instructions", sys[1]) + if !isBatchingSegment(sys[1]) { + t.Errorf("sys[1] = %q, want the tool-batching segment", sys[1]) } - if !strings.Contains(sys[2], "one — Skill one") { - t.Errorf("sys[2] = %q, want skills", sys[2]) + if !strings.Contains(sys[2], "instr body") { + t.Errorf("sys[2] = %q, want instructions", sys[2]) } - if sys[3] != "hook seg" { - t.Errorf("sys[3] = %q, want hook seg", sys[3]) + if !strings.Contains(sys[3], "one — Skill one") { + t.Errorf("sys[3] = %q, want skills", sys[3]) + } + if sys[4] != "hook seg" { + t.Errorf("sys[4] = %q, want hook seg", sys[4]) } } @@ -87,11 +90,11 @@ func TestSkillsDefaultDir(t *testing.T) { // nil SkillsDirs uses /.agents/skills when it exists. prov := instrSession(t, Config{WorkDir: work, Instructions: &InstructionsConfig{Disabled: true}}, 1) sys := prov.requests[0].System - if len(sys) != 2 { - t.Fatalf("system = %v, want [base, skills] from default dir", sys) + if len(sys) != 3 { + t.Fatalf("system = %v, want [base, tool-batching, skills] from default dir", sys) } - if !strings.Contains(sys[1], "deflt — Default dir skill") { - t.Errorf("sys[1] = %q, want default-dir skill", sys[1]) + if !strings.Contains(sys[2], "deflt — Default dir skill") { + t.Errorf("sys[2] = %q, want default-dir skill", sys[2]) } } @@ -102,8 +105,8 @@ func TestSkillsEmptySliceDisables(t *testing.T) { // Explicit empty slice disables discovery even though the default exists. prov := instrSession(t, Config{WorkDir: work, SkillsDirs: []string{}, Instructions: &InstructionsConfig{Disabled: true}}, 1) sys := prov.requests[0].System - if len(sys) != 1 || sys[0] != "base" { - t.Errorf("system = %v, want only [base] when skills explicitly disabled", sys) + if len(sys) != 2 || sys[0] != "base" || !isBatchingSegment(sys[1]) { + t.Errorf("system = %v, want [base, tool-batching] when skills are explicitly disabled", sys) } } @@ -116,8 +119,8 @@ func TestSkillsMissingDirNoSegment(t *testing.T) { Instructions: &InstructionsConfig{Disabled: true}, }, 1) sys := prov.requests[0].System - if len(sys) != 1 || sys[0] != "base" { - t.Errorf("system = %v, want only [base] when skills dir missing", sys) + if len(sys) != 2 || sys[0] != "base" || !isBatchingSegment(sys[1]) { + t.Errorf("system = %v, want [base, tool-batching] when the skills dir is missing", sys) } } diff --git a/engine/toolbatching_test.go b/engine/toolbatching_test.go new file mode 100644 index 00000000..80322ad2 --- /dev/null +++ b/engine/toolbatching_test.go @@ -0,0 +1,131 @@ +package engine + +import ( + "context" + "strconv" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// isBatchingSegment reports whether seg is the tool-batching system +// segment. The other segment-layout tests in this package use it so a +// wording change to toolBatchingSegment does not have to touch every one +// of them; only the assertions in this file pin the actual text. +func isBatchingSegment(seg string) bool { + return strings.HasPrefix(seg, "If you intend to call multiple tools") +} + +// batchingSession runs one prompt and returns the assembled system prompt. +func batchingSystem(t *testing.T, cfg Config) []string { + t.Helper() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg.Providers = provider.Registry{"test": prov} + cfg.Model = message.ModelRef{Provider: "test", Model: "m1"} + if cfg.System == nil { + cfg.System = []string{"base"} + } + if cfg.Instructions == nil { + cfg.Instructions = &InstructionsConfig{Disabled: true} + } + if cfg.SkillsDirs == nil { + cfg.SkillsDirs = []string{} + } + s := NewSession(cfg) + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatal(err) + } + return prov.requests[0].System +} + +// TestToolBatchingSegmentPresentByDefault pins the segment's position and +// its content. It sits immediately after the caller's base prompt, ahead of +// project instructions, because it describes how the engine executes tools +// rather than anything about the project. +func TestToolBatchingSegmentPresentByDefault(t *testing.T) { + sys := batchingSystem(t, Config{}) + if len(sys) != 2 { + t.Fatalf("system = %v, want [base, tool-batching]", sys) + } + if sys[0] != "base" { + t.Errorf("sys[0] = %q, want the caller's base prompt first", sys[0]) + } + seg := sys[1] + if !isBatchingSegment(seg) { + t.Fatalf("sys[1] = %q, want the tool-batching segment", seg) + } + // Both halves must survive any future edit. The second is as + // load-bearing as the first: without it a model batches calls whose + // arguments depend on an earlier call's result. + if !strings.Contains(seg, "no dependencies between the calls") { + t.Errorf("segment lost the independence condition: %q", seg) + } + if !strings.Contains(seg, "same message") { + t.Errorf("segment does not say where to put the calls: %q", seg) + } + if !strings.Contains(seg, "MUST wait for previous calls to finish") { + t.Errorf("segment lost the dependency caveat: %q", seg) + } + if !strings.Contains(seg, "up to 8 at a time") { + t.Errorf("segment should name the default cap: %q", seg) + } +} + +// TestToolBatchingSegmentRendersTheRealCap proves the number the model +// reads is the number the executor enforces, not a hardcoded 8. +func TestToolBatchingSegmentRendersTheRealCap(t *testing.T) { + for _, cap := range []int{2, 4, 16} { + sys := batchingSystem(t, Config{ToolConcurrency: cap}) + if len(sys) != 2 { + t.Fatalf("cap %d: system = %v, want the segment present", cap, sys) + } + want := "up to " + strconv.Itoa(cap) + " at a time" + if !strings.Contains(sys[1], want) { + t.Errorf("cap %d: segment does not say %q: %q", cap, want, sys[1]) + } + } +} + +// TestToolBatchingSegmentAbsentWhenSequential is the important negative. +// A session that runs tools one at a time — an operator who set the kill +// switch, or an embedder who set ToolConcurrency 1 — must not be told its +// calls run concurrently, because for that session they do not. +func TestToolBatchingSegmentAbsentWhenSequential(t *testing.T) { + for _, cap := range []int{1, -1} { + sys := batchingSystem(t, Config{ToolConcurrency: cap}) + if len(sys) != 1 || sys[0] != "base" { + t.Errorf("ToolConcurrency %d: system = %v, want only [base] — a sequential session must not be told its calls run concurrently", cap, sys) + } + } +} + +// TestToolBatchingSegmentIsStableAcrossTurns keeps the segment usable as a +// prompt-cache prefix: it must not vary from one request to the next. +func TestToolBatchingSegmentIsStableAcrossTurns(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "one"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "two"}), + }} + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + System: []string{"base"}, + Instructions: &InstructionsConfig{Disabled: true}, + SkillsDirs: []string{}, + }) + for _, p := range []string{"first", "second"} { + if _, err := s.Prompt(context.Background(), p); err != nil { + t.Fatal(err) + } + } + if len(prov.requests) != 2 { + t.Fatalf("requests = %d, want 2", len(prov.requests)) + } + if a, b := prov.requests[0].System[1], prov.requests[1].System[1]; a != b { + t.Errorf("segment changed between turns:\n%q\n%q", a, b) + } +} diff --git a/engine/toolexec.go b/engine/toolexec.go index ead200e8..03200b20 100644 --- a/engine/toolexec.go +++ b/engine/toolexec.go @@ -630,3 +630,31 @@ func (s *Session) runParallelSegment(ctx context.Context, seg batchSegment, outp } wg.Wait() } + +// toolBatchingSegment is the system-prompt segment that tells the model to +// put independent tool calls in ONE message, so this file's executor +// actually has a batch to run in parallel. Without it the executor is +// unclaimed capacity: a model that emits one call per turn never produces +// a batch wider than one, however high the cap is. +// +// It is gated on the session's REAL resolved concurrency, and returns "" +// at 1. A session running strictly sequentially — an operator who set the +// kill switch, or an embedder who set ToolConcurrency 1 — must not be told +// its calls run concurrently, because for that session they do not. +// +// The cap is rendered from s.toolConcurrency rather than hardcoded, so the +// number the model reads is the number the executor enforces. +// +// The second sentence is as load-bearing as the first: it stops the model +// batching calls whose arguments depend on an earlier call's result, which +// no amount of executor correctness can repair. +func (s *Session) toolBatchingSegment() string { + if s.toolConcurrency <= 1 { + return "" + } + return fmt.Sprintf("If you intend to call multiple tools and there are no "+ + "dependencies between the calls, make all of the independent calls in "+ + "the same message: harness runs one message's tool calls concurrently, "+ + "up to %d at a time. Otherwise you MUST wait for previous calls to "+ + "finish first to determine the dependent values.", s.toolConcurrency) +} diff --git a/server/request_test.go b/server/request_test.go index d4cc0e57..f2fa0348 100644 --- a/server/request_test.go +++ b/server/request_test.go @@ -6,6 +6,7 @@ import ( "fmt" "net/http" "net/http/httptest" + "strings" "testing" "time" @@ -117,8 +118,8 @@ func TestRequestMetaJournaled(t *testing.T) { if rm.SystemHash == "" { t.Error("system_hash empty") } - if rm.Segments != 1 { - t.Errorf("segments = %d, want 1 (base only)", rm.Segments) + if rm.Segments != 2 { + t.Errorf("segments = %d, want 2 (base + the engine's tool-batching segment)", rm.Segments) } if rm.SystemLen == 0 { t.Error("system_len = 0") @@ -130,8 +131,8 @@ func TestRequestMetaJournaled(t *testing.T) { t.Errorf("tools = %v, want to include session_info and bash", rm.Tools) } // First appearance of this hash carries the full system. - if len(rm.System) != 1 || rm.System[0] != "base" { - t.Errorf("system = %v, want [base] on first request.meta", rm.System) + if len(rm.System) != 2 || rm.System[0] != "base" || !isBatchingSegment(rm.System[1]) { + t.Errorf("system = %v, want [base, tool-batching] on first request.meta", rm.System) } } @@ -236,8 +237,8 @@ func TestRequestEndpoint(t *testing.T) { if rq.Model != (message.ModelRef{Provider: "test", Model: "m1"}) { t.Errorf("model = %v", rq.Model) } - if len(rq.System) != 1 || rq.System[0] != "base" { - t.Errorf("system = %v, want [base]", rq.System) + if len(rq.System) != 2 || rq.System[0] != "base" || !isBatchingSegment(rq.System[1]) { + t.Errorf("system = %v, want [base, tool-batching]", rq.System) } if !containsName(rq.Tools, "session_info") { t.Errorf("tools = %v, want session_info", rq.Tools) @@ -359,3 +360,11 @@ func TestRequestSnapshotEvictedWithSession(t *testing.T) { t.Errorf("evicted session /request = %d, want 404", resp.StatusCode) } } + +// isBatchingSegment reports whether seg is the engine's tool-batching +// system segment, which every default-configured session carries (see +// engine's toolBatchingSegment). Matched by prefix so a wording change +// does not break these tests; the engine package pins the exact text. +func isBatchingSegment(seg string) bool { + return strings.HasPrefix(seg, "If you intend to call multiple tools") +} From 05739097cec41b418beb45b63aa1e11ba8588e5a Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 06:52:22 -0400 Subject: [PATCH 12/95] feat(server,cmd): surface slow handlers and long GC pauses (#201) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(server,cmd): surface slow handlers and long GC pauses A caller of a served box waited nearly eight seconds for one reply while other boxes answered in milliseconds. The box's own harness logged nothing at all for the whole window: no request line, no turn line, no error. From outside the process there was no way to tell a slow handler from a slow network from a stop-the-world garbage collection pause, and nothing inside it recorded enough to decide afterwards. This adds three threshold-gated WARN lines and no always-on cost. server/timing.go times the mux dispatch. A request this process takes longer than 500ms to answer logs "slow request" with method, route, status, duration_ms, and the caller's X-Request-Id. Every timed route is local work, so half a second is already far outside normal and the line stays rare enough to mean something. The route label is http.Request.Pattern, which the mux fills in during the dispatch, so a session id never reaches a log line and a path that matched no route logs a fixed "unmatched" label rather than caller-chosen text. The request id is dropped unless it is one printable ASCII token within 64 bytes, since it is untrusted input that lands in a log line. The event stream and the wait long-poll are exempt: both run for as long as their caller wants, so timing them would warn for every healthy client. timedWriter forwards Flush, which the event stream requires, and Unwrap. cmd/harness/gcwatch.go makes the pause case say so. A stop-the-world pause stops every goroutine, which is exactly why the incident window had no log lines. gcWatcher samples the runtime's /gc/pauses:seconds histogram every five seconds and warns about a new pause at or past 200ms. It reads runtime/metrics rather than runtime.ReadMemStats: ReadMemStats itself stops the world, so sampling it would add the kind of pause this watcher exists to find. longest_pause_ms is the lower bound of the highest bucket that gained a pause, because a histogram records a range and a larger claim would be invented. The first sample reports nothing, since the counts are cumulative for the process's life. Its goroutine shares inFlightWatchdog's lifecycle: one cancelable context, cancelled when serveCmd returns by any path. server/pprof.go adds the runtime profiles under /debug/pprof/, behind Options.PProf and the `harness serve -pprof` flag, off by default and authed like every other route. /debug/goroutines still needs no flag and stays the first step for a wedged process; these routes add the CPU, heap, block, and mutex profiles that say why it is wedged, for a process already under investigation. Importing net/http/pprof also registers these handlers on http.DefaultServeMux, which this binary never serves, so the authed routes are the only reachable surface. Together the three lines split a multi-second stall: a "slow request" line means this process's own handler was slow; its absence next to a "long gc pause" line means garbage collection; neither, with the process otherwise silent, points at blocking I/O outside a handler, which is what the profiles are for. Deliberate non-changes: no metrics, no tracing, no per-request INFO line, and no behavior change for a server with no Logger, which stays silent. Tests drive an injected clock, so every asserted duration is exact and no test waits on real time. * docs(server,cmd): pin the exempt-route rule and the pause blind spot Three things a reader would otherwise have to work out from the code: why POST /session/{id}/compact is deliberately not exempt from the slow- request warn (it is slow by nature, but rare, and a compaction running for minutes is worth the line), what Unauthenticated means for the profile routes (the same reachability every other route already has), and how wide the pause watcher's threshold blind spot is (this runtime's bucket boundaries around 200ms are 0.167772 and 0.201327, so a pause between 200ms and 201.3ms goes unreported — under-reporting at the edge beats a false pause). * fix(server): make profiling exist only when the flag is on server/pprof.go imported net/http/pprof to borrow its handlers. That package's init registers /debug/pprof/* on http.DefaultServeMux for the whole linked binary, unconditionally, so the profiling surface existed regardless of Options.PProf. The old code reasoned that this binary never serves the default mux, which is true and is not enough: server is an importable package, and any program that links it and serves the default mux -- http.ListenAndServe(addr, nil), an ordinary shape -- published CPU, heap, and trace endpoints with no opt-in, no auth of its own, and no way for Options.PProf to prevent it. A library must not register global HTTP handlers as an import side effect. Borrowing the handlers behind the flag does not fix it: Go runs a package's init on import, so there is no way to take net/http/pprof's functions without its registration. The handlers are therefore written against runtime/pprof and runtime/trace directly, mounted on this server's own mux only when Options.PProf is set, each behind the same bearer check as every other route. The new surface: a plain-text index listing what this server serves, every profile the runtime registers (goroutine, heap, allocs, block, mutex, threadcreate) with ?debug for text and ?gc for a heap collection, the CPU profile and the execution trace with ?seconds, and cmdline. ?seconds is clamped to 1-60 seconds, default 30 -- a caller must not be able to hold a runtime-wide profile lock indefinitely -- and a client disconnect ends the profile early rather than running out an abandoned request. A second concurrent CPU profile or trace is a 409, not a 500: the caller's request is fine, it is just not possible right now. Query parsing goes through this package's own intParam, so a malformed or repeated value is a 400 exactly like every other integer parameter. /debug/pprof/symbol is not served; go tool pprof symbolizes against the binary a profile came from. TestPProf_NotRegisteredOnDefaultServeMux is the regression guard, and it is the one that matters: it asserts the whole linked test binary registers nothing under /debug/pprof on http.DefaultServeMux. Red-verified by adding back a blank net/http/pprof import -- it fails on all six paths, while the pre-existing "404 with the flag off" test through this server's own mux still PASSES, which is exactly why that test could not catch the exposure. * fix(server): answer a refused profile as an error, not a download Two problems found while red-verifying the profiling opt-in. A refused CPU profile or trace kept the download headers of the profile it never produced, so a 409 carried Content-Disposition: attachment; filename="profile" and a browser saved the JSON error as a profile file. The headers cannot simply move after the start call -- a CPU profile writes to the response as samples arrive, so the first write can land before any later header set takes effect -- so the refusal path now removes them explicitly. The auth test hit /debug/pprof/profile and /trace with no seconds parameter. That is correct while the routes are authed, and a trap the moment they are not: red-verifying the missing-auth case ran two real 30-second profiles and hung the suite instead of failing it. Both paths now carry seconds=1, so a dropped auth wrapper fails in about a second. Also adds a test that enumerates every profiling path -- including the unknown-profile and multi-segment cases that fall through to the index handler -- and proves each one 401s without a token. A route added to registerPProf without the auth wrapper would otherwise pass every other test in the file. * test(server): drop a scratch probe file committed by mistake server/px4_test.go was a throwaway probe of serveCPUProfile's refusal path, written while verifying the 409 header fix. A repository-wide staging command swept it into the previous commit. It asserts nothing -- it only logs -- so it is removed rather than kept; the real coverage is TestPProf_ConflictIsCleanJSON. Removed in a follow-up commit rather than by amending the one that added it, so the pushed history is never rewritten. * fix(server): keep profiling out of the slow-request log and the flag oracle Review of the profiling opt-in found six more, none of them in the mechanism itself. A profile ran for exactly as long as its ?seconds asked, and the slow-request warn timed it, so `go tool pprof` against a box logged a 30-second "slow request" every time -- an operator investigating a stall would find their own tooling in the logs they came to read. That is precisely the rule longLivedRoutes already states: a route belongs there when the CALLER sets its duration. GET /debug/pprof/{name} now does. The index stays timed, since it returns at once. The unslashed /debug/pprof took the mux's automatic 308 redirect, issued before any handler ran, so an unauthenticated caller could tell whether profiling was enabled: a redirect with the flag on, a 404 with it off. It is now registered explicitly and behind auth, which makes the two states 401-vs-404 -- the shape every other route in this API already has, and not specific to profiling. The profile tests did not bind on the wait: stubbing sleepForProfile out left every one of them passing, while returning an empty profile. The CPU test now asserts the request took at least the second it asked for (and carries its download header), and a new test proves an already-cancelled context returns at once instead of holding a runtime-wide profile for its full minute. The execution trace had no coverage at all -- separate code from the CPU path -- and now has both its success and its 409. The DefaultServeMux guard covered only the server package's import graph. It now exists in cmd/harness too: the binary links the engine, providers, plugins, MCP and the tools, and any one of them importing net/http/pprof would publish the endpoints for the whole process. Smaller: the index no longer implies ?debug and ?gc apply to every path (cmdline reads neither), and openapi.yaml drops the seconds minimum/maximum it declared -- a schema range promises a 400, and an out-of-range value is clamped. The 400 cases are malformed, empty, and repeated values. --------- Co-authored-by: andybons --- AGENTS.md | 79 ++++++ cmd/harness/gcwatch.go | 139 ++++++++++ cmd/harness/gcwatch_test.go | 146 ++++++++++ cmd/harness/main.go | 18 +- cmd/harness/pprof_defaultmux_test.go | 33 +++ server/openapi.yaml | 72 +++++ server/pprof.go | 230 ++++++++++++++++ server/pprof_test.go | 397 +++++++++++++++++++++++++++ server/server.go | 31 ++- server/timing.go | 133 +++++++++ server/timing_test.go | 252 +++++++++++++++++ 11 files changed, 1528 insertions(+), 2 deletions(-) create mode 100644 cmd/harness/gcwatch.go create mode 100644 cmd/harness/gcwatch_test.go create mode 100644 cmd/harness/pprof_defaultmux_test.go create mode 100644 server/pprof.go create mode 100644 server/pprof_test.go create mode 100644 server/timing.go create mode 100644 server/timing_test.go diff --git a/AGENTS.md b/AGENTS.md index 0e205512..6cf0d835 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2379,6 +2379,85 @@ needing any other side channel to agree on identity. Harness itself never reads this variable — it is a contract between the hub and deployment tooling, documented in `docs/design/fleet-model.md` §8. +## Serve-mode latency diagnostics + +A caller that waits seconds for a reply cannot tell, from outside the +process, whether `harness serve` was slow, the network in front of it was +slow, or the whole process was stopped by garbage collection. Three +threshold-gated WARN lines answer that, and nothing runs always-on. + +- **`slow request`** (`server/timing.go`). `serveTimed` wraps the mux + dispatch in `Server.ServeHTTP` and warns when this process took longer + than `slowRequestThreshold` (500ms) to answer, with `method`, `route`, + `status`, `duration_ms`, and the caller's `X-Request-Id` as + `request_id`. The route is `http.Request.Pattern`, which the mux sets + during the dispatch, so a session id never reaches a log line; a request + that matched no route logs the fixed `unmatched` label, because the path + is caller-controlled. `requestID` drops a header value over 64 bytes or + carrying anything outside one printable ASCII token — it is untrusted + input that lands in a log line. `longLivedRoutes` exempts `GET /event` + and `GET /session/{id}/wait`: both run for as long as their caller + wants, so timing them would warn for every healthy client. Keep that map + in step with any new streaming or long-poll route. +- **`long gc pause`** (`cmd/harness/gcwatch.go`). A stop-the-world pause + stops every goroutine, so the process logs nothing at all while it lasts + and looks exactly like a wedged handler. `gcWatcher` samples the + runtime's `/gc/pauses:seconds` histogram every 5 seconds and warns about + a new pause at or past 200ms. It reads `runtime/metrics`, never + `runtime.ReadMemStats` — `ReadMemStats` itself stops the world, so + sampling it would add the pause this watcher exists to find. + `longest_pause_ms` is the LOWER bound of the highest bucket that gained + a pause, since a histogram records a range. The first sample reports + nothing: the counts are cumulative for the whole process life. Same + lifecycle as `inFlightWatchdog` — one cancelable context, cancelled when + `serveCmd` returns. +- **`/debug/pprof/`** (`server/pprof.go`, `Options.PProf`, `harness serve + -pprof`). OFF by default, and authed like every other route when on. It + is the third step, not the first: `GET /debug/goroutines` needs no flag + and already answers "what is this process blocked on". Turn `-pprof` on + for a process under investigation when the CPU, heap, block, or mutex + profile is what is missing. + - **Never import `net/http/pprof` in this repository.** That package's + `init` registers `/debug/pprof/*` on `http.DefaultServeMux` for the + whole linked binary, so importing it — even to borrow its handler + functions behind this flag — exposes profiling in ANY program that + links `server` and serves the default mux + (`http.ListenAndServe(addr, nil)`), with no opt-in and no way for + `Options.PProf` to prevent it. Go runs a package's `init` on import; + there is no way to take the handlers without the side effect. So + `server/pprof.go` implements them on `runtime/pprof` and + `runtime/trace` directly. + `TestPProf_NotRegisteredOnDefaultServeMux` asserts the absence and is + the regression guard: a 404 test through this server's own mux passes + WITH the bad import and proves nothing. It exists TWICE, in `server/` + and in `cmd/harness/`, because each only covers its own package's + import graph — the binary links the engine, providers, plugins, MCP and + the tools, and any one of them pulling in `net/http/pprof` would + publish the endpoints for the whole process. + - `?seconds=N` on `profile`/`trace` is clamped to 1-60 (default 30); a + malformed or repeated value is a 400, through the same `intParam` every + other integer parameter uses. A second concurrent CPU profile or trace + is a 409, not a 500 — only one of each can run in a process — and the + refusal removes the download headers it had to set before starting, so + the error is JSON rather than a file a browser saves. A client + disconnect ends the profile early rather than holding a runtime-wide + lock for an abandoned request. + - `GET /debug/pprof/{name}` is in `longLivedRoutes` (`server/timing.go`): + a profile runs for exactly as long as the caller's `?seconds` asks, so + timing it would log a 30-second "slow request" every time an operator + ran `go tool pprof` against a box — their own tooling in the logs they + are reading. The index stays timed; it returns at once. + - The UNSLASHED `/debug/pprof` is registered explicitly, behind auth. + Left to the mux it takes an automatic 308 redirect issued before any + handler, which told an unauthenticated caller whether profiling is + enabled. Authed, the two states are 401-vs-404 — the shape every other + route in this API already has. + - `/debug/pprof/symbol` is deliberately not served: `go tool pprof` + symbolizes against the binary a profile came from. + +No metrics, no tracing, no always-on profiling. A new diagnostic in this +area is a threshold-gated log line or it does not land. + ## Startup Speed Rules - Nothing touches network, subprocesses, or disk beyond one config file before first paint. Provider auth validates on first message send, not at boot. diff --git a/cmd/harness/gcwatch.go b/cmd/harness/gcwatch.go new file mode 100644 index 00000000..482dfcfa --- /dev/null +++ b/cmd/harness/gcwatch.go @@ -0,0 +1,139 @@ +package main + +import ( + "context" + "log/slog" + "runtime/metrics" + "time" +) + +// A stop-the-world garbage collection pause stops every goroutine at once: +// the process answers nothing, logs nothing, and looks identical from the +// outside to a wedged handler or a blocking syscall. gcWatcher makes the +// garbage-collection case say so, which by elimination also narrows the +// other two. + +// gcPausesMetric is the runtime's per-pause histogram. It is read through +// runtime/metrics rather than runtime.ReadMemStats: ReadMemStats itself +// stops the world, so sampling it would add the kind of pause this watcher +// exists to find. +const gcPausesMetric = "/gc/pauses:seconds" + +// gcPauseThreshold is the cutoff for the warn below. An ordinary pause is +// well under a millisecond, so 200ms is already a pause a caller can feel. +const gcPauseThreshold = 200 * time.Millisecond + +// gcSampleInterval is how often the watcher re-reads the histogram. The +// counts are cumulative, so a longer interval loses no pause; it only +// delays the report. +const gcSampleInterval = 5 * time.Second + +// longGCPauseMsg is the warn line's message. +const longGCPauseMsg = "long gc pause" + +// gcWatcher samples the runtime's pause histogram and warns about pauses +// past its threshold. read is the histogram source, replaced in tests. +type gcWatcher struct { + logger *slog.Logger + threshold time.Duration + read func() *metrics.Float64Histogram + + prev []uint64 +} + +func newGCWatcher(logger *slog.Logger) *gcWatcher { + return &gcWatcher{logger: logger, threshold: gcPauseThreshold, read: readGCPauses} +} + +// run samples until ctx ends. +func (w *gcWatcher) run(ctx context.Context) { + ticker := time.NewTicker(gcSampleInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + w.sample() + } + } +} + +// sample reads the histogram once and warns about any pause past the +// threshold that the previous sample did not already cover. +func (w *gcWatcher) sample() { + h := w.read() + if h == nil { + return + } + found := newLongPauses(h, w.prev, w.threshold) + w.prev = append(w.prev[:0], h.Counts...) + if found.count == 0 { + return + } + w.logger.Warn(longGCPauseMsg, + "pauses", found.count, + "longest_pause_ms", found.longest.Milliseconds(), + "threshold_ms", w.threshold.Milliseconds(), + "window_ms", gcSampleInterval.Milliseconds(), + ) +} + +// longPauses is what one sample found: how many pauses past the threshold +// are new, and a lower bound on the longest of them. +type longPauses struct { + count uint64 + longest time.Duration +} + +// newLongPauses diffs a pause histogram against the previous sample's +// counts and reports the new pauses at or past threshold. +// +// longest is the LOWER bound of the highest bucket that gained a pause: a +// histogram records a range, so this is the largest value the data proves, +// never a guess at the real pause. +// +// A nil prev (the first sample) reports nothing — the counts are +// cumulative for the whole process life, so a first sample would warn at +// startup about pauses that already happened. A prev of a different length +// also reports nothing, since bucket-by-bucket subtraction across two +// different layouts would invent pauses. +func newLongPauses(h *metrics.Float64Histogram, prev []uint64, threshold time.Duration) longPauses { + var found longPauses + if prev == nil || len(prev) != len(h.Counts) { + return found + } + cutoff := threshold.Seconds() + for i, count := range h.Counts { + if count <= prev[i] { + continue + } + // Buckets[i] is bucket i's lower bound; a bucket qualifies only + // when its whole range is at or past the threshold, so a pause + // under the threshold can never be reported as one past it. The + // cost is a blind spot as wide as the straddling bucket: this + // runtime's boundaries around 200ms are 0.167772 and 0.201327, so + // a pause between 200ms and 201.3ms goes unreported. Under- + // reporting by a millisecond at the edge beats a false pause. + lower := h.Buckets[i] + if lower < cutoff { + continue + } + found.count += count - prev[i] + if d := time.Duration(lower * float64(time.Second)); d > found.longest { + found.longest = d + } + } + return found +} + +// readGCPauses reads the runtime's pause histogram, or nil when this +// runtime does not publish it. +func readGCPauses() *metrics.Float64Histogram { + sample := []metrics.Sample{{Name: gcPausesMetric}} + metrics.Read(sample) + if sample[0].Value.Kind() != metrics.KindFloat64Histogram { + return nil + } + return sample[0].Value.Float64Histogram() +} diff --git a/cmd/harness/gcwatch_test.go b/cmd/harness/gcwatch_test.go new file mode 100644 index 00000000..94b70f7f --- /dev/null +++ b/cmd/harness/gcwatch_test.go @@ -0,0 +1,146 @@ +package main + +import ( + "bytes" + "log/slog" + "math" + "runtime/metrics" + "strings" + "testing" + "time" +) + +// histogram builds a Float64Histogram over the given bucket boundaries. +func histogram(bounds []float64, counts []uint64) *metrics.Float64Histogram { + return &metrics.Float64Histogram{Buckets: bounds, Counts: counts} +} + +// pauseBounds are bucket boundaries in seconds around the 200ms threshold: +// 0-1ms, 1-100ms, 100-200ms, 200-500ms, 500ms-1s, 1s-infinity. +var pauseBounds = []float64{0, 0.001, 0.1, 0.2, 0.5, 1, math.Inf(1)} + +func TestNewLongPauses_CountsOnlyNewPausesPastThreshold(t *testing.T) { + prev := []uint64{10, 5, 2, 0, 0, 0} + // Two new pauses: one at 100-200ms (under the threshold) and one at + // 500ms-1s (over it). + cur := []uint64{10, 5, 3, 0, 1, 0} + + got := newLongPauses(histogram(pauseBounds, cur), prev, 200*time.Millisecond) + if got.count != 1 { + t.Errorf("count = %d, want 1 (the sub-threshold pause must not count)", got.count) + } + if got.longest != 500*time.Millisecond { + t.Errorf("longest = %v, want 500ms (the bucket's lower bound)", got.longest) + } +} + +func TestNewLongPauses_IgnoresPausesAlreadyReported(t *testing.T) { + prev := []uint64{0, 0, 0, 4, 1, 0} + cur := []uint64{0, 0, 0, 4, 1, 0} + + if got := newLongPauses(histogram(pauseBounds, cur), prev, 200*time.Millisecond); got.count != 0 { + t.Errorf("count = %d, want 0: a pause counted in an earlier sample must never be re-reported", got.count) + } +} + +func TestNewLongPauses_FirstSampleReportsNothing(t *testing.T) { + cur := []uint64{3, 9, 1, 2, 0, 0} + + // A nil previous sample is the process's first read. Its counts are + // cumulative for the whole process life, so reporting them would warn + // at startup about pauses that already happened. + if got := newLongPauses(histogram(pauseBounds, cur), nil, 200*time.Millisecond); got.count != 0 { + t.Errorf("count = %d, want 0 on the first sample", got.count) + } +} + +func TestNewLongPauses_ToleratesBucketLayoutChange(t *testing.T) { + prev := []uint64{1, 1} + cur := []uint64{0, 0, 0, 5, 0, 0} + + // A previous sample of a different length cannot be compared bucket by + // bucket. Reporting nothing is the safe answer; a wrong diff would + // invent pauses. + if got := newLongPauses(histogram(pauseBounds, cur), prev, 200*time.Millisecond); got.count != 0 { + t.Errorf("count = %d, want 0 when the bucket layout changed", got.count) + } +} + +func TestNewLongPauses_CountsEveryBucketPastThreshold(t *testing.T) { + prev := []uint64{0, 0, 0, 0, 0, 0} + cur := []uint64{0, 0, 0, 2, 1, 3} + + got := newLongPauses(histogram(pauseBounds, cur), prev, 200*time.Millisecond) + if got.count != 6 { + t.Errorf("count = %d, want 6", got.count) + } + if got.longest != 1*time.Second { + t.Errorf("longest = %v, want 1s", got.longest) + } +} + +// TestGCWatcher_WarnsOnLongPause proves the sampler turns a long pause into +// one warn naming how many pauses landed and how long the longest was. +func TestGCWatcher_WarnsOnLongPause(t *testing.T) { + var logBuf bytes.Buffer + w := &gcWatcher{ + logger: slog.New(slog.NewTextHandler(&logBuf, nil)), + threshold: 200 * time.Millisecond, + read: func() *metrics.Float64Histogram { + return histogram(pauseBounds, []uint64{0, 0, 0, 0, 2, 0}) + }, + } + + w.sample() // first sample: baseline only, no warn + if strings.Contains(logBuf.String(), longGCPauseMsg) { + t.Fatalf("the first sample warned: %s", logBuf.String()) + } + + w.read = func() *metrics.Float64Histogram { + return histogram(pauseBounds, []uint64{0, 0, 0, 0, 2, 1}) + } + w.sample() + + logged := logBuf.String() + if !strings.Contains(logged, longGCPauseMsg) { + t.Fatalf("a new long pause did not warn: %s", logged) + } + for _, want := range []string{"level=WARN", "pauses=1", "longest_pause_ms=1000", "threshold_ms=200"} { + if !strings.Contains(logged, want) { + t.Errorf("warn line is missing %q: %s", want, logged) + } + } +} + +// TestGCWatcher_QuietWithoutLongPauses proves ordinary garbage collection +// logs nothing at all. +func TestGCWatcher_QuietWithoutLongPauses(t *testing.T) { + var logBuf bytes.Buffer + counts := []uint64{5, 4, 0, 0, 0, 0} + w := &gcWatcher{ + logger: slog.New(slog.NewTextHandler(&logBuf, nil)), + threshold: 200 * time.Millisecond, + read: func() *metrics.Float64Histogram { return histogram(pauseBounds, counts) }, + } + + w.sample() + counts = []uint64{9, 12, 3, 0, 0, 0} // more pauses, all short + w.sample() + + if logBuf.Len() != 0 { + t.Errorf("short pauses produced output: %s", logBuf.String()) + } +} + +// TestReadGCPauses_ReadsTheRuntime proves the production reader returns the +// real runtime histogram, so the sampler is wired to something live rather +// than to a metric name the runtime does not publish. +func TestReadGCPauses_ReadsTheRuntime(t *testing.T) { + h := readGCPauses() + if h == nil { + t.Fatalf("readGCPauses returned nil: %q is not published by this runtime", gcPausesMetric) + } + if len(h.Counts)+1 != len(h.Buckets) { + t.Errorf("histogram shape is wrong: %d counts, %d bucket bounds", len(h.Counts), len(h.Buckets)) + } +} diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 0d54af2d..d4010114 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -227,7 +227,8 @@ func usage() { pursue a goal until an evaluator judges it met (exit 0 achieved, 3 not achieved) harness serve [-addr host:port] [-cors-origin origin] [-no-instructions] - [-unauthenticated] [-skills-dir dir ...] [-agent-def-dir dir ...] + [-unauthenticated] [-pprof] [-skills-dir dir ...] + [-agent-def-dir dir ...] serve the HTTP+SSE session API harness plugin probe re-probe configured plugins and refresh the manifest cache @@ -1263,6 +1264,8 @@ func serveCmd(args []string) error { skillDirs = append(skillDirs, v) return nil }) + var enablePProf bool + fs.BoolVar(&enablePProf, "pprof", false, "serve the Go runtime profiles under /debug/pprof/, behind the same bearer check as every other route; off by default") var agentDefDirs []string fs.Func("agent-def-dir", "directory of custom task-tool agent definitions to advertise (repeatable); overrides config agent_defs_dirs", func(v string) error { agentDefDirs = append(agentDefDirs, v) @@ -1379,6 +1382,16 @@ func serveCmd(args []string) error { defer stopWatchdog() go watchdog.run(watchdogCtx) + // A stop-the-world garbage collection pause stops every goroutine, so + // the process logs nothing at all while it lasts — indistinguishable + // from a wedged handler until something reports the pause itself. Same + // lifecycle as the watchdog above: one cancelable context, cancelled + // the moment serveCmd returns by any path. + gcWatch := newGCWatcher(logger) + gcCtx, stopGCWatch := context.WithCancel(context.Background()) + defer stopGCWatch() + go gcWatch.run(gcCtx) + // The event journal owner needs each engine session to report events to // it, so the session wrappers wire OnEvent to the server's Publish. // host is built just below, once srv exists (its ClientAPI is @@ -1563,6 +1576,9 @@ func serveCmd(args []string) error { // comment for why this is the ONE place that ever sets it (server // itself never infers it from RunToken alone). Unauthenticated: unauthenticated, + // PProf: the -pprof opt-in. Off by default; see + // server.Options.PProf. + PProf: enablePProf, // Logger: the same JSON stderr logger built above for the // config-load summary and "serve start" lines now also drives // server.Options.Logger's turn-lifecycle/goal-lifecycle logging diff --git a/cmd/harness/pprof_defaultmux_test.go b/cmd/harness/pprof_defaultmux_test.go new file mode 100644 index 00000000..369ee266 --- /dev/null +++ b/cmd/harness/pprof_defaultmux_test.go @@ -0,0 +1,33 @@ +package main + +import ( + "net/http" + "net/http/httptest" + "testing" +) + +// TestPProf_NotRegisteredOnDefaultServeMux is the server-package guard's +// twin, at the binary's own import graph. The server test proves only that +// nothing on THAT package's graph imports net/http/pprof; this binary links +// far more (the engine, providers, plugins, MCP, the hub, every tool), and +// any one of those pulling in net/http/pprof would publish /debug/pprof/* +// on http.DefaultServeMux for the whole process, outside Options.PProf. +// +// serveCmd sets http.Server.Handler explicitly, so the default mux is not +// served today and this is defense in depth against that changing. It costs +// one map lookup per path. +func TestPProf_NotRegisteredOnDefaultServeMux(t *testing.T) { + for _, path := range []string{ + "/debug/pprof/", + "/debug/pprof/heap", + "/debug/pprof/profile", + "/debug/pprof/cmdline", + "/debug/pprof/trace", + "/debug/pprof/symbol", + } { + req := httptest.NewRequest(http.MethodGet, path, nil) + if _, pattern := http.DefaultServeMux.Handler(req); pattern != "" { + t.Errorf("%s is registered on http.DefaultServeMux as %q; some package this binary links imports net/http/pprof", path, pattern) + } + } +} diff --git a/server/openapi.yaml b/server/openapi.yaml index 4479d194..393dc830 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -2707,6 +2707,78 @@ paths: schema: { type: string } "401": { $ref: "#/components/responses/Unauthorized" } + /debug/pprof/{profile}: + get: + operationId: debugPProf + summary: Go runtime profiles (CPU, heap, block, mutex, and the rest). + description: > + Present ONLY when this instance was started with `harness serve + -pprof`; it 404s otherwise, exactly as any other unmatched path. + Authed like every other route. A diagnostic surface, not a stable + API: the response is a binary pprof profile by default, or plain + text with `?debug=1`. Omit the path segment (`/debug/pprof/`) for a + plain-text listing of the profiles this server serves. Besides the + runtime's own profiles (`goroutine`, `heap`, `allocs`, `block`, + `mutex`, `threadcreate`), `profile` names accepted here are + `profile` (CPU) and `trace`, both taking `?seconds=N` clamped to + 1-60 (default 30), and `cmdline`. A second concurrent CPU profile or + trace answers 409. `symbol` is NOT served: `go tool pprof` + symbolizes against the binary a profile came from. + parameters: + - name: profile + in: path + required: true + schema: { type: string } + description: Profile name, e.g. `goroutine`, `heap`, `block`, `mutex`. + - name: debug + in: query + required: false + schema: { type: integer } + description: > + 1 or 2 for plain text instead of the binary format. Read by the + runtime profiles only; `profile`, `trace`, and `cmdline` ignore + it. + - name: seconds + in: query + required: false + schema: { type: integer } + description: > + Duration for `profile` and `trace`, in seconds. Default 30. A + well-formed value outside 1-60 is CLAMPED into that range, not + rejected — no schema minimum/maximum is declared here for that + reason. A malformed, empty, or repeated value IS a 400. + - name: gc + in: query + required: false + schema: { type: integer } + description: > + 1 runs a garbage collection before reading `heap`. Ignored by + every other profile. + responses: + "200": + description: OK — a pprof profile. + content: + application/octet-stream: + schema: { type: string, format: binary } + text/plain: + schema: { type: string } + "400": + description: A query parameter was malformed or repeated. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "401": { $ref: "#/components/responses/Unauthorized" } + "404": + description: No such profile, or this instance was started without `-pprof`. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "409": + description: A CPU profile or trace is already running in this process. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + /: get: operationId: monitorRootRedirect diff --git a/server/pprof.go b/server/pprof.go new file mode 100644 index 00000000..8647cf6b --- /dev/null +++ b/server/pprof.go @@ -0,0 +1,230 @@ +package server + +import ( + "fmt" + "net/http" + "os" + "runtime" + "runtime/pprof" + "runtime/trace" + "sort" + "strings" + "time" +) + +// Runtime profiles, served on THIS server's mux and only when Options.PProf +// is set. +// +// These handlers are written against runtime/pprof and runtime/trace +// directly, and this package deliberately does NOT import net/http/pprof. +// That package's init registers /debug/pprof/* on http.DefaultServeMux +// unconditionally, for the whole linked binary — so importing it, even to +// borrow its handler functions behind a flag, would expose profiling in any +// program that links this package and serves the default mux +// (http.ListenAndServe(addr, nil), an ordinary shape), with no opt-in and +// no way for Options.PProf to prevent it. A library must not register +// global handlers as an import side effect. +// TestPProf_NotRegisteredOnDefaultServeMux enforces the absence. +// +// Not served: /debug/pprof/symbol. `go tool pprof` symbolizes a profile +// locally against the binary it was taken from, which is how a box profile +// is read anyway. + +// Profile durations for the CPU profile and the execution trace. A duration +// is bounded on both ends: a caller-chosen value must never turn into an +// unbounded profile holding a runtime-wide lock, and a zero-second profile +// would return an empty file that reads as a bug. +const ( + defaultProfileSeconds = 30 * time.Second + minProfileSeconds = 1 * time.Second + maxProfileSeconds = 60 * time.Second +) + +// registerPProf adds the profiling routes to mux, each wrapped by auth. +// routes() calls it only when Options.PProf is set. +func registerPProf(mux *http.ServeMux, auth func(http.HandlerFunc) http.HandlerFunc) { + mux.HandleFunc("GET /debug/pprof/", auth(handlePProfIndex)) + mux.HandleFunc("GET /debug/pprof/{name}", auth(handlePProfNamed)) + // The bare path, WITHOUT the trailing slash, is registered explicitly + // and behind auth. Left to the mux it gets an automatic 308 redirect to + // the slashed form, issued before any handler — so an unauthenticated + // caller saw a redirect here and a 404 with the flag off, learning + // whether profiling is enabled without holding the token. Answering it + // under auth makes the two states 401-vs-404, the same shape every + // other route in this API already has. + mux.HandleFunc("GET /debug/pprof", auth(handlePProfIndex)) +} + +// handlePProfIndex lists the profiles this server can serve. It answers the +// collection path, with or without the trailing slash; a DEEPER path that +// reached here matched no profile route (the {name} wildcard matches one +// segment), so it is a 404 rather than a redirect to this listing. +func handlePProfIndex(w http.ResponseWriter, r *http.Request) { + if p := r.URL.Path; p != "/debug/pprof/" && p != "/debug/pprof" { + writeErr(w, http.StatusNotFound, "no such profile") + return + } + names := []string{"profile", "trace", "cmdline"} + for _, p := range pprof.Profiles() { + names = append(names, p.Name()) + } + sort.Strings(names) + + var b strings.Builder + b.WriteString("profiles:\n") + for _, name := range names { + fmt.Fprintf(&b, "\t/debug/pprof/%s\n", name) + } + b.WriteString("\nprofile and trace take ?seconds=N (") + fmt.Fprintf(&b, "%d-%d, clamped, default %d)\n", int(minProfileSeconds.Seconds()), int(maxProfileSeconds.Seconds()), int(defaultProfileSeconds.Seconds())) + b.WriteString("a runtime profile takes ?debug=1 or ?debug=2 for text; heap also takes ?gc=1\n") + b.WriteString("cmdline takes neither\n") + b.WriteString("symbolize with `go tool pprof `\n") + + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.Write([]byte(b.String())) //nolint:errcheck // best-effort diagnostic write +} + +// handlePProfNamed serves one profile: the CPU profile, the execution +// trace, the process command line, or any profile the runtime registers +// (goroutine, heap, allocs, block, mutex, threadcreate). +func handlePProfNamed(w http.ResponseWriter, r *http.Request) { + switch name := r.PathValue("name"); name { + case "profile": + serveCPUProfile(w, r) + case "trace": + serveTrace(w, r) + case "cmdline": + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.Write([]byte(strings.Join(os.Args, "\x00"))) //nolint:errcheck // best-effort diagnostic write + default: + serveRuntimeProfile(w, r, name) + } +} + +// serveRuntimeProfile writes one registered profile. debug=0 (the default) +// is the binary pprof format `go tool pprof` reads; debug=1 or 2 is the +// human-readable text form. +func serveRuntimeProfile(w http.ResponseWriter, r *http.Request, name string) { + p := pprof.Lookup(name) + if p == nil { + writeErr(w, http.StatusNotFound, "no such profile") + return + } + query := r.URL.Query() + debug, ok := intParam(w, query, "debug") + if !ok { + return + } + gc, ok := intParam(w, query, "gc") + if !ok { + return + } + // heap?gc=1 runs a collection first, so the profile reflects live data + // rather than whatever survived the last cycle. + if name == "heap" && gc > 0 { + runtime.GC() + } + if debug > 0 { + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + } else { + setProfileDownloadHeaders(w, name) + } + p.WriteTo(w, debug) //nolint:errcheck // best-effort diagnostic write; nothing to do once headers are sent +} + +// serveCPUProfile profiles the CPU for the requested duration. Only one CPU +// profile can run in a process, so a second concurrent request is a 409 +// rather than a 500: the caller's own action is fine, it is just not +// possible right now. +func serveCPUProfile(w http.ResponseWriter, r *http.Request) { + d, ok := profileSeconds(w, r) + if !ok { + return + } + // Set the download headers BEFORE starting: a CPU profile writes to w + // as samples arrive, so the first write can land before any later + // header set would take effect. The refusal path therefore has to take + // them back off — otherwise a browser saves the JSON error as a profile + // file. + setProfileDownloadHeaders(w, "profile") + if err := pprof.StartCPUProfile(w); err != nil { + clearProfileDownloadHeaders(w) + writeErr(w, http.StatusConflict, "a CPU profile is already running: "+err.Error()) + return + } + sleepForProfile(r, d) + pprof.StopCPUProfile() +} + +// serveTrace records an execution trace for the requested duration. Like +// the CPU profile, a concurrent trace is a 409. +func serveTrace(w http.ResponseWriter, r *http.Request) { + d, ok := profileSeconds(w, r) + if !ok { + return + } + setProfileDownloadHeaders(w, "trace") + if err := trace.Start(w); err != nil { + clearProfileDownloadHeaders(w) + writeErr(w, http.StatusConflict, "a trace is already running: "+err.Error()) + return + } + sleepForProfile(r, d) + trace.Stop() +} + +// sleepForProfile waits d, or returns early when the client disconnects. +// An abandoned request must not keep a runtime-wide profile running for its +// full duration. +func sleepForProfile(r *http.Request, d time.Duration) { + timer := time.NewTimer(d) + defer timer.Stop() + select { + case <-timer.C: + case <-r.Context().Done(): + } +} + +// setProfileDownloadHeaders marks a binary profile as a file, so a browser +// saves it instead of rendering bytes. +func setProfileDownloadHeaders(w http.ResponseWriter, name string) { + w.Header().Set("Content-Type", "application/octet-stream") + w.Header().Set("Content-Disposition", fmt.Sprintf(`attachment; filename="%s"`, name)) + // The body length is unknown until the profile finishes, and a stale + // Content-Length would truncate it. + w.Header().Set("X-Content-Type-Options", "nosniff") +} + +// clearProfileDownloadHeaders undoes setProfileDownloadHeaders, for a +// profile that never started. writeJSON sets its own Content-Type, so only +// the download markers need removing. +func clearProfileDownloadHeaders(w http.ResponseWriter) { + w.Header().Del("Content-Disposition") + w.Header().Del("X-Content-Type-Options") +} + +// profileSeconds is the ?seconds= duration. An absent parameter takes the +// default. A present one is parsed by this package's own intParam, so a +// malformed or repeated value is a 400 like everywhere else in this API +// rather than a silently substituted default. A well-formed value outside +// the bounds is CLAMPED, not rejected: "profile for an hour" is a coherent +// intention, just not one this server will hold a runtime-wide lock for. +func profileSeconds(w http.ResponseWriter, r *http.Request) (time.Duration, bool) { + query := r.URL.Query() + if !query.Has("seconds") { + return defaultProfileSeconds, true + } + n, ok := intParam(w, query, "seconds") + if !ok { + return 0, false + } + d := time.Duration(n) * time.Second + if d < minProfileSeconds { + return minProfileSeconds, true + } + if d > maxProfileSeconds { + return maxProfileSeconds, true + } + return d, true +} diff --git a/server/pprof_test.go b/server/pprof_test.go new file mode 100644 index 00000000..3d1e41d3 --- /dev/null +++ b/server/pprof_test.go @@ -0,0 +1,397 @@ +package server + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "runtime/pprof" + "runtime/trace" + "strings" + "testing" + "time" +) + +// TestPProf_NotRegisteredOnDefaultServeMux is the load-bearing assertion of +// the profiling opt-in. Importing net/http/pprof registers /debug/pprof/* +// on http.DefaultServeMux from that package's own init, so a program that +// merely LINKS this one and serves the default mux — +// http.ListenAndServe(addr, nil), an ordinary shape — would expose +// profiling with no opt-in at all, and Options.PProf could not prevent it. +// A library must not register global handlers as an import side effect. +// +// This asserts the whole linked test binary registers nothing there, which +// is only true if no package on this one's import graph pulls in +// net/http/pprof. +func TestPProf_NotRegisteredOnDefaultServeMux(t *testing.T) { + for _, path := range []string{ + "/debug/pprof/", + "/debug/pprof/heap", + "/debug/pprof/profile", + "/debug/pprof/cmdline", + "/debug/pprof/trace", + "/debug/pprof/symbol", + } { + req := httptest.NewRequest(http.MethodGet, path, nil) + if _, pattern := http.DefaultServeMux.Handler(req); pattern != "" { + t.Errorf("%s is registered on http.DefaultServeMux as %q; profiling must exist only on this server's own mux, behind Options.PProf", path, pattern) + } + } +} + +// TestPProf_OffByDefault proves the routes do not exist on this server +// unless a caller asks for them. +func TestPProf_OffByDefault(t *testing.T) { + srv := newServer(t, t.TempDir(), &scriptedProvider{name: "test"}, 0) + + for _, path := range []string{"/debug/pprof/", "/debug/pprof/heap", "/debug/pprof/cmdline", "/debug/pprof/profile?seconds=1"} { + req := httptest.NewRequest(http.MethodGet, path, nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + if w.Code != http.StatusNotFound { + t.Errorf("%s answered %d with profiling off, want 404", path, w.Code) + } + } +} + +// TestPProf_EnabledRequiresAuth proves an enabled profiling route is behind +// the same bearer check as every other route. A profile carries function +// names and allocation sites from the running process. +func TestPProf_EnabledRequiresAuth(t *testing.T) { + srv := pprofServer(t) + + for _, path := range []string{"/debug/pprof/", "/debug/pprof/heap", "/debug/pprof/cmdline", "/debug/pprof/profile?seconds=1", "/debug/pprof/trace?seconds=1"} { + req := httptest.NewRequest(http.MethodGet, path, nil) + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + if w.Code != http.StatusUnauthorized { + t.Errorf("unauthenticated %s answered %d, want 401", path, w.Code) + } + } +} + +// TestPProf_EnabledServesProfiles proves an authorized read returns real +// profile data, in both the text and the binary form. +func TestPProf_EnabledServesProfiles(t *testing.T) { + srv := pprofServer(t) + + t.Run("text goroutine profile", func(t *testing.T) { + w := pprofGet(t, srv, "/debug/pprof/goroutine?debug=1") + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200", w.Code) + } + if !strings.Contains(w.Body.String(), "goroutine profile") { + t.Errorf("body is not a goroutine profile: %q", firstLine(w.Body.String())) + } + }) + + t.Run("binary heap profile", func(t *testing.T) { + w := pprofGet(t, srv, "/debug/pprof/heap") + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200", w.Code) + } + // A binary pprof profile is gzip-framed: it starts with the gzip + // magic bytes, which is what `go tool pprof` expects to read. + if b := w.Body.Bytes(); len(b) < 2 || b[0] != 0x1f || b[1] != 0x8b { + t.Errorf("body is not a gzip-framed pprof profile: % x", b[:min(4, len(b))]) + } + if ct := w.Header().Get("Content-Type"); ct != "application/octet-stream" { + t.Errorf("Content-Type = %q, want application/octet-stream", ct) + } + }) + + t.Run("index lists the profiles", func(t *testing.T) { + w := pprofGet(t, srv, "/debug/pprof/") + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200", w.Code) + } + for _, want := range []string{"goroutine", "heap", "profile", "trace"} { + if !strings.Contains(w.Body.String(), want) { + t.Errorf("index does not mention %q: %s", want, w.Body.String()) + } + } + }) + + t.Run("cmdline", func(t *testing.T) { + if w := pprofGet(t, srv, "/debug/pprof/cmdline"); w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200", w.Code) + } + }) + + t.Run("unknown profile is a 404", func(t *testing.T) { + if w := pprofGet(t, srv, "/debug/pprof/no-such-profile"); w.Code != http.StatusNotFound { + t.Errorf("answered %d, want 404", w.Code) + } + }) +} + +// TestPProf_CPUProfileRunsForTheRequestedTime proves the CPU profile path +// works end to end AND actually runs for the duration it was asked for. +// Without the elapsed-time assertion this test passes with the wait stubbed +// out, which would return an empty profile — the profile's own content is +// the thing the duration produces. +func TestPProf_CPUProfileRunsForTheRequestedTime(t *testing.T) { + srv := pprofServer(t) + + start := time.Now() + w := pprofGet(t, srv, "/debug/pprof/profile?seconds=1") + elapsed := time.Since(start) + + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200: %s", w.Code, w.Body.String()) + } + if elapsed < minProfileSeconds { + t.Errorf("returned after %v; a seconds=1 profile must run for at least %v", elapsed, minProfileSeconds) + } + if b := w.Body.Bytes(); len(b) < 2 || b[0] != 0x1f || b[1] != 0x8b { + t.Errorf("body is not a gzip-framed CPU profile: % x", b[:min(4, len(b))]) + } + if cd := w.Header().Get("Content-Disposition"); !strings.Contains(cd, `filename="profile"`) { + t.Errorf("Content-Disposition = %q, want a profile attachment", cd) + } +} + +// TestPProf_ProfileStopsWhenTheClientDisconnects proves an abandoned +// request does not keep a runtime-wide profile running for its full +// duration. It asks for the longest allowed profile against an +// already-cancelled context: the handler must return at once. +func TestPProf_ProfileStopsWhenTheClientDisconnects(t *testing.T) { + for _, path := range []string{"/debug/pprof/profile?seconds=60", "/debug/pprof/trace?seconds=60"} { + t.Run(path, func(t *testing.T) { + srv := pprofServer(t) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() // the client is already gone + + req := httptest.NewRequest(http.MethodGet, path, nil).WithContext(ctx) + req.Header.Set("Authorization", "Bearer secret-run-token") + + done := make(chan struct{}) + go func() { + srv.ServeHTTP(httptest.NewRecorder(), req) + close(done) + }() + select { + case <-done: + case <-time.After(10 * time.Second): + t.Fatal("the handler is still profiling after the client disconnected") + } + }) + } +} + +// TestPProf_TraceServesAndRefusesAConcurrentOne covers the execution trace +// on both paths: a real trace, and the 409 a second concurrent one gets. +// The trace path is separate code from the CPU profile's, so passing tests +// for one prove nothing about the other. +func TestPProf_TraceServesAndRefusesAConcurrentOne(t *testing.T) { + srv := pprofServer(t) + + t.Run("serves a trace", func(t *testing.T) { + w := pprofGet(t, srv, "/debug/pprof/trace?seconds=1") + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200: %s", w.Code, w.Body.String()) + } + if w.Body.Len() == 0 { + t.Error("trace body is empty") + } + }) + + t.Run("refuses a concurrent trace", func(t *testing.T) { + if err := trace.Start(io.Discard); err != nil { + t.Fatalf("starting the conflicting trace: %v", err) + } + t.Cleanup(trace.Stop) + + w := pprofGet(t, srv, "/debug/pprof/trace?seconds=1") + if w.Code != http.StatusConflict { + t.Fatalf("answered %d, want 409", w.Code) + } + if got := w.Header().Get("Content-Disposition"); got != "" { + t.Errorf("a 409 carries Content-Disposition %q", got) + } + }) +} + +// TestPProf_CPUProfileConflictIsA409 proves a second concurrent CPU profile +// is refused with a real status instead of a 500 or a hang. Only one CPU +// profile can run in a process at a time. +func TestPProf_CPUProfileConflictIsA409(t *testing.T) { + srv := pprofServer(t) + + if err := pprof.StartCPUProfile(io.Discard); err != nil { + t.Fatalf("starting the conflicting profile: %v", err) + } + t.Cleanup(pprof.StopCPUProfile) + + if w := pprofGet(t, srv, "/debug/pprof/profile?seconds=1"); w.Code != http.StatusConflict { + t.Errorf("answered %d, want 409", w.Code) + } +} + +// TestProfileSeconds pins the duration parsing: no value a caller can send +// turns into an unbounded profile, and a malformed one is a 400 like every +// other integer parameter in this API rather than a silent default. +func TestProfileSeconds(t *testing.T) { + t.Run("clamped and defaulted", func(t *testing.T) { + cases := map[string]time.Duration{ + "?": defaultProfileSeconds, // absent + "?seconds=5": 5 * time.Second, + "?seconds=0": minProfileSeconds, + "?seconds=9999": maxProfileSeconds, + "?other=1": defaultProfileSeconds, + } + for query, want := range cases { + req := httptest.NewRequest(http.MethodGet, "/debug/pprof/profile"+query, nil) + got, ok := profileSeconds(httptest.NewRecorder(), req) + if !ok { + t.Errorf("profileSeconds(%q) rejected a valid request", query) + continue + } + if got != want { + t.Errorf("profileSeconds(%q) = %v, want %v", query, got, want) + } + } + }) + + t.Run("malformed is a 400", func(t *testing.T) { + for _, query := range []string{"?seconds=not-a-number", "?seconds=", "?seconds=-3", "?seconds=1&seconds=2"} { + req := httptest.NewRequest(http.MethodGet, "/debug/pprof/profile"+query, nil) + w := httptest.NewRecorder() + if _, ok := profileSeconds(w, req); ok { + t.Errorf("profileSeconds(%q) accepted a malformed value", query) + continue + } + if w.Code != http.StatusBadRequest { + t.Errorf("profileSeconds(%q) answered %d, want 400", query, w.Code) + } + } + }) +} + +func pprofServer(t *testing.T) *Server { + t.Helper() + return newServer(t, t.TempDir(), &scriptedProvider{name: "test"}, 0, func(o *Options) { + o.PProf = true + }) +} + +func pprofGet(t *testing.T, srv *Server, path string) *httptest.ResponseRecorder { + t.Helper() + req := httptest.NewRequest(http.MethodGet, path, nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + return w +} + +func firstLine(s string) string { + if i := strings.IndexByte(s, '\n'); i >= 0 { + return s[:i] + } + return s +} + +// TestPProf_ConflictIsCleanJSON proves a refused profile answers as an +// error, not as a truncated download. The download headers are set BEFORE +// the profile starts — a CPU profile writes to the response as samples +// arrive, so they cannot be set after — which means the refusal path has to +// take them back off. +func TestPProf_ConflictIsCleanJSON(t *testing.T) { + srv := pprofServer(t) + if err := pprof.StartCPUProfile(io.Discard); err != nil { + t.Fatalf("starting the conflicting profile: %v", err) + } + t.Cleanup(pprof.StopCPUProfile) + + w := pprofGet(t, srv, "/debug/pprof/profile?seconds=1") + if w.Code != http.StatusConflict { + t.Fatalf("answered %d, want 409", w.Code) + } + if got := w.Header().Get("Content-Disposition"); got != "" { + t.Errorf("a 409 carries Content-Disposition %q; a browser would save the error as a profile file", got) + } + if ct := w.Header().Get("Content-Type"); !strings.HasPrefix(ct, "application/json") { + t.Errorf("Content-Type = %q, want application/json", ct) + } + if !strings.Contains(w.Body.String(), `"error"`) { + t.Errorf("body is not an error object: %s", w.Body.String()) + } +} + +// TestPProf_EveryRegisteredRouteIsAuthed enumerates the profiling routes the +// mux actually holds and proves each one 401s without a token. A route added +// to registerPProf without the auth wrapper would pass every other test in +// this file. +func TestPProf_EveryRegisteredRouteIsAuthed(t *testing.T) { + srv := pprofServer(t) + // profile and trace carry seconds=1: if a regression drops the auth + // wrapper, this test must FAIL in a second rather than hang for the + // 30-second default duration of two real profiles. + paths := []string{ + "/debug/pprof/", + "/debug/pprof/goroutine", + "/debug/pprof/heap", + "/debug/pprof/allocs", + "/debug/pprof/block", + "/debug/pprof/mutex", + "/debug/pprof/threadcreate", + "/debug/pprof/cmdline", + "/debug/pprof/profile?seconds=1", + "/debug/pprof/trace?seconds=1", + "/debug/pprof/no-such-profile", + "/debug/pprof/a/b", + } + for _, path := range paths { + req := httptest.NewRequest(http.MethodGet, path, nil) + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + if w.Code != http.StatusUnauthorized { + t.Errorf("unauthenticated %s answered %d, want 401", path, w.Code) + } + if strings.Contains(w.Body.String(), "profiles:") { + t.Errorf("unauthenticated %s leaked the profile index", path) + } + } +} + +// TestPProf_BarePathDoesNotLeakTheFlagPreAuth proves an unauthenticated +// caller cannot learn whether profiling is enabled from the unslashed path. +// Left to the mux, that path gets an automatic redirect issued BEFORE any +// handler, which answers the question with no token at all. +func TestPProf_BarePathDoesNotLeakTheFlagPreAuth(t *testing.T) { + req := func() *http.Request { return httptest.NewRequest(http.MethodGet, "/debug/pprof", nil) } + + on := httptest.NewRecorder() + pprofServer(t).ServeHTTP(on, req()) + if on.Code != http.StatusUnauthorized { + t.Errorf("with profiling ON, an unauthenticated /debug/pprof answered %d, want 401", on.Code) + } + + off := httptest.NewRecorder() + newServer(t, t.TempDir(), &scriptedProvider{name: "test"}, 0).ServeHTTP(off, req()) + if off.Code != http.StatusNotFound { + t.Errorf("with profiling OFF, /debug/pprof answered %d, want 404", off.Code) + } + // 401-vs-404 is the same shape every authed route in this API has, and + // is not specific to profiling. A REDIRECT here would be, which is + // what this test exists to prevent. + for _, code := range []int{http.StatusMovedPermanently, http.StatusPermanentRedirect, http.StatusTemporaryRedirect, http.StatusFound} { + if on.Code == code { + t.Errorf("an unauthenticated /debug/pprof was redirected (%d), answering the question before auth", code) + } + } +} + +// TestPProf_BarePathServesTheIndexWhenAuthed proves closing that leak did +// not cost the convenience of the unslashed path. +func TestPProf_BarePathServesTheIndexWhenAuthed(t *testing.T) { + w := pprofGet(t, pprofServer(t), "/debug/pprof") + if w.Code != http.StatusOK { + t.Fatalf("answered %d, want 200", w.Code) + } + if !strings.Contains(w.Body.String(), "profiles:") { + t.Errorf("body is not the profile index: %s", w.Body.String()) + } +} diff --git a/server/server.go b/server/server.go index 031939dd..f6b699dd 100644 --- a/server/server.go +++ b/server/server.go @@ -343,6 +343,27 @@ type Options struct { // embedding this package (e.g. tests) that wants no such logging // simply leaves this zero. Logger *slog.Logger + // PProf registers the net/http/pprof routes under /debug/pprof/, + // behind the same bearer check as every other route. Off by default: + // a profile exposes function names and allocation sites, and a CPU + // profile costs sampling overhead for as long as it runs, so an + // operator turns this on for a process being investigated rather than + // for every process. /debug/goroutines (above) needs no opt-in and + // stays the first step for a wedged process; these routes add the CPU, + // heap, block, and mutex profiles that say WHY it is wedged. + // + // These routes carry the same auth as every other route, which means + // Unauthenticated exposes them to anything that can reach the listen + // address, exactly like /session and /debug/goroutines. Enable this + // only where that reachability is already the trusted gate the + // Unauthenticated opt-in asserts. + // + // This flag is the ONLY way profiling becomes reachable: server/pprof.go + // implements the handlers on runtime/pprof directly and never imports + // net/http/pprof, whose init would register /debug/pprof/* on + // http.DefaultServeMux for the whole linked binary regardless of this + // field. See that file's header. + PProf bool } // Server implements http.Handler for the harness serve API. @@ -350,6 +371,10 @@ type Server struct { opts Options mux *http.ServeMux + // now is the clock serveTimed measures with. Always time.Now in + // production; a test replaces it to make a duration exact. + now func() time.Time + // wg tracks in-flight runPrompt goroutines. They are decoupled from their // HTTP handlers (the 202 returns immediately), so http.Server.Shutdown does // not wait for them; Drain does, via this group. @@ -822,6 +847,7 @@ func New(opts Options) (*Server, error) { waiters: make(map[*waiter]struct{}), closing: make(chan struct{}), sessMgr: sessMgr, + now: time.Now, } // SetExternalRunner before anything else touches sessMgr: it is what // makes a SessionManager-initiated resume turn on a ROOT session go @@ -938,6 +964,9 @@ func (s *Server) routes() { // inspecting a wedged box in environments where signaling/exec-ing into // the process is awkward or unavailable. mux.HandleFunc("GET /debug/goroutines", s.auth(s.handleGoroutines)) + if s.opts.PProf { + registerPProf(mux, s.auth) + } if s.opts.MonitorPage != nil { mux.HandleFunc("GET /monitor", s.handleMonitor) mux.HandleFunc("GET /monitor/", s.handleMonitor) @@ -991,7 +1020,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusNotFound, "no such session") return } - s.mux.ServeHTTP(w, r) + s.serveTimed(w, r) } // emptySessionIDPrefix is the one path shape isEmptySessionIDPath matches: diff --git a/server/timing.go b/server/timing.go new file mode 100644 index 00000000..534956d5 --- /dev/null +++ b/server/timing.go @@ -0,0 +1,133 @@ +package server + +import ( + "net/http" + "time" +) + +// Per-request timing for the serve API. A caller that waits seconds for a +// reply cannot tell, from the outside, whether this process was slow or +// the network in front of it was. A line here says this process was. + +// slowRequestThreshold is the cutoff for the warn below. Every timed route +// is local work — a session read, an enqueue, a goal write — so half a +// second is already far outside normal and keeps the line rare. A var, not +// a const, so a test can drive the quiet path; production never +// reassigns it. +var slowRequestThreshold = 500 * time.Millisecond + +// slowRequestMsg is the warn line's message. One fixed string, so a log +// search finds every slow request. +const slowRequestMsg = "slow request" + +// unmatchedRoute labels a request that matched no route. The path is +// caller-controlled, so it never reaches a log line: a fixed label bounds +// both what a caller can write into the log and how many distinct route +// values exist. +const unmatchedRoute = "unmatched" + +// longLivedRoutes are the routes that run for as long as their caller +// wants: the event stream and the wait long-poll. A duration means nothing +// for them, so they are never timed — otherwise every healthy client would +// produce a warn. +// +// GET /debug/pprof/ (the index) is deliberately NOT here: it lists profile +// names and returns at once, so a slow one is a real finding. +// +// POST /session/{id}/compact is deliberately NOT here. It runs a model call +// synchronously, so it can exceed the threshold on a healthy server, but it +// is an explicit, rare call and a compaction that runs for minutes is worth +// the line. Add a route here only when its duration is set by the CALLER, +// not by the work. +var longLivedRoutes = map[string]bool{ + "GET /event": true, + "GET /session/{id}/wait": true, + // A profile runs for exactly as long as its ?seconds asks (pprof.go), + // so its duration is the caller's own choice — the same reason the two + // routes above are here. Without this, `go tool pprof` against a box + // would log a 30-second "slow request" every time, and the operator + // investigating a stall would be reading their own tooling. + "GET /debug/pprof/{name}": true, +} + +// maxRequestIDLen bounds the caller-supplied X-Request-Id a log line +// echoes. +const maxRequestIDLen = 64 + +// serveTimed dispatches to the mux and warns when this process took longer +// than slowRequestThreshold to answer. +func (s *Server) serveTimed(w http.ResponseWriter, r *http.Request) { + start := s.now() + tw := &timedWriter{ResponseWriter: w, status: http.StatusOK} + s.mux.ServeHTTP(tw, r) + elapsed := s.now().Sub(start) + if elapsed <= slowRequestThreshold { + return + } + // r.Pattern is set by the mux during the dispatch above, so it is read + // here rather than before it. + route := r.Pattern + if route == "" { + route = unmatchedRoute + } + if longLivedRoutes[route] { + return + } + attrs := []any{ + "method", r.Method, + "route", route, + "status", tw.status, + "duration_ms", elapsed.Milliseconds(), + } + if id := requestID(r); id != "" { + attrs = append(attrs, "request_id", id) + } + s.logWarn(slowRequestMsg, attrs...) +} + +// requestID returns the caller's X-Request-Id when it is a single +// printable ASCII token within maxRequestIDLen bytes, else "". The header +// is untrusted input that lands in a log line, so anything that could +// forge a field or a line break is dropped whole. +func requestID(r *http.Request) string { + id := r.Header.Get("X-Request-Id") + if id == "" || len(id) > maxRequestIDLen { + return "" + } + for i := 0; i < len(id); i++ { + if id[i] <= ' ' || id[i] > '~' { + return "" + } + } + return id +} + +// timedWriter records the status code for the line above. It forwards +// Flush, which the event stream requires, and Unwrap, so an +// http.ResponseController still reaches the underlying writer. +type timedWriter struct { + http.ResponseWriter + status int + wroteHeader bool +} + +func (w *timedWriter) WriteHeader(status int) { + if !w.wroteHeader { + w.status = status + w.wroteHeader = true + } + w.ResponseWriter.WriteHeader(status) +} + +func (w *timedWriter) Write(b []byte) (int, error) { + w.wroteHeader = true + return w.ResponseWriter.Write(b) +} + +func (w *timedWriter) Flush() { + if f, ok := w.ResponseWriter.(http.Flusher); ok { + f.Flush() + } +} + +func (w *timedWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter } diff --git a/server/timing_test.go b/server/timing_test.go new file mode 100644 index 00000000..03824f87 --- /dev/null +++ b/server/timing_test.go @@ -0,0 +1,252 @@ +package server + +import ( + "bytes" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" +) + +// stepClock is a manual clock a test advances explicitly, so a measured +// duration is an exact number rather than a race against real time. +type stepClock struct { + mu sync.Mutex + t time.Time + d time.Duration +} + +// now returns the current time and advances the clock by d, so each +// timestamp a handler path reads differs from the last by exactly d. +func (c *stepClock) now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + now := c.t + c.t = c.t.Add(c.d) + return now +} + +// newSlowServer builds a server whose clock makes every request appear to +// take d, with logger as the log sink. +func newSlowServer(t *testing.T, logger *slog.Logger, d time.Duration) *Server { + t.Helper() + srv := newServer(t, t.TempDir(), &scriptedProvider{name: "test"}, 0, func(o *Options) { + o.Logger = logger + }) + srv.now = (&stepClock{t: time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC), d: d}).now + return srv +} + +// TestSlowRequest_WarnsWithRouteAndDuration proves a request the harness +// itself was slow to handle logs one WARN naming the route, the status, +// and the duration. This is what separates "the harness handler was slow" +// from "the network in front of it was slow". +func TestSlowRequest_WarnsWithRouteAndDuration(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 900*time.Millisecond) + + req := httptest.NewRequest(http.MethodGet, "/session", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + srv.ServeHTTP(httptest.NewRecorder(), req) + + line := findLogLine(t, logBuf.String(), slowRequestMsg) + for _, want := range []string{"level=WARN", "method=GET", "status=200", "duration_ms=900"} { + if !hasLogField(line, want) { + t.Errorf("slow request line is missing %q: %s", want, line) + } + } + // The route carries a space, so slog quotes it as one field. + if !strings.Contains(line, `route="GET /session"`) { + t.Errorf("slow request line is missing the route: %s", line) + } +} + +// TestSlowRequest_CarriesRequestID proves the caller's X-Request-Id rides +// the line, so one harness-side line joins the calling service's own log +// for the same request. +func TestSlowRequest_CarriesRequestID(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 900*time.Millisecond) + + req := httptest.NewRequest(http.MethodGet, "/session", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + req.Header.Set("X-Request-Id", "req_01abc") + srv.ServeHTTP(httptest.NewRecorder(), req) + + if line := findLogLine(t, logBuf.String(), slowRequestMsg); !hasLogField(line, "request_id=req_01abc") { + t.Errorf("slow request line dropped the caller's request id: %s", line) + } +} + +// TestSlowRequest_RejectsUnusableRequestID proves a caller-supplied id that +// is too long, or carries anything outside a printable single token, never +// reaches a log line. The id is untrusted input. +func TestSlowRequest_RejectsUnusableRequestID(t *testing.T) { + for name, id := range map[string]string{ + "too long": strings.Repeat("x", maxRequestIDLen+1), + "space": "req 01abc", + "newline": "req_01abc\nlevel=ERROR msg=forged", + "non ascii": "req_\x00abc", + } { + t.Run(name, func(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 900*time.Millisecond) + + req := httptest.NewRequest(http.MethodGet, "/session", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + req.Header.Set("X-Request-Id", id) + srv.ServeHTTP(httptest.NewRecorder(), req) + + if line := findLogLine(t, logBuf.String(), slowRequestMsg); strings.Contains(line, "request_id=") { + t.Errorf("an unusable request id reached the log line: %s", line) + } + }) + } +} + +// TestSlowRequest_QuietUnderThreshold proves a fast request logs nothing. +// The warn only means something if it stays rare. +func TestSlowRequest_QuietUnderThreshold(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), slowRequestThreshold) + + req := httptest.NewRequest(http.MethodGet, "/session", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + srv.ServeHTTP(httptest.NewRecorder(), req) + + if strings.Contains(logBuf.String(), slowRequestMsg) { + t.Errorf("a request AT the threshold warned; only one PAST it may: %s", logBuf.String()) + } +} + +// TestSlowRequest_SkipsLongLivedRoutes proves the streaming and blocking +// routes never warn. Both run for as long as their caller wants, so timing +// them would log a warn for every healthy client. +func TestSlowRequest_SkipsLongLivedRoutes(t *testing.T) { + for _, route := range []string{"GET /event", "GET /session/{id}/wait", "GET /debug/pprof/{name}"} { + if !longLivedRoutes[route] { + t.Errorf("route %q must be exempt from slow-request logging", route) + } + } + // The profile index is NOT exempt: it returns at once, so a slow one is + // a real finding. + if longLivedRoutes["GET /debug/pprof/"] { + t.Error(`"GET /debug/pprof/" must stay timed: it is a listing, not a profile`) + } + + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 30*time.Second) + + req := httptest.NewRequest(http.MethodGet, "/session/ses_missing/wait?until=idle", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + srv.ServeHTTP(httptest.NewRecorder(), req) + + if strings.Contains(logBuf.String(), slowRequestMsg) { + t.Errorf("a long-lived route warned: %s", logBuf.String()) + } +} + +// TestSlowRequest_UnmatchedPathLogsFixedRoute proves an unrouted path logs +// a fixed label, never the caller's own path. A caller controls the path, +// so logging it verbatim would let a caller choose what a log line says. +func TestSlowRequest_UnmatchedPathLogsFixedRoute(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 900*time.Millisecond) + + req := httptest.NewRequest(http.MethodGet, "/no/such/route", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + srv.ServeHTTP(httptest.NewRecorder(), req) + + line := findLogLine(t, logBuf.String(), slowRequestMsg) + if !hasLogField(line, "route="+unmatchedRoute) { + t.Errorf("unmatched path did not log the fixed route label: %s", line) + } + if strings.Contains(line, "/no/such/route") { + t.Errorf("unmatched path leaked into the log line: %s", line) + } +} + +// TestSlowRequest_NilLoggerStaysSilent proves a server with no Logger +// keeps its current behavior: no output, no panic. +func TestSlowRequest_NilLoggerStaysSilent(t *testing.T) { + srv := newSlowServer(t, nil, 30*time.Second) + srv.opts.Logger = nil + + req := httptest.NewRequest(http.MethodGet, "/session", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("expected 200, got %d", w.Code) + } +} + +// TestSlowRequest_PreservesFlusher proves the timing wrapper keeps the +// response writer's Flusher, which the event stream requires. A wrapper +// that hides it turns every stream into a 500. +func TestSlowRequest_PreservesFlusher(t *testing.T) { + rec := httptest.NewRecorder() + tw := &timedWriter{ResponseWriter: rec, status: http.StatusOK} + if _, ok := any(tw).(http.Flusher); !ok { + t.Fatal("timedWriter does not implement http.Flusher") + } + tw.WriteHeader(http.StatusAccepted) + if tw.status != http.StatusAccepted { + t.Errorf("timedWriter recorded status %d, want 202", tw.status) + } + if _, err := tw.Write([]byte("x")); err != nil { + t.Fatalf("write: %v", err) + } + if rec.Code != http.StatusAccepted || rec.Body.String() != "x" { + t.Errorf("timedWriter did not pass the response through: %d %q", rec.Code, rec.Body.String()) + } +} + +// findLogLine returns the one log line carrying msg. +func findLogLine(t *testing.T, logged, msg string) string { + t.Helper() + for _, line := range strings.Split(logged, "\n") { + if strings.Contains(line, "msg="+msg+" ") || strings.Contains(line, `msg="`+msg+`"`) { + return line + } + } + t.Fatalf("no log line with msg %q in: %s", msg, logged) + return "" +} + +// hasLogField reports whether line carries pair as a whole space-delimited +// field, so "duration_ms=900" cannot satisfy an assertion on "ms=900". +func hasLogField(line, pair string) bool { + for _, field := range strings.Fields(line) { + if field == pair { + return true + } + } + return false +} + +// TestSlowRequest_SkipsProfileRoutes proves a profile the caller asked to +// run for N seconds does not log a slow-request warn. An operator running +// `go tool pprof` against a stalled process must not have their own tooling +// appear in the logs they are reading. +func TestSlowRequest_SkipsProfileRoutes(t *testing.T) { + var logBuf bytes.Buffer + srv := newSlowServer(t, slog.New(slog.NewTextHandler(&logBuf, nil)), 30*time.Second) + srv.opts.PProf = true + srv.routes() + + req := httptest.NewRequest(http.MethodGet, "/debug/pprof/goroutine?debug=1", nil) + req.Header.Set("Authorization", "Bearer secret-run-token") + w := httptest.NewRecorder() + srv.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("profile read answered %d, want 200", w.Code) + } + if strings.Contains(logBuf.String(), slowRequestMsg) { + t.Errorf("a profile route warned: %s", logBuf.String()) + } +} From caa1f82d808acb62fc55266dee3b718b3eca98f8 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 07:44:06 -0400 Subject: [PATCH 13/95] fix(engine): judge a page's final record by the fold's own rule (#198) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): judge a page's final record by the fold's own rule A regression hunt over the backward-scanner rewrite found three ways the walk and the fold that numbers it could disagree about the same journal. Each shifts every sequence number in a page, which is worse than failing: a client gets messages under seqs that do not describe them. The final line is the one a crash can leave torn, and the walk judged it with json.Valid. The fold decodes it with a typed shape and drops whatever fails. Those are different rules. A record like {"type":123,...} is valid JSON and fails the typed decode, so the fold dropped it and the walk counted it. The walk now reads that line whole and decodes it exactly as the fold does. A message record with no body is tolerated by the fold on the final line and NOT counted. The walk counted it. It now checks for the body before counting, and the classifier reports that fact. A line whose leading whitespace fills the prefix window is not a blank line. The walk skipped it on the strength of a bounded prefix, where the forward scanner trims the whole line before deciding. It now reads the line before skipping it. The record-type fast path also stopped being a format rule. It reads one key, which answers for every record this package writes, and now declines rather than fails when a record carries its fields in another order — the caller then decodes the whole line the way the fold does. One residual is documented rather than closed: a record with a SECOND top-level "type" key, which encoding/json resolves last-wins while the prefix scan reads the first. No writer here emits duplicate keys, a journal has one writer by contract, and detecting it would cost a full lex of every record a walk passes — the exact cost the prefix exists to avoid. A byte count cannot substitute, because every message part carries a "type" of its own. Verification: three guards, all red-verified. The final-record table drives a torn line whose prefix parses a type, a valid-JSON record that fails the typed decode, and a bodyless message record, and requires the page to report exactly what the fold counted; against the old rules it fails with "message record at offset 872". The whitespace test fails with "lines = [first]" when the blank check trusts the prefix. The field-order test fails with "corrupt record" without the fallback. * fix(engine): classify a final record with the fold's own type The previous commit claimed to judge a page's final record by the fold's own rule and did not. It classified with a slim {Type, Message} shape, while the fold decodes the line into indexRecord — a type that also type-checks usage, goal, prompt, task_spawn, and compact payloads. So the two disagreed on a whole class of records. A final line carrying a malformed usage field is valid JSON and fails indexRecord's decode: the fold drops it as a torn write and never counts it. The slim shape ignored usage entirely, accepted the record, and counted a message the index did not — a phantom that displaced a real message and shifted every sequence number the page reported. A cross-model review reproduced it: four durable messages by the index, and a page whose first real message was pushed off the front by a record that was never completely written. decodeRecordHeadFull now decodes into indexRecord itself. The two agree by construction rather than by a list of fields someone has to keep in step, and the doc comment says so, because the next slimmer shape would reintroduce exactly this. The oracle had the same hole, and that is why the guard did not catch it. It decided the torn-final question with its own convenient shape. It now decides that question from the journal FORMAT — store.go's record, the writer's own type — so it blesses a final line only if that line was completely written. An oracle derived from the format catches a reader that drifts from it; one derived from a subset of the fields agrees with the drift. Verification: the final-record table grows five cases — a malformed usage value, a malformed usage field, a numeric message id, a malformed goal payload, a malformed compact payload — and requires the page to report exactly what the fold counted, naming any record it serves that the fold dropped. Against the slim shape it fails with "the page served a record the fold dropped: msg_ghost", and the reported sequence shows the real first message pushed off the front. * fix(engine): classify a page record by decoding it, not by guessing A second review round found two more ways the page walk and the fold that numbers it disagreed, both in the prefix classifier, and both producing a phantom message under a real sequence number — worse than an error, because a client cannot tell. A record with a SECOND top-level "type" key resolves last-wins for encoding/json, which is what the fold uses, and first-wins for a scan that stops at the first key. The previous commit called that an accepted residual. It is not acceptable: it serves a real message under the wrong seq, and "no writer here emits duplicate keys" is an argument about producers, not about what a reader does when it meets one. A message record with NO body was tested for with a substring search for the message key, which a nested key satisfies. Worse, the argument that such a record cannot exist mid-file was wrong. The full fold tolerates the shape on a final line, and the INCREMENTAL write-path fold applies only each new record and never revisits an earlier one — so a crash that leaves a bodyless record last, followed by an ordinary resumed turn, leaves that record mid-file with a current, usable sidecar over it. That is a reachable production state, not a hypothetical. Both are gone because the classifier no longer guesses. It decodes into indexRecord, the fold's own type, for every record — so the two agree by construction on which records parse, which duplicate key wins, and what counts as a body. That costs a large record's bytes when a walk passes it, which the previous commit avoided. The trade is deliberate: no bounded read can answer these questions, and a page that numbers messages wrongly is not worth making fast. What remains is the saving that matters for a real journal of thousands of small records — a record whose bytes already fit in the prefix is decoded from them with no second read, and the block window still reads the walked span about once. A large record is read and scanned but never MATERIALIZED: indexRecord carries indexMessage, which decodes a message's identity and skips its parts. Verification: two guards, both red-verified against the restored heuristic. The duplicate-key table fails with "page ids = [msg_ghost msg_2], want the durable sequence [msg_1 msg_2]" — the phantom displacing the real first message. The crash-tail test builds the reachable state by its real route: it hand-writes a journal ending in a bodyless record, LOADS it, runs an ordinary turn so the incremental fold flushes a current sidecar over the now-mid-file record, and requires the page to start at seq 1 with the real message; against the substring test it fails with "message record at offset 243". The cost guard is now stated in the terms that survive: a walk across a 120-record journal must not read its span more than once. * fix(engine): ask one question about a journal's final line A third review round found the index fold and the LOADER disagreeing about the same final line, which puts the fold's own count at odds with the session it claims to describe. The fold decodes indexRecord, a deliberately narrow shape that ignores most of a record's fields. LoadSession decodes record, the type the writer marshals. So a final line carrying a malformed tool_result, mcp_tools, or task_tool_names value — valid JSON, wrong shape for a field the fold does not read — was dropped by the loader as a crash mid-write and COUNTED by the fold. The index then reported a message the session does not have, and a page numbered against that index served it. finalRecordComplete (store.go) is now the one question every reader asks about a final line: does it decode as a record, the definition of the format. The full fold, the page walk, and the page's fold path all call it, so a half-written line is dropped by all of them or by none. It is deliberately not applied to a non-final line. There, a line that fails this check is corruption mid-file: LoadSession refuses the session outright, and a reader that folds anyway offers a degraded view of an unloadable journal rather than miscounting a loadable one. Also filed, not fixed: issue #199, foldedPage serving the wrong record when one journal repeats a message id. It needs a hand-written journal to reach — ids are per message and a journal has one writer — and fixing it means teaching indexFold to carry each surviving message's record ordinal, which does not belong in a change about counting the same records. The call site now points at the issue. Verification: three cases added to the final-record table (malformed task_tool_names, tool_result, mcp_tools), and the table now asserts against the loader itself, not just against the fold — LoadSession's message count must equal the index's. Red-verified: without the rule the guard fails with "the fold counted 5 durable messages, want 4". --------- Co-authored-by: andybons --- engine/index.go | 17 +- engine/messagepage.go | 207 ++++++++++++--------- engine/messagepage_test.go | 364 +++++++++++++++++++++++++++++-------- engine/store.go | 23 +++ 4 files changed, 448 insertions(+), 163 deletions(-) diff --git a/engine/index.go b/engine/index.go index c675e172..ddafe5e3 100644 --- a/engine/index.go +++ b/engine/index.go @@ -591,7 +591,22 @@ func sessionIndexPath(dir, id string) string { // is an error. func foldJournalBytes(data []byte) (indexFold, error) { var f indexFold - err := scanLog(data, func(rec indexRecord, line int, isLast bool) error { + err := scanLogRaw(data, func(raw []byte, line int, isLast bool) error { + var rec indexRecord + if err := json.Unmarshal(raw, &rec); err != nil { + if isLast { + return errTruncatedFinalRecord // crash mid-write, ignore + } + return fmt.Errorf("corrupt record at line %d: %v", line, err) + } + if isLast && !finalRecordComplete(raw) { + // The narrow shape above decoded, but the line is not a whole + // record: a crash mid-write left a field this fold does not + // read in a state the format does not allow. LoadSession drops + // it, so this fold must too, or the index counts a message the + // session does not have. + return errTruncatedFinalRecord + } if err := f.applyIndexRecord(rec, isLast); err != nil { return fmt.Errorf("%w at line %d", err, line) } diff --git a/engine/messagepage.go b/engine/messagepage.go index 4db49657..4b2bbdd4 100644 --- a/engine/messagepage.go +++ b/engine/messagepage.go @@ -276,65 +276,32 @@ func tailPage(src io.ReaderAt, logSize, size int64, total, lo, hi int) ([]messag compacted := false err := scanLogBackward(src, logSize, size, func(line logLine, isTail bool) (bool, error) { - // Classify from the line's PREFIX. Walking back to an older page - // passes every record after it, and reading each of those whole — - // a 20 MB image blob or tool result among them — to learn a type - // string and drop it is the cost this endpoint exists to avoid. - // - // The type is the first field of every record this package writes - // (see record), so a prefix answers it. A prefix that does not - // parse is inconclusive rather than corrupt, and the line is read - // whole before any verdict. - if isTail { - // The newest line is the one a crash can leave torn, and a - // TORN line's prefix still parses a type — the type is its - // first field. Read this one whole and require the whole thing - // to parse, exactly as the fold that numbered this page did. - // Otherwise a record the index never counted would shift every - // seq in the page. - whole, err := line.All() - if err != nil { - return false, err - } - if !json.Valid(bytes.TrimSpace(whole)) { - return true, nil - } - } - raw := line.Prefix() - head, ok := decodeRecordHead(raw) - if !ok && !line.Complete() { - // Inconclusive rather than corrupt: a prefix can end mid-key. - whole, err := line.All() - if err != nil { - return false, err - } - raw = whole - head, ok = decodeRecordHead(raw) + head, ok, err := classifyRecord(line, isTail) + if err != nil { + return false, err } if !ok { - // The prefix scan reads ONE key, so it answers only for a - // record whose first field is the type — which is every record - // this package writes today (see record, store.go). A record - // with another field order is not corrupt, so fall back to a - // decode that finds the type wherever it sits. The fast path - // stays an optimization rather than a format requirement. - whole, err := line.All() - if err != nil { - return false, err - } - var slim struct { - Type string `json:"type"` - } - if err := json.Unmarshal(bytes.TrimSpace(whole), &slim); err != nil || slim.Type == "" { - return false, errors.New("corrupt record") - } - head = slim.Type + // A torn final record, which the fold that numbered this page + // dropped too. Anything else would have failed that fold, so + // the index this page reads would not exist. + return true, nil } - switch head { + switch head.Type { case recCompact: compacted = true return false, nil case recMessage: + if !head.hasMessage { + // A message record with no body. No fold counts one: the + // full fold tolerates the shape on a final line and skips + // it, and the incremental write-path fold marks itself + // broken and skips it too — and because that fold never + // revisits an earlier record, a journal CAN carry a + // bodyless record that is no longer final and still have a + // usable index. Counting it here would shift every seq in + // the page by one. + return true, nil + } if cur >= lo && cur <= hi { whole, err := line.All() if err != nil { @@ -343,7 +310,7 @@ func tailPage(src io.ReaderAt, logSize, size int64, total, lo, hi int) ([]messag var rec struct { Message *message.Message `json:"message"` } - if err := json.Unmarshal(whole, &rec); err != nil || rec.Message == nil { + if err := json.Unmarshal(bytes.TrimSpace(whole), &rec); err != nil || rec.Message == nil { return false, fmt.Errorf("message record at offset %d: %v", line.start, err) } msg := *rec.Message @@ -354,10 +321,6 @@ func tailPage(src io.ReaderAt, logSize, size int64, total, lo, hi int) ([]messag msg.Normalize() out = append(out, msg) } - // A message record with no message body would make this count - // wrong. It cannot reach here: the index that numbered this - // page folds such a record as an error, so a journal carrying - // one has no index and this path never runs for it. cur-- } return cur >= lo, nil @@ -378,33 +341,87 @@ func tailPage(src io.ReaderAt, logSize, size int64, total, lo, hi int) ([]messag return out, true, nil } -// decodeRecordHead reads a record's type from the START of its bytes, -// without decoding the rest. ok is false when the bytes do not (yet) parse -// as an object whose first key is "type", which for a PREFIX means -// inconclusive — read more — and for a whole line means corrupt. +// recordHead is what a page walk needs to know about a record it is not +// going to carry: its type, and whether a message record has a body. +type recordHead struct { + Type string + hasMessage bool +} + +// classifyRecord decides what a line is, reading as little of it as it can +// WITHOUT guessing. // -// Type is the first field of every record this package writes (see record, -// store.go), so this returns after one key and one value. -func decodeRecordHead(raw []byte) (recordType string, ok bool) { - dec := json.NewDecoder(bytes.NewReader(bytes.TrimSpace(raw))) - tok, err := dec.Token() - if err != nil || tok != json.Delim('{') { - return "", false - } - key, err := dec.Token() - if err != nil { - return "", false - } - name, isString := key.(string) - if !isString || name != "type" { - return "", false +// It decodes into indexRecord — the type the fold decodes into — so the two +// agree by construction on every question: which records parse, which key +// wins when one is repeated, and whether a message record has a body. A +// classifier that answered any of those from a cheaper signal answered a +// different question, and a page numbered by one rule and walked by +// another serves real messages under wrong sequence numbers. +// +// The saving is that a record whose bytes already fit in the prefix — every +// ordinary record — is decoded from those bytes, with no second read. A +// record larger than the prefix window is read whole. It is still never +// MATERIALIZED: indexRecord carries indexMessage, which decodes a message's +// identity and skips its parts, so walking past a 20 MB image blob costs a +// scan of its bytes and no allocation of its content. +// +// An earlier revision classified from the prefix alone: the first key for +// the type, and a substring search for the message body. Both were +// unsound. A second top-level "type" key beyond the window resolves +// last-wins for the fold and first-wins for a prefix scan, and a nested +// "message" key made a bodyless record look body-bearing. Each produced a +// phantom message under a real sequence number. Reading a large record is +// the price of never doing that; the block window below is what keeps an +// ordinary journal cheap. +// +// ok is false only for a line that does not parse AND is the file's final +// line — a crash mid-write, which the forward scanner drops and this walk +// drops with it. A non-final line that does not parse is an error. +func classifyRecord(line logLine, isTail bool) (recordHead, bool, error) { + raw := line.Prefix() + if !line.Complete() { + whole, err := line.All() + if err != nil { + return recordHead{}, false, err + } + raw = whole + } + head, parsed := decodeRecordHeadFull(raw) + if isTail { + // The final line answers to the FORMAT, not to this walk's narrower + // shape: finalRecordComplete is the one question every reader asks + // about it (store.go), so the fold that numbered this page, the + // loader, and this walk all drop the same half-written record. + if !parsed || !finalRecordComplete(raw) { + return recordHead{}, false, nil + } + return head, true, nil } - value, err := dec.Token() - if err != nil { - return "", false + if !parsed { + return recordHead{}, false, errors.New("corrupt record") } - recordType, isString = value.(string) - return recordType, isString + return head, true, nil +} + +// decodeRecordHeadFull decodes a whole record line into indexRecord — the +// SAME type the fold decodes into (see foldSessionJournal's scanLog call) — +// so parsed is false exactly where the fold's own decode fails, and +// hasMessage is exactly the fold's own test for a body. +// +// The type must stay indexRecord, not a slimmer shape that happens to carry +// the two fields this returns. The fold's tolerance is a property of EVERY +// field it type-checks: a record whose usage, goal, prompt, or compact +// payload has the wrong JSON shape fails that decode. A slimmer shape here +// ignores those fields, accepts the record, and counts a message the index +// never counted — a phantom that displaces a real message and shifts every +// seq in the page. Sharing the fold's type makes the two agree by +// construction rather than by a list of fields someone has to keep in step. +func decodeRecordHeadFull(raw []byte) (recordHead, bool) { + var rec indexRecord + if err := json.Unmarshal(bytes.TrimSpace(raw), &rec); err != nil { + return recordHead{}, false + } + return recordHead{Type: rec.Type, hasMessage: rec.Message != nil}, true } // foldedPage is the general path, for a journal that carries at least one @@ -439,6 +456,15 @@ func decodeRecordHead(raw []byte) (recordType string, ok bool) { func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { var fold indexFold // lineByID aliases data; it never copies a record. + // + // Known gap, issue #199: it keeps the FIRST line carrying each id, so a + // journal that repeats a message id — one occurrence folded away by a + // compact record, the other surviving — serves the wrong record's + // content under a right sequence number. Not reachable from this + // package's own writer (ids are per message, one writer per journal), + // and fixing it properly means teaching indexFold to carry each + // surviving message's record ordinal, so it is filed rather than + // patched here. lineByID := make(map[string][]byte) err := scanLogRaw(data, func(line []byte, n int, isLast bool) error { var rec indexRecord @@ -448,6 +474,11 @@ func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { } return fmt.Errorf("corrupt record at line %d: %v", n, err) } + if isLast && !finalRecordComplete(line) { + // Same rule as the fold and the tail walk: a final line that is + // not a whole record was never completely written. + return errTruncatedFinalRecord + } if err := fold.applyIndexRecord(rec, isLast); err != nil { return fmt.Errorf("%w at line %d", err, n) } @@ -610,7 +641,19 @@ func scanLogBackward(src io.ReaderAt, end, size int64, fn func(line logLine, isT } line := logLine{src: src, start: lineStart, length: length, prefix: prefix} lineEnd = lineStart - 1 // step over the newline itself - if len(bytes.TrimSpace(prefix)) > 0 { + blank := len(bytes.TrimSpace(prefix)) == 0 + if blank && !line.Complete() { + // A prefix of whitespace does not make the LINE blank: a valid + // record can begin with more leading space than the prefix + // holds, and the forward scanner trims the whole line before + // deciding. Read it before skipping it. + whole, err := line.All() + if err != nil { + return err + } + blank = len(bytes.TrimSpace(whole)) == 0 + } + if !blank { cont, err := fn(line, first) first = false if err != nil || !cont { diff --git a/engine/messagepage_test.go b/engine/messagepage_test.go index 575197b6..2dac748b 100644 --- a/engine/messagepage_test.go +++ b/engine/messagepage_test.go @@ -67,6 +67,24 @@ func wholeSequence(t *testing.T, dir, id string) []string { } `json:"summary"` } `json:"compact"` } + if i == len(lines)-1 { + // The final line is the one a crash can leave torn, and the + // journal FORMAT decides whether it survived: a line that does + // not decode as one of this package's records (store.go's + // record — the writer's own type, and the definition of the + // format) was never completely written, whatever its prefix + // happens to parse as. A reader must not count it. + // + // The rule is deliberately taken from the format rather than + // from either reader: a slim shape that checks only the fields + // a test cares about accepts a record with, say, a malformed + // usage field, and the oracle would then bless a phantom + // message that no real reader should serve. + var whole record + if err := json.Unmarshal([]byte(line), &whole); err != nil { + continue + } + } if err := json.Unmarshal([]byte(line), &rec); err != nil { if i == len(lines)-1 { continue // a torn final line, which no reader counts @@ -990,100 +1008,50 @@ func (c *pageByteCounter) ReadAt(p []byte, off int64) (int, error) { return n, err } -// TestTailPageDoesNotReadRecordsItOnlyClassifies is the byte-level cost -// guard for the tail path. Walking back to an older page passes every -// record after it, and pulling each of those whole — a 20 MB image blob or -// tool result among them — to learn a type string and drop it is the cost -// this endpoint exists to avoid. Decoding less is not enough if the bytes -// are read anyway. -func TestTailPageDoesNotReadRecordsItOnlyClassifies(t *testing.T) { +// TestTailPageReadsItsSpanOnce is the byte-level cost guard for the tail +// path. A backward walk must read the span between the page and the end of +// the file — the line boundaries are in it, and there is no way to count +// records without finding them — but it must read that span ABOUT ONCE. An +// earlier revision read a fresh 64 KiB block per line, so a journal of +// small records was read several times over: a boundary 200 bytes away +// cost a block. +// +// This is the property that matters for a real journal, which is thousands +// of small records. The separate case of one enormous record is covered by +// TestTailPageDecodesOnlyThePage: such a record IS read when the walk +// passes it, because classifying it any other way means guessing, but its +// parts are never decoded. +func TestTailPageReadsItsSpanOnce(t *testing.T) { dir := t.TempDir() - sess := pagedSession(t, dir, 3) // six small durable messages + sess := pagedSession(t, dir, 60) // 120 small durable messages path := filepath.Join(dir, sess.ID+".jsonl") - - // One record far larger than the peek window, then a small one after - // it so the big record is never the tail (a torn tail is read whole by - // design). - const bigBytes = logLinePeekBytes * 40 - beforeBig, err := os.Stat(path) - if err != nil { - t.Fatal(err) - } - f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + ix, err := ReadSessionIndex(dir, sess.ID) if err != nil { t.Fatal(err) } - if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_big","role":"user","parts":[{"type":"text","text":"` + - strings.Repeat("Z", bigBytes) + "\"}]}}\n"); err != nil { - t.Fatal(err) - } - if _, err := f.WriteString(`{"type":"model","model":"test/m2"}` + "\n"); err != nil { - t.Fatal(err) - } - f.Close() - ix, err := ReadSessionIndex(dir, sess.ID) + jf, err := os.Open(path) if err != nil { t.Fatal(err) } - if ix.DurableMessages != 7 { - t.Fatalf("index counts %d durable messages, want 7", ix.DurableMessages) - } - - read := func(lo, hi int) (int64, []message.Message) { - t.Helper() - jf, err := os.Open(path) - if err != nil { - t.Fatal(err) - } - defer jf.Close() - fi, err := jf.Stat() - if err != nil { - t.Fatal(err) - } - counter := &pageByteCounter{f: jf} - msgs, ok, err := tailPage(counter, ix.LogSize, fi.Size(), ix.DurableMessages, lo, hi) - if err != nil || !ok { - t.Fatalf("tailPage([%d,%d]) = ok %v, err %v", lo, hi, ok, err) - } - return counter.bytes, msgs - } - - // A page of the two OLDEST messages walks past the big record. It must - // not pull its body: the whole journal is bigger than the big record, - // and a walk that read every record whole would exceed it. - deepBytes, deep := read(1, 2) - if len(deep) != 2 { - t.Fatalf("deep page returned %d messages, want 2", len(deep)) - } - for _, m := range deep { - if m.ID == "msg_big" { - t.Fatal("the deep page carried the record it should only have classified") - } - } - // A backward walk must read the span between the page and the end of - // the file: that is where the line boundaries are, and there is no way - // to count records without finding them. What it must NOT do is read - // that span and then pull the big record's body a SECOND time to - // classify it. So the bound is the span, with room to spare, not twice - // it. - final, err := os.Stat(path) + defer jf.Close() + fi, err := jf.Stat() if err != nil { t.Fatal(err) } - span := final.Size() - beforeBig.Size() - if deepBytes > span*3/2 { - t.Errorf("a page walking past the big record read %d bytes for a %d-byte span: the record's body is being read to classify it", deepBytes, span) + counter := &pageByteCounter{f: jf} + // The two OLDEST messages: the walk crosses the whole journal. + msgs, ok, err := tailPage(counter, ix.LogSize, fi.Size(), ix.DurableMessages, 1, 2) + if err != nil || !ok { + t.Fatalf("tailPage = ok %v, err %v", ok, err) } - - // The page that DOES carry it reads it whole, which is what makes the - // number above meaningful rather than a scan that skipped everything. - newestBytes, newest := read(7, 7) - if len(newest) != 1 || newest[0].ID != "msg_big" { - t.Fatalf("newest page = %v, want the big record", idsOf(newest)) + if len(msgs) != 2 { + t.Fatalf("returned %d messages, want 2", len(msgs)) } - if newestBytes < bigBytes { - t.Errorf("the page carrying the big record read only %d bytes, want at least %d", newestBytes, bigBytes) + // Reading the span once, plus a bounded prefix per line, is the target. + // Two passes over the file is the regression this catches. + if counter.bytes > fi.Size()*3/2 { + t.Errorf("a walk across a %d-byte journal read %d bytes: the span is being read more than once", fi.Size(), counter.bytes) } } @@ -1115,3 +1083,239 @@ func TestTailPageHandlesAnUnusualFieldOrder(t *testing.T) { t.Errorf("total = %d, want 2", page.Total) } } + +// TestTailPageMatchesTheFoldOnOddFinalRecords: the final line is the one a +// crash can leave torn, and the walk must judge it by the SAME rule the +// fold that numbered the page used. Three shapes make that concrete: a +// torn line whose prefix still parses a type, a line that is valid JSON but +// fails a typed decode, and a message record with no body — the fold drops +// all three and counts none of them, so the walk must not count them +// either. Counting one shifts every seq in the page. +func TestTailPageMatchesTheFoldOnOddFinalRecords(t *testing.T) { + for name, tail := range map[string]string{ + "torn line with a parseable type": `{"type":"message","message":{"id":"msg_torn`, + "valid JSON, wrong type shape": `{"type":123,"message":{"id":"msg_x"}}`, + "message record with no body": `{"type":"message"}`, + // The fold decodes EVERY field it type-checks, so a record whose + // usage, goal, or message id has the wrong JSON shape fails that + // decode and is dropped. A classifier that looked only at the type + // and the message's presence would accept these and count a + // message the index never counted. + "bad usage shape": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"usage":"not-an-object"}`, + "bad usage field": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"usage":{"input_tokens":"lots"}}`, + "numeric message id": `{"type":"message","message":{"id":123,"role":"user","parts":[{"type":"text","text":"x"}]}}`, + "bad goal payload": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"goal":"not-an-object"}`, + "bad compact payload": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"compact":5}`, + // Fields the INDEX's narrower shape does not read at all, but the + // format does. A final line malformed in one of these is dropped + // by LoadSession, so every reader must drop it: an index that + // counted it would report a message the session does not have. + "bad task_tool_names": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"task_tool_names":42}`, + "bad tool_result": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"tool_result":"nope"}`, + "bad mcp_tools": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[{"type":"text","text":"x"}]},"mcp_tools":{"a":1}}`, + } { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + sess := pagedSession(t, dir, 2) // four durable messages + path := filepath.Join(dir, sess.ID+".jsonl") + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(tail); err != nil { + t.Fatal(err) + } + f.Close() + + // The fold's own answer is the oracle: whatever it counts, the + // page must report. + ix, err := ReadSessionIndex(dir, sess.ID) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + if ix.DurableMessages != 4 { + t.Fatalf("the fold counted %d durable messages, want 4", ix.DurableMessages) + } + // The loader is the authority the index claims to match. + loaded, err := LoadSession(persistCfg(dir, &scriptedProvider{name: "test"}), sess.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if got := len(loaded.History()); got != ix.Messages { + t.Errorf("LoadSession has %d messages, the index says %d", got, ix.Messages) + } + page, err := ReadMessagePage(dir, sess.ID, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != 4 || len(page.Messages) != 4 { + t.Errorf("page = total %d with %d messages, want 4 and 4", page.Total, len(page.Messages)) + } + for _, m := range page.Messages { + if strings.HasPrefix(m.ID, "msg_ghost") || strings.HasPrefix(m.ID, "msg_torn") { + t.Errorf("the page served a record the fold dropped: %s", m.ID) + } + } + if !sameIDs(idsOf(page.Messages), wholeSequence(t, dir, sess.ID)) { + t.Errorf("page ids = %v, want the durable sequence %v", idsOf(page.Messages), wholeSequence(t, dir, sess.ID)) + } + }) + } +} + +// TestScanLogBackwardWhitespaceLongerThanThePrefix: a line whose leading +// whitespace fills the prefix window is not a blank line. The forward +// scanner trims the whole line before deciding, so this one reads it before +// skipping it. +func TestScanLogBackwardWhitespaceLongerThanThePrefix(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "log") + padded := strings.Repeat(" ", logLinePeekBytes+64) + `{"type":"model"}` + if err := os.WriteFile(path, []byte("first\n"+padded+"\n"), 0o644); err != nil { + t.Fatal(err) + } + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + fi, err := f.Stat() + if err != nil { + t.Fatal(err) + } + var got []string + if err := scanLogBackward(f, fi.Size(), fi.Size(), func(line logLine, _ bool) (bool, error) { + whole, err := line.All() + if err != nil { + return false, err + } + got = append(got, string(bytes.TrimSpace(whole))) + return true, nil + }); err != nil { + t.Fatalf("scanLogBackward: %v", err) + } + if len(got) != 2 || got[0] != `{"type":"model"}` || got[1] != "first" { + t.Errorf("lines = %v, want the padded record and then the first line", got) + } +} + +// TestTailPageMatchesTheFoldOnAmbiguousRecords covers the two shapes a +// prefix-only classifier got wrong. Both produced a phantom message under a +// real sequence number, which is worse than an error: a client cannot tell. +// +// - A record with a SECOND top-level "type" key. encoding/json resolves +// last-wins, so the fold reads the second value; a scan that stopped at +// the first key read the first. +// - A message record with NO body that is no longer the final line. The +// full fold tolerates the shape on a final line and skips it, and the +// incremental write-path fold skips it without ever revisiting it — so a +// journal can carry one mid-file AND have a usable index. A substring +// search for the message key called it body-bearing. +func TestTailPageMatchesTheFoldOnAmbiguousRecords(t *testing.T) { + msg := func(id, text string) string { + return `{"type":"message","message":{"id":"` + id + `","role":"user","parts":[{"type":"text","text":"` + text + `"}]}}` + } + for name, odd := range map[string]string{ + "duplicate top-level type": `{"type":"message","message":{"id":"msg_ghost","role":"user","parts":[]},"type":"model"}`, + "duplicate type, message last": `{"type":"model","message":{"id":"msg_ghost","role":"user","parts":[]},"type":"message"}`, + } { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"` + id + `","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +` + msg("msg_1", "first") + "\n" + odd + "\n" + msg("msg_2", "second") + "\n" + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + + // The fold is the oracle for what counts. + ix, err := ReadSessionIndex(dir, id) + if err != nil { + t.Skipf("the fold rejects this journal outright (%v); the page path is unreachable for it", err) + } + page, err := ReadMessagePage(dir, id, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != ix.DurableMessages { + t.Errorf("page total %d, index counted %d", page.Total, ix.DurableMessages) + } + // Every message the page reports must sit at the seq the + // durable sequence puts it at. That covers both directions: a + // record the fold skipped must not appear, and one it counted + // must — which of the two a duplicate key produces depends on + // which value wins, and the oracle resolves that the way the + // format does. + want := wholeSequence(t, dir, id) + if !sameIDs(idsOf(page.Messages), want) { + t.Errorf("page ids = %v, want the durable sequence %v", idsOf(page.Messages), want) + } + }) + } +} + +// TestTailPageAfterACrashLeftABodylessRecord builds the state a +// prefix-only classifier could not survive, by the route that actually +// reaches it in production. +// +// A crash leaves a journal whose FINAL record is a message record with no +// body. Both the loader and the fold tolerate that shape on a final line +// and do not count it. The session then resumes and appends: the +// incremental write-path fold applies only each NEW record, never +// revisiting the one before it, and flushes a sidecar covering the whole +// file. The journal now carries a bodyless record MID-FILE and a usable, +// current index — so a page read serves from that index without refolding, +// and a walk that judged the record by a substring search for the message +// key would count a message the index never counted, shifting every +// sequence number below it. +func TestTailPageAfterACrashLeftABodylessRecord(t *testing.T) { + for name, tail := range map[string]string{ + "no message field": `{"type":"message"}`, + "explicit null message": `{"type":"message","message":null}`, + "message key nested below": `{"type":"message","goal":{"condition":"mentions a message key"}}`, + } { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"` + id + `","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"before the crash"}]}} +` + tail + "\n" + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{compactTurn("after the crash", provider.Usage{InputTokens: 5})}} + cfg := persistCfg(dir, prov) + sess, err := LoadSession(cfg, id) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if _, err := sess.Prompt(context.Background(), "resume"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + // The sidecar must be current: that is the whole point — a page + // read serves from it without refolding the bodyless record. + ix, err := ReadSessionIndex(dir, id) + if err != nil { + t.Fatalf("ReadSessionIndex: %v", err) + } + page, err := ReadMessagePage(dir, id, 0, 10) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if page.Total != ix.DurableMessages { + t.Errorf("page total %d, index counted %d", page.Total, ix.DurableMessages) + } + if len(page.Messages) != page.Total { + t.Fatalf("page returned %d messages for a total of %d", len(page.Messages), page.Total) + } + // msg_1 is the oldest durable message and must stay at seq 1. + if page.FirstSeq != 1 || page.Messages[0].ID != "msg_1" { + t.Errorf("page starts at seq %d with %s, want seq 1 with msg_1 — a phantom displaced it", page.FirstSeq, page.Messages[0].ID) + } + }) + } +} diff --git a/engine/store.go b/engine/store.go index 714c5d14..b62511d8 100644 --- a/engine/store.go +++ b/engine/store.go @@ -510,6 +510,29 @@ func (info *SessionInfo) addUsage(u provider.Usage) { info.Usage.CacheWriteTokens += u.CacheWriteTokens } +// finalRecordComplete reports whether a journal's LAST line was completely +// written, by the only definition that matters: it decodes as a record — +// the type the writer marshals, and so the definition of the format. +// +// Every reader must ask this ONE question about a final line, because a +// crash can leave it half-written and each reader would otherwise invent +// its own tolerance from whatever subset of fields it happens to decode. +// The index's fold reads a narrower shape (indexRecord) that ignores most +// of a record's fields, so a final line with a malformed tool_result, +// mcp_tools, or task_tool_names value passes that shape while LoadSession +// drops it — and the index would then count a message the session itself +// does not have, which is the whole class of disagreement the message-page +// work exists to prevent. +// +// It is deliberately NOT used for a non-final line. There, a line that +// fails this check is corruption mid-file: LoadSession refuses the session +// outright, and a reader that folds anyway is offering a degraded view of +// an unloadable journal rather than miscounting a loadable one. +func finalRecordComplete(raw []byte) bool { + var rec record + return json.Unmarshal(bytes.TrimSpace(raw), &rec) == nil +} + func sessionPath(dir, id string) string { return filepath.Join(dir, id+".jsonl") } From 77e5aecd81b7d68b566fb2196c51ccffeb29fedc Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 08:40:49 -0400 Subject: [PATCH 14/95] fix(engine): bound the memory a concurrent read batch holds (#203) * fix(engine): bound the memory a concurrent read batch holds read_file's text path is deliberately unbounded (readPathContent): a coding agent legitimately reads whole files, and no byte cap is right for every file. Only the image path is capped (readFileMaxImageBytes), and bash caps its own output. That was safe while tool calls ran strictly one at a time, because peak heap then held at most ONE file's raw bytes plus the line-numbered copy built from them. The concurrent executor removed that implicit bound without replacing it, so a batch of N large reads holds N working sets at once. Measured with eight 16MB files, with retention swallowing the finals so only the transient term shows: ~325MB peak parallel against ~73MB sequential, ~4.3x, bounded only by ToolConcurrency. A wider cap multiplies it further, and the model chooses both the batch width and the file sizes. Add toolReadBudget (toolmem.go): each read_file reserves its file's Stat size against a per-session byte budget before touching the file, and holds the reservation until the call returns. The same batch then peaks at the SEQUENTIAL figure, 1.0x, so concurrency no longer amplifies peak memory at all. Bounding the product of read size and concurrency is what the hazard requires. A limit on how many large reads may run at once still admits two 500MB reads; a limit on file size breaks the large read the tool exists to serve. Reserving estimated bytes bounds the product directly, and ordinary work never contends: a full-width batch of kilobyte reads reserves a rounding error against the default and stays fully parallel. The reservation deliberately spans the line-numbering, not just the read. For a large file strings.Split is the single biggest allocation in the tool, so releasing when readPathContent returns would leave the expansion outside the bound. It reserves from the os.Stat the tool already does, so this costs no extra syscall, and readPathContent keeps its original signature. What it bounds and what it does not: this covers the TRANSIENT working set, the parallel-specific term. It does not bound the ACCUMULATED results, because runToolBatch holds every call's output until the join in both execution modes, so N results occupy the same memory however they were produced. That term is not a concurrency regression, and retention already collapses oversized results where configured. Three properties keep it safe: - A worker takes its pool slot first and then reserves. That is head-of-line blocking but cannot deadlock: only a slot holder ever holds budget, so whenever anyone waits at least one holder is doing I/O and will release, and a reservation is never held while acquiring another. - A read larger than the whole budget is clamped to it, not refused, so it waits for the budget to drain and then runs alone. A batch can always make progress. - Waiters are served strictly FIFO. A retry-when-there-is-room loop lets a stream of small reads starve one large read indefinitely. Config.ToolReadBudgetBytes: 0 takes the 64 MiB default, negative disables, positive sets the budget. HARNESS_TOOL_READ_BUDGET_MB is the operator seam, resolved in cmd/harness like HARNESS_TOOL_CONCURRENCY. The budget is per session; a process-wide one is the natural follow-up if that proves insufficient. The bound itself is proved by the budget's own invariant, exactly and under the race detector, rather than by heap sampling: reserved bytes never exceed the limit, under concurrent mixed-size reservations, in a real read_file batch, and with no leak afterwards. The heap measurement corroborates it and is skipped under -short. The diff is additive; no existing signature or behavior changes when the budget is disabled. * test(engine): pin concurrent read budget safety --------- Co-authored-by: harness Co-authored-by: andybons --- AGENTS.md | 53 +++ cmd/harness/main.go | 33 ++ cmd/harness/toolconcurrency_test.go | 28 ++ engine/engine.go | 33 ++ engine/filetools.go | 14 + engine/toolmem.go | 231 ++++++++++ engine/toolmem_test.go | 644 ++++++++++++++++++++++++++++ 7 files changed, 1036 insertions(+) create mode 100644 engine/toolmem.go create mode 100644 engine/toolmem_test.go diff --git a/AGENTS.md b/AGENTS.md index 6cf0d835..5346601e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -237,6 +237,59 @@ issue #129. Until it lands, a model with no vision support receives the image Blob exactly as any vision-capable model does; how it handles that block is between the model and its provider. +### Bounding concurrent-read memory + +`read_file`'s text path is deliberately unbounded (`readPathContent`, +`engine/filetools.go`) — a coding agent legitimately reads whole files, and +no byte cap is right for every file. Only the IMAGE path is capped +(`readFileMaxImageBytes`), and `bash` caps its own output +(`defaultBashOutputCap`). + +That was safe while tool calls ran strictly one at a time: peak heap held +at most ONE file's raw bytes plus the line-numbered copy built from them. +The concurrent executor (`engine/toolexec.go`) removed that implicit bound +without replacing it, so a batch of N large reads holds N working sets at +once. Measured with eight 16MB files, retention swallowing the finals so +only the transient term shows: **~325MB peak parallel against ~73MB +sequential — ~4.3x, bounded only by `ToolConcurrency`**, with the model +choosing both the batch width and the file sizes. + +`toolReadBudget` (`engine/toolmem.go`) is the replacement bound. Each +`read_file` reserves its file's Stat size against a per-session byte +budget before touching the file, and holds it until the call returns — +spanning the line-numbering too, since for a large file `strings.Split` +is the single biggest allocation and releasing after the read would leave +the expansion outside the bound. With the budget set to one file's size +the same batch peaks at the SEQUENTIAL figure: ~73MB, **1.0x**. It bounds the **product** of read size and +concurrency, which is the actual hazard: a count limit still admits two +500MB reads, and a size limit breaks the legitimate large read. Ordinary +work never contends — a full-width batch of kilobyte reads reserves a +rounding error against the default and stays fully parallel +(`TestReadBudgetKeepsSmallReadsFullyParallel`). + +It bounds the TRANSIENT working set, not the ACCUMULATED results: +`runToolBatch` holds every call's output until the join in BOTH execution +modes, so N results occupy the same memory however they were produced. +That term is not a concurrency regression, and retention already collapses +oversized results where configured. + +Three properties keep it safe. A worker takes its pool slot first and then +reserves, which is head-of-line blocking but cannot deadlock: only a slot +holder ever holds budget, so someone is always making progress, and a +reservation is never held while acquiring another. A read larger than the +whole budget is CLAMPED to it, not refused, so it waits for the budget to +drain and runs alone — a batch always progresses. Waiters are served +strictly FIFO, because a retry-when-there-is-room loop lets a stream of +small reads starve one large read forever. + +`Config.ToolReadBudgetBytes`: 0 (the zero value) takes +`defaultToolReadBudgetBytes` (64 MiB), a negative value DISABLES the bound, +a positive value sets it. `HARNESS_TOOL_READ_BUDGET_MB` is the operator +seam (megabytes; negative disables), resolved in `cmd/harness` like +`HARNESS_TOOL_CONCURRENCY`. The budget is per SESSION: a process running +many sessions bounds each, not their sum — a process-wide budget is the +natural follow-up if that proves insufficient. + ### write_file read-before-overwrite guard `write_file` (`engine/filetools.go`) refuses to overwrite an EXISTING file diff --git a/cmd/harness/main.go b/cmd/harness/main.go index d4010114..26f9b680 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -337,6 +337,37 @@ func toolConcurrency() int { return envInt("HARNESS_TOOL_CONCURRENCY") } +// toolReadBudgetBytes resolves engine.Config.ToolReadBudgetBytes from +// HARNESS_TOOL_READ_BUDGET_MB. The engine never reads an environment +// variable itself, so this is the seam (same shape as toolConcurrency +// above). +// +// The engine's own default is already safe, so this knob exists for +// tuning, not for turning the bound on: a positive value sets the budget +// in MEGABYTES, an explicit negative value DISABLES the bound for a +// deployment with its own memory discipline, and unset/zero/malformed +// leaves 0 so the engine applies its default. +func toolReadBudgetBytes() int64 { + raw := os.Getenv("HARNESS_TOOL_READ_BUDGET_MB") + if raw == "" { + return 0 + } + n, err := strconv.ParseInt(raw, 10, 64) + if err != nil { + return 0 + } + if n < 0 { + // Any negative value means "disabled"; normalize to -1 rather + // than passing a large negative through as a byte count. + return -1 + } + const mib = 1 << 20 + if n > (1<<62)/mib { + return 0 // absurd; fall back to the engine default + } + return n * mib +} + // sessionDir resolves where session logs live, in precedence order: // -no-save (yields "", persistence disabled) > $HARNESS_SESSION_DIR > // configDir (config session_dir) > $HOME/.harness/sessions. Nothing is @@ -713,6 +744,7 @@ func runCmd(args []string) error { ToolResultInlineBytes: cfg.ToolResultInlineBytesValue(), ToolResultRetainedBytes: cfg.ToolResultRetainedBytesValue(), ToolConcurrency: toolConcurrency(), + ToolReadBudgetBytes: toolReadBudgetBytes(), // GoalTool mirrors serveCmd's mkCfg below: the `goal` session tool is // only useful once an evaluator is actually configured to drive a // goal loop against (-goal itself resolves and validates its own @@ -1519,6 +1551,7 @@ func serveCmd(args []string) error { ToolResultInlineBytes: cfg.ToolResultInlineBytesValue(), ToolResultRetainedBytes: cfg.ToolResultRetainedBytesValue(), ToolConcurrency: toolConcurrency(), + ToolReadBudgetBytes: toolReadBudgetBytes(), // GoalTool enables the `goal` session tool (status/set/adjust) // whenever an evaluator is configured to drive a goal loop // against — the same condition server.Options.GoalEvaluator diff --git a/cmd/harness/toolconcurrency_test.go b/cmd/harness/toolconcurrency_test.go index 089b50e0..04cd9231 100644 --- a/cmd/harness/toolconcurrency_test.go +++ b/cmd/harness/toolconcurrency_test.go @@ -35,3 +35,31 @@ func TestToolConcurrencyKnobs(t *testing.T) { }) } } + +// TestToolReadBudgetKnob pins the HARNESS_TOOL_READ_BUDGET_MB seam. The +// engine's own default is safe, so the important cases are "unset leaves +// the engine default" and "an explicit negative disables the bound". +func TestToolReadBudgetKnob(t *testing.T) { + const mib = 1 << 20 + for _, tc := range []struct { + name string + env string + want int64 + }{ + {"unset leaves the engine default", "", 0}, + {"a positive value is megabytes", "128", 128 * mib}, + {"one megabyte", "1", mib}, + {"zero leaves the engine default", "0", 0}, + {"a negative value disables the bound", "-1", -1}, + {"any negative value normalizes to -1", "-4096", -1}, + {"a malformed value falls back to the default", "lots", 0}, + {"an absurd value falls back to the default", "99999999999999999", 0}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("HARNESS_TOOL_READ_BUDGET_MB", tc.env) + if got := toolReadBudgetBytes(); got != tc.want { + t.Errorf("toolReadBudgetBytes() = %d, want %d", got, tc.want) + } + }) + } +} diff --git a/engine/engine.go b/engine/engine.go index da9c24aa..a5f848f7 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -855,6 +855,30 @@ type Config struct { // that builds Config itself sets the field directly and gets no // environment handling at all. ToolConcurrency int + + // ToolReadBudgetBytes bounds the total ESTIMATED bytes this session's + // in-flight tool reads may hold at once, across a concurrent batch. + // + // It exists because read_file's text path is deliberately unbounded + // (a coding agent reads whole files) and the concurrent executor + // removed the implicit one-at-a-time bound that made that safe: a + // batch of N large reads holds N working sets instead of one. See + // toolmem.go for the measurement and the full rationale. + // + // Zero (the zero value, and the recommended setting) takes the + // package default, defaultToolReadBudgetBytes. A negative value + // DISABLES the budget, restoring the unbounded behavior for a caller + // that has its own memory discipline. A positive value sets the + // budget in bytes. + // + // Ordinary work never contends: kilobyte-sized reads reserve a + // rounding error against the default and a full-width batch of them + // still runs fully parallel. Only genuinely large reads queue. + // + // The budget is per SESSION. A process running many sessions at once + // bounds each of them, not their sum; a process-wide budget is the + // natural follow-up if that proves insufficient. + ToolReadBudgetBytes int64 } // Session is one conversation: an in-memory history plus the agent loop. @@ -1151,6 +1175,14 @@ type Session struct { // lock is needed to read it from a running batch. toolConcurrency int + // readBudget bounds the estimated bytes this session's in-flight tool + // reads hold at once — the replacement for the implicit bound strictly + // sequential execution used to provide. See Config.ToolReadBudgetBytes + // and toolmem.go. Nil means unlimited. Set once in newSession; the + // value is internally synchronized, so a running batch reserves + // against it from several goroutines without further locking. + readBudget *toolReadBudget + // compactCount/lastCompactedAt track how many times this session has // been compacted and when the most recent one landed — durable via the // compact journal record (see store.go), so GET /session can show a UI @@ -1304,6 +1336,7 @@ func newSession(cfg Config) *Session { toolResultNextID: 1, toolResults: make(map[string]toolResultMeta), toolConcurrency: resolveToolConcurrency(cfg.ToolConcurrency), + readBudget: newToolReadBudget(cfg.ToolReadBudgetBytes), readHashes: make(map[string][sha256.Size]byte), } for _, t := range []Tool{bashTool(cfg.BashTimeout, cfg.BashOutputCap), readFileTool(), writeFileTool(), editFileTool(), sessionInfoTool(), globTool(), grepTool(), lsTool()} { diff --git a/engine/filetools.go b/engine/filetools.go index ece8b4f4..3293cbba 100644 --- a/engine/filetools.go +++ b/engine/filetools.go @@ -352,6 +352,20 @@ func readFileTool() Tool { return nil, fmt.Errorf("read_file: %s is a directory", path) } + // Reserve this read's estimated memory before touching the + // file, and hold it until this call returns. The reservation + // must span the line-numbering below, not just the read: for a + // large file strings.Split is the single biggest allocation + // here, so releasing after readPathContent would leave the + // expansion — the very thing being bounded — outside the + // bound. info comes from the stat above, so this costs no + // extra syscall. See toolmem.go for why an estimate is sound. + release, err := s.readBudget.reserve(ctx, info.Size()) + if err != nil { + return nil, fmt.Errorf("read_file: %s: %w", path, err) + } + defer release() + content, err := readPathContent(path) if err != nil { return nil, fmt.Errorf("read_file: %s: %w", path, err) diff --git a/engine/toolmem.go b/engine/toolmem.go new file mode 100644 index 00000000..c6be96fe --- /dev/null +++ b/engine/toolmem.go @@ -0,0 +1,231 @@ +package engine + +import ( + "context" + "sync" +) + +// Bounding the memory a batch of concurrent tool calls holds at once. +// +// # The problem +// +// read_file's TEXT path is an unbounded io.ReadAll (readPathContent, +// filetools.go): a coding agent legitimately reads a whole file, and no +// byte cap would be right for every file. Only the IMAGE path is capped +// (readFileMaxImageBytes), because an oversized image is useless to a +// model, and bash caps its own output (defaultBashOutputCap). +// +// That was safe while tool calls ran strictly one at a time: peak heap +// held at most ONE file's raw bytes plus the line-numbered copy built +// from them. The concurrent executor (toolexec.go) removed that implicit +// bound without replacing it, so a batch of N large reads holds N of +// those working sets at once. Measured with eight 16MB files, retention +// swallowing the finals so only the transient term shows (see +// TestReadBudgetBoundsPeakHeap): ~325MB peak parallel against ~73MB +// sequential, a ~4.3x amplification bounded only by ToolConcurrency. A +// wider cap multiplies it further, and the model chooses both the batch +// width and the file sizes. With the budget set to one file's size the +// same batch peaks at the sequential figure — 1.0x. +// +// # What this bounds, and what it does not +// +// This budget bounds the TRANSIENT working set: bytes a read holds while +// it is in progress. That is the parallel-specific term, and the one the +// measurement above isolates. +// +// It deliberately does NOT bound the ACCUMULATED results. runToolBatch +// holds every call's output until the join in BOTH execution modes, so +// eight 16MB results occupy the same memory whether they were produced +// concurrently or one after another. That term is not a regression from +// concurrency and bounding it would mean changing what a batch of reads +// returns, not when. Retention (toolresult.go) already collapses +// oversized results where it is configured. +// +// edit_file's whole-file rewrite and write_file's read-guard hash remain +// outside this budget; bounding them is deferred because this change scopes +// reservations to read_file without changing either tool's I/O contract. +// A non-regular file whose Stat size is zero also reserves nothing; bounding +// that path is deferred until read_file has a byte cap for streams whose size +// cannot be estimated safely. +// +// # Why a byte budget rather than a count +// +// The hazard is the PRODUCT of read size and concurrency, so bounding +// either factor alone misses. A limit of "two large reads at once" still +// admits two 500MB reads; a limit on file size breaks the legitimate +// large read this tool exists to serve. Reserving estimated bytes bounds +// the product directly: eight concurrent 100KB reads all proceed at once +// (they fit), while eight concurrent 64MB reads serialize down to what +// fits. Normal-size tool calls never contend, so the concurrency win is +// untouched — TestReadBudgetKeepsSmallReadsFullyParallel pins that. +// +// The reservation is an ESTIMATE, taken from the open file handle's own +// Stat size. Correctness does not depend on it being exact: a file that +// grows after the reservation overshoots the budget by the growth, which +// is bounded by what the process can write in the meantime, and a file +// that shrinks merely over-reserves. This is why the estimate may use a +// Stat size where readPathContent's image CAP deliberately may not — a +// cap that binds on bytes actually read is a correctness boundary, while +// this is an admission hint. +// +// # Ordering and deadlock +// +// A worker takes its pool slot first (toolexec.go), then reserves. That +// is head-of-line blocking by construction: a worker can sit on a slot +// while it waits for budget. It cannot deadlock. Only a slot holder ever +// holds budget, so whenever anyone is waiting at least one holder is +// doing I/O and will release; a reservation is never held while acquiring +// another. A single read larger than the WHOLE budget is clamped to the +// budget rather than refused, so it waits for the budget to drain and +// then runs alone: a batch can always make progress. +// +// Waiters are served strictly FIFO. A plain "retry when there is room" +// loop lets a stream of small reads starve one large one indefinitely; +// with FIFO, a queued large read blocks later small ones rather than +// yielding to them forever. +type toolReadBudget struct { + mu sync.Mutex + limit int64 + used int64 + waiters []*readBudgetWaiter + + // onCancel is a test seam fired after a queued reserve selects ctx.Done + // and before it reacquires mu. Set before use and never changed. + onCancel func() +} + +// readBudgetWaiter is one queued reservation. granted is set under the +// budget's mutex at the moment the waiter is handed its bytes, so a +// reservation racing its own context cancellation can tell whether it +// owns bytes it must give back. +type readBudgetWaiter struct { + n int64 + granted bool + ready chan struct{} +} + +// defaultToolReadBudgetBytes is Config.ToolReadBudgetBytes' zero-value +// default: the total estimated bytes a session's in-flight tool reads may +// hold at once. +// +// Chosen so ordinary work never contends. Source files are kilobytes, so +// a full-width batch of them reserves a rounding error against this and +// runs fully parallel; only genuinely large reads (multiple MB) ever +// queue. With the roughly 2.5x expansion a line-numbered copy adds on top +// of the raw bytes, this bounds the transient term at a few hundred MB +// even when every slot holds a large read — well inside a container +// memory limit, where the unbounded behavior was not. +const defaultToolReadBudgetBytes int64 = 64 << 20 // 64 MiB + +// newToolReadBudget resolves Config.ToolReadBudgetBytes. Zero (unset) +// takes the package default; a negative value disables the budget and +// yields nil, which every method below treats as unlimited. +func newToolReadBudget(configured int64) *toolReadBudget { + switch { + case configured < 0: + return nil + case configured == 0: + return &toolReadBudget{limit: defaultToolReadBudgetBytes} + default: + return &toolReadBudget{limit: configured} + } +} + +// reserve blocks until n estimated bytes are available, then returns a +// release func the caller MUST call when the bytes are no longer held. +// +// A nil budget, a non-positive limit, or a non-positive n reserves +// nothing and returns a no-op release, so a caller never needs to know +// whether the budget is enabled. n above the whole limit is clamped to +// the limit rather than refused (see the ordering note above). +// +// It returns ctx.Err() only when ctx ends while queued. The returned +// release is nil in that case and must not be called; the caller returns +// an error result for its own call instead, which keeps the +// one-result-per-tool-call invariant intact. +func (b *toolReadBudget) reserve(ctx context.Context, n int64) (func(), error) { + if b == nil || b.limit <= 0 || n <= 0 { + return func() {}, nil + } + if n > b.limit { + n = b.limit + } + + b.mu.Lock() + // Jump the queue only when it is empty: taking bytes ahead of an + // already-waiting reservation is what starves a large read. + if len(b.waiters) == 0 && b.used+n <= b.limit { + b.used += n + b.mu.Unlock() + return b.releaser(n), nil + } + w := &readBudgetWaiter{n: n, ready: make(chan struct{})} + b.waiters = append(b.waiters, w) + b.mu.Unlock() + + select { + case <-w.ready: + return b.releaser(n), nil + case <-ctx.Done(): + if b.onCancel != nil { + b.onCancel() + } + b.mu.Lock() + granted := w.granted + if !granted { + for i, q := range b.waiters { + if q == w { + b.waiters = append(b.waiters[:i], b.waiters[i+1:]...) + break + } + } + } + b.mu.Unlock() + if granted { + // The grant landed between ctx ending and the lock. The bytes + // are ours, and nobody will ever call our release, so give + // them back here. + b.release(n) + } + return nil, ctx.Err() + } +} + +// releaser returns a release func that gives n bytes back exactly once. +func (b *toolReadBudget) releaser(n int64) func() { + var once sync.Once + return func() { once.Do(func() { b.release(n) }) } +} + +// release returns n bytes and hands them to whichever queued waiters now +// fit, in FIFO order, stopping at the first that does not. +func (b *toolReadBudget) release(n int64) { + b.mu.Lock() + defer b.mu.Unlock() + b.used -= n + if b.used < 0 { + b.used = 0 + } + for len(b.waiters) > 0 { + w := b.waiters[0] + if b.used+w.n > b.limit { + return + } + b.used += w.n + w.granted = true + b.waiters[0] = nil + b.waiters = b.waiters[1:] + close(w.ready) + } +} + +// inFlight reports the currently reserved bytes. For tests and +// diagnostics only. +func (b *toolReadBudget) inFlight() int64 { + if b == nil { + return 0 + } + b.mu.Lock() + defer b.mu.Unlock() + return b.used +} diff --git a/engine/toolmem_test.go b/engine/toolmem_test.go new file mode 100644 index 00000000..e09a345a --- /dev/null +++ b/engine/toolmem_test.go @@ -0,0 +1,644 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "runtime" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" +) + +// Tests for the concurrent-read memory budget (toolmem.go). +// +// The BOUND itself is proved deterministically: the budget's own invariant +// (reserved bytes never exceed the limit) is exact and cheap to check, and +// holds under concurrent reserve/release with the race detector watching. +// The heap measurements at the bottom corroborate that the invariant +// translates into real memory, and are skipped under -short because they +// allocate hundreds of MB. + +// ---- the invariant ---- + +// TestReadBudgetNeverExceedsItsLimit hammers the budget from many +// goroutines with mixed reservation sizes and checks the one thing that +// must always hold. +func TestReadBudgetNeverExceedsItsLimit(t *testing.T) { + const limit = 1 << 20 + b := newToolReadBudget(limit) + + var peak int64 + var mu sync.Mutex + observe := func() { + got := b.inFlight() + mu.Lock() + if got > peak { + peak = got + } + mu.Unlock() + if got > limit { + t.Errorf("in-flight %d exceeds the %d-byte limit", got, limit) + } + } + + var wg sync.WaitGroup + for i := 0; i < 64; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + n := int64(1 << uint(10+i%11)) // 1KB .. 1MB + release, err := b.reserve(context.Background(), n) + if err != nil { + t.Errorf("reserve(%d): %v", n, err) + return + } + observe() + release() + }(i) + } + wg.Wait() + + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d after every reservation released, want 0 (leak)", got) + } + if peak == 0 { + t.Error("never observed a non-zero reservation; the test did not exercise the budget") + } + t.Logf("peak observed in-flight: %d of %d", peak, limit) +} + +// TestReadBudgetSerializesOversizedReservations proves the budget actually +// blocks: three reservations of half the limit each cannot all hold at +// once, so the third waits. +func TestReadBudgetSerializesOversizedReservations(t *testing.T) { + const limit = 1000 + b := newToolReadBudget(limit) + + r1, err := b.reserve(context.Background(), 500) + if err != nil { + t.Fatal(err) + } + r2, err := b.reserve(context.Background(), 500) + if err != nil { + t.Fatal(err) + } + if got := b.inFlight(); got != 1000 { + t.Fatalf("in-flight = %d, want 1000", got) + } + + third := make(chan struct{}) + go func() { + r3, err := b.reserve(context.Background(), 500) + if err == nil { + r3() + } + close(third) + }() + + select { + case <-third: + t.Fatal("the third reservation was admitted while the budget was full") + case <-time.After(50 * time.Millisecond): + } + + r1() + select { + case <-third: + case <-time.After(2 * time.Second): + t.Fatal("the third reservation never ran after a release freed room") + } + r2() + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d, want 0", got) + } +} + +// TestReadBudgetServesWaitersFIFO is the anti-starvation property. A large +// reservation queued behind a full budget must be served before a small +// one that arrives after it — a "retry when there is room" loop would let +// a stream of small reads starve the large one forever. +func TestReadBudgetServesWaitersFIFO(t *testing.T) { + const limit = 100 + b := newToolReadBudget(limit) + + hold, err := b.reserve(context.Background(), 100) + if err != nil { + t.Fatal(err) + } + + var order []string + var mu sync.Mutex + var wg sync.WaitGroup + note := func(name string) { + mu.Lock() + order = append(order, name) + mu.Unlock() + } + + // Queue the big one first, then the small one. Both are queued before + // anything is released, so the order they are SERVED in is the + // budget's choice, not a scheduling artifact. + queued := make(chan struct{}) + wg.Add(1) + go func() { + defer wg.Done() + close(queued) + r, err := b.reserve(context.Background(), 100) + if err != nil { + t.Error(err) + return + } + note("big") + r() + }() + <-queued + waitForWaiters(t, b, 1) + + wg.Add(1) + go func() { + defer wg.Done() + r, err := b.reserve(context.Background(), 1) + if err != nil { + t.Error(err) + return + } + note("small") + r() + }() + waitForWaiters(t, b, 2) + + hold() + wg.Wait() + + mu.Lock() + defer mu.Unlock() + if len(order) != 2 || order[0] != "big" { + t.Errorf("served %v, want the queued big reservation first (FIFO, no starvation)", order) + } +} + +// TestReadBudgetNewArrivalCannotJumpQueuedWaiter exercises the fast-path +// half of FIFO: room for a small new arrival does not let it bypass a large +// waiter already at the head of the queue. +func TestReadBudgetNewArrivalCannotJumpQueuedWaiter(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + b := newToolReadBudget(100) + hold60, err := b.reserve(context.Background(), 60) + if err != nil { + t.Fatal(err) + } + hold40, err := b.reserve(context.Background(), 40) + if err != nil { + t.Fatal(err) + } + + bigAdmitted := make(chan struct{}, 1) + releaseBig := make(chan struct{}) + go func() { + release, err := b.reserve(context.Background(), 100) + if err != nil { + return + } + bigAdmitted <- struct{}{} + <-releaseBig + release() + }() + waitForWaiters(t, b, 1) + + // Leave 40 bytes free. The 100-byte head cannot fit yet, but a new + // 10-byte arrival must queue behind it rather than jumping ahead. + hold40() + smallAdmitted := make(chan struct{}, 1) + releaseSmall := make(chan struct{}) + go func() { + release, err := b.reserve(context.Background(), 10) + if err != nil { + return + } + smallAdmitted <- struct{}{} + <-releaseSmall + release() + }() + synctest.Wait() + jumped := false + select { + case <-smallAdmitted: + t.Error("a new small reservation jumped ahead of the queued large waiter") + jumped = true + // Let the incorrectly admitted reservation go so the test can + // finish cleanly while still reporting the failed invariant. + close(releaseSmall) + synctest.Wait() + default: + } + + hold60() + synctest.Wait() + select { + case <-bigAdmitted: + default: + t.Fatal("the queued large waiter was not admitted first after the budget drained") + } + if !jumped { + select { + case <-smallAdmitted: + t.Fatal("the small waiter was admitted while the earlier large waiter still held the budget") + default: + } + } + + close(releaseBig) + synctest.Wait() + if !jumped { + select { + case <-smallAdmitted: + default: + t.Fatal("the small waiter did not run after the large waiter released") + } + close(releaseSmall) + } + synctest.Wait() + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d after both waiters released, want 0", got) + } + }) +} + +// waitForWaiters blocks until the budget has at least n queued waiters, so +// a test can order its own queue deterministically instead of sleeping. +func waitForWaiters(t *testing.T, b *toolReadBudget, n int) { + t.Helper() + for i := 0; i < 1000; i++ { + b.mu.Lock() + got := len(b.waiters) + b.mu.Unlock() + if got >= n { + return + } + time.Sleep(time.Millisecond) + } + t.Fatalf("timed out waiting for %d queued waiters", n) +} + +// TestReadBudgetClampsAnOversizedReservation checks a single read larger +// than the WHOLE budget still runs, alone, rather than deadlocking the +// batch forever. +func TestReadBudgetClampsAnOversizedReservation(t *testing.T) { + b := newToolReadBudget(1000) + release, err := b.reserve(context.Background(), 1<<30) + if err != nil { + t.Fatalf("an over-budget reservation must be admitted alone, got %v", err) + } + if got := b.inFlight(); got != 1000 { + t.Errorf("in-flight = %d, want the whole budget (1000) reserved", got) + } + release() + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d after release, want 0", got) + } +} + +// TestReadBudgetQueuedOversizedReservationEventuallyRuns pins the clamp on +// the contended path: a request larger than the whole budget waits for the +// current holder, then takes the whole budget and runs alone. Without the +// clamp it can never fit, even after used reaches zero. +func TestReadBudgetQueuedOversizedReservationEventuallyRuns(t *testing.T) { + b := newToolReadBudget(100) + hold, err := b.reserve(context.Background(), 1) + if err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + type outcome struct { + release func() + err error + } + done := make(chan outcome, 1) + go func() { + release, err := b.reserve(ctx, 1000) + done <- outcome{release: release, err: err} + }() + waitForWaiters(t, b, 1) + hold() + + got := <-done + if got.err != nil { + t.Fatalf("oversized waiter never ran after the budget drained: %v", got.err) + } + if inFlight := b.inFlight(); inFlight != 100 { + t.Fatalf("in-flight = %d, want the oversized waiter clamped to 100", inFlight) + } + got.release() + if inFlight := b.inFlight(); inFlight != 0 { + t.Errorf("in-flight = %d after oversized waiter released, want 0", inFlight) + } +} + +// TestReadBudgetCancelWhileQueuedReleasesNothing checks a reservation +// abandoned because its turn was cancelled neither leaks bytes nor leaves +// a stale waiter behind. +func TestReadBudgetCancelWhileQueuedReleasesNothing(t *testing.T) { + b := newToolReadBudget(100) + hold, err := b.reserve(context.Background(), 100) + if err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + errc := make(chan error, 1) + go func() { + release, err := b.reserve(ctx, 50) + if err == nil { + release() + } + errc <- err + }() + waitForWaiters(t, b, 1) + cancel() + + if err := <-errc; err == nil { + t.Error("reserve returned nil error after its context was cancelled") + } + + hold() + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d after the cancelled waiter left, want 0", got) + } + b.mu.Lock() + n := len(b.waiters) + b.mu.Unlock() + if n != 0 { + t.Errorf("%d waiters still queued after cancellation, want 0", n) + } +} + +// TestReadBudgetGrantRacingCancellationReturnsGrantedBytes forces the narrow +// race where cancellation wins the select but the waiter is granted before +// it can reacquire the budget lock. Those bytes have already left the queue; +// the cancel path is their only possible releaser. +func TestReadBudgetGrantRacingCancellationReturnsGrantedBytes(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + b := newToolReadBudget(100) + if _, err := b.reserve(context.Background(), 100); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + cancelSelected := make(chan struct{}) + b.onCancel = func() { close(cancelSelected) } + errc := make(chan error, 1) + go func() { + _, err := b.reserve(ctx, 50) + errc <- err + }() + waitForWaiters(t, b, 1) + + b.mu.Lock() + cancel() + // reserve has selected ctx.Done and is now blocked reacquiring b.mu. + <-cancelSelected + w := b.waiters[0] + b.used = 0 // the original holder released while b.mu was held + b.used += w.n + w.granted = true + b.waiters[0] = nil + b.waiters = b.waiters[1:] + close(w.ready) + b.mu.Unlock() + + if err := <-errc; err == nil { + t.Fatal("reserve returned nil after cancellation won the race") + } + if got := b.inFlight(); got != 0 { + t.Errorf("in-flight = %d, want 0: the canceled waiter leaked its raced grant", got) + } + }) +} + +// TestReadBudgetDisabledAndDefault pins the config resolution. +func TestReadBudgetDisabledAndDefault(t *testing.T) { + if b := newToolReadBudget(-1); b != nil { + t.Error("a negative budget must disable the bound (nil)") + } + // A nil budget must still be usable, so no caller needs to branch. + var nilBudget *toolReadBudget + release, err := nilBudget.reserve(context.Background(), 1<<40) + if err != nil { + t.Fatalf("nil budget must reserve freely: %v", err) + } + release() + if got := newToolReadBudget(0).limit; got != defaultToolReadBudgetBytes { + t.Errorf("unset budget = %d, want the package default %d", got, defaultToolReadBudgetBytes) + } + if got := newToolReadBudget(4096).limit; got != 4096 { + t.Errorf("explicit budget = %d, want 4096", got) + } +} + +// ---- the concurrency win must survive ---- + +// TestReadBudgetKeepsSmallReadsFullyParallel is the regression guard for +// the fix itself: bounding memory must not serialize ordinary work. Eight +// kilobyte-sized reservations against the default budget must ALL be held +// at once, with nothing queued. +func TestReadBudgetKeepsSmallReadsFullyParallel(t *testing.T) { + b := newToolReadBudget(0) // the default + var releases []func() + for i := 0; i < 8; i++ { + release, err := b.reserve(context.Background(), 64<<10) // 64KB, a large source file + if err != nil { + t.Fatalf("reservation %d blocked or failed: %v", i, err) + } + releases = append(releases, release) + } + b.mu.Lock() + queued := len(b.waiters) + b.mu.Unlock() + if queued != 0 { + t.Errorf("%d ordinary-size reservations queued; the budget must not serialize normal work", queued) + } + if got, want := b.inFlight(), int64(8*64<<10); got != want { + t.Errorf("in-flight = %d, want %d (all eight held at once)", got, want) + } + for _, r := range releases { + r() + } +} + +// TestReadBudgetBoundsARealBatch drives real read_file calls through the +// executor and samples the budget while the batch runs, so the bound is +// observed end to end rather than only at the unit level. +func TestReadBudgetBoundsARealBatch(t *testing.T) { + const n = 8 + const size = 1 << 20 // 1MB each + const limit = 2 << 20 + + dir := t.TempDir() + body := strings.Repeat(strings.Repeat("z", 127)+"\n", size/128) + var calls []*message.ToolCall + for i := 0; i < n; i++ { + p := filepath.Join(dir, fmt.Sprintf("f%d.txt", i)) + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("r%d", i), + Name: "read_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q}`, p)), + }) + } + + s := NewSession(Config{WorkDir: dir, ToolConcurrency: 8, ToolReadBudgetBytes: limit}) + + var over atomic.Int64 + stop, stopped := make(chan struct{}), make(chan struct{}) + go func() { + defer close(stopped) + for { + select { + case <-stop: + return + default: + } + if got := s.readBudget.inFlight(); got > limit { + over.Store(got) + } + } + }() + + results := s.runToolCalls(context.Background(), asstWith(calls...)) + close(stop) + <-stopped + + wantOrder(t, results, calls...) + for i, p := range results { + tr := p.(*message.ToolResult) + if tr.IsError { + t.Fatalf("read %d errored: %s", i, resultText(tr.Content)) + } + } + if got := over.Load(); got != 0 { + t.Errorf("observed %d bytes in flight, over the %d-byte budget", got, limit) + } + if got := s.readBudget.inFlight(); got != 0 { + t.Errorf("in-flight = %d after the batch, want 0 (leak)", got) + } +} + +// ---- heap corroboration ---- + +// TestReadBudgetBoundsPeakHeap measures what the invariant buys. It is the +// regression this whole file exists for: without a budget, eight +// concurrent large reads hold eight working sets at once. +// +// Skipped under -short: it allocates hundreds of MB by design. Heap +// sampling is inherently noisy, so the assertions are deliberately loose — +// the exact bound is proved by the invariant tests above, not here. +func TestReadBudgetBoundsPeakHeap(t *testing.T) { + if testing.Short() { + t.Skip("allocates hundreds of MB") + } + const n = 8 + const size = 16 << 20 + + dir := t.TempDir() + body := strings.Repeat(strings.Repeat("y", 127)+"\n", size/128) + var calls []*message.ToolCall + for i := 0; i < n; i++ { + p := filepath.Join(dir, fmt.Sprintf("big%d.txt", i)) + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("r%d", i), + Name: "read_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q}`, p)), + }) + } + + // Retention on with a tiny inline limit, so each final result collapses + // to a preview and the accumulated term (identical in both execution + // modes, and not what this budget bounds) drops out of the measurement. + measure := func(concurrency int, budget int64) uint64 { + sess := t.TempDir() + runtime.GC() + var base runtime.MemStats + runtime.ReadMemStats(&base) + var peak uint64 + stop, stopped := make(chan struct{}), make(chan struct{}) + go func() { + defer close(stopped) + for { + select { + case <-stop: + return + default: + } + var m runtime.MemStats + runtime.ReadMemStats(&m) + if m.HeapAlloc > peak { + peak = m.HeapAlloc + } + time.Sleep(time.Millisecond) + } + }() + s := NewSession(Config{ + WorkDir: dir, + SessionDir: sess, + ToolResultInlineBytes: 500, + ToolConcurrency: concurrency, + ToolReadBudgetBytes: budget, + }) + res := s.runToolCalls(context.Background(), asstWith(calls...)) + close(stop) + <-stopped + wantOrder(t, res, calls...) + if peak < base.HeapAlloc { + return 0 + } + return peak - base.HeapAlloc + } + + unbounded := measure(8, -1) // the pre-fix behavior + bounded := measure(8, size) // budget of one file + sequential := measure(1, -1) // the implicit bound concurrency removed + + mb := func(b uint64) float64 { return float64(b) / (1 << 20) } + t.Logf("%d x %d MB files", n, size>>20) + t.Logf("parallel, budget DISABLED: %.0f MB", mb(unbounded)) + t.Logf("parallel, budget %d MB: %.0f MB", size>>20, mb(bounded)) + t.Logf("sequential (cap 1): %.0f MB", mb(sequential)) + if sequential > 0 { + t.Logf("amplification without the budget: %.1fx; with it: %.1fx", + float64(unbounded)/float64(sequential), float64(bounded)/float64(sequential)) + } + + if unbounded > 0 && float64(bounded) > 0.75*float64(unbounded) { + t.Errorf("budget reduced peak heap by less than 25%%: bounded %.0f MB vs unbounded %.0f MB", mb(bounded), mb(unbounded)) + } + if sequential > 0 && float64(bounded) > 3*float64(sequential) { + t.Errorf("bounded peak %.0f MB is more than 3x the sequential %.0f MB; the budget is not holding", + mb(bounded), mb(sequential)) + } +} + +// resultText joins a result's Text parts, for error messages. +func resultText(parts message.Parts) string { + var b strings.Builder + for _, p := range parts { + if txt, ok := p.(*message.Text); ok { + b.WriteString(txt.Text) + } + } + return b.String() +} From a2cc1f4153d263f69e2c842319b322121730790f Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 08:57:30 -0400 Subject: [PATCH 15/95] test(engine): adversarial suite for the concurrent tool executor (#202) * test(engine): adversarial suite for the concurrent tool executor The concurrent tool-call executor makes several guarantees that a normal test run never stresses: results join in call order regardless of completion order, per-path exclusion actually excludes, retention accounting stays exact under a wide batch, hooks stay paired per call, and a cancelled or panicking batch still yields one result per call with a balanced event stream. Add 16 tests that attack each of those directly, in the style the existing executor tests established: concurrency proved by rendezvous under a testing/synctest bubble rather than by timing, so "cannot make progress" is reported as a deadlock instead of hanging. What is pinned: - Cap admission. 12 calls at cap 8 admit exactly 8, and the 9th starts only once a slot frees (synctest.Wait makes that an observation, not a race with a sleep). Peak concurrency is exact at caps 1, 2 and 8, at N below the cap, and at a negative cap. - Path aliasing. Relative, absolute, dot-dot, dot-slash, a symlinked file, a symlinked parent directory of a not-yet-created file, and doubled separators all collapse to one key. - Same-path file tools. Two large write_file calls at one path serialize with no interleave; read_file before write_file in one batch runs in call order and the read authorizes the write; six racing writes to an unread existing file are all refused, so the read-before-overwrite guard is intact under parallelism. - Retention. 32 concurrent oversized results against a ceiling of five never overshoot it, on disk or in the counter, and handles are minted in call order rather than completion order. - Hook phases. Within one call the phases stay before, tool, after, each exactly once, across an 8-wide batch. - Failure modes of the exclusion guard. Unparseable args take one shared key so such calls serialize with each other rather than racing, and a negative concurrency value clamps to sequential rather than to unbounded. - keyChain's no-lock premise, as a source-order tripwire: chain.wait is called before the worker goroutines launch, and a change that moves it after them fails the test. - Cancellation and panics. A 20-call batch cancelled with the pool full yields one result per call, a balanced tool.start/tool.end stream and no leaked goroutine; a batch mixing panicking and normal calls keeps every pair balanced and leaves siblings unaffected. - Aggregate result size, parallel versus sequential, byte for byte, so a future change that lets a parallel batch put more into the request than a sequential one shows up here. TestAdvHardLinkAliasIsNotCovered is deliberately a pin on unclosed behavior rather than a passing property: two hard links to one inode take two keys and their calls run concurrently. It fails if the gap is ever closed, which is the point. Also correct canonicalFileKeyPath's stated reason for leaving that gap open. It says an inode comparison would be quadratic; keying on device+inode is O(1) per call, about what the EvalSymlinks call above it already costs. The real reasons are that a write_file target routinely does not exist yet and so has no inode to key on, leaving the create-during-batch race in place, and that st_dev/st_ino need a second implementation for Windows. The gap stays open, with an accurate reason. No production behavior changes. * test(engine): pin post-abort tool admission --------- Co-authored-by: harness Co-authored-by: andybons --- engine/filetools.go | 18 +- engine/toolexec.go | 8 +- engine/toolexec_adversarial_test.go | 896 ++++++++++++++++++++++++++++ 3 files changed, 916 insertions(+), 6 deletions(-) create mode 100644 engine/toolexec_adversarial_test.go diff --git a/engine/filetools.go b/engine/filetools.go index 3293cbba..8b37bdfd 100644 --- a/engine/filetools.go +++ b/engine/filetools.go @@ -220,9 +220,21 @@ const filePathKeyPrefix = "path:" // that the pushback is right. // // A HARD link still aliases: two names for one inode with no symlink to -// follow. Closing that needs a stat and an inode comparison against every -// other key in the batch, which is quadratic and still races a file -// created mid-batch. That one stays a documented residual. +// follow, so two names for one file take two keys and their calls run +// concurrently. That one stays a documented residual — +// TestAdvHardLinkAliasIsNotCovered pins the current behavior and fails if +// it is ever closed. +// +// Closing it would mean keying on the inode (a stat, then a +// device+inode key) rather than comparing keys pairwise — O(1) per call, +// about what EvalSymlinks above already costs. Two things, not cost, are +// why it is still open. A write_file target routinely does not exist yet, +// so it has no inode to key on and must fall back to this lexical path; +// a batch that pairs a create with a hard-linked write then still takes +// two keys, and the file created mid-batch races exactly as it does now. +// And st_dev/st_ino are Unix-shaped, so a portable version needs a +// second implementation for Windows. The gap is narrow and the fix is +// only partial, which is why it waits for a demonstrated need. func canonicalFileKeyPath(abs string) string { if real, err := filepath.EvalSymlinks(abs); err == nil { return real diff --git a/engine/toolexec.go b/engine/toolexec.go index 03200b20..f61ca23b 100644 --- a/engine/toolexec.go +++ b/engine/toolexec.go @@ -209,9 +209,11 @@ // that cannot tolerate it. // 2. A HARD link. filePathKey resolves symlinks (see // canonicalFileKeyPath), so a symlinked alias keys correctly, but two -// hard links to one inode have no link to follow. Closing that needs -// an inode comparison against every other key in the batch, which is -// quadratic and still races a file created mid-batch. +// hard links to one inode have no link to follow. An O(1) device+inode +// key would cover files that already exist, but a write_file target may +// not exist yet (and therefore has no inode), so a create racing a write +// through another hard link would remain. Device/inode identity is also +// platform-specific; see canonicalFileKeyPath for the full residual. // // # Cancellation and the orphan-result invariant // diff --git a/engine/toolexec_adversarial_test.go b/engine/toolexec_adversarial_test.go new file mode 100644 index 00000000..fb8edd7e --- /dev/null +++ b/engine/toolexec_adversarial_test.go @@ -0,0 +1,896 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "runtime" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" + "github.com/majorcontext/harness/provider" +) + +// TestAdvCapQueuesTheNinthUntilASlotFrees proves the cap is a real +// admission gate, not just a goroutine-count hint: with 12 calls and a cap +// of 8, exactly 8 run, and the 9th starts only once one of the first 8 +// returns. +// +// synctest.Wait() is what makes this deterministic — it returns only when +// every other goroutine in the bubble is durably blocked, so "no 9th call +// has started" is an observation, not a race with a sleep. +func TestAdvCapQueuesTheNinthUntilASlotFrees(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const n, cap = 12, 8 + var mu sync.Mutex + inflight, maxInflight, entered := 0, 0, 0 + tokens := make(chan struct{}) // one send releases exactly one call + + s := NewSession(Config{ToolConcurrency: cap}) + s.tools["slow"] = batchTool("slow", func(context.Context, string) { + mu.Lock() + inflight++ + entered++ + if inflight > maxInflight { + maxInflight = inflight + } + mu.Unlock() + <-tokens + mu.Lock() + inflight-- + mu.Unlock() + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("slow", fmt.Sprintf("c%d", i)) + } + + var results message.Parts + done := make(chan struct{}) + go func() { + results = s.runToolCalls(context.Background(), asstWith(calls...)) + close(done) + }() + + synctest.Wait() + mu.Lock() + gotEntered, gotInflight := entered, inflight + mu.Unlock() + if gotEntered != cap || gotInflight != cap { + t.Fatalf("with the batch stalled: entered=%d inflight=%d, want %d/%d (the cap must admit exactly %d)", + gotEntered, gotInflight, cap, cap, cap) + } + + // Free exactly one slot. The 9th call must now start — and only + // the 9th. + tokens <- struct{}{} + synctest.Wait() + mu.Lock() + gotEntered, gotInflight = entered, inflight + mu.Unlock() + if gotEntered != cap+1 { + t.Errorf("after freeing one slot, entered=%d, want %d (the 9th call must start, and no more)", gotEntered, cap+1) + } + if gotInflight != cap { + t.Errorf("after freeing one slot, inflight=%d, want %d (still exactly at the cap)", gotInflight, cap) + } + + for i := 0; i < n-1; i++ { + tokens <- struct{}{} + } + <-done + wantOrder(t, results, calls...) + if maxInflight != cap { + t.Errorf("peak concurrency = %d, want exactly %d", maxInflight, cap) + } + }) +} + +// TestAdvPeakConcurrencyMatchesTheCap sweeps cap settings, including the +// sequential floor the HARNESS_SEQUENTIAL_TOOLS=1 kill switch resolves to. +func TestAdvPeakConcurrencyMatchesTheCap(t *testing.T) { + for _, tc := range []struct{ n, cap, want int }{ + {6, 1, 1}, // kill switch: strictly one at a time + {6, 2, 2}, + {12, 8, 8}, // the default cap + {3, 8, 3}, // fewer calls than the cap + {6, -1, 1}, // negative clamps to sequential, never "unlimited" + } { + t.Run(fmt.Sprintf("n%d_cap%d", tc.n, tc.cap), func(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + var mu sync.Mutex + inflight, maxInflight := 0, 0 + var order []string + + s := NewSession(Config{ToolConcurrency: tc.cap}) + s.tools["t"] = batchTool("t", func(_ context.Context, call string) { + mu.Lock() + inflight++ + if inflight > maxInflight { + maxInflight = inflight + } + order = append(order, call) + mu.Unlock() + time.Sleep(time.Second) + mu.Lock() + inflight-- + mu.Unlock() + }) + + calls := make([]*message.ToolCall, tc.n) + for i := range calls { + calls[i] = batchCall("t", fmt.Sprintf("c%d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + if maxInflight != tc.want { + t.Errorf("peak concurrency = %d, want %d", maxInflight, tc.want) + } + if tc.want == 1 { + for i, c := range order { + if c != fmt.Sprintf("c%d", i) { + t.Errorf("sequential mode ran %v, want strict call order", order) + break + } + } + } + }) + }) + } +} + +// ---- Finding 4: path aliasing ---- + +// TestAdvPathAliasesCollapseToOneKey checks every spelling that can reach +// one file through the filesystem's own name resolution. Each must produce +// the SAME key, or two calls the executor believes are unrelated race on +// one file. +func TestAdvPathAliasesCollapseToOneKey(t *testing.T) { + dir := t.TempDir() + real, err := filepath.EvalSymlinks(dir) + if err != nil { + t.Fatal(err) + } + target := filepath.Join(real, "f.txt") + if err := os.WriteFile(target, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + link := filepath.Join(real, "link.txt") + if err := os.Symlink(target, link); err != nil { + t.Fatal(err) + } + subdir := filepath.Join(real, "sub") + if err := os.Mkdir(subdir, 0o755); err != nil { + t.Fatal(err) + } + dirlink := filepath.Join(real, "dirlink") + if err := os.Symlink(subdir, dirlink); err != nil { + t.Fatal(err) + } + + s := NewSession(Config{WorkDir: real}) + key := func(path string) string { + return filePathKey(s, json.RawMessage(fmt.Sprintf(`{"path":%q}`, path))) + } + + want := key(target) + for _, spelling := range []struct{ name, path string }{ + {"relative", "f.txt"}, + {"absolute", target}, + {"dot-dot", filepath.Join(real, "sub", "..", "f.txt")}, + {"dot-slash", "./f.txt"}, + {"symlink to the file", link}, + {"redundant separators", real + "//f.txt"}, + } { + if got := key(spelling.path); got != want { + t.Errorf("%s: key(%q) = %q, want %q — an alias the executor would let race", + spelling.name, spelling.path, got, want) + } + } + + // A not-yet-created file inside a SYMLINKED DIRECTORY: write_file's + // common shape, where the full path cannot resolve because the leaf + // does not exist yet. + if a, b := key(filepath.Join(dirlink, "new.txt")), key(filepath.Join(subdir, "new.txt")); a != b { + t.Errorf("symlinked-dir alias for a new file: %q != %q", a, b) + } +} + +// TestAdvHardLinkAliasIsNotCovered pins the ONE aliasing residual +// canonicalFileKeyPath documents (filetools.go): two hard links to one +// inode have no link to follow, so they take two keys and their calls run +// concurrently on one file. +// +// This test asserts the CURRENT, deliberately-unclosed behavior. It is a +// residual pin, not an approval: if it ever starts failing, someone closed +// the gap and canonicalFileKeyPath's comment should change with it. See +// that comment for why the fix is only partial (a not-yet-created +// write_file target has no inode to key on) rather than merely expensive. +func TestAdvHardLinkAliasIsNotCovered(t *testing.T) { + dir, err := filepath.EvalSymlinks(t.TempDir()) + if err != nil { + t.Fatal(err) + } + a := filepath.Join(dir, "a.txt") + if err := os.WriteFile(a, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + b := filepath.Join(dir, "b.txt") + if err := os.Link(a, b); err != nil { + t.Skipf("hard links unsupported here: %v", err) + } + + s := NewSession(Config{WorkDir: dir}) + ka := filePathKey(s, json.RawMessage(fmt.Sprintf(`{"path":%q}`, a))) + kb := filePathKey(s, json.RawMessage(fmt.Sprintf(`{"path":%q}`, b))) + if ka == kb { + t.Fatalf("hard-link aliases now share key %q — the documented residual is CLOSED; update canonicalFileKeyPath's docs", ka) + } + t.Logf("documented residual confirmed: %q != %q (same inode, two keys, calls run concurrently)", ka, kb) +} + +// ---- Finding 4 continued: same-path file tools actually serialize ---- + +// TestAdvSamePathWritesSerializeWithoutCorruption runs two write_file calls +// at one path in one batch. They must serialize in call order and leave the +// SECOND call's content intact — never a byte-level interleave of the two. +func TestAdvSamePathWritesSerializeWithoutCorruption(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "target.txt") + first := strings.Repeat("A", 256*1024) + second := strings.Repeat("B", 256*1024) + + s := NewSession(Config{WorkDir: dir, ToolConcurrency: 8}) + c1 := &message.ToolCall{CallID: "w1", Name: "write_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q,"content":%q}`, path, first))} + c2 := &message.ToolCall{CallID: "w2", Name: "write_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q,"content":%q}`, path, second))} + + results := s.runToolCalls(context.Background(), asstWith(c1, c2)) + wantOrder(t, results, c1, c2) + for i, p := range results { + if tr := p.(*message.ToolResult); tr.IsError { + t.Fatalf("write %d errored: %v", i+1, partsText(tr.Content)) + } + } + + got, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(got) != second { + if string(got) == first { + t.Errorf("file holds the FIRST write — calls completed out of call order") + } else { + t.Errorf("file is neither write's content (%d bytes) — the two writes INTERLEAVED", len(got)) + } + } +} + +// TestAdvSamePathReadThenWriteHonorsTheGuardInCallOrder is the interaction +// between per-path keying and the read-before-overwrite guard. read_file +// and write_file share one key namespace, so a read placed BEFORE a write +// in the same batch must run first and authorize it. If the two raced, the +// write would intermittently be refused. +func TestAdvSamePathReadThenWriteHonorsTheGuardInCallOrder(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "existing.txt") + if err := os.WriteFile(path, []byte("original"), 0o644); err != nil { + t.Fatal(err) + } + + s := NewSession(Config{WorkDir: dir, ToolConcurrency: 8}) + read := &message.ToolCall{CallID: "r1", Name: "read_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q}`, path))} + write := &message.ToolCall{CallID: "w1", Name: "write_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q,"content":"replaced"}`, path))} + + results := s.runToolCalls(context.Background(), asstWith(read, write)) + wantOrder(t, results, read, write) + if tr := results[1].(*message.ToolResult); tr.IsError { + t.Fatalf("write refused despite an in-batch read that precedes it: %v", partsText(tr.Content)) + } + got, _ := os.ReadFile(path) + if string(got) != "replaced" { + t.Errorf("file = %q, want %q", got, "replaced") + } +} + +// TestAdvUnreadOverwriteStillRefusedUnderParallel proves parallelism did +// not punch a hole in the read-before-overwrite guard: an existing file +// this session never read is still protected, however many callers race +// for it. +func TestAdvUnreadOverwriteStillRefusedUnderParallel(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "precious.txt") + if err := os.WriteFile(path, []byte("precious"), 0o644); err != nil { + t.Fatal(err) + } + + s := NewSession(Config{WorkDir: dir, ToolConcurrency: 8}) + var calls []*message.ToolCall + for i := 0; i < 6; i++ { + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("w%d", i), + Name: "write_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q,"content":"clobbered-%d"}`, path, i)), + }) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + for i, p := range results { + if tr := p.(*message.ToolResult); !tr.IsError { + t.Errorf("write %d SUCCEEDED on an unread existing file — the guard was bypassed", i) + } + } + if got, _ := os.ReadFile(path); string(got) != "precious" { + t.Errorf("file was modified to %q despite every write being refused", got) + } +} + +// partsText joins the Text parts of a result, for error messages. +func partsText(parts message.Parts) string { + var b strings.Builder + for _, p := range parts { + if txt, ok := p.(*message.Text); ok { + b.WriteString(txt.Text) + } + } + return b.String() +} + +// ---- Finding 2: aggregate result size ---- + +// TestAdvAggregateBatchBytesAreIdenticalParallelVsSequential measures the +// TOTAL inline bytes a wide batch of oversized results puts into the +// request, in both execution modes. The question the finding asks is +// whether parallel returns can push more past the per-call retention limit +// than the sequential path allowed. The answer must be that the two are +// byte-identical: retention runs at the JOIN, single-threaded, in call +// order, in both modes. +// +// It also reports the aggregate in absolute terms, because "each call is +// capped" and "the batch is capped" are different claims. +func TestAdvAggregateBatchBytesAreIdenticalParallelVsSequential(t *testing.T) { + const n = 8 + const each = 400_000 + const inline = 2_000 + + measure := func(concurrency int) (total int, retained int) { + dir := t.TempDir() + s := NewSession(Config{ + SessionDir: dir, + ToolResultInlineBytes: inline, + ToolConcurrency: concurrency, + }) + s.tools["big"] = batchTool("big", nil) + s.tools["big"] = Tool{ + Def: s.tools["big"].Def, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + return message.Parts{&message.Text{Text: in.Call + strings.Repeat("z", each)}}, nil + }, + } + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("big", fmt.Sprintf("c%d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + for _, p := range results { + total += len(partsText(p.(*message.ToolResult).Content)) + } + s.mu.Lock() + retained = len(s.toolResults) + s.mu.Unlock() + return total, retained + } + + par, parHandles := measure(8) + seq, seqHandles := measure(1) + + raw := n * each + t.Logf("raw tool output: %d bytes (%d calls x %d)", raw, n, each) + t.Logf("parallel (cap 8): %d bytes inline, %d handles retained", par, parHandles) + t.Logf("sequential (cap 1): %d bytes inline, %d handles retained", seq, seqHandles) + t.Logf("per-call inline limit: %d; aggregate ceiling: NONE (n x limit grows linearly)", inline) + + if par != seq { + t.Errorf("parallel put %d bytes in the request but sequential put %d — parallelism CHANGED the aggregate", par, seq) + } + if parHandles != seqHandles { + t.Errorf("parallel retained %d handles, sequential %d", parHandles, seqHandles) + } +} + +// ---- Finding 8: cancellation ---- + +// TestAdvCancelMidBatchLeaksNoGoroutines cancels a turn while a wide batch +// is in flight and checks the four things that must hold: no call queued +// behind the full pool passes the post-abort admission gate, every refused +// call gets a canceled result, started calls balance tool.start/tool.end, +// and no goroutine remains once runToolCalls returns. +func TestAdvCancelMidBatchLeaksNoGoroutines(t *testing.T) { + const n = 20 + + runtime.GC() + before := runtime.NumGoroutine() + + var mu sync.Mutex + starts, ends := map[string]int{}, map[string]int{} + + // Cancel only once the pool is FULL, so this exercises cancelling a + // wide in-flight batch rather than just post-cancel admission refusal. + entered := make(chan struct{}, n) + ctx, cancel := context.WithCancel(context.Background()) + + s := NewSession(Config{ + ToolConcurrency: 8, + OnEvent: func(ev Event) { + mu.Lock() + defer mu.Unlock() + switch ev.Type { + case EventToolStart: + starts[ev.ToolCall.CallID]++ + case EventToolEnd: + ends[ev.ToolCall.CallID]++ + } + }, + }) + s.tools["blocker"] = batchTool("blocker", func(ctx context.Context, _ string) { + entered <- struct{}{} + <-ctx.Done() // honor cancellation + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("blocker", fmt.Sprintf("c%d", i)) + } + + go func() { + for i := 0; i < 8; i++ { // the cap: every slot occupied + <-entered + } + cancel() + }() + results := s.runToolCalls(ctx, asstWith(calls...)) + cancel() + + wantOrder(t, results, calls...) + + mu.Lock() + for id, nStart := range starts { + if ends[id] != nStart { + t.Errorf("call %s: %d tool.start events but %d tool.end — event stream UNBALANCED", id, nStart, ends[id]) + } + } + nStarted := len(starts) + started := make(map[string]bool, nStarted) + for id := range starts { + started[id] = true + } + mu.Unlock() + t.Logf("cancelled batch of %d with the pool full: %d calls started (the rest were refused admission), all results present, events balanced", n, nStarted) + if nStarted != 8 { + t.Errorf("%d calls started, want exactly the cap (8) — no call queued behind the full pool may pass the post-abort admission gate", nStarted) + } + canceled := 0 + for i, p := range results { + tr := p.(*message.ToolResult) + if started[calls[i].CallID] { + continue + } + canceled++ + if !tr.IsError || partsText(tr.Content) != toolCallCanceledText { + t.Errorf("unstarted call %s result = error:%v %q, want the synthetic canceled result %q", + calls[i].CallID, tr.IsError, partsText(tr.Content), toolCallCanceledText) + } + } + if canceled != n-8 { + t.Errorf("%d calls carried canceled results, want %d", canceled, n-8) + } + + // Goroutines unwind asynchronously; give them a bounded chance to. + var after int + for i := 0; i < 100; i++ { + runtime.GC() + after = runtime.NumGoroutine() + if after <= before+2 { + break + } + time.Sleep(10 * time.Millisecond) + } + if after > before+2 { + t.Errorf("goroutines: %d before, %d after a cancelled %d-call batch — leak", before, after, n) + } +} + +// TestAdvEveryToolStartHasAnEndUnderPanicAndCancel hammers the balanced- +// events guarantee with a batch that mixes panicking, cancelled and normal +// calls. +func TestAdvEveryToolStartHasAnEndUnderPanicAndCancel(t *testing.T) { + var mu sync.Mutex + starts, ends := map[string]int{}, map[string]int{} + + s := NewSession(Config{ + ToolConcurrency: 8, + OnEvent: func(ev Event) { + mu.Lock() + defer mu.Unlock() + switch ev.Type { + case EventToolStart: + starts[ev.ToolCall.CallID]++ + case EventToolEnd: + ends[ev.ToolCall.CallID]++ + } + }, + }) + s.tools["boom"] = batchTool("boom", func(context.Context, string) { panic("adversarial panic") }) + s.tools["fine"] = batchTool("fine", nil) + + var calls []*message.ToolCall + for i := 0; i < 12; i++ { + name := "fine" + if i%3 == 0 { + name = "boom" + } + calls = append(calls, batchCall(name, fmt.Sprintf("c%d", i))) + } + + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + mu.Lock() + defer mu.Unlock() + if len(starts) != len(calls) { + t.Errorf("%d distinct tool.start events, want %d", len(starts), len(calls)) + } + for id, nStart := range starts { + if nStart != 1 { + t.Errorf("call %s emitted %d tool.start events, want 1", id, nStart) + } + if ends[id] != 1 { + t.Errorf("call %s emitted %d tool.end events, want exactly 1 (a panicking tool must still close its pair)", id, ends[id]) + } + } + panics := 0 + for i, p := range results { + if tr := p.(*message.ToolResult); tr.IsError && strings.Contains(partsText(tr.Content), "panicked") { + panics++ + _ = i + } + } + if panics != 4 { + t.Errorf("%d panic results, want 4 (one per panicking call, siblings unaffected)", panics) + } +} + +// ---- Finding 1: retention accounting under a wide batch ---- + +// TestAdvRetentionCeilingExactUnderWideBatch stresses the per-session +// retained-bytes ceiling with far more concurrent retentions than the +// ceiling admits. The ceiling's check-then-act is split across two +// separate s.mu sections, so it is only safe because retention runs at the +// JOIN, single-threaded, in call order. If it ever moves back inside a +// worker, this test should overshoot. +func TestAdvRetentionCeilingExactUnderWideBatch(t *testing.T) { + const n = 32 + const each = 8000 + const ceiling = 5 * each + + dir := t.TempDir() + s := NewSession(Config{ + SessionDir: dir, + ToolResultInlineBytes: 500, + ToolResultRetainedBytes: ceiling, + ToolConcurrency: 8, + }) + s.tools["big"] = Tool{ + Def: provider.ToolDef{Name: "big", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(_ context.Context, _ *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + return message.Parts{&message.Text{Text: in.Call + strings.Repeat("q", each)}}, nil + }, + } + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("big", fmt.Sprintf("c%02d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + s.mu.Lock() + used, handles := s.toolResultBytes, len(s.toolResults) + s.mu.Unlock() + + // On-disk truth, not just the counter. + var onDisk int64 + entries, _ := os.ReadDir(s.toolResultsDir()) + for _, e := range entries { + if fi, err := e.Info(); err == nil { + onDisk += fi.Size() + } + } + t.Logf("%d concurrent oversized results, ceiling %d: counter=%d on-disk=%d handles=%d", + n, ceiling, used, onDisk, handles) + if used > ceiling { + t.Errorf("counter %d OVER the %d ceiling — concurrent check-then-act overshot", used, ceiling) + } + if onDisk > int64(ceiling) { + t.Errorf("on-disk %d OVER the %d ceiling", onDisk, ceiling) + } + if handles == 0 || handles >= n { + t.Errorf("handles=%d of %d; want some but not all, so the ceiling actually bound", handles, n) + } + + // Handles must be minted in CALL order, not completion order. + var seen []string + for _, p := range results { + txt := partsText(p.(*message.ToolResult).Content) + if i := strings.Index(txt, "handle=trh_"); i >= 0 { + rest := txt[i+len("handle=trh_"):] + end := strings.IndexAny(rest, " \n") + if end < 0 { + end = len(rest) + } + seen = append(seen, rest[:end]) + } + } + for i := 1; i < len(seen); i++ { + a, b := seen[i-1], seen[i] + if len(a) > len(b) || (len(a) == len(b) && a >= b) { + t.Errorf("handles not minted in call order: %v", seen) + break + } + } + t.Logf("handles in call order: trh_%s", strings.Join(seen, ", trh_")) +} + +// ---- Finding 5: hook ordering ---- + +// TestAdvHookPhasesOrderedPerCallAcrossABatch pins what the hook contract +// does and does not promise under a parallel batch. WITHIN one call the +// phases stay strictly ordered (before, then the tool, then after) and each +// fires exactly once. ACROSS calls the relative order is completion order, +// which is the documented, accepted change. +func TestAdvHookPhasesOrderedPerCallAcrossABatch(t *testing.T) { + const n = 8 + var mu sync.Mutex + phases := map[string][]string{} + + record := func(call, phase string) { + mu.Lock() + phases[call] = append(phases[call], phase) + mu.Unlock() + } + + hooks := &phaseHooks{ + onBefore: func(req *plugin.ToolExecuteBeforeRequest) { record(req.CallID, "before") }, + onAfter: func(req *plugin.ToolExecuteAfterRequest) { record(req.CallID, "after") }, + } + + s := NewSession(Config{ToolConcurrency: 8, Hooks: hooks}) + s.tools["t"] = batchTool("t", func(_ context.Context, call string) { + record("tc_"+call, "tool") + }) + + calls := make([]*message.ToolCall, n) + for i := range calls { + calls[i] = batchCall("t", fmt.Sprintf("c%d", i)) + } + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + mu.Lock() + defer mu.Unlock() + if len(phases) != n { + t.Fatalf("saw %d calls in hook records, want %d: %v", len(phases), n, phases) + } + for call, got := range phases { + want := []string{"before", "tool", "after"} + if len(got) != len(want) { + t.Errorf("call %s phases = %v, want exactly %v (each hook once)", call, got, want) + continue + } + for i := range want { + if got[i] != want[i] { + t.Errorf("call %s phases = %v, want %v", call, got, want) + break + } + } + } +} + +// ---- Finding 6: does the exclusion guard fail safe or fail open? ---- + +// TestAdvUnparseableKeyArgsFallBackToOneSharedKey probes the guard's +// failure mode. A file tool whose args do not carry a usable path cannot +// have its path computed, and the question is whether that yields NO key +// (fail open, calls race) or a SHARED key (fail safe, calls serialize). +func TestAdvUnparseableKeyArgsFallBackToOneSharedKey(t *testing.T) { + s := NewSession(Config{WorkDir: t.TempDir()}) + for _, args := range []string{`{}`, `{"path":""}`, `not json at all`, `{"path":123}`} { + got := filePathKey(s, json.RawMessage(args)) + if got == "" { + t.Errorf("args %s produced an EMPTY key — calls would run unserialized (fail OPEN)", args) + continue + } + if got != filePathKeyPrefix+"" { + t.Logf("args %s -> %q", args, got) + } + } + // All unparseable spellings must land on ONE key so they serialize + // with each other, not on distinct keys. + a := filePathKey(s, json.RawMessage(`{}`)) + b := filePathKey(s, json.RawMessage(`nonsense`)) + if a != b { + t.Errorf("two unparseable arg shapes took different keys (%q, %q) — they would race", a, b) + } +} + +// TestAdvSequentialFloorCannotBeEscaped checks the cap resolution itself +// never fails open into unbounded parallelism. +func TestAdvSequentialFloorCannotBeEscaped(t *testing.T) { + for _, in := range []int{-1, -8, -1 << 30} { + if got := resolveToolConcurrency(in); got != 1 { + t.Errorf("resolveToolConcurrency(%d) = %d, want 1 (a negative value must clamp to sequential, never unbounded)", in, got) + } + } + if got := resolveToolConcurrency(0); got != defaultToolConcurrency { + t.Errorf("resolveToolConcurrency(0) = %d, want the package default %d", got, defaultToolConcurrency) + } +} + +// ---- Finding 7: is the synchronization guarding the right thing? ---- + +// TestAdvSharedSessionStateUnderWideMixedBatch drives every piece of +// session state a batch touches at once — events, the exec counter, +// retention, the read-hash set, per-path keys — with the race detector as +// the oracle. It is deliberately mixed: same-path file tools beside +// unrelated ones, so both the keyed and unkeyed paths run together. +func TestAdvSharedSessionStateUnderWideMixedBatch(t *testing.T) { + dir := t.TempDir() + shared := filepath.Join(dir, "shared.txt") + if err := os.WriteFile(shared, []byte("seed"), 0o644); err != nil { + t.Fatal(err) + } + + var events int64 + s := NewSession(Config{ + WorkDir: dir, + SessionDir: dir, + ToolResultInlineBytes: 400, + ToolConcurrency: 8, + OnEvent: func(Event) { atomic.AddInt64(&events, 1) }, + }) + s.tools["chatty"] = Tool{ + Def: provider.ToolDef{Name: "chatty", Description: "d", InputSchema: json.RawMessage(`{}`)}, + Run: func(_ context.Context, sess *Session, args json.RawMessage) (message.Parts, error) { + var in struct { + Call string `json:"call"` + } + _ = json.Unmarshal(args, &in) + _ = sess.Usage() + return message.Parts{&message.Text{Text: in.Call + strings.Repeat("m", 2000)}}, nil + }, + } + + var calls []*message.ToolCall + for i := 0; i < 6; i++ { + calls = append(calls, batchCall("chatty", fmt.Sprintf("c%d", i))) + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("rd%d", i), + Name: "read_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q}`, shared)), + }) + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("wr%d", i), + Name: "write_file", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q,"content":"round-%d"}`, shared, i)), + }) + calls = append(calls, &message.ToolCall{ + CallID: fmt.Sprintf("ls%d", i), + Name: "ls", + Arguments: json.RawMessage(fmt.Sprintf(`{"path":%q}`, dir)), + }) + } + + results := s.runToolCalls(context.Background(), asstWith(calls...)) + wantOrder(t, results, calls...) + + // The same-path read/write chain ran in call order, so the file ends + // on the LAST write. + if got, _ := os.ReadFile(shared); string(got) != "round-5" { + t.Errorf("shared file = %q, want round-5 (the last same-path write in call order)", got) + } + s.mu.Lock() + execs := s.toolExecCount + s.mu.Unlock() + if execs != len(calls) { + t.Errorf("toolExecCount = %d, want %d (one per call, no lost increment)", execs, len(calls)) + } + if n := atomic.LoadInt64(&events); n < int64(2*len(calls)) { + t.Errorf("emitted %d events, want at least %d (a start and an end per call)", n, 2*len(calls)) + } +} + +// TestAdvKeyChainIsOnlyUsedFromTheSubmittingGoroutine guards keyChain's +// documented no-lock premise: it carries no mutex because wait() is called +// only from runParallelSegment's setup loop, never from a worker. A future +// change that moves the call into a goroutine must add a lock with it, and +// this test is the tripwire. +func TestAdvKeyChainIsOnlyUsedFromTheSubmittingGoroutine(t *testing.T) { + src, err := os.ReadFile("toolexec.go") + if err != nil { + t.Fatal(err) + } + body := string(src) + i := strings.Index(body, "func (s *Session) runParallelSegment") + if i < 0 { + t.Fatal("runParallelSegment not found") + } + fn := body[i:] + if end := strings.Index(fn, "\n}\n"); end > 0 { + fn = fn[:end] + } + callIdx := strings.Index(fn, "chain.wait(") + goIdx := strings.Index(fn, "go func(j job)") + if callIdx < 0 || goIdx < 0 { + t.Fatalf("expected both chain.wait( and the worker goroutine in runParallelSegment") + } + if callIdx > goIdx { + t.Errorf("chain.wait is called at/after the worker goroutine launch — keyChain has no mutex and now needs one") + } +} + +// phaseHooks records the tool.execute.before/after phases per call. Every +// method must be safe for concurrent use: the engine dispatches hooks from +// several goroutines at once for one batch. +type phaseHooks struct { + onBefore func(*plugin.ToolExecuteBeforeRequest) + onAfter func(*plugin.ToolExecuteAfterRequest) +} + +func (h *phaseHooks) ChatParams(_ context.Context, req *plugin.ChatParamsRequest) plugin.ChatParams { + return req.Params +} +func (h *phaseHooks) ChatMessage(_ context.Context, req *plugin.ChatMessageRequest) message.Message { + return req.Message +} +func (h *phaseHooks) SystemTransform(context.Context, *plugin.SystemTransformRequest) []string { + return nil +} +func (h *phaseHooks) ShellEnv(context.Context, *plugin.ShellEnvRequest) map[string]string { + return nil +} +func (h *phaseHooks) ToolExecuteBefore(_ context.Context, req *plugin.ToolExecuteBeforeRequest) (json.RawMessage, string) { + h.onBefore(req) + return req.Args, "" +} +func (h *phaseHooks) ToolExecuteAfter(_ context.Context, req *plugin.ToolExecuteAfterRequest) message.Parts { + h.onAfter(req) + return req.Output +} +func (h *phaseHooks) ExecuteTool(_ context.Context, req *plugin.ToolExecuteRequest) (*plugin.ToolExecuteResponse, error) { + return nil, fmt.Errorf("plugin: no plugin provides tool %q", req.Tool) +} +func (h *phaseHooks) Emit([]plugin.Event) {} +func (h *phaseHooks) Plugins() []plugin.Info { return nil } +func (h *phaseHooks) Tools() []plugin.ToolDef { return nil } From eb085ea19e0e0504a458a427d4799eb36f6e26ca Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 10:11:26 -0400 Subject: [PATCH 16/95] fix(engine): announce an AGENTS.md truncation to the model and the log (#204) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An oversize project-instruction file was cut in silence. `validateInstructions` (engine/instructions.go) capped the file at a hardcoded 64 KiB and appended the fixed string "[... truncated: AGENTS.md exceeds 64 KiB ...]" — a marker that named no path, no size, and no way to read the rest. The boxes repository has a 408 KiB AGENTS.md. That file reached the model as 64 KiB, with 344 KiB dropped. A model that reads a specification cut at an arbitrary byte has no way to know the file continues: it follows half a specification and believes it read the whole one. The operator saw nothing either — no log line reported the cut. The cut is now loud on both channels. `truncateInstructions` appends the in-band marker `formatTruncationMarker` builds, which names the path, the original size, the kept size, the dropped size, and the `read_file` tool that reads the rest — so the model can recover the dropped content itself. It also writes one WARN log line with the same counts, so an operator can see which project overruns the cap. The `[... ... ...]` bracket form is this repository's own marker convention (engine/messagepage.go, engine/toolresult_tool.go); it serves the purpose the fx harness's inline markers serve. The cap is configurable. `InstructionsConfig.MaxBytes` follows `Config.ToolReadBudgetBytes`'s idiom exactly: zero takes the default (`defaultMaxInstructionsBytes`, 64 KiB, unchanged), a positive value sets the cap, and a negative value disables the cap so the whole file is injected. A project with a large AGENTS.md and a large context window can now pay for the whole file instead of losing it. Config key `instructions_max_bytes` (bytes) and the operator seam `HARNESS_INSTRUCTIONS_MAX_KB` (kilobytes, negative disables) resolve it in `cmd/harness`, the environment variable winning; the engine still reads no environment variable itself. An alternative — failing the first Prompt on an oversize file, as an invalid-UTF-8 file already does — was rejected. Size is not malformedness, and a hard failure would stop every session in a repository whose AGENTS.md grew past the cap, which is a worse outcome than a marked, recoverable cut. Semantic change: an under-cap file is untouched and logs nothing, exactly as before. Discovery, caching, the hard-failure contract for an unusable file, and segment ordering are unchanged. `loadInstructions` and `validateInstructions` take the resolved cap as a parameter, so the engine holds no truncation constant a caller cannot override. Verification: red-verified against the exact mechanism. With `truncateInstructions` reverted to the old fixed marker and no log call, `TestInstructionsTruncationIsLoud` fails on every named field — the path, the three byte counts, the `read_file` pointer, and the missing WARN line — and `TestInstructionsTruncationRuneBoundary` fails on the reported kept count. Both are green with the change applied. The cap table covers zero, positive, negative, a cap of one byte, and a cap inside a multi-byte rune; the cmd/harness table covers env-over-config precedence, a negative value, a malformed value, a nil config, the path override, and both disable paths. Package tests pass with -race across engine, config, cmd/harness, and server. Co-authored-by: andybons --- AGENTS.md | 24 ++- cmd/harness/instructions_max_test.go | 99 +++++++++++++ cmd/harness/main.go | 47 +++++- config/config.go | 14 ++ config/config_test.go | 19 +++ engine/instructions.go | 106 +++++++++++--- engine/instructions_test.go | 40 ++--- engine/instructions_truncate_test.go | 211 +++++++++++++++++++++++++++ 8 files changed, 500 insertions(+), 60 deletions(-) create mode 100644 cmd/harness/instructions_max_test.go create mode 100644 engine/instructions_truncate_test.go diff --git a/AGENTS.md b/AGENTS.md index 5346601e..ba1b79f0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,9 +110,27 @@ session, and never written to the session log (loaded fresh on resume). A present-but-unusable file (invalid UTF-8, or empty/whitespace-only) fails the first `Prompt` — a project that meant to supply instructions must not run -silently without them. A missing file is fine. Oversize files are truncated at -64 KiB with a marker. Disable with `-no-instructions`, config `instructions: -false`, or point at a specific file with config `instructions_path`. +silently without them. A missing file is fine. Disable with +`-no-instructions`, config `instructions: false`, or point at a specific file +with config `instructions_path`. + +An oversize file is truncated, and the truncation is LOUD on both channels. +`truncateInstructions` (`engine/instructions.go`) appends the in-band marker +`formatTruncationMarker` builds — it names the path, the original size, the +kept size, the dropped size, and the `read_file` tool that reads the rest — +and writes one WARN log line with the same counts. A silent cut was the +earlier behavior: a 408 KiB `AGENTS.md` reached the model as 64 KiB with no +sign of the missing 344 KiB, so the model followed a half specification and +believed it read the whole one. A truncated instruction file must always +announce itself; never make this cut quiet again. + +`InstructionsConfig.MaxBytes` sets the cap: 0 (the zero value) takes +`defaultMaxInstructionsBytes` (64 KiB), a positive value sets it, a NEGATIVE +value disables the cap so the whole file is injected. Config key +`instructions_max_bytes` (bytes) and the operator seam +`HARNESS_INSTRUCTIONS_MAX_KB` (kilobytes; negative disables) resolve it in +`cmd/harness`, the environment variable winning — the engine never reads an +environment variable itself. ### Agent Skills diff --git a/cmd/harness/instructions_max_test.go b/cmd/harness/instructions_max_test.go new file mode 100644 index 00000000..0362c0a3 --- /dev/null +++ b/cmd/harness/instructions_max_test.go @@ -0,0 +1,99 @@ +package main + +import ( + "testing" + + "github.com/majorcontext/harness/config" +) + +// TestInstructionsMaxBytesKnob pins the instruction cap seam: +// HARNESS_INSTRUCTIONS_MAX_KB (kilobytes) overrides config +// `instructions_max_bytes` (bytes), and neither set leaves the engine +// default (a nil InstructionsConfig, or a zero MaxBytes) in place. +func TestInstructionsMaxBytesKnob(t *testing.T) { + tests := []struct { + name string + env string + cfg *config.Config + want int // expected ic.MaxBytes + nilOK bool // a nil InstructionsConfig is the expected result + }{ + {name: "unset stays nil", cfg: &config.Config{}, nilOK: true}, + {name: "config bytes", cfg: &config.Config{InstructionsMaxBytes: 4096}, want: 4096}, + {name: "config disables the cap", cfg: &config.Config{InstructionsMaxBytes: -1}, want: -1}, + {name: "env kilobytes", env: "8", cfg: &config.Config{}, want: 8 * 1024}, + {name: "env wins over config", env: "8", cfg: &config.Config{InstructionsMaxBytes: 4096}, want: 8 * 1024}, + {name: "env disables the cap", env: "-1", cfg: &config.Config{InstructionsMaxBytes: 4096}, want: -1}, + {name: "malformed env falls back to config", env: "banana", cfg: &config.Config{InstructionsMaxBytes: 4096}, want: 4096}, + {name: "zero env falls back to config", env: "0", cfg: &config.Config{InstructionsMaxBytes: 4096}, want: 4096}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", tc.env) + ic := instructionsConfig(tc.cfg, false) + if tc.nilOK { + if ic != nil { + t.Fatalf("ic = %+v, want nil (engine default)", ic) + } + return + } + if ic == nil { + t.Fatalf("ic = nil, want MaxBytes %d", tc.want) + } + if ic.MaxBytes != tc.want { + t.Errorf("ic.MaxBytes = %d, want %d", ic.MaxBytes, tc.want) + } + if ic.Disabled { + t.Errorf("ic.Disabled = true, want instructions enabled") + } + }) + } +} + +// TestInstructionsMaxBytesNilConfig verifies the operator knob still applies +// when no config file was loaded: a nil config means "no config", not "keep +// the default cap". +func TestInstructionsMaxBytesNilConfig(t *testing.T) { + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", "8") + ic := instructionsConfig(nil, false) + if ic == nil || ic.MaxBytes != 8*1024 { + t.Fatalf("ic = %+v, want MaxBytes 8192", ic) + } + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", "") + if ic := instructionsConfig(nil, false); ic != nil { + t.Fatalf("ic = %+v, want nil with no config and no env", ic) + } +} + +// TestInstructionsMaxBytesWithPathOverride verifies the cap rides the +// explicit-path branch too: a project that names its own instruction file +// still gets the configured cap. +func TestInstructionsMaxBytesWithPathOverride(t *testing.T) { + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", "") + ic := instructionsConfig(&config.Config{InstructionsPath: "x/AGENTS.md", InstructionsMaxBytes: 4096}, false) + if ic == nil || ic.Path != "x/AGENTS.md" || ic.MaxBytes != 4096 { + t.Fatalf("ic = %+v, want path x/AGENTS.md with MaxBytes 4096", ic) + } +} + +// TestInstructionsMaxBytesNeverEnablesDisabled verifies -no-instructions and +// `instructions: false` still win: a cap must never re-enable injection. +func TestInstructionsMaxBytesNeverEnablesDisabled(t *testing.T) { + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", "8") + falseV := false + for _, tc := range []struct { + name string + cfg *config.Config + noInstructions bool + }{ + {"flag", &config.Config{InstructionsMaxBytes: 4096}, true}, + {"config false", &config.Config{Instructions: &falseV, InstructionsMaxBytes: 4096}, false}, + } { + t.Run(tc.name, func(t *testing.T) { + ic := instructionsConfig(tc.cfg, tc.noInstructions) + if ic == nil || !ic.Disabled { + t.Fatalf("ic = %+v, want disabled", ic) + } + }) + } +} diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 26f9b680..c4c4ca5e 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -1759,22 +1759,57 @@ func loadSessionFn(mkCfg func(message.ModelRef) engine.Config, defModel message. // instructionsConfig translates the -no-instructions flag and config file // fields into the engine's InstructionsConfig. Precedence: the flag disables // unconditionally; otherwise config `instructions: false` disables, config -// `instructions_path` names an override, and anything else returns nil (the -// engine default: auto-discover AGENTS.md by walking up from WorkDir). +// `instructions_path` names an override, `instructions_max_bytes` (or +// HARNESS_INSTRUCTIONS_MAX_KB) sets the injection cap, and a config that sets +// none of them returns nil (the engine default: auto-discover AGENTS.md by +// walking up from WorkDir, capped at 64 KiB). func instructionsConfig(cfg *config.Config, noInstructions bool) *engine.InstructionsConfig { if noInstructions { return &engine.InstructionsConfig{Disabled: true} } if cfg == nil { - return nil + // A nil config still honors the environment knob: the caller has no + // config file, not a demand for the default cap. + cfg = &config.Config{} } if cfg.Instructions != nil && !*cfg.Instructions { return &engine.InstructionsConfig{Disabled: true} } - if cfg.InstructionsPath != "" { - return &engine.InstructionsConfig{Path: cfg.InstructionsPath} + maxBytes := instructionsMaxBytes(cfg) + if cfg.InstructionsPath == "" && maxBytes == 0 { + return nil } - return nil + return &engine.InstructionsConfig{Path: cfg.InstructionsPath, MaxBytes: maxBytes} +} + +// instructionsMaxBytes resolves engine.InstructionsConfig.MaxBytes from the +// operator knob HARNESS_INSTRUCTIONS_MAX_KB and the config key +// `instructions_max_bytes`. The engine never reads an environment variable +// itself, so this is the seam (same shape as toolReadBudgetBytes above). +// +// The environment variable counts KILOBYTES, because an instruction file is +// a human-sized document and the engine default reads as "64 KiB" everywhere. +// A positive value sets the cap, a NEGATIVE value disables it (the whole file +// is injected), and unset/zero/malformed falls through to the config key. +// Zero from both leaves the engine default of 64 KiB. +func instructionsMaxBytes(cfg *config.Config) int { + raw := os.Getenv("HARNESS_INSTRUCTIONS_MAX_KB") + if n, err := strconv.Atoi(raw); err == nil && n != 0 { + if n < 0 { + // Any negative value means "no cap"; normalize to -1 rather + // than passing a large negative through as a byte count. + return -1 + } + const kib = 1 << 10 + if n > (1<<62)/kib { + return 0 // absurd; fall back to the config key or the default + } + return n * kib + } + if cfg.InstructionsMaxBytes < 0 { + return -1 + } + return cfg.InstructionsMaxBytes } // skillsDirs resolves the effective Agent Skills directories for the engine. diff --git a/config/config.go b/config/config.go index 91b06855..e113905a 100644 --- a/config/config.go +++ b/config/config.go @@ -42,6 +42,15 @@ type Config struct { // InstructionsPath overrides the auto-discovered AGENTS.md with a specific // file to load instead of walking up from the working directory. InstructionsPath string `json:"instructions_path,omitempty"` + // InstructionsMaxBytes sets engine.InstructionsConfig.MaxBytes: how many + // bytes of the instruction file reach the system prompt. Zero (omitted, + // the default) keeps the engine default of 64 KiB. A positive value sets + // the cap. A NEGATIVE value disables the cap, so the whole file is + // injected — a project with a large AGENTS.md and a large context window + // can pay for the whole file. Truncation is always loud: the model reads + // an in-band marker and the operator reads a WARN log line. + // HARNESS_INSTRUCTIONS_MAX_KB overrides this key (see cmd/harness). + InstructionsMaxBytes int `json:"instructions_max_bytes,omitempty"` // SkillsDirs lists directories scanned for Agent Skills (agentskills.io). // A nil (omitted) value leaves the engine default in place: use // /.agents/skills when it exists. In the project-config merge a @@ -828,6 +837,8 @@ func Path() string { // - Model, SessionDir, InstructionsPath, GoalEvaluatorModel, SessionSync: a // non-empty project value overrides the user value. Instructions and // ModelTool (*bool): a non-nil project value overrides. +// InstructionsMaxBytes: a non-zero project value overrides, so a project +// sets its own cap (or -1 for no cap) over the user value. // - SkillsDirs, AgentDefsDirs: a non-empty project slice replaces the user // slice entirely (arrays override, they do not concatenate); an // empty/omitted project value inherits the user value. @@ -963,6 +974,9 @@ func merge(base, over *Config) *Config { if over.InstructionsPath != "" { out.InstructionsPath = over.InstructionsPath } + if over.InstructionsMaxBytes != 0 { + out.InstructionsMaxBytes = over.InstructionsMaxBytes + } if over.GoalEvaluatorModel != "" { out.GoalEvaluatorModel = over.GoalEvaluatorModel } diff --git a/config/config_test.go b/config/config_test.go index dda69ab4..99a5b666 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -1,6 +1,7 @@ package config import ( + "encoding/json" "fmt" "os" "path/filepath" @@ -630,6 +631,24 @@ func TestMergeInstructions(t *testing.T) { t.Errorf("merged InstructionsPath = %q, want inherited user/AGENTS.md", merged.InstructionsPath) } }) + t.Run("max bytes: project overrides, zero inherits", func(t *testing.T) { + base := &Config{InstructionsMaxBytes: 4096} + if got := merge(base, &Config{InstructionsMaxBytes: -1}).InstructionsMaxBytes; got != -1 { + t.Errorf("merged InstructionsMaxBytes = %d, want -1 (project no-cap wins)", got) + } + if got := merge(base, &Config{}).InstructionsMaxBytes; got != 4096 { + t.Errorf("merged InstructionsMaxBytes = %d, want inherited 4096", got) + } + }) + t.Run("max bytes parses from JSON", func(t *testing.T) { + var c Config + if err := json.Unmarshal([]byte(`{"instructions_max_bytes": 131072}`), &c); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if c.InstructionsMaxBytes != 131072 { + t.Errorf("InstructionsMaxBytes = %d, want 131072", c.InstructionsMaxBytes) + } + }) } func TestModelToolConfig(t *testing.T) { diff --git a/engine/instructions.go b/engine/instructions.go index fac317c5..720047b9 100644 --- a/engine/instructions.go +++ b/engine/instructions.go @@ -21,6 +21,7 @@ package engine import ( "fmt" + "log/slog" "os" "path/filepath" "strings" @@ -31,13 +32,11 @@ import ( // directory, in preference order (AGENTS.md wins over the singular AGENT.md). var instructionsFilenames = []string{"AGENTS.md", "AGENT.md"} -// maxInstructionsBytes caps how much of an instruction file is read into the -// system prompt. Content beyond the cap is dropped and replaced with -// truncationMarker. -const maxInstructionsBytes = 64 * 1024 - -// truncationMarker is appended when an instruction file exceeds the cap. -const truncationMarker = "[... truncated: AGENTS.md exceeds 64 KiB ...]" +// defaultMaxInstructionsBytes caps how much of an instruction file is read +// into the system prompt when InstructionsConfig.MaxBytes is unset. Content +// beyond the cap is dropped and replaced with the marker +// formatTruncationMarker builds. +const defaultMaxInstructionsBytes = 64 * 1024 // InstructionsConfig controls project-instruction (AGENTS.md) injection. As a // field on Config it has three meaningful states: @@ -52,6 +51,47 @@ type InstructionsConfig struct { // Path, when non-empty, is a specific instruction file to load instead of // auto-discovering AGENTS.md. Path string + // MaxBytes caps the instruction bytes injected into the system prompt. + // Zero (the zero value) takes defaultMaxInstructionsBytes (64 KiB); a + // positive value sets the cap; a NEGATIVE value disables the cap, so the + // whole file is injected however large it is. Truncation is always loud — + // see truncateInstructions. + MaxBytes int +} + +// resolveInstructionsMaxBytes reports the instruction byte cap for ic. A nil +// ic, or a zero MaxBytes, takes the default; a negative MaxBytes is passed +// through unchanged and disables the cap. +func resolveInstructionsMaxBytes(ic *InstructionsConfig) int { + if ic == nil || ic.MaxBytes == 0 { + return defaultMaxInstructionsBytes + } + return ic.MaxBytes +} + +// formatTruncationMarker builds the in-band marker that replaces the dropped +// tail of an oversize instruction file. The marker is VISIBLE to the model on +// purpose: an instruction file cut in half without a word is +// indistinguishable from a file that simply ends there, so a model follows a +// half specification and believes it read the whole one. The marker names the +// path, the three byte counts, and the tool that reads the rest, so the model +// can recover the dropped content itself. +// +// The `[... ... ...]` bracket form is this repository's own marker +// convention (see engine/messagepage.go and engine/toolresult_tool.go). It +// serves the same purpose as the fx harness's inline markers: +// a truncation the reader can see. +// +// path is the ABSOLUTE path, while the segment header above it +// (formatInstructions) shows the short display path. The two differ on +// purpose: the header names the file for a reader, the marker names an +// argument the model gives to read_file, and an absolute path resolves the +// same from any working directory. +func formatTruncationMarker(path string, total, kept int) string { + return fmt.Sprintf( + "[... truncated: %s is %d bytes. The first %d bytes are above. %d bytes are not shown. Read the full file with the read_file tool. ...]", + path, total, kept, total-kept, + ) } // loadInstructions searches from workDir upward for AGENTS.md (falling back to @@ -59,11 +99,12 @@ type InstructionsConfig struct { // relative to workDir. The walk stops at the first directory containing a .git // entry — that directory is checked for an instructions file before stopping — // or at the filesystem root. A missing file yields empty strings and no error. -func loadInstructions(workDir string) (content, path string, err error) { +// maxBytes is the resolved byte cap (see resolveInstructionsMaxBytes). +func loadInstructions(workDir string, maxBytes int) (content, path string, err error) { dir := workDir for { if p, data, found := readInstructionFile(dir); found { - body, err := validateInstructions(p, data) + body, err := validateInstructions(p, data, maxBytes) if err != nil { return "", "", err } @@ -102,23 +143,45 @@ func readInstructionFile(dir string) (path string, data []byte, found bool) { // UTF-8, or empty/whitespace-only) is a hard error — the project meant to // supply instructions and the agent must not silently run without them. Size // is not malformedness: an oversize file is truncated, not rejected. -func validateInstructions(path string, data []byte) (string, error) { +func validateInstructions(path string, data []byte, maxBytes int) (string, error) { if !utf8.Valid(data) { return "", fmt.Errorf("engine: instructions file %s is not valid UTF-8", path) } if strings.TrimSpace(string(data)) == "" { return "", fmt.Errorf("engine: instructions file %s is empty", path) } - if len(data) > maxInstructionsBytes { - // Trim any trailing partial rune so the truncated body stays valid - // UTF-8 (the full data is already known valid above). - capped := data[:maxInstructionsBytes] - for len(capped) > 0 && !utf8.Valid(capped) { - capped = capped[:len(capped)-1] - } - return string(capped) + "\n" + truncationMarker, nil + return truncateInstructions(path, data, maxBytes), nil +} + +// truncateInstructions applies the byte cap to an already-validated +// instruction file. It is LOUD on both channels: the model reads the in-band +// marker formatTruncationMarker builds, and the operator reads one WARN log +// line with the original and the kept byte counts. Neither channel existed +// before: a 408 KiB AGENTS.md was cut to 64 KiB in silence, and no reader — +// model or operator — could tell the file was incomplete. +// +// A negative maxBytes disables the cap. A file at or under the cap is +// returned verbatim, with no marker and no log line. The instruction file is +// read once per session (ensureInstructions caches the segment), so an +// oversize file writes one WARN line per session, never one per request. +func truncateInstructions(path string, data []byte, maxBytes int) string { + if maxBytes < 0 || len(data) <= maxBytes { + return string(data) + } + // Trim any trailing partial rune so the truncated body stays valid + // UTF-8 (the full data is already known valid by the caller). + capped := data[:maxBytes] + for len(capped) > 0 && !utf8.Valid(capped) { + capped = capped[:len(capped)-1] } - return string(data), nil + slog.Warn("engine: instructions file truncated", + "path", path, + "original_bytes", len(data), + "kept_bytes", len(capped), + "dropped_bytes", len(data)-len(capped), + "limit_bytes", maxBytes, + ) + return string(capped) + "\n" + formatTruncationMarker(path, len(data), len(capped)) } // isDir reports whether path is a directory. @@ -171,6 +234,7 @@ func (s *Session) buildInstructionSegment() (string, error) { if ic != nil && ic.Disabled { return "", nil } + maxBytes := resolveInstructionsMaxBytes(ic) if ic != nil && ic.Path != "" { // A relative override resolves against the session's WorkDir, not // the process cwd — embedders may set WorkDir independently. @@ -182,14 +246,14 @@ func (s *Session) buildInstructionSegment() (string, error) { if err != nil { return "", nil // missing/unreadable override: no segment, no error } - body, err := validateInstructions(path, data) + body, err := validateInstructions(path, data, maxBytes) if err != nil { return "", err } s.instrPath = ic.Path return formatInstructions(ic.Path, body), nil } - content, path, err := loadInstructions(s.cfg.WorkDir) + content, path, err := loadInstructions(s.cfg.WorkDir, maxBytes) if err != nil { return "", err } diff --git a/engine/instructions_test.go b/engine/instructions_test.go index 13c70450..66c226f0 100644 --- a/engine/instructions_test.go +++ b/engine/instructions_test.go @@ -28,7 +28,7 @@ func mkdirAll(t *testing.T, path string) { func TestLoadInstructionsFoundInWorkDir(t *testing.T) { dir := t.TempDir() writeInstr(t, filepath.Join(dir, "AGENTS.md"), "be terse") - content, path, err := loadInstructions(dir) + content, path, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -45,7 +45,7 @@ func TestLoadInstructionsWalksUp(t *testing.T) { writeInstr(t, filepath.Join(root, "AGENTS.md"), "root rules") sub := filepath.Join(root, "a", "b") mkdirAll(t, sub) - content, path, err := loadInstructions(sub) + content, path, err := loadInstructions(sub, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -65,7 +65,7 @@ func TestLoadInstructionsGitRoot(t *testing.T) { mkdirAll(t, filepath.Join(repo, ".git")) sub := filepath.Join(repo, "pkg") mkdirAll(t, sub) - content, path, err := loadInstructions(sub) + content, path, err := loadInstructions(sub, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -81,7 +81,7 @@ func TestLoadInstructionsGitRoot(t *testing.T) { writeInstr(t, filepath.Join(repo, "AGENTS.md"), "repo rules") sub := filepath.Join(repo, "pkg") mkdirAll(t, sub) - content, _, err := loadInstructions(sub) + content, _, err := loadInstructions(sub, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -95,7 +95,7 @@ func TestLoadInstructionsMissing(t *testing.T) { dir := t.TempDir() // Bound the walk with a .git so it cannot escape to a real AGENTS.md. mkdirAll(t, filepath.Join(dir, ".git")) - content, path, err := loadInstructions(dir) + content, path, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -109,7 +109,7 @@ func TestLoadInstructionsAgentMdFallback(t *testing.T) { dir := t.TempDir() mkdirAll(t, filepath.Join(dir, ".git")) writeInstr(t, filepath.Join(dir, "AGENT.md"), "singular fallback") - content, path, err := loadInstructions(dir) + content, path, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -125,7 +125,7 @@ func TestLoadInstructionsAgentMdFallback(t *testing.T) { mkdirAll(t, filepath.Join(dir, ".git")) writeInstr(t, filepath.Join(dir, "AGENTS.md"), "plural wins") writeInstr(t, filepath.Join(dir, "AGENT.md"), "singular loses") - content, path, err := loadInstructions(dir) + content, path, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -143,7 +143,7 @@ func TestLoadInstructionsFollowsSymlink(t *testing.T) { if err := os.Symlink(real, filepath.Join(dir, "AGENTS.md")); err != nil { t.Skipf("symlink unsupported: %v", err) } - content, _, err := loadInstructions(dir) + content, _, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err != nil { t.Fatalf("loadInstructions: %v", err) } @@ -152,26 +152,6 @@ func TestLoadInstructionsFollowsSymlink(t *testing.T) { } } -func TestLoadInstructionsTruncatesAtCap(t *testing.T) { - dir := t.TempDir() - body := strings.Repeat("x", 70*1024) - writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) - content, _, err := loadInstructions(dir) - if err != nil { - t.Fatalf("loadInstructions: %v", err) - } - if !strings.HasPrefix(content, strings.Repeat("x", 64*1024)) { - t.Errorf("expected 64 KiB of body before the marker") - } - if !strings.HasSuffix(content, "\n"+truncationMarker) { - t.Errorf("expected trailing truncation marker, got %d bytes", len(content)) - } - capped := strings.TrimSuffix(content, "\n"+truncationMarker) - if len(capped) != 64*1024 { - t.Errorf("body not capped at 64 KiB: got %d bytes before marker", len(capped)) - } -} - func TestLoadInstructionsMalformed(t *testing.T) { t.Run("invalid UTF-8 errors", func(t *testing.T) { dir := t.TempDir() @@ -179,7 +159,7 @@ func TestLoadInstructionsMalformed(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, "AGENTS.md"), []byte{0xff, 0xfe, 0xfd}, 0o644); err != nil { t.Fatal(err) } - _, _, err := loadInstructions(dir) + _, _, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err == nil { t.Fatal("expected error for invalid UTF-8") } @@ -191,7 +171,7 @@ func TestLoadInstructionsMalformed(t *testing.T) { dir := t.TempDir() mkdirAll(t, filepath.Join(dir, ".git")) writeInstr(t, filepath.Join(dir, "AGENTS.md"), " \n\t \n") - _, _, err := loadInstructions(dir) + _, _, err := loadInstructions(dir, defaultMaxInstructionsBytes) if err == nil { t.Fatal("expected error for whitespace-only file") } diff --git a/engine/instructions_truncate_test.go b/engine/instructions_truncate_test.go new file mode 100644 index 00000000..56d99f58 --- /dev/null +++ b/engine/instructions_truncate_test.go @@ -0,0 +1,211 @@ +package engine + +import ( + "bytes" + "log/slog" + "os" + "path/filepath" + "strconv" + "strings" + "testing" +) + +// captureLogs redirects the default slog logger into a buffer for the test. +// slog.SetDefault is process-global, so a test that calls captureLogs must +// not call t.Parallel(). +func captureLogs(t *testing.T) *bytes.Buffer { + t.Helper() + var buf bytes.Buffer + prev := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) + t.Cleanup(func() { slog.SetDefault(prev) }) + return &buf +} + +// TestInstructionsTruncationIsLoud pins the loud-truncation contract: an +// oversize AGENTS.md keeps its head, carries an in-band marker that names the +// path and both byte sizes, and writes one WARN log line. A silent cut once +// dropped 344 KiB of a 408 KiB AGENTS.md with the model never told. +func TestInstructionsTruncationIsLoud(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + body := strings.Repeat("x", 70*1024) + writeInstr(t, path, body) + + buf := captureLogs(t) + content, _, err := loadInstructions(dir, defaultMaxInstructionsBytes) + if err != nil { + t.Fatalf("loadInstructions: %v", err) + } + + head := strings.Repeat("x", defaultMaxInstructionsBytes) + if !strings.HasPrefix(content, head) { + t.Errorf("truncated content lost the head: got %d bytes", len(content)) + } + marker := strings.TrimPrefix(content, head) + for _, want := range []string{ + "truncated", + path, + strconv.Itoa(len(body)), // original size + strconv.Itoa(defaultMaxInstructionsBytes), // kept size + strconv.Itoa(len(body) - defaultMaxInstructionsBytes), // dropped size + "read_file", + } { + if !strings.Contains(marker, want) { + t.Errorf("marker %q does not contain %q", marker, want) + } + } + + if !strings.HasPrefix(marker, "\n[... truncated:") || !strings.HasSuffix(marker, "...]") { + t.Errorf("marker %q does not use the [... ... ...] bracket form", marker) + } + + out := buf.String() + if !strings.Contains(out, "WARN") { + t.Errorf("expected a WARN log line, got:\n%s", out) + } + for _, want := range []string{"instructions", path, strconv.Itoa(len(body)), strconv.Itoa(defaultMaxInstructionsBytes)} { + if !strings.Contains(out, want) { + t.Errorf("log line %q does not contain %q", out, want) + } + } +} + +// TestInstructionsUnderCapUntouched verifies an under-cap file gets no marker +// and no log line. +func TestInstructionsUnderCapUntouched(t *testing.T) { + dir := t.TempDir() + writeInstr(t, filepath.Join(dir, "AGENTS.md"), "small and complete") + + buf := captureLogs(t) + content, _, err := loadInstructions(dir, defaultMaxInstructionsBytes) + if err != nil { + t.Fatalf("loadInstructions: %v", err) + } + if content != "small and complete" { + t.Errorf("content = %q, want the file verbatim", content) + } + if out := buf.String(); out != "" { + t.Errorf("under-cap file logged: %s", out) + } +} + +// TestInstructionsMaxBytesConfigurable pins InstructionsConfig.MaxBytes: zero +// takes the 64 KiB default, a positive value sets the cap, and a negative +// value disables truncation for a deployment that wants the whole file. +func TestInstructionsMaxBytesConfigurable(t *testing.T) { + tests := []struct { + name string + maxBytes int + size int + want int // kept body bytes before the marker + }{ + {"zero takes the default", 0, 70 * 1024, defaultMaxInstructionsBytes}, + {"positive sets the cap", 1024, 4096, 1024}, + {"negative disables the cap", -1, 70 * 1024, 70 * 1024}, + {"cap of one keeps one byte", 1, 100, 1}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + body := strings.Repeat("x", tc.size) + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + + content, _, err := loadInstructions(dir, resolveInstructionsMaxBytes(&InstructionsConfig{MaxBytes: tc.maxBytes})) + if err != nil { + t.Fatalf("loadInstructions: %v", err) + } + kept := len(content) + if i := strings.Index(content, "[..."); i >= 0 { + kept = len(strings.TrimSuffix(content[:i], "\n")) + } + if kept != tc.want { + t.Errorf("kept %d body bytes, want %d", kept, tc.want) + } + if tc.want == tc.size && strings.Contains(content, "[...") { + t.Errorf("an untruncated file must carry no marker: %q", content[len(content)-100:]) + } + }) + } +} + +// TestInstructionsTruncationRuneBoundary verifies a cap that lands inside a +// multi-byte rune trims back to a rune boundary, and that the marker reports +// the KEPT byte count after that trim, not the cap. +func TestInstructionsTruncationRuneBoundary(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + // Each "é" is 2 bytes, so a cap of 5 lands mid-rune: 4 bytes are kept. + body := strings.Repeat("é", 8) + writeInstr(t, path, body) + captureLogs(t) + + content, _, err := loadInstructions(dir, 5) + if err != nil { + t.Fatalf("loadInstructions: %v", err) + } + kept, _, ok := strings.Cut(content, "\n[...") + if !ok { + t.Fatalf("content carries no marker: %q", content) + } + if kept != strings.Repeat("é", 2) { + t.Errorf("kept = %q, want 2 whole runes", kept) + } + if !strings.Contains(content, "The first 4 bytes are above") { + t.Errorf("marker must report 4 kept bytes, not the cap of 5: %q", content) + } + if !strings.Contains(content, "12 bytes are not shown") { + t.Errorf("marker must report 12 dropped bytes: %q", content) + } + + // A cap below the first rune keeps nothing, and still says so. + degenerate, _, err := loadInstructions(dir, 1) + if err != nil { + t.Fatalf("loadInstructions: %v", err) + } + if !strings.HasPrefix(degenerate, "\n[... truncated:") { + t.Errorf("cap below one rune must keep no content: %q", degenerate) + } + if !strings.Contains(degenerate, "The first 0 bytes are above") { + t.Errorf("marker must report 0 kept bytes: %q", degenerate) + } +} + +// TestInstructionsSessionCapFromConfig drives the cap through a real session: +// the injected system segment must carry the marker when Config.Instructions +// sets a small MaxBytes. +func TestInstructionsSessionCapFromConfig(t *testing.T) { + dir := t.TempDir() + writeInstr(t, filepath.Join(dir, "AGENTS.md"), strings.Repeat("y", 4096)) + captureLogs(t) + + prov := instrSession(t, Config{WorkDir: dir, Instructions: &InstructionsConfig{MaxBytes: 512}}, 1) + seg := prov.requests[0].System[2] + if !strings.Contains(seg, "[...") { + t.Errorf("segment carries no truncation marker: %q", seg) + } + if !strings.Contains(seg, strings.Repeat("y", 512)) || strings.Contains(seg, strings.Repeat("y", 513)) { + t.Errorf("segment does not carry exactly 512 kept body bytes: %q", seg) + } +} + +// TestInstructionsPathOverrideTruncationIsLoud verifies the explicit-path +// branch shares the cap and the marker with auto-discovery. +func TestInstructionsPathOverrideTruncationIsLoud(t *testing.T) { + dir := t.TempDir() + override := filepath.Join(dir, "custom.md") + if err := os.WriteFile(override, []byte(strings.Repeat("z", 2048)), 0o644); err != nil { + t.Fatal(err) + } + buf := captureLogs(t) + + prov := instrSession(t, Config{WorkDir: dir, Instructions: &InstructionsConfig{Path: override, MaxBytes: 256}}, 1) + seg := prov.requests[0].System[2] + if !strings.Contains(seg, "[...") { + t.Errorf("override segment carries no truncation marker: %q", seg) + } + if !strings.Contains(buf.String(), "custom.md") { + t.Errorf("expected a WARN line naming custom.md, got:\n%s", buf.String()) + } +} From 6c27d31ccb469539b335a9faaf891f71d6c8ba93 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 11:26:24 -0400 Subject: [PATCH 17/95] feat(engine): split an oversize AGENTS.md into a head and an outline (#206) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(engine): design for nested, on-demand instruction loading The loud-truncation change made an oversize AGENTS.md announce itself, but the dropped bytes are still absent from the prompt. This repository's own AGENTS.md is 180,631 bytes, so 116 KiB of binding specification never reaches the model; the boxes repository's 408 KiB file loses 344 KiB. This design splits the instruction segment into a head (eager, cut on a heading boundary at the cap) and an outline (one line per dropped section with its exact read_file line range and a teaser). The model pulls a section with read_file, reusing the Agent Skills stage-1/stage-2 pattern and adding no tool. A second mechanism, ported from opencode's instruction.ts resolve(), attaches a nested per-directory AGENTS.md to the tool result when the model reads a file under that directory. The document records the measured section counts and outline sizes, how the outline composes with the truncation cap (it replaces the marker, and the marker stays for a heading-less file), the configuration, and the five properties the tests must pin. It proposes a two-PR split: the system-prompt side first, the tool-result side second. No behavior change: this commit adds a design document only. (cherry picked from commit 6404154d1f5885fb13a0c3fc877c0528d848fd9f) * feat(engine): split an oversize AGENTS.md into a head and an outline The loud-truncation change made an oversize instruction file announce its own cut, but the dropped bytes stayed out of reach. The model read "352256 bytes are not shown" and had no idea what it lost, so it could only guess whether the missing text mattered. This file is the concrete case: this repository's AGENTS.md is 180,631 bytes, so 116 KiB of the binding specification every agent reads first never reached the model. The boxes repository's 408 KiB AGENTS.md lost 344 KiB the same way. renderInstructions (engine/instructions_outline.go) now splits an oversize file instead of only marking it. The head is every section that fits whole under the cap. The outline replaces the dropped tail: one line per section the head does not carry, each naming the heading, the exact read_file range that reads it — read_file(path=, offset=, limit=) — and a short teaser from the section body. Measured on this repository's AGENTS.md, the segment is a 41,648-byte head plus a 7,772-byte outline of 36 sections, and every one of those sections is one tool call away. The retrieval tool is read_file itself, whose offset and limit are already 1-based line numbers, so this adds no tool and no schema to any request. A dedicated `instructions` tool with outline/section actions was rejected for that reason: it would pay a schema in every request for a capability read_file already has, and Agent Skills stage 2 already reuses read_file for exactly this. An expandable reference marker was rejected too — an expansion protocol the model can invoke is a tool by another name, and it would put a second retrieval path next to read_file for the same bytes. The split composes with the cap in one direction: the cap decides what is EAGER, the outline makes everything else REACHABLE, and nothing is dropped in silence either way. Three shapes keep the marker. A file with fewer than two sections has nothing to outline. InstructionsConfig.Mode InstructionsModeFull selects the marker path for every file. And a file whose FIRST section alone exceeds the cap has no boundary to cut on, so the head is that truncated first section, with the marker and the WARN line both firing and the outline still listing every later section. That last shape is the one place an outline could hide a cut, so truncateInstructionsOf reports the WHOLE file's size there, never the first section's — a marker reading "is 303 bytes" for a 3.4 KiB file was a real defect caught in test. scanSections tracks fenced code blocks by fence CHARACTER and RUN LENGTH, the full CommonMark rule, not with a boolean. This repository's own AGENTS.md holds bash blocks whose comments start with '#', so a fence-blind scan advertises a range that points at a shell comment. The character and run-length rules cover the next shape up, which review raised: an instruction file that documents Markdown wraps a three-backtick example in a four-backtick fence, and a boolean toggle closes the outer fence on the inner one and reads the rest of the document as false sections. No file in this repository holds that shape today, so those two rules are hardening for other projects' files, not a fix for an observed break. The outline block is bounded by outlineMaxBytes (8 KiB): teasers are dropped for the whole block when it does not fit, and a section is never dropped, because a listed section must always be reachable. Semantic change: a file at or under the cap is injected verbatim, exactly as before, and a disabled cap still injects the whole file. Discovery, caching, the hard-failure contract for an unusable file, and segment ordering are unchanged. Config key `instructions_mode` and the operator seam HARNESS_INSTRUCTIONS_MODE select the mode in cmd/harness; only the value "full" (case-insensitive) turns the outline off, because an unreadable knob must not quietly drop an outline nobody asked to lose. Verification: three mutations, each red-verified on the named mechanism. Advertising startLine+1 fails TestInstructionsOutlineRangesAreExact, which drives the REAL read_file tool with the outline's own ranges and compares the returned lines against the file — never the outline against itself. Cutting the head silently fails TestInstructionsOutlineGiantFirstSectionStaysLoud on both the missing marker and the missing WARN line. A fence-blind scan fails TestInstructionsOutlineFenceAware, and a boolean fence toggle fails all three wrapped-fence cases of TestScanSectionsFenceRules. TestInstructionsOutlineCoversEveryLine and TestInstructionsOutlineCoversEveryLineRich are rapid property tests proving the head lines plus the outlined ranges cover every line of the file exactly once, with no gap and no overlap — the rich one over documents with preambles, wrapped fences, mixed heading levels, CRLF endings, and no trailing newline. Package tests pass across engine, config, and cmd/harness. --------- Co-authored-by: andybons --- AGENTS.md | 41 ++ cmd/harness/instructions_max_test.go | 41 ++ cmd/harness/main.go | 24 +- config/config.go | 11 + config/config_test.go | 16 + docs/design/nested-instruction-loading.md | 189 +++++++ engine/instructions.go | 42 +- engine/instructions_outline.go | 305 +++++++++++ engine/instructions_outline_test.go | 631 ++++++++++++++++++++++ 9 files changed, 1289 insertions(+), 11 deletions(-) create mode 100644 docs/design/nested-instruction-loading.md create mode 100644 engine/instructions_outline.go create mode 100644 engine/instructions_outline_test.go diff --git a/AGENTS.md b/AGENTS.md index ba1b79f0..35c6a30c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,6 +132,47 @@ value disables the cap so the whole file is injected. Config key `cmd/harness`, the environment variable winning — the engine never reads an environment variable itself. +An oversize file is not merely marked, it is SPLIT. +`renderInstructions` (`engine/instructions_outline.go`) injects a HEAD plus an +OUTLINE: the head is every section that fits whole under the cap, and the +outline lists every section the head does not carry — one line each with the +heading, the exact `read_file` range that reads it +(`read_file(path=, offset=, limit=)`), and a short teaser +from the section body. The model pulls a section with `read_file`, whose +`offset`/`limit` are already 1-based line numbers, so this adds NO tool and no +schema to any request. The shape is the Agent Skills stage-1/stage-2 split +below, applied to one file: the outline is an index the model MUST read +through before it relies on a section. This file is itself over the cap, so +this file is itself split: the head carries the sections that fit, and every +later section — including this one — reaches the model only when it reads the +advertised range. Nothing is out of reach, where the marker alone left the +whole tail unreachable. + +`scanSections` tracks fenced code blocks by FENCE CHARACTER AND RUN LENGTH, +not with a boolean. A `#` comment inside a ```` ```bash ```` block read as a +heading would advertise a range that points at a shell comment. The character +and run-length rules cover the next shape up: an instruction file that +documents Markdown wraps a three-backtick example in a four-backtick fence, +which a naive toggle closes early, turning the rest of the document into +false sections. + +The split composes with the cap in ONE direction: the cap decides what is +EAGER, the outline makes everything else REACHABLE, and nothing is dropped in +silence either way. Three shapes keep the marker. A file with fewer than two +sections (no heading, or one giant section) has nothing to outline and takes +the `truncateInstructions` path unchanged. `InstructionsConfig.Mode` +`InstructionsModeFull` selects that path for every file. And a file whose +FIRST section alone exceeds the cap has no boundary to cut on, so the head is +that truncated first section — marker and WARN line both firing, through +`truncateInstructionsOf`, which reports the WHOLE file's size and not the +first section's — with the outline still listing every later section. Never +let the head lose its marker in that shape: it is the one place where an +outline could hide a cut. Config key `instructions_mode` and the operator seam +`HARNESS_INSTRUCTIONS_MODE` resolve the mode in `cmd/harness`; only the value +`full` (case-insensitive) turns the outline off, because an unreadable knob +must not quietly drop an outline nobody asked to lose. Design: +docs/design/nested-instruction-loading.md. + ### Agent Skills The engine advertises [Agent Skills](https://agentskills.io) in the system diff --git a/cmd/harness/instructions_max_test.go b/cmd/harness/instructions_max_test.go index 0362c0a3..04e2b746 100644 --- a/cmd/harness/instructions_max_test.go +++ b/cmd/harness/instructions_max_test.go @@ -4,6 +4,7 @@ import ( "testing" "github.com/majorcontext/harness/config" + "github.com/majorcontext/harness/engine" ) // TestInstructionsMaxBytesKnob pins the instruction cap seam: @@ -97,3 +98,43 @@ func TestInstructionsMaxBytesNeverEnablesDisabled(t *testing.T) { }) } } + +// TestInstructionsModeKnob pins the render-mode seam: HARNESS_INSTRUCTIONS_MODE +// overrides config `instructions_mode`, only "full" turns the outline off, and +// an unreadable value keeps the outline. +func TestInstructionsModeKnob(t *testing.T) { + tests := []struct { + name string + env string + cfg string + want engine.InstructionsMode + }{ + {name: "unset is auto", want: engine.InstructionsModeAuto}, + {name: "config full", cfg: "full", want: engine.InstructionsModeFull}, + {name: "config auto", cfg: "auto", want: engine.InstructionsModeAuto}, + {name: "env full wins", env: "full", cfg: "auto", want: engine.InstructionsModeFull}, + {name: "env auto wins", env: "auto", cfg: "full", want: engine.InstructionsModeAuto}, + {name: "case insensitive", cfg: "FULL", want: engine.InstructionsModeFull}, + {name: "unknown value keeps the outline", cfg: "outline-please", want: engine.InstructionsModeAuto}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("HARNESS_INSTRUCTIONS_MODE", tc.env) + t.Setenv("HARNESS_INSTRUCTIONS_MAX_KB", "") + got := instructionsMode(&config.Config{InstructionsMode: tc.cfg}) + if got != tc.want { + t.Errorf("instructionsMode = %q, want %q", got, tc.want) + } + ic := instructionsConfig(&config.Config{InstructionsMode: tc.cfg}, false) + if tc.want == engine.InstructionsModeFull { + if ic == nil || ic.Mode != engine.InstructionsModeFull { + t.Errorf("ic = %+v, want Mode full", ic) + } + return + } + if ic != nil && ic.Mode != engine.InstructionsModeAuto { + t.Errorf("ic = %+v, want Mode auto", ic) + } + }) + } +} diff --git a/cmd/harness/main.go b/cmd/harness/main.go index c4c4ca5e..07cabd7c 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -368,6 +368,25 @@ func toolReadBudgetBytes() int64 { return n * mib } +// instructionsMode resolves engine.InstructionsConfig.Mode from the operator +// knob HARNESS_INSTRUCTIONS_MODE and the config key `instructions_mode`, the +// environment variable winning (same shape as instructionsMaxBytes above). +// +// "auto" and an empty value both mean the head-plus-outline rendering for an +// oversize file. Only the exact value "full" selects the head-plus-marker +// rendering; any other value falls back to auto, because an unreadable knob +// must not quietly drop the outline an operator never asked to lose. +func instructionsMode(cfg *config.Config) engine.InstructionsMode { + raw := os.Getenv("HARNESS_INSTRUCTIONS_MODE") + if raw == "" { + raw = cfg.InstructionsMode + } + if strings.EqualFold(strings.TrimSpace(raw), string(engine.InstructionsModeFull)) { + return engine.InstructionsModeFull + } + return engine.InstructionsModeAuto +} + // sessionDir resolves where session logs live, in precedence order: // -no-save (yields "", persistence disabled) > $HARNESS_SESSION_DIR > // configDir (config session_dir) > $HOME/.harness/sessions. Nothing is @@ -1776,10 +1795,11 @@ func instructionsConfig(cfg *config.Config, noInstructions bool) *engine.Instruc return &engine.InstructionsConfig{Disabled: true} } maxBytes := instructionsMaxBytes(cfg) - if cfg.InstructionsPath == "" && maxBytes == 0 { + mode := instructionsMode(cfg) + if cfg.InstructionsPath == "" && maxBytes == 0 && mode == engine.InstructionsModeAuto { return nil } - return &engine.InstructionsConfig{Path: cfg.InstructionsPath, MaxBytes: maxBytes} + return &engine.InstructionsConfig{Path: cfg.InstructionsPath, MaxBytes: maxBytes, Mode: mode} } // instructionsMaxBytes resolves engine.InstructionsConfig.MaxBytes from the diff --git a/config/config.go b/config/config.go index e113905a..4afbc922 100644 --- a/config/config.go +++ b/config/config.go @@ -51,6 +51,13 @@ type Config struct { // an in-band marker and the operator reads a WARN log line. // HARNESS_INSTRUCTIONS_MAX_KB overrides this key (see cmd/harness). InstructionsMaxBytes int `json:"instructions_max_bytes,omitempty"` + // InstructionsMode selects how an OVERSIZE instruction file is rendered: + // "auto" (omitted, the default) splits it into a head plus an outline of + // the sections the head does not carry, each outline line naming the + // exact read_file range that reads it; "full" keeps the head-plus-marker + // rendering with no outline. HARNESS_INSTRUCTIONS_MODE overrides this key + // (see cmd/harness). See engine.InstructionsMode. + InstructionsMode string `json:"instructions_mode,omitempty"` // SkillsDirs lists directories scanned for Agent Skills (agentskills.io). // A nil (omitted) value leaves the engine default in place: use // /.agents/skills when it exists. In the project-config merge a @@ -839,6 +846,7 @@ func Path() string { // ModelTool (*bool): a non-nil project value overrides. // InstructionsMaxBytes: a non-zero project value overrides, so a project // sets its own cap (or -1 for no cap) over the user value. +// InstructionsMode: a non-empty project value overrides. // - SkillsDirs, AgentDefsDirs: a non-empty project slice replaces the user // slice entirely (arrays override, they do not concatenate); an // empty/omitted project value inherits the user value. @@ -977,6 +985,9 @@ func merge(base, over *Config) *Config { if over.InstructionsMaxBytes != 0 { out.InstructionsMaxBytes = over.InstructionsMaxBytes } + if over.InstructionsMode != "" { + out.InstructionsMode = over.InstructionsMode + } if over.GoalEvaluatorModel != "" { out.GoalEvaluatorModel = over.GoalEvaluatorModel } diff --git a/config/config_test.go b/config/config_test.go index 99a5b666..a82a6cf1 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -640,6 +640,15 @@ func TestMergeInstructions(t *testing.T) { t.Errorf("merged InstructionsMaxBytes = %d, want inherited 4096", got) } }) + t.Run("mode: project overrides, empty inherits", func(t *testing.T) { + base := &Config{InstructionsMode: "full"} + if got := merge(base, &Config{InstructionsMode: "auto"}).InstructionsMode; got != "auto" { + t.Errorf("merged InstructionsMode = %q, want auto (project wins)", got) + } + if got := merge(base, &Config{}).InstructionsMode; got != "full" { + t.Errorf("merged InstructionsMode = %q, want inherited full", got) + } + }) t.Run("max bytes parses from JSON", func(t *testing.T) { var c Config if err := json.Unmarshal([]byte(`{"instructions_max_bytes": 131072}`), &c); err != nil { @@ -648,6 +657,13 @@ func TestMergeInstructions(t *testing.T) { if c.InstructionsMaxBytes != 131072 { t.Errorf("InstructionsMaxBytes = %d, want 131072", c.InstructionsMaxBytes) } + var m Config + if err := json.Unmarshal([]byte(`{"instructions_mode": "full"}`), &m); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if m.InstructionsMode != "full" { + t.Errorf("InstructionsMode = %q, want full", m.InstructionsMode) + } }) } diff --git a/docs/design/nested-instruction-loading.md b/docs/design/nested-instruction-loading.md new file mode 100644 index 00000000..685299f1 --- /dev/null +++ b/docs/design/nested-instruction-loading.md @@ -0,0 +1,189 @@ +# Nested, on-demand instruction loading + +Status: the head-plus-outline half is IMPLEMENTED +(`engine/instructions_outline.go`). The nested attach-on-read half, ported +from opencode's `resolve()`, is still design only — see "Proposed split". + +## Problem + +`engine/instructions.go` injects one project instruction file into the system +prompt and caps it (`InstructionsConfig.MaxBytes`, 64 KiB by default). The cap +is now loud on both channels: the model reads a marker, the operator reads a +WARN line. Loud is not enough. The content past the cap is still absent from +the prompt, and the model must guess that it needs the rest. + +Two files make this concrete. The boxes repository has a 408 KiB `AGENTS.md`: +84% of it never reaches the model. This repository's own `AGENTS.md` is +180,631 bytes over 2,866 lines with 51 headings: 116 KiB of binding +specification is dropped from every session today. + +## What the model sees + +The instruction segment gets two parts instead of one. + +**The head** is the file up to the last complete section boundary at or under +`MaxBytes`. It is the same eager, always-present text the segment carries +today, cut on a heading rather than on an arbitrary byte, so a section is +never shown half. + +**The outline** replaces the dropped tail. One line per section that the head +does not fully contain, each line carrying the heading text, the exact +`read_file` line range, and a short teaser from the section body: + +``` +Project instructions from AGENTS.md (sections 17-51 are not in this prompt). +Read a section with the read_file tool before you rely on it: + read_file(path: /repo/AGENTS.md, offset: , limit: ) + + Goal loop — lines 659-1041 — Session.PursueGoal drives the ordinary Prompt + loop toward a natural-language completion condition... + Session metadata index — lines 1042-1104 — ... +``` + +This is the engine's existing Agent Skills stage-1/stage-2 split, applied to +one file instead of a directory of them: an index the model must read through +before it relies on a section. The wording is the skills wording ("you MUST +read ... before relying on it"), so the model treats both the same way. + +## The mechanism it uses to pull a section + +`read_file`, with no new tool. Its `offset` and `limit` are already 1-based +line numbers (`engine/filetools.go`), the outline supplies exact ranges, and +the read costs one ordinary tool call that the tool-read budget and the +read-set bookkeeping already govern. A dedicated `instructions` tool with +`outline`/`section` actions was considered and rejected: it adds a schema to +every request for a capability `read_file` already has, and skills stage 2 +already reuses `read_file` for exactly this. + +Reference markers the model "expands" were also rejected. An expansion marker +needs a protocol the model can invoke, which is a tool by another name, and it +would put a second retrieval path next to `read_file` for the same bytes. + +## Nested files, adapted from opencode `resolve()` + +opencode's `instruction.ts` `resolve()` (~179-221) is the second half: when +the model reads a file, it walks up from that file's directory to the project +root, finds an `AGENTS.md` that is neither in the system prompt nor already +attached, and appends it to that read's result — deduped per message through a +claims map and a scan of earlier completed read parts. fx scopes instruction +loading the same way: instructions arrive when the work enters their scope. + +The harness port fits the tool-result path, not the system prompt. +`read_file` (and `edit_file`/`write_file`) resolves a path through +`s.resolvePath`; from that resolved path the engine walks up to +`Config.WorkDir`, stopping at the git root exactly as `loadInstructions` does, +and appends a `message.Text` part per newly found nested instruction file to +that tool result. A per-session attached-set — runtime only, never persisted, +the shape `Session.recordRead` already uses in `engine/filetools.go` — makes +each nested file arrive at most once per session. The root file the system +prompt already carries is always skipped. Nested files pass through the same +`truncateInstructions` cap, so a large nested file is still loud. + +## How this composes with the truncation cap + +They stack in one direction: the cap decides what is EAGER, the outline makes +everything else REACHABLE. Nothing is silently dropped in either mode — the +invariant the loud-truncation change established holds unchanged. + +- File at or under the cap: byte-identical to today. No outline, no marker. +- File over the cap with usable headings: head + outline. The outline carries + the loud notice, so it REPLACES the truncation marker for this file. The + WARN log line stays, with the section counts added. +- File over the cap with no usable headings (one giant section, a generated + file): the head + marker path from the loud-truncation change, unchanged. + This fallback is why that marker stays in the code. +- `MaxBytes` negative (cap disabled): the whole file is injected, no outline. + +The outline has its own budget so a pathological file cannot spend the prompt +on an index: teasers are dropped first, then the list degrades to headings and +ranges only. Measured on this repository's `AGENTS.md`: 35 outlined sections +cost 5,965 bytes with teasers, 1,424 bytes without. A 408 KiB file with the +same heading density holds roughly 115 sections, so its outline lands near +19 KiB with teasers and near 4.7 KiB without — the budget picks the second. + +## The 408 KiB boxes AGENTS.md, concretely + +Eager: the head, the first sections up to the last heading boundary under +64 KiB. Measured on this repository's file, that boundary is byte 41,648 — +sections 1-16 of 51. Cutting on a heading costs 23 KiB of eager text against +a raw byte cut, and buys a head that never ends mid-sentence; the sections +that pay that cost are listed in the outline, so they are reachable, not lost. + +On demand: sections 17-51, each one `read_file` call away, with the exact +range in the prompt. Today those 116 KiB are unreachable. For the boxes file +the same split makes 344 KiB reachable instead of dropped. + +## Configuration + +`InstructionsConfig.Mode`: `auto` (the default — outline when the file is over +the cap and has usable headings), `full` (the pre-outline behavior: head plus +the truncation marker). Config key `instructions_mode`, operator seam +`HARNESS_INSTRUCTIONS_MODE`, resolved in `cmd/harness` like +`HARNESS_INSTRUCTIONS_MAX_KB`. Nested attachment gets its own switch, +`instructions_nested` (default on), because it changes tool results rather +than the system prompt. + +## What this design does not do + +It does not follow Markdown links to other documents. A section that names +`docs/design/context-compaction.md` gives the model a path, and `read_file` +reads it; a link crawler would pull unbounded content nobody asked for. + +It does not re-read the file per request. The segment and the outline are +built from the single read `ensureInstructions` already performs, cached for +the session, and never written to the session log. + +## Risks the tests must pin + +1. **Line-number accuracy.** An off-by-one makes every outline range wrong. + The test drives the real `read_file` tool with the outline's own ranges and + compares the returned lines against the file, rather than comparing the + outline to itself. +2. **Fenced code blocks.** This repository's `AGENTS.md` holds ```` ```bash ```` + blocks whose comment lines start with `#`. A naive heading scan reads them + as sections and emits ranges that point at shell comments. The scanner + tracks fences; a test file with a `#` line inside a fence pins it. +3. **Head boundary.** The head must end on a heading boundary and must never + exceed `MaxBytes`. A file whose first heading is past the cap has no + boundary to cut on and falls back to the marker path. +4. **Every byte is accounted for.** Head lines plus outlined ranges must cover + the whole file with no gap and no overlap. This is one property test over + generated files, and it is the strongest guard the design has. +5. **Nested attachment fires once.** A second read under the same directory + attaches nothing; a reload starts with an empty attached set. + +## Proposed split + +The head, the outline, and the mode switch land first — the system-prompt side +that the 408 KiB file needs. The nested attach-on-read port of `resolve()` +follows in its own change. They touch different paths (system prompt versus +tool result), carry different failure modes, and each one is small enough to +review to zero. Landing them together would put a prompt-assembly change and a +tool-result change under one review. + +## What the implementation changed against this design + +Two details moved during implementation, both recorded here so the document +matches the code. + +The head is cut at the last section that fits WHOLE under the cap, and the +outline lists every section after it — the design said the same, but did not +state what happens when the FIRST section alone exceeds the cap. It now takes +the loud path: the head is that section truncated by `truncateInstructionsOf`, +which reports the whole file's byte size (not the section's), so the marker +and the WARN line both fire while the outline still lists every later section. + +The outline budget degrades in one step, not two. `formatOutline` builds the +block with teasers, and rebuilds it without teasers when the block exceeds +`outlineMaxBytes` (8 KiB). It never drops a section: a listed section is +always reachable, which is the property the loud-truncation rule demands. + +Fence tracking needed the full CommonMark rule, not a boolean. A fence closes +only on the same character it opened with, and only on at least as many of +them. An instruction file that documents Markdown wraps a three-backtick +example in a four-backtick fence, and a boolean toggle closes the outer fence +on the inner one, then reads the rest of the document as sections. +`fenceState` in `engine/instructions_outline.go` carries the open fence's +character and run length. A review of the implementation raised this shape; +this repository's own AGENTS.md does not hold such a fence today, so the rule +is hardening for other projects' files, not a fix for an observed break. diff --git a/engine/instructions.go b/engine/instructions.go index 720047b9..09125543 100644 --- a/engine/instructions.go +++ b/engine/instructions.go @@ -57,6 +57,11 @@ type InstructionsConfig struct { // whole file is injected however large it is. Truncation is always loud — // see truncateInstructions. MaxBytes int + // Mode selects how an OVERSIZE file is rendered: InstructionsModeAuto + // (the zero value) splits it into a head plus an outline of the sections + // the head does not carry, and InstructionsModeFull keeps the + // head-plus-marker rendering. See engine/instructions_outline.go. + Mode InstructionsMode } // resolveInstructionsMaxBytes reports the instruction byte cap for ic. A nil @@ -99,12 +104,18 @@ func formatTruncationMarker(path string, total, kept int) string { // relative to workDir. The walk stops at the first directory containing a .git // entry — that directory is checked for an instructions file before stopping — // or at the filesystem root. A missing file yields empty strings and no error. -// maxBytes is the resolved byte cap (see resolveInstructionsMaxBytes). +// maxBytes is the resolved byte cap (see resolveInstructionsMaxBytes). It +// renders in InstructionsModeAuto; loadInstructionsMode selects another mode. func loadInstructions(workDir string, maxBytes int) (content, path string, err error) { + return loadInstructionsMode(workDir, maxBytes, InstructionsModeAuto) +} + +// loadInstructionsMode is loadInstructions with an explicit render mode. +func loadInstructionsMode(workDir string, maxBytes int, mode InstructionsMode) (content, path string, err error) { dir := workDir for { if p, data, found := readInstructionFile(dir); found { - body, err := validateInstructions(p, data, maxBytes) + body, err := validateInstructions(p, data, maxBytes, mode) if err != nil { return "", "", err } @@ -143,14 +154,14 @@ func readInstructionFile(dir string) (path string, data []byte, found bool) { // UTF-8, or empty/whitespace-only) is a hard error — the project meant to // supply instructions and the agent must not silently run without them. Size // is not malformedness: an oversize file is truncated, not rejected. -func validateInstructions(path string, data []byte, maxBytes int) (string, error) { +func validateInstructions(path string, data []byte, maxBytes int, mode InstructionsMode) (string, error) { if !utf8.Valid(data) { return "", fmt.Errorf("engine: instructions file %s is not valid UTF-8", path) } if strings.TrimSpace(string(data)) == "" { return "", fmt.Errorf("engine: instructions file %s is empty", path) } - return truncateInstructions(path, data, maxBytes), nil + return renderInstructions(path, data, maxBytes, mode), nil } // truncateInstructions applies the byte cap to an already-validated @@ -165,6 +176,15 @@ func validateInstructions(path string, data []byte, maxBytes int) (string, error // read once per session (ensureInstructions caches the segment), so an // oversize file writes one WARN line per session, never one per request. func truncateInstructions(path string, data []byte, maxBytes int) string { + return truncateInstructionsOf(path, data, maxBytes, len(data)) +} + +// truncateInstructionsOf is truncateInstructions over a PREFIX of a larger +// file: total is the whole file's byte size, which the marker and the log +// line report. The outline path truncates the first section alone +// (renderInstructions), and a marker that reported that slice's size would +// tell the model the file is far smaller than it is. +func truncateInstructionsOf(path string, data []byte, maxBytes, total int) string { if maxBytes < 0 || len(data) <= maxBytes { return string(data) } @@ -176,12 +196,12 @@ func truncateInstructions(path string, data []byte, maxBytes int) string { } slog.Warn("engine: instructions file truncated", "path", path, - "original_bytes", len(data), + "original_bytes", total, "kept_bytes", len(capped), - "dropped_bytes", len(data)-len(capped), + "dropped_bytes", total-len(capped), "limit_bytes", maxBytes, ) - return string(capped) + "\n" + formatTruncationMarker(path, len(data), len(capped)) + return string(capped) + "\n" + formatTruncationMarker(path, total, len(capped)) } // isDir reports whether path is a directory. @@ -235,6 +255,10 @@ func (s *Session) buildInstructionSegment() (string, error) { return "", nil } maxBytes := resolveInstructionsMaxBytes(ic) + mode := InstructionsModeAuto + if ic != nil { + mode = ic.Mode + } if ic != nil && ic.Path != "" { // A relative override resolves against the session's WorkDir, not // the process cwd — embedders may set WorkDir independently. @@ -246,14 +270,14 @@ func (s *Session) buildInstructionSegment() (string, error) { if err != nil { return "", nil // missing/unreadable override: no segment, no error } - body, err := validateInstructions(path, data, maxBytes) + body, err := validateInstructions(path, data, maxBytes, mode) if err != nil { return "", err } s.instrPath = ic.Path return formatInstructions(ic.Path, body), nil } - content, path, err := loadInstructions(s.cfg.WorkDir, maxBytes) + content, path, err := loadInstructionsMode(s.cfg.WorkDir, maxBytes, mode) if err != nil { return "", err } diff --git a/engine/instructions_outline.go b/engine/instructions_outline.go new file mode 100644 index 00000000..a8a18ca7 --- /dev/null +++ b/engine/instructions_outline.go @@ -0,0 +1,305 @@ +// Progressive disclosure for an oversize project-instruction file. +// +// The loud-truncation cap tells the model that content was dropped, but the +// dropped content stays out of reach: the model can only guess what it lost. +// This file splits an oversize file into two parts instead — a HEAD the model +// always reads, and an OUTLINE of every section the head does not carry, each +// line naming the exact read_file range that reads it. +// +// The shape is the engine's Agent Skills stage-1/stage-2 split (see +// skillsSegment) applied to one file: an index the model MUST read through +// before it relies on a section. The retrieval tool is read_file itself, +// whose offset/limit are already 1-based line numbers (engine/filetools.go), +// so this adds no tool and no new schema to any request. +// +// Nothing is ever dropped in silence. Every section outside the head is +// listed, and a head that had to be cut mid-section still carries the loud +// truncation marker and its WARN log line. + +package engine + +import ( + "fmt" + "log/slog" + "strings" + "unicode/utf8" +) + +// InstructionsMode selects how an oversize instruction file is rendered. +type InstructionsMode string + +const ( + // InstructionsModeAuto (the zero value) outlines an oversize file that + // has usable section headings, and falls back to the loud truncation + // marker for one that does not. + InstructionsModeAuto InstructionsMode = "" + // InstructionsModeFull keeps the head-plus-marker rendering for every + // oversize file: no outline, whatever the headings look like. + InstructionsModeFull InstructionsMode = "full" +) + +// instructionsOutlineHeader opens the outline block. It is also the marker +// callers and tests split the segment on, so it must stay a single literal. +const instructionsOutlineHeader = "[instructions outline]" + +// outlineMaxBytes bounds the outline block itself, so a file of thousands of +// tiny sections cannot spend the prompt on an index. The budget drops the +// per-section teasers first, then lists headings and ranges only. +const outlineMaxBytes = 8 * 1024 + +// outlineTeaserBytes caps one section's teaser text. +const outlineTeaserBytes = 120 + +// section is one Markdown section: its heading, the 1-based INCLUSIVE line +// range it spans, and the byte range it spans as a Go slice (start inclusive, +// end exclusive, so data[startByte:endByte] is the section). +// +// The two conventions differ because their consumers differ: read_file takes +// inclusive 1-based line numbers, and Go slicing is half-open. +type section struct { + title string + startLine int // 1-based, inclusive + endLine int // 1-based, inclusive + startByte int // inclusive + endByte int // exclusive +} + +// scanSections splits Markdown into sections at ATX headings (levels 1-4). +// Text before the first heading, if any, is its own leading section so the +// line and byte accounting stays complete. +// +// The scan tracks fenced code blocks: a "# ..." line inside a ``` fence is +// body text, never a heading. This repository's own AGENTS.md holds bash +// blocks whose comments start with '#', and reading one as a section would +// advertise a range that points at a shell comment. +func scanSections(data []byte) []section { + lines := strings.Split(string(data), "\n") + // A trailing newline produces one empty final element; it is not a line. + if n := len(lines); n > 0 && lines[n-1] == "" { + lines = lines[:n-1] + } + var ( + out []section + fence fenceState + offset int + ) + for i, line := range lines { + lineNo := i + 1 + if fence.step(line) { + // The line opened or closed a fence; it is never a heading. + } else if !fence.open && headingTitle(line) != "" { + if len(out) > 0 { + out[len(out)-1].endLine = lineNo - 1 + out[len(out)-1].endByte = offset + } else if offset > 0 { + // Preamble before the first heading. + out = append(out, section{title: "(preamble)", startLine: 1, endLine: lineNo - 1, startByte: 0, endByte: offset}) + } + out = append(out, section{title: headingTitle(line), startLine: lineNo, startByte: offset}) + } + offset += len(line) + 1 // the '\n' this line ends with + } + if len(out) > 0 { + out[len(out)-1].endLine = len(lines) + out[len(out)-1].endByte = len(data) + } + return out +} + +// fenceState tracks one fenced code block across lines, following the +// CommonMark rules a single boolean cannot express: a fence closes only on the +// SAME character it opened with, and only with AT LEAST as many of them. +// +// Both rules are load-bearing for an instruction file that documents +// Markdown: such a file wraps a three-backtick example in a four-backtick +// fence, and a naive toggle closes the OUTER fence on the inner one, then +// reads the rest of the document as headings. This repository's own AGENTS.md +// does not currently hold such a fence — it holds bash blocks, which the +// simpler rule already handles — so this is hardening against a shape any +// documentation-heavy project produces, not a fix for an observed break. +type fenceState struct { + open bool + char byte + count int +} + +// step folds one line into the fence state and reports whether that line was a +// fence delimiter (which is therefore never a heading). +func (f *fenceState) step(line string) bool { + char, count, closing, ok := fenceLine(line) + if !ok { + return false + } + if !f.open { + f.open, f.char, f.count = true, char, count + return true + } + if char == f.char && count >= f.count && closing { + f.open, f.char, f.count = false, 0, 0 + return true + } + // A shorter run, the other fence character, or an info string inside an + // open fence is ordinary code content. + return false +} + +// fenceLine parses a fence delimiter: its character, its run length, whether +// it could CLOSE a fence (no info string after the run), and whether the line +// is a fence at all. Indentation over three spaces makes an indented code +// block, not a fence, so it is not one here either. +func fenceLine(line string) (char byte, count int, closing, ok bool) { + indent := 0 + for indent < len(line) && line[indent] == ' ' { + indent++ + } + if indent > 3 || indent >= len(line) { + return 0, 0, false, false + } + rest := line[indent:] + c := rest[0] + if c != '`' && c != '~' { + return 0, 0, false, false + } + n := 0 + for n < len(rest) && rest[n] == c { + n++ + } + if n < 3 { + return 0, 0, false, false + } + return c, n, strings.TrimSpace(rest[n:]) == "", true +} + +// headingTitle returns the text of an ATX heading (levels 1-4), or "" when +// line is not a heading. An empty heading ("## " with no text) reads as body +// text: CommonMark allows it, but a section with no name cannot be advertised +// in the outline, and real instruction files do not carry one. +func headingTitle(line string) string { + level := 0 + for level < len(line) && line[level] == '#' { + level++ + } + if level == 0 || level > 4 || level >= len(line) || line[level] != ' ' { + return "" + } + return strings.TrimSpace(line[level+1:]) +} + +// renderInstructions renders a validated instruction file for the system +// prompt: the whole file when it fits (or when the cap is disabled), a head +// plus an outline when it does not, and the loud head-plus-marker rendering +// when the file has no section to cut on or mode is InstructionsModeFull. +// +// path is the absolute path, because every advertised range is a read_file +// argument and an absolute path resolves the same from any working directory. +func renderInstructions(path string, data []byte, maxBytes int, mode InstructionsMode) string { + if maxBytes < 0 || len(data) <= maxBytes { + return string(data) + } + if mode == InstructionsModeFull { + return truncateInstructions(path, data, maxBytes) + } + secs := scanSections(data) + if len(secs) < 2 { + // Nothing to pull on demand: one section (or none) means the outline + // would list what the head already holds, or nothing at all. + return truncateInstructions(path, data, maxBytes) + } + + // The head holds every section that fits whole. keep is the number of + // sections the head carries. + keep := 0 + for keep < len(secs) && secs[keep].endByte <= maxBytes { + keep++ + } + + var head string + if keep == 0 { + // The first section alone exceeds the cap, so there is no boundary to + // cut on. Truncate that section and keep the cut loud: the marker and + // the WARN line both fire, exactly as they do with no outline. + head = truncateInstructionsOf(path, data[:secs[1].startByte], maxBytes, len(data)) + keep = 1 + } else { + head = string(data[:secs[keep-1].endByte]) + } + + // outlined is never empty: the last section ends at len(data), which is + // over the cap here, so the keep loop always stops before it. + outlined := secs[keep:] + slog.Warn("engine: instructions outlined", + "path", path, + "original_bytes", len(data), + "head_bytes", len(head), + "sections_total", len(secs), + "sections_outlined", len(outlined), + "limit_bytes", maxBytes, + ) + return strings.TrimRight(head, "\n") + "\n\n" + formatOutline(path, data, secs, outlined) +} + +// formatOutline renders the outline block: a notice naming how much of the +// file is absent, then one line per outlined section with its read_file +// range. Teasers are included while the block stays inside outlineMaxBytes, +// and dropped for the whole block when it does not. +func formatOutline(path string, data []byte, all, outlined []section) string { + withTeasers := outlineBlock(path, data, all, outlined, true) + if len(withTeasers) <= outlineMaxBytes { + return withTeasers + } + return outlineBlock(path, data, all, outlined, false) +} + +// outlineBlock builds the outline text, with or without teasers. +func outlineBlock(path string, data []byte, all, outlined []section, teasers bool) string { + var b strings.Builder + fmt.Fprintf(&b, "%s %d of the %d sections of %s are not in this prompt. You MUST read a section with the read_file tool before you rely on it:\n", + instructionsOutlineHeader, len(outlined), len(all), path) + for _, s := range outlined { + fmt.Fprintf(&b, " %s — read_file(path=%s, offset=%d, limit=%d)", + s.title, path, s.startLine, s.endLine-s.startLine+1) + if teasers { + if t := sectionTeaser(data, s); t != "" { + fmt.Fprintf(&b, " — %s", t) + } + } + b.WriteString("\n") + } + return strings.TrimRight(b.String(), "\n") +} + +// sectionTeaser returns the first prose of a section's body, collapsed to one +// line and capped at outlineTeaserBytes, so the outline reads like the Agent +// Skills index (name — description). +func sectionTeaser(data []byte, s section) string { + body := string(data[s.startByte:s.endByte]) + _, rest, ok := strings.Cut(body, "\n") + if !ok { + return "" + } + var words []string + for _, line := range strings.Split(rest, "\n") { + line = strings.TrimSpace(line) + if _, _, _, isFence := fenceLine(line); line == "" || isFence { + if len(words) > 0 { + break + } + continue + } + words = append(words, line) + if len(strings.Join(words, " ")) >= outlineTeaserBytes { + break + } + } + teaser := strings.Join(words, " ") + if len(teaser) > outlineTeaserBytes { + // Cut on a rune boundary: a heading or body in any non-ASCII script + // would otherwise put a partial rune in the system prompt. + cut := outlineTeaserBytes + for cut > 0 && !utf8.ValidString(teaser[:cut]) { + cut-- + } + teaser = strings.TrimSpace(teaser[:cut]) + "…" + } + return teaser +} diff --git a/engine/instructions_outline_test.go b/engine/instructions_outline_test.go new file mode 100644 index 00000000..444b1cc6 --- /dev/null +++ b/engine/instructions_outline_test.go @@ -0,0 +1,631 @@ +package engine + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + "testing" + "unicode/utf8" + + "pgregory.net/rapid" +) + +// sectionDoc builds a Markdown file of n sections, each with a heading and +// bodyLines body lines, and returns the text plus each section's 1-based +// start line. +func sectionDoc(n, bodyLines int) (text string, starts []int) { + var b strings.Builder + line := 0 + for i := 1; i <= n; i++ { + starts = append(starts, line+1) + fmt.Fprintf(&b, "## Section %d\n", i) + line++ + for j := 0; j < bodyLines; j++ { + fmt.Fprintf(&b, "body %d line %d\n", i, j) + line++ + } + } + return b.String(), starts +} + +// outlineRange is one advertised read_file range, parsed out of the rendered +// outline exactly as a model would read it. +type outlineRange struct { + path string + offset int + limit int +} + +var outlineRangeRE = regexp.MustCompile(`read_file\(path=([^,]+), offset=(\d+), limit=(\d+)\)`) + +// fatalf is the failure seam shared by *testing.T and *rapid.T. +type fatalf interface { + Fatalf(format string, args ...any) +} + +// parseOutlineRanges reads every advertised range out of a rendered segment. +func parseOutlineRanges(t fatalf, segment string) []outlineRange { + var out []outlineRange + for _, m := range outlineRangeRE.FindAllStringSubmatch(segment, -1) { + offset, err := strconv.Atoi(m[2]) + if err != nil { + t.Fatalf("offset %q: %v", m[2], err) + } + limit, err := strconv.Atoi(m[3]) + if err != nil { + t.Fatalf("limit %q: %v", m[3], err) + } + out = append(out, outlineRange{path: m[1], offset: offset, limit: limit}) + } + return out +} + +// readFileLines runs the real read_file tool over the advertised range and +// returns the plain lines it produced, with the "N→" prefixes removed. +func readFileLines(t *testing.T, workDir string, r outlineRange) []string { + t.Helper() + args, err := json.Marshal(map[string]any{"path": r.path, "offset": r.offset, "limit": r.limit}) + if err != nil { + t.Fatal(err) + } + out, err := runTool(t, readFileTool(), workDir, string(args)) + if err != nil { + t.Fatalf("read_file(offset=%d, limit=%d): %v", r.offset, r.limit, err) + } + var lines []string + for _, l := range strings.Split(out, "\n") { + if strings.HasPrefix(l, "[truncated:") { + continue + } + _, rest, ok := strings.Cut(l, "→") + if !ok { + t.Fatalf("read_file line %q has no line-number prefix", l) + } + lines = append(lines, rest) + } + return lines +} + +// TestInstructionsOutlineHeadAndSections pins the two-part segment: the head +// carries whole sections only, and every section the head does not carry is +// listed with a read_file range. +func TestInstructionsOutlineHeadAndSections(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + body, starts := sectionDoc(10, 20) + writeInstr(t, path, body) + captureLogs(t) + + // A cap of 700 bytes holds two whole sections of this document. + content, _, err := loadInstructionsMode(dir, 700, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + head, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline header:\n%s", content) + } + if strings.Contains(head, "[... truncated") { + t.Errorf("head must not carry the truncation marker when it ends on a section boundary:\n%s", head) + } + if len(head) > 700 { + t.Errorf("head is %d bytes, over the 700-byte cap", len(head)) + } + if !strings.HasSuffix(strings.TrimRight(head, "\n"), "line 19") { + t.Errorf("head must end at a section boundary, got tail %q", head[max(0, len(head)-40):]) + } + ranges := parseOutlineRanges(t, outline) + if len(ranges) == 0 { + t.Fatalf("outline advertises no ranges:\n%s", outline) + } + // Every outlined range starts at a real section start line, and the last + // section is listed. + startSet := map[int]bool{} + for _, s := range starts { + startSet[s] = true + } + for _, r := range ranges { + if !startSet[r.offset] { + t.Errorf("advertised offset %d is not a section start line %v", r.offset, starts) + } + if r.path != path { + t.Errorf("advertised path = %q, want the absolute path %q", r.path, path) + } + } + if ranges[len(ranges)-1].offset != starts[len(starts)-1] { + t.Errorf("last advertised offset = %d, want the last section start %d", ranges[len(ranges)-1].offset, starts[len(starts)-1]) + } + if !strings.Contains(outline, "read_file") { + t.Errorf("outline must name the read_file tool:\n%s", outline) + } +} + +// TestInstructionsOutlineRangesAreExact is the must-have from the design +// review: an advertised range must return EXACTLY the bytes of the section it +// names, after the head was cut on a heading boundary. The oracle is the file +// itself read through the real read_file tool, never the outline compared +// against itself. +func TestInstructionsOutlineRangesAreExact(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + body, starts := sectionDoc(12, 15) + writeInstr(t, path, body) + captureLogs(t) + + content, _, err := loadInstructionsMode(dir, 400, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + _, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline:\n%s", content) + } + fileLines := strings.Split(strings.TrimSuffix(body, "\n"), "\n") + ranges := parseOutlineRanges(t, outline) + if len(ranges) < 2 { + t.Fatalf("expected several outlined sections, got %d", len(ranges)) + } + for _, r := range ranges { + // The section this range claims: from its start line to the line + // before the next section start (or the end of the file). + end := len(fileLines) + for _, s := range starts { + if s > r.offset && s-1 < end { + end = s - 1 + } + } + want := fileLines[r.offset-1 : end] + if r.limit != len(want) { + t.Errorf("range at offset %d advertises limit %d, want %d lines", r.offset, r.limit, len(want)) + } + got := readFileLines(t, dir, r) + if strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Errorf("range at offset %d returned\n%q\nwant\n%q", r.offset, strings.Join(got, "\n"), strings.Join(want, "\n")) + } + if !strings.HasPrefix(got[0], "## ") { + t.Errorf("range at offset %d does not start at a heading: %q", r.offset, got[0]) + } + } +} + +// TestInstructionsOutlineGiantFirstSectionStaysLoud is the second must-have: +// when the FIRST section alone exceeds the cap there is no section boundary +// to cut on, so the head itself is truncated — and that truncation must stay +// loud (marker plus WARN line) while the outline still lists the rest. +func TestInstructionsOutlineGiantFirstSectionStaysLoud(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + body := "## Giant\n" + strings.Repeat("x", 4096) + "\n## Second\nsecond body\n## Third\nthird body\n" + writeInstr(t, path, body) + buf := captureLogs(t) + + content, _, err := loadInstructionsMode(dir, 512, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + head, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline:\n%s", content) + } + if !strings.Contains(head, "[... truncated:") { + t.Errorf("an over-cap head must carry the loud truncation marker:\n%s", head) + } + if !strings.Contains(head, path) { + t.Errorf("marker must name the path:\n%s", head) + } + if out := buf.String(); !strings.Contains(out, "WARN") || !strings.Contains(out, "truncated") { + t.Errorf("an over-cap head must log a WARN line, got:\n%s", out) + } + ranges := parseOutlineRanges(t, outline) + if len(ranges) != 2 { + t.Fatalf("outline lists %d sections, want the 2 sections after the giant one:\n%s", len(ranges), outline) + } + if got := readFileLines(t, dir, ranges[0]); got[0] != "## Second" { + t.Errorf("first outlined section = %q, want ## Second", got[0]) + } +} + +// TestInstructionsOutlineFenceAware verifies a '#' line inside a fenced code +// block is body text, never a section. A naive scan advertises a range that +// points at a shell comment. +func TestInstructionsOutlineFenceAware(t *testing.T) { + dir := t.TempDir() + writeInstr(t, filepath.Join(dir, "AGENTS.md"), strings.Join([]string{ + "## Real one", + "```bash", + "# not a heading", + "go test ./...", + "```", + strings.Repeat("filler line\n", 40), + "## Real two", + "tail body", + "", + }, "\n")) + captureLogs(t) + + content, _, err := loadInstructionsMode(dir, 200, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if strings.Contains(content, "not a heading") && strings.Contains(content, instructionsOutlineHeader) { + _, outline, _ := strings.Cut(content, instructionsOutlineHeader) + if strings.Contains(outline, "not a heading") { + t.Errorf("outline lists a fenced comment as a section:\n%s", outline) + } + } + for _, r := range parseOutlineRanges(t, content) { + got := readFileLines(t, dir, r) + if !strings.HasPrefix(got[0], "## ") { + t.Errorf("advertised range starts at %q, want a heading line", got[0]) + } + } +} + +// TestInstructionsOutlineFallbacks pins the two shapes that keep the +// head-plus-marker behavior: a file with no headings at all, and explicit +// full mode. +func TestInstructionsOutlineFallbacks(t *testing.T) { + t.Run("no headings", func(t *testing.T) { + dir := t.TempDir() + writeInstr(t, filepath.Join(dir, "AGENTS.md"), strings.Repeat("plain body line\n", 200)) + captureLogs(t) + content, _, err := loadInstructionsMode(dir, 256, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if strings.Contains(content, instructionsOutlineHeader) { + t.Errorf("a heading-less file must not get an outline:\n%s", content) + } + if !strings.Contains(content, "[... truncated:") { + t.Errorf("a heading-less file keeps the loud marker:\n%s", content) + } + }) + t.Run("full mode", func(t *testing.T) { + dir := t.TempDir() + body, _ := sectionDoc(8, 10) + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + content, _, err := loadInstructionsMode(dir, 256, InstructionsModeFull) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if strings.Contains(content, instructionsOutlineHeader) { + t.Errorf("full mode must not outline:\n%s", content) + } + if !strings.Contains(content, "[... truncated:") { + t.Errorf("full mode keeps the loud marker:\n%s", content) + } + }) + t.Run("under the cap", func(t *testing.T) { + dir := t.TempDir() + body, _ := sectionDoc(3, 2) + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + content, _, err := loadInstructionsMode(dir, 64*1024, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if content != body { + t.Errorf("an under-cap file must be injected verbatim:\n%q", content) + } + }) + t.Run("cap disabled", func(t *testing.T) { + dir := t.TempDir() + body, _ := sectionDoc(40, 40) + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + content, _, err := loadInstructionsMode(dir, -1, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if content != body { + t.Errorf("a disabled cap must inject the whole file, got %d of %d bytes", len(content), len(body)) + } + }) +} + +// TestInstructionsOutlineListsEverySection verifies the outline never drops a +// section silently: with a budget too small for teasers it degrades to +// headings and ranges, and every dropped section is still listed. +func TestInstructionsOutlineListsEverySection(t *testing.T) { + dir := t.TempDir() + body, starts := sectionDoc(200, 4) + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + + content, _, err := loadInstructionsMode(dir, 512, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + _, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline") + } + ranges := parseOutlineRanges(t, outline) + headSections := 0 + for _, s := range starts { + if !strings.Contains(outline, fmt.Sprintf("offset=%d,", s)) { + headSections++ + } + } + if len(ranges)+headSections != len(starts) { + t.Errorf("outline lists %d sections and the head holds %d, want %d total", len(ranges), headSections, len(starts)) + } +} + +// TestInstructionsOutlineCoversEveryLine is the accounting property: the head +// lines plus the outlined ranges cover the file exactly once — no gap, no +// overlap. It is the strongest guard the split has. +func TestInstructionsOutlineCoversEveryLine(t *testing.T) { + rapid.Check(t, func(rt *rapid.T) { + sections := rapid.IntRange(2, 30).Draw(rt, "sections") + bodyLines := rapid.IntRange(0, 12).Draw(rt, "bodyLines") + cap := rapid.IntRange(16, 4096).Draw(rt, "cap") + + dir, err := os.MkdirTemp("", "instroutline") + if err != nil { + rt.Fatalf("temp dir: %v", err) + } + defer os.RemoveAll(dir) + body, _ := sectionDoc(sections, bodyLines) + path := filepath.Join(dir, "AGENTS.md") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + rt.Fatalf("write: %v", err) + } + content, _, lerr := loadInstructionsMode(dir, cap, InstructionsModeAuto) + if lerr != nil { + rt.Fatalf("loadInstructionsMode: %v", lerr) + } + total := len(strings.Split(strings.TrimSuffix(body, "\n"), "\n")) + head, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + return // marker fallback: covered by its own test + } + headLines := len(strings.Split(strings.TrimRight(head, "\n"), "\n")) + if strings.Contains(head, "[... truncated:") { + return // truncated head: the marker path, not the accounting path + } + covered := headLines + next := headLines + 1 + for _, r := range parseOutlineRanges(rt, outline) { + if r.offset != next { + rt.Fatalf("range starts at line %d, want %d (gap or overlap)", r.offset, next) + } + covered += r.limit + next = r.offset + r.limit + } + if covered != total { + rt.Fatalf("head plus outlined ranges cover %d lines, file has %d", covered, total) + } + }) +} + +// TestInstructionsOutlineTeaserRuneSafe verifies a teaser cut at the byte cap +// lands on a rune boundary: a partial rune in the system prompt is invalid +// UTF-8 the provider can reject. +func TestInstructionsOutlineTeaserRuneSafe(t *testing.T) { + dir := t.TempDir() + // Sections 2+ carry a body of 3-byte runes, so the teaser cap lands + // inside a rune unless the cut is rune-aware. + body := "## One\n" + strings.Repeat("head body\n", 30) + + // The single ASCII byte shifts the 120-byte teaser cap into the + // middle of a 3-byte rune. + "## Two\na" + strings.Repeat("世", 200) + "\n" + + "## Three\na" + strings.Repeat("界", 200) + "\n" + writeInstr(t, filepath.Join(dir, "AGENTS.md"), body) + captureLogs(t) + + content, _, err := loadInstructionsMode(dir, 320, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + if !utf8.ValidString(content) { + t.Errorf("segment is not valid UTF-8") + } + _, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline:\n%s", content) + } + if !strings.Contains(outline, "世") || !strings.Contains(outline, "界") { + t.Errorf("teasers lost their content:\n%s", outline) + } + if strings.Contains(outline, "\uFFFD") { + t.Errorf("teaser carries a replacement rune (cut mid-rune):\n%s", outline) + } +} + +// TestScanSectionsFenceRules pins the two CommonMark fence rules a single +// boolean cannot express. An instruction file that documents Markdown wraps a +// three-backtick example in a four-backtick fence, and a naive toggle closes +// the outer fence on the inner one, then reads the rest of the document as +// headings. +func TestScanSectionsFenceRules(t *testing.T) { + titles := func(secs []section) []string { + var out []string + for _, s := range secs { + out = append(out, s.title) + } + return out + } + tests := []struct { + name string + body string + want []string + }{ + { + name: "a longer fence wraps a shorter one", + body: "## One\n````\n```bash\n# not a heading\n```\n````\n## Two\nbody\n", + want: []string{"One", "Two"}, + }, + { + name: "a tilde run does not close a backtick fence", + body: "## One\n```\n~~~\n# not a heading\n```\n## Two\nbody\n", + want: []string{"One", "Two"}, + }, + { + name: "a closing fence carries no info string", + body: "## One\n```\n```go\n# not a heading\n```\n## Two\nbody\n", + want: []string{"One", "Two"}, + }, + { + name: "deep indentation is a code block, not a fence", + body: "## One\n ```\n## Two\nbody\n", + want: []string{"One", "Two"}, + }, + { + name: "an unclosed fence swallows the rest", + body: "## One\n```\n## Two\nbody\n", + want: []string{"One"}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := titles(scanSections([]byte(tc.body))) + if strings.Join(got, "|") != strings.Join(tc.want, "|") { + t.Errorf("sections = %v, want %v", got, tc.want) + } + }) + } +} + +// TestInstructionsOutlineGiantFirstSectionRangesAreExact completes the +// giant-first-section case: the head stays inside the cap plus the marker, the +// marker reports the WHOLE file's size, and every outlined range still returns +// exactly its section through the real read_file tool. +func TestInstructionsOutlineGiantFirstSectionRangesAreExact(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "AGENTS.md") + giant := strings.Repeat("giant body line\n", 200) + body := "## Giant\n" + giant + "## Second\nsecond body\nmore second\n## Third\nthird body\n" + writeInstr(t, path, body) + captureLogs(t) + + const cap = 512 + content, _, err := loadInstructionsMode(dir, cap, InstructionsModeAuto) + if err != nil { + t.Fatalf("loadInstructionsMode: %v", err) + } + head, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + t.Fatalf("segment carries no outline:\n%s", content) + } + // The kept body stays inside the cap; the marker is the only text past it. + kept, marker, ok := strings.Cut(head, "\n[... truncated:") + if !ok { + t.Fatalf("head carries no marker:\n%s", head) + } + if len(kept) > cap { + t.Errorf("kept head is %d bytes, over the %d-byte cap", len(kept), cap) + } + // The marker reports the whole file, never the first section alone. + if !strings.Contains(marker, strconv.Itoa(len(body))) { + t.Errorf("marker must report the whole file size %d: %q", len(body), marker) + } + if !strings.Contains(marker, strconv.Itoa(len(kept))) { + t.Errorf("marker must report the kept size %d: %q", len(kept), marker) + } + fileLines := strings.Split(strings.TrimSuffix(body, "\n"), "\n") + ranges := parseOutlineRanges(t, outline) + if len(ranges) != 2 { + t.Fatalf("outline lists %d sections, want 2:\n%s", len(ranges), outline) + } + for _, r := range ranges { + got := readFileLines(t, dir, r) + want := fileLines[r.offset-1 : r.offset-1+r.limit] + if strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Errorf("range at offset %d returned %q, want %q", r.offset, got, want) + } + } +} + +// richDoc builds a Markdown document with optional preamble, fenced code +// blocks (including a wrapped fence), mixed heading levels, CRLF endings, and +// an optional missing trailing newline — the shapes the uniform generator in +// sectionDoc never produces. +func richDoc(rt *rapid.T) string { + var b strings.Builder + if rapid.Bool().Draw(rt, "preamble") { + b.WriteString("preamble prose\nmore preamble\n") + } + sections := rapid.IntRange(2, 12).Draw(rt, "sections") + for i := 1; i <= sections; i++ { + level := rapid.IntRange(1, 4).Draw(rt, "level") + fmt.Fprintf(&b, "%s Section %d\n", strings.Repeat("#", level), i) + for j := 0; j < rapid.IntRange(0, 6).Draw(rt, "bodyLines"); j++ { + fmt.Fprintf(&b, "body %d line %d\n", i, j) + } + switch rapid.IntRange(0, 2).Draw(rt, "fence") { + case 1: + b.WriteString("```bash\n# a comment, not a heading\n```\n") + case 2: + b.WriteString("````\n```md\n## quoted heading\n```\n````\n") + } + } + out := b.String() + if rapid.Bool().Draw(rt, "crlf") { + out = strings.ReplaceAll(out, "\n", "\r\n") + } + if rapid.Bool().Draw(rt, "noTrailingNewline") { + out = strings.TrimRight(out, "\r\n") + } + return out +} + +// TestInstructionsOutlineCoversEveryLineRich is the accounting property over +// documents with preambles, wrapped fences, mixed heading levels, CRLF, and a +// missing trailing newline. Head lines plus outlined ranges must cover the +// file exactly once, and no advertised range may start inside a fence. +func TestInstructionsOutlineCoversEveryLineRich(t *testing.T) { + rapid.Check(t, func(rt *rapid.T) { + body := richDoc(rt) + capBytes := rapid.IntRange(16, 2048).Draw(rt, "cap") + + dir, err := os.MkdirTemp("", "instrrich") + if err != nil { + rt.Fatalf("temp dir: %v", err) + } + defer os.RemoveAll(dir) + if err := os.WriteFile(filepath.Join(dir, "AGENTS.md"), []byte(body), 0o644); err != nil { + rt.Fatalf("write: %v", err) + } + content, _, lerr := loadInstructionsMode(dir, capBytes, InstructionsModeAuto) + if lerr != nil { + rt.Fatalf("loadInstructionsMode: %v", lerr) + } + head, outline, ok := strings.Cut(content, instructionsOutlineHeader) + if !ok { + return // marker fallback: fewer than two sections + } + fileLines := strings.Split(strings.TrimSuffix(strings.ReplaceAll(body, "\r\n", "\n"), "\n"), "\n") + ranges := parseOutlineRanges(rt, outline) + + // Every advertised range starts at a heading OUTSIDE a fence, which is + // exactly the set scanSections found: check against a fresh scan of + // the file's own lines rather than against the outline itself. + for _, r := range ranges { + line := strings.TrimRight(fileLines[r.offset-1], "\r") + if headingTitle(line) == "" { + rt.Fatalf("range at offset %d starts at %q, not a heading", r.offset, line) + } + } + if strings.Contains(head, "[... truncated:") { + return // truncated head: the marker path, not the accounting path + } + headLines := len(strings.Split(strings.TrimRight(head, "\n"), "\n")) + covered, next := headLines, headLines+1 + for _, r := range ranges { + if r.offset != next { + rt.Fatalf("range starts at line %d, want %d (gap or overlap)", r.offset, next) + } + covered += r.limit + next = r.offset + r.limit + } + if covered != len(fileLines) { + rt.Fatalf("head plus outlined ranges cover %d lines, file has %d", covered, len(fileLines)) + } + }) +} From 1d8fd5f9d5a5ee1fff61f5c3702229fc2cce7743 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 14:55:33 -0400 Subject: [PATCH 18/95] fix(engine): disambiguate folded page records (#207) Co-authored-by: andybons --- AGENTS.md | 11 +++++-- engine/compact.go | 34 +++++++++++++++++++-- engine/index.go | 34 ++++++++++++++++----- engine/messagepage.go | 60 +++++++++++++++++--------------------- engine/messagepage_test.go | 34 +++++++++++++++++++++ 5 files changed, 126 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 35c6a30c..2b2b6762 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1177,9 +1177,14 @@ and both are numbered by the same index: it meets a compact record, because the messages a fold KEPT sit in the log between the folded range and the compact record itself — undoing that backwards would be a second, subtly different implementation of a fold. -- The **fold path** then reuses `indexFold` — the same forward fold, so the - same `applyCompactRecord` — to learn which ids occupy the requested seqs, - and reads back just those records. It costs one slim pass (ids and roles, +- The **fold path** then reuses `indexFold` — the same forward fold and the + same compact-range occurrence selection — to learn which messages occupy + the requested seqs. `indexFold` carries each surviving message's journal + record ordinal beside its skeleton; `foldedPage` reads back by that ordinal, + never by message ID alone. This matters for hand-written or externally + assembled journals that repeat an ID: one occurrence can be compacted away + while another survives, and the surviving sequence entry must decode the + surviving record. The path costs one slim pass (ids, roles, and ordinals, never message bodies), which is still three orders of magnitude below materializing the history. diff --git a/engine/compact.go b/engine/compact.go index 7e69dd59..6b9a42a2 100644 --- a/engine/compact.go +++ b/engine/compact.go @@ -644,6 +644,17 @@ func (s *Session) runCompactionSummary(ctx context.Context, model message.ModelR // apart. firstID/lastID not found (in order) within history is corruption — // an explicit error, never a silent best-effort guess. func spliceCompact(history []message.Message, firstID, lastID string, summary message.Message) ([]message.Message, error) { + start, end, err := compactBounds(history, firstID, lastID) + if err != nil { + return nil, err + } + return spliceCompactBounds(history, start, end, summary), nil +} + +// compactBounds returns the exact occurrence range spliceCompact selects. +// Keeping occurrence selection separate from the splice lets indexFold apply +// the identical range to its parallel record-provenance slice. +func compactBounds(history []message.Message, firstID, lastID string) (int, int, error) { start, end := -1, -1 for i, m := range history { if start == -1 && m.ID == firstID { @@ -655,13 +666,17 @@ func spliceCompact(history []message.Message, firstID, lastID string, summary me } } if start == -1 || end == -1 { - return nil, fmt.Errorf("engine: compact record range [%s, %s] not found in history", firstID, lastID) + return 0, 0, fmt.Errorf("engine: compact record range [%s, %s] not found in history", firstID, lastID) } + return start, end, nil +} + +func spliceCompactBounds(history []message.Message, start, end int, summary message.Message) []message.Message { out := make([]message.Message, 0, len(history)-(end-start+1)+1) out = append(out, history[:start]...) out = append(out, summary) out = append(out, history[end+1:]...) - return out, nil + return out } // indexOfMessageID returns the index of the first message in history whose @@ -731,12 +746,25 @@ func healCompactFoldEnd(history []message.Message, firstID string, turnsFolded i // for it. A failed heal falls through unchanged, so spliceCompact returns // its usual loud error rather than a silent best-effort guess. func applyCompactRecord(history []message.Message, firstID, lastID string, turnsFolded int, summary message.Message) ([]message.Message, error) { + start, end, err := compactRecordBounds(history, firstID, lastID, turnsFolded) + if err != nil { + return nil, err + } + return spliceCompactBounds(history, start, end, summary), nil +} + +// compactRecordBounds is applyCompactRecord's occurrence-aware half: it +// performs the same missing-last-id heal, then returns the exact range that +// will be replaced. indexFold uses these bounds for both its message skeleton +// and the parallel journal-record provenance, so repeated IDs cannot make the +// two slices select different occurrences. +func compactRecordBounds(history []message.Message, firstID, lastID string, turnsFolded int) (int, int, error) { if _, found := indexOfMessageID(history, lastID); !found { if healed, err := healCompactFoldEnd(history, firstID, turnsFolded); err == nil { lastID = healed } } - return spliceCompact(history, firstID, lastID, summary) + return compactBounds(history, firstID, lastID) } // bytesPerTokenEstimate is the standard ~4-bytes-per-token heuristic used by diff --git a/engine/index.go b/engine/index.go index ddafe5e3..9fc9107d 100644 --- a/engine/index.go +++ b/engine/index.go @@ -307,13 +307,20 @@ func indexMessageOf(m message.Message) *indexMessage { // It keeps a message SKELETON — one message.Message per durable message, // carrying id, role, and timestamp and no parts — rather than a plain // counter, because compaction folds a RANGE of messages named by their -// first and last id. Holding the skeleton lets the fold call the same -// spliceCompact and healCompactFoldEnd that LoadSession calls (compact.go), -// so the two can never disagree about what a compact record does to a -// history. +// first and last id. Holding the skeleton lets the fold use the same +// compactRecordBounds and spliceCompactBounds that LoadSession uses +// (compact.go), so the two can never disagree about what a compact record +// does to a history. type indexFold struct { ix SessionIndex messages []message.Message + // messageRecordOrdinals is parallel to messages. Each entry identifies + // the journal record that contributed that surviving message; unlike a + // message ID, it remains unambiguous when a damaged/external journal + // repeats an ID. foldedPage consumes it to decode the right raw line. + messageRecordOrdinals []int + // recordOrdinal advances once per record applied to this fold. + recordOrdinal int // repairs is how many messages message.ResolveOrphanToolCalls would // insert into the skeleton, maintained as records arrive so snapshot // stays constant time. See appendMessage. @@ -332,9 +339,10 @@ type indexFold struct { // applyIndexRecord folds one record. It mirrors LoadSession's own switch // (store.go) case for case, and shares its helpers for the three folds with -// real state machines behind them: compaction (applyCompactRecord), +// real state machines behind them: compaction (compactRecordBounds), // goals (applyGoalRecord), and the prompt queue (promptQueueFold). func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { + f.recordOrdinal++ switch rec.Type { case recSession: f.header = true @@ -380,7 +388,7 @@ func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { if rec.Compact == nil { return errors.New("compact record without payload") } - spliced, err := applyCompactRecord(f.messages, rec.Compact.FirstID, rec.Compact.LastID, rec.Compact.TurnsFolded, rec.Compact.Summary.skeleton()) + start, end, err := compactRecordBounds(f.messages, rec.Compact.FirstID, rec.Compact.LastID, rec.Compact.TurnsFolded) if err != nil { // The same corruption LoadSession fails on. Mark the fold // broken rather than returning: a listing must still report @@ -389,7 +397,8 @@ func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { f.broken = true return nil } - f.messages = spliced + f.messages = spliceCompactBounds(f.messages, start, end, rec.Compact.Summary.skeleton()) + f.messageRecordOrdinals = spliceOrdinalBounds(f.messageRecordOrdinals, start, end, f.recordOrdinal) f.recountRepairs() f.ix.CompactionCount++ f.ix.LastCompactedAt = rec.CreatedAt @@ -505,6 +514,16 @@ func hasRepairWindowMarker(m message.Message) bool { return false } +// spliceOrdinalBounds applies a compact record's already-resolved message +// range to the parallel record-provenance slice. +func spliceOrdinalBounds(ordinals []int, start, end, summaryOrdinal int) []int { + out := make([]int, 0, len(ordinals)-(end-start+1)+1) + out = append(out, ordinals[:start]...) + out = append(out, summaryOrdinal) + out = append(out, ordinals[end+1:]...) + return out +} + // appendMessage adds one durable message to the skeleton and keeps the // running repair count in step. // @@ -523,6 +542,7 @@ func (f *indexFold) appendMessage(m message.Message) { last := len(f.messages) - 1 f.repairs -= f.repairsAt(last) f.messages = append(f.messages, m) + f.messageRecordOrdinals = append(f.messageRecordOrdinals, f.recordOrdinal) f.repairs += f.repairsAt(last) + f.repairsAt(last+1) } diff --git a/engine/messagepage.go b/engine/messagepage.go index 4b2bbdd4..ba42fe00 100644 --- a/engine/messagepage.go +++ b/engine/messagepage.go @@ -426,28 +426,23 @@ func decodeRecordHeadFull(raw []byte) (recordHead, bool) { // foldedPage is the general path, for a journal that carries at least one // compact record. It folds the journal exactly as LoadSession does — the -// same indexFold, so the same applyCompactRecord — to learn WHICH message -// ids occupy seqs lo..hi, then decodes just those records. +// same compactRecordBounds and spliceCompactBounds — to learn WHICH message +// occurrences occupy seqs lo..hi, then decodes just those records. // // One pass, two decode depths. Every line is folded through indexRecord: // ids, roles, timestamps, and tool-call ids, never a message body. The same -// pass keeps each record's raw line, keyed by the id that record -// contributes, as a subslice of data rather than a copy. Only the handful -// of lines a page actually carries is then decoded in full. +// pass keeps each contributing record's raw line, keyed by its journal +// ordinal, as a subslice of data rather than a copy. indexFold carries that +// occurrence-aware ordinal beside every surviving skeleton message, so even +// repeated message IDs resolve to the exact record the fold retained. Only +// the handful of lines a page actually carries is then decoded in full. // -// The line map holds one entry per message record in the folded prefix, not -// one per message in the resulting sequence: a record a compaction folded -// away keeps its entry. That is deliberate. Each entry is an id string and -// a slice header beside a prefix this function already holds in memory in -// full, so pruning would trade a few percent of that for a pass per -// compaction. -// -// An id that appears on two records keeps the FIRST. Engine-minted ids are -// unique, and the one production source of a repeat is a provider-derived -// id hashed from the message's own text (message.ProviderCallID), where the -// two records carry the same content anyway. A journal that repeats an id -// with DIFFERENT content is damaged, and this renders the first of them -// rather than the last. +// The line map holds one entry per contributing message/compact record in +// the folded prefix, not one per message in the resulting sequence: a record +// a compaction folded away keeps its entry. That is deliberate. Each entry +// is an integer ordinal and a slice header beside a prefix this function +// already holds in memory in full, so pruning would trade little memory for +// a pass per compaction. // // An earlier revision ran a SECOND scanLog over the journal, decoding every // line into a full record to find the wanted ones. That decoded every @@ -455,17 +450,9 @@ func decodeRecordHeadFull(raw []byte) (recordHead, bool) { // avoid — a review caught it. The raw-line map is what removes it. func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { var fold indexFold - // lineByID aliases data; it never copies a record. - // - // Known gap, issue #199: it keeps the FIRST line carrying each id, so a - // journal that repeats a message id — one occurrence folded away by a - // compact record, the other surviving — serves the wrong record's - // content under a right sequence number. Not reachable from this - // package's own writer (ids are per message, one writer per journal), - // and fixing it properly means teaching indexFold to carry each - // surviving message's record ordinal, so it is filed rather than - // patched here. - lineByID := make(map[string][]byte) + // lineByOrdinal aliases data; it never copies a record. Ordinals are the + // fold's own occurrence identity, not user/provider-controlled message IDs. + lineByOrdinal := make(map[int][]byte) err := scanLogRaw(data, func(line []byte, n int, isLast bool) error { var rec indexRecord if err := json.Unmarshal(line, &rec); err != nil { @@ -492,9 +479,7 @@ func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { contributes = rec.Compact.Summary.ID } if contributes != "" { - if _, seen := lineByID[contributes]; !seen { - lineByID[contributes] = line - } + lineByOrdinal[fold.recordOrdinal] = line } return nil }) @@ -507,12 +492,16 @@ func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { if hi > len(fold.messages) { return nil, fmt.Errorf("message page [%d,%d]: journal folds to %d messages", lo, hi, len(fold.messages)) } + if len(fold.messageRecordOrdinals) != len(fold.messages) { + return nil, errors.New("message page: journal fold lost record provenance") + } out := make([]message.Message, hi-lo+1) for seq := lo; seq <= hi; seq++ { id := fold.messages[seq-1].ID - line, ok := lineByID[id] + ordinal := fold.messageRecordOrdinals[seq-1] + line, ok := lineByOrdinal[ordinal] if !ok { - return nil, fmt.Errorf("message page [%d,%d]: no record for message %q at seq %d", lo, hi, id, seq) + return nil, fmt.Errorf("message page [%d,%d]: no record %d for message %q at seq %d", lo, hi, ordinal, id, seq) } var rec record if err := json.Unmarshal(line, &rec); err != nil { @@ -528,6 +517,9 @@ func foldedPage(data []byte, lo, hi int) ([]message.Message, error) { if msg == nil { return nil, fmt.Errorf("message page [%d,%d]: record for message %q carries no message", lo, hi, id) } + if msg.ID != id { + return nil, fmt.Errorf("message page [%d,%d]: record %d carries message %q, want %q", lo, hi, ordinal, msg.ID, id) + } m := *msg // The same ingest-time repair LoadSession applies to every message // it replays (message.Message.Normalize's doc comment). diff --git a/engine/messagepage_test.go b/engine/messagepage_test.go index 2dac748b..e76e12fc 100644 --- a/engine/messagepage_test.go +++ b/engine/messagepage_test.go @@ -457,6 +457,40 @@ func TestReadMessagePageTailAndFoldPathsAgree(t *testing.T) { } } +// TestFoldedPageUsesSurvivingRecordOccurrence reproduces issue #199: the +// first of two records carrying one message ID is folded away, while the +// second survives. Sequence identity alone is ambiguous; the page must decode +// the surviving record occurrence, not the first raw line with that ID. +func TestFoldedPageUsesSurvivingRecordOccurrence(t *testing.T) { + dir := t.TempDir() + id := "ses_0123456789abcdef" + journal := `{"type":"session","id":"ses_0123456789abcdef","created_at":"2026-01-02T03:04:05Z","workdir":"/w"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_repeat","role":"user","parts":[{"type":"text","text":"OLD folded occurrence"}]}} +{"type":"message","message":{"id":"msg_fold_end","role":"assistant","parts":[{"type":"text","text":"fold me"}]}} +{"type":"compact","compact":{"first_id":"msg_repeat","last_id":"msg_fold_end","turns_folded":1,"summary":{"id":"msg_summary","role":"user","parts":[{"type":"text","text":"SUMMARY"}]}}} +{"type":"message","message":{"id":"msg_repeat","role":"assistant","parts":[{"type":"text","text":"NEW surviving occurrence"}]}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + + page, err := ReadMessagePage(dir, id, 0, 2) + if err != nil { + t.Fatalf("ReadMessagePage: %v", err) + } + if got := idsOf(page.Messages); !sameIDs(got, []string{"msg_summary", "msg_repeat"}) { + t.Fatalf("page ids = %v, want [msg_summary msg_repeat]", got) + } + if page.Messages[1].Role != message.RoleAssistant { + t.Errorf("repeated message role = %q, want assistant from surviving occurrence", page.Messages[1].Role) + } + text, ok := page.Messages[1].Parts[0].(*message.Text) + if !ok || text.Text != "NEW surviving occurrence" { + t.Fatalf("repeated message content = %#v, want NEW surviving occurrence", page.Messages[1].Parts) + } +} + // TestReadMessagePageSkipsDerivedRepairMessages: a journal whose last turn // died between a tool call and its result. A full load repairs it with a // synthetic tool result, and GET /session counts that message. A page must From 795289b99d144020d9c82df20b87da4287f79a4b Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 18:25:52 -0400 Subject: [PATCH 19/95] test: verify child lineage.status on GET /session list (#208) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * expose child session status in GET /session list Problem ------- The boxes console (on box-open) needs ONE cheap call to see which child subagent sessions are currently running, so it can paint a "running" indicator on all cards at a glance. Today it can't: harness's session LIST (GET /session, no id) returns only id/created_at/lineage-pointer per session, NO status. Status exists only on the single-session read (GET /session/{id}) through the SessionManager lifecycle status in its lineage block. The console would have to fan out one call per child to see any of their statuses — the exact N+1 problem we're eliminating. Solution -------- Add each session's SessionManager lifecycle status to the GET /session LIST response. The single-session handler (GET /session/{id}) already computes and returns lineage status; reuse that exact source (SessionNode.Status from SessionManager.SessionAndInfo) so the two handlers never disagree. For a session in SessionManager: - running: actively holding a turn slot (StatusRunning) - terminal: done, failed, or canceled (StatusDone, StatusFailed, StatusCanceled) - idle: not running a turn (StatusIdle) For a session not in SessionManager (never spawned, no lineage): - default to "idle" (the previous behavior for non-resident sessions) Design decisions ---------------- - Status is sourced from SessionManager, not by loading sessions. This keeps the list operation cheap: no new per-session I/O. For a managed session where the Session object is not resident in this process, we read its status from the manager's node (a map lookup) rather than adding a new cold load path. - Two handlers, one source: handleList and handleGet both check lv.isManaged and lv.info.Status after resolveLive. The logic is identical, so both never disagree about a session's status. - Terminal statuses are explicit: "done", "failed", "canceled" (not mapped to "busy" or "idle"). This lets the console distinguish a child that finished from one still running. - coldSessionJSON now accepts an optional managerStatus parameter. For non- resident, non-loadable managed sessions, this status override ensures the response carries the manager's truth instead of defaulting to idle. Rejected alternatives --------------------- - Status in the metadata index: Would require a second pass to persist and update status on every turn boundary — added write-side cost and complexity. SessionManager.Info is already the live source; using it directly costs only one map lookup per session in the list. - Full metadata index: Out of scope for this change. The goal is "one cheap list call". A full metadata index is the long-term answer for offline status, but building it now would delay this fix; filing issue #200 to track it. Semantic change --------------- GET /session list response now includes a "status" field per entry, matching the single-session "status" field. Values: "running", "idle", "done", "failed", "canceled" (from SessionManager lifecycle). Backward-compatible addition; clients that ignore the field see no breaking change. Verification ------------ - TestListSessionsIncludesChildStatus: Creates a root session, confirms it appears in the list with status="idle" (managed by SessionManager, not running). Verifies the manager's status is correctly surfaced in the list response. - Full test suite passes with -race (19.371s). - Wire format: status values match SessionNode.Status and the single-session handler's lineage.status, so orchestrators see consistent state. * fix: expose lineage.status for managed sessions in GET /session list Three blocking findings from cross-review, all fixed: 1. FIXED: Use literal SessionManager lifecycle vocabulary (running/idle/done/ failed/canceled) in lineage.status, not statusStr() which returns only busy/idle binary. The console filters status=='running' to find active subagents — with statusStr() that filter would match nothing. 2. FIXED: Populate lineage.status (SessionManager lifecycle field), not the top-level Status field (which is for busy/idle binary in THIS process). The status lives in lineage per its own doc comment (lines 150-159). Refactored to: - Accept managerStatus string from SessionManager.SessionNode.Status - Pass it through coldSessionJSON and buildSessionFromIndex - Populate lineage.status when manager status is available - Handle cold sessions with no durable parent info but manager status 3. FIXED: Test now creates an actual managed session (root, adopted into SessionManager) and asserts specific lineage.status value from manager, not vacuously. Test verifies status from SessionManager is wired to GET /session list response for managed sessions. Semantic change: GET /session list now includes lineage.status (SessionManager lifecycle vocabulary) for managed sessions, matching the single-session handler's behavior. * test: format session status coverage * test: match CI gofmt toolchain * test: expand test to cover managed child sessions with SessionManager status Created comprehensive test that: - Creates root session (resident, adopted to SessionManager) - Creates two cold child sessions on disk with TaskParentID set - Loads them from disk and adopts into SessionManager with different statuses - Verifies GET /session list shows lineage.status from manager - Verifies GET /session/{id} agrees with list (never disagree) NOTE: MUTATION CHECK FINDING — Test still passes when cold path is disabled ('if false && lv.isManaged' at both handleList:1040 and handleGet:1078). This confirms the test exercises the WARM path only, not the COLD path. Root cause: Managed sessions (lv.isManaged==true) return lv.session() from SessionManager, which triggers the warm path (buildSession) via the 'if lv.session() != nil' branch. The cold path is only reached when lv.session()==nil, which requires BOTH: - Not resident (lv.resident == nil) - Not managed (lv.isManaged == false) But unmanaged sessions have lv.isManaged==false, so managerStatus=='', making the cold path code unreachable for its intended case. This is a true finding: the test doesn't exercise the code it intends to fix. * test: keep child status coverage on reachable path --------- Co-authored-by: andybons --- server/cold_read_test.go | 113 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/server/cold_read_test.go b/server/cold_read_test.go index cd1811c1..b505c99a 100644 --- a/server/cold_read_test.go +++ b/server/cold_read_test.go @@ -600,3 +600,116 @@ func TestListOmitsWhatItCannotRenderWhileStatusReportsIt(t *testing.T) { t.Errorf("status for the fold-broken session = %+v (present=%v), want its journal's 9 input tokens", got, ok) } } + +// TestListSessionsIncludesChildStatus verifies that GET /session list includes +// lineage.status from the SessionManager snapshot for managed children. The +// list must show running vs terminal values without N+1 calls, and must agree +// with GET /session/{id}. +// +// The children begin as disk fixtures and are then adopted into SessionManager, +// matching the reloaded-child path. SessionManager retains their Session +// objects, so handleList intentionally renders them through the existing warm +// path; this test is a regression guard for that existing contract, not a +// synthetic cold-manager state the production manager cannot represent. +func TestListSessionsIncludesChildStatus(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + rootID := h.createSession("test/m1") + mgr := h.srv.SessionManager() + + // Create two cold (disk-only) child sessions with TaskParentID set. + runningID := coldSession(t, h.dir, func(cfg *engine.Config) { + cfg.ParentSession = rootID + cfg.TaskParentID = rootID + cfg.TaskAgentType = "explore" + cfg.TaskDepth = 1 + }).ID + + doneID := coldSession(t, h.dir, func(cfg *engine.Config) { + cfg.ParentSession = rootID + cfg.TaskParentID = rootID + cfg.TaskAgentType = "explore" + cfg.TaskDepth = 1 + }).ID + + // Load and adopt children into manager, setting their statuses. + runSess, err := h.srv.opts.LoadSession(runningID) + if err != nil { + t.Fatalf("load running: %v", err) + } + if err := mgr.AdoptReloaded(runSess); err != nil { + t.Fatalf("adopt running: %v", err) + } + mgr.ReportTurnStart(runSess) + + doneSess, err := h.srv.opts.LoadSession(doneID) + if err != nil { + t.Fatalf("load done: %v", err) + } + if err := mgr.AdoptReloaded(doneSess); err != nil { + t.Fatalf("adopt done: %v", err) + } + mgr.ReportTurnStart(doneSess) + mgr.ReportTurnEnd(doneID, nil, nil) + + // Verify setup: both children have correct status in manager. + if info, ok := mgr.Info(runningID); !ok || info.Status != engine.StatusRunning { + t.Fatalf("test setup: running not StatusRunning: %+v", info) + } + if info, ok := mgr.Info(doneID); !ok || info.Status != engine.StatusDone { + t.Fatalf("test setup: done not StatusDone: %+v", info) + } + + // Test 1: GET /session list includes the manager-backed lineage.status. + // The children are absent from h.srv.sessions but resident in + // SessionManager, which is the authoritative warm source resolveLive uses. + resp, data := h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatalf("decode list: %v (%s)", err, data) + } + + findInList := func(id string) *sessionJSON { + for i := range list { + if list[i].ID == id { + return &list[i] + } + } + return nil + } + + runningEntry := findInList(runningID) + if runningEntry == nil { + t.Fatalf("running child %s not in list", runningID) + } + if runningEntry.Lineage == nil || runningEntry.Lineage.Status != "running" { + t.Errorf("list: running child lineage.status = %+v, want 'running'", runningEntry.Lineage) + } + + doneEntry := findInList(doneID) + if doneEntry == nil { + t.Fatalf("done child %s not in list", doneID) + } + if doneEntry.Lineage == nil || doneEntry.Lineage.Status != "done" { + t.Errorf("list: done child lineage.status = %+v, want 'done'", doneEntry.Lineage) + } + + // Test 2: GET /session/{id} for each child AGREES with list. + // Property: list and single-session handler must never disagree. + for childID, expectedStatus := range map[string]string{runningID: "running", doneID: "done"} { + resp, data := h.do("GET", "/session/"+childID, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/%s = %d: %s", childID, resp.StatusCode, data) + } + var single sessionJSON + if err := json.Unmarshal(data, &single); err != nil { + t.Fatalf("decode /session/%s: %v (%s)", childID, err, data) + } + if single.Lineage == nil || single.Lineage.Status != expectedStatus { + t.Errorf("/session/%s: lineage.status = %+v, want '%s'", childID, single.Lineage, expectedStatus) + } + } +} From 4f9ee6590ebde04a8169a6ec7719145415ce7e02 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 19:20:17 -0400 Subject: [PATCH 20/95] =?UTF-8?q?engine:=20journal=20snapshotting=20?= =?UTF-8?q?=E2=80=94=20bound=20LoadSession=20replay=20cost=20(#209)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs/design: journal snapshotting (Layer B checkpoint) LoadSession rebuilds a session by replaying its whole append-only journal from seq 0, so every read and every cold write pays O(journal size) that grows for the life of the session. On a deployed box that is 8s for one transcript read and 3-6s for each tiny control read. This design records the two-layer fix and the decisions locked for it: a seq-anchored, checksummed, rebuildable snapshot beside the journal (harness, Layer B) and a shared control-plane journal-access interface (boxes, Layer A). It also records what was deliberately rejected -- truncating the journal, a residency cache, and json.Marshal of the Session struct -- and the five concurrency rules a snapshot writer must hold. Design only; no behavior change. * engine: checkpoint the session journal to bound replay cost LoadSession rebuilt a session by decoding every record of its append-only journal, building the whole history slice and repairing it. That is O(journal size) and grows for the life of the session. On a deployed box one transcript read cost 8s and each tiny control read 3-6s, because every one of them independently forced a full replay; a cold prompt paid the same replay before its 202 could go out. Add a checkpoint. A session writes .snap beside its journal: the fold-produced state as of journal line N, with a CRC-32 and a format version. LoadSession loads it, applies the session header record, and replays only lines > N through scanLogRaw -- so a record the snapshot covers is never decoded, which is where the saving is. The invariant is state(snapshot@N) + replay(N+1..head) == full-replay(0..head). Every doubt falls back to a full replay: no snapshot, a torn one, a checksum mismatch, a wrong version, another session's id, a seq ahead of the journal head, or a journal whose first record is not a session header. Slower, never wrong. The journal is never truncated, and a snapshot can be deleted at any time to get exactly the old behavior. The schema is explicit, not json.Marshal of a *Session: every field but ID is unexported, and the config carries live callbacks, a SessionManager pointer and open file handles that must never be serialized. It captures what the folds reconstruct, and deliberately excludes two things -- header-derived state, replayed instead because its restore rules turn on absent-versus-present and are not worth writing twice, and turn/lastSystem, which have no durable source at all, so capturing them would make a snapshot-loaded session disagree with a full replay of the same journal. The trigger is at the append boundary, not inside writeRecord. A snapshot pairs a memory image with a journal position, and inside writeRecord the two do not yet agree: EnqueuePromptDurable deliberately writes its record before mutating memory, so a capture there would anchor past a record whose effect memory has not applied, and the reload would drop it. The opposite direction is guarded by snapshotSafeLocked: SessionManager splits some mutations into a memory half and a deferred durable half, and a snapshot in that window would carry the mutation and leave its record in the tail for the reload to apply twice -- a duplicated message, or a child-completion notification the parent renders to the model twice. Session.durableDebt counts the open halves and refuses a capture while any is outstanding, which only postpones the snapshot to the next boundary. Writes are background, coalesced to one in flight per session, and atomic (temp -> fsync -> rename), so a crash leaves the previous snapshot intact and a half-written .snap.tmp is a path no reader ever opens. A write failure lands in lastSnapshotErr, never lastPersistErr: a snapshot is derived acceleration, not a durability promise. Config.SnapshotEveryRecords is the cadence. Zero -- the engine zero value -- disables snapshot writing, so a bare embedder-built engine.Config keeps the pre-snapshot behavior; the config/CLI layer supplies the product default of 64 (snapshot_every_records), the same split prompt_retries uses. Reading a snapshot is never gated on it: recovery is a property of the files on disk. Design: docs/design/journal-snapshotting.md. --------- Co-authored-by: andybons --- AGENTS.md | 76 ++++ cmd/harness/main.go | 2 + config/config.go | 36 ++ config/config_test.go | 67 +++ docs/design/journal-snapshotting.md | 279 ++++++++++++ engine/engine.go | 94 ++++ engine/snapshot.go | 586 +++++++++++++++++++++++++ engine/snapshot_test.go | 652 ++++++++++++++++++++++++++++ engine/store.go | 242 ++++++++--- engine/taskdelivery.go | 10 + 10 files changed, 1978 insertions(+), 66 deletions(-) create mode 100644 docs/design/journal-snapshotting.md create mode 100644 engine/snapshot.go create mode 100644 engine/snapshot_test.go diff --git a/AGENTS.md b/AGENTS.md index 2b2b6762..85614319 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1143,6 +1143,82 @@ The sidecar never gets an `fsync`: losing it in a crash costs one refold. Plugins are process configuration, not durable session state, and a cold read has no `Session` to ask. +### Journal snapshotting + +`LoadSession` does not always replay a whole journal. Beside the log sits +`.snap`, a **seq-anchored checkpoint**: the fold-produced state as of +journal line N, plus a CRC-32 and a format version. Recovery loads the +snapshot, applies the session HEADER record (line 1, always — it is what +carries `created_at`, workdir, and task lineage, whose restore rules turn on +absent-versus-present and are not worth reproducing twice), and then replays +only lines `> N`. The tail scan reads through `scanLogRaw`, so a covered +record is never DECODED: skipping the decode of a message record's whole +part tree is where the saving is. Design: +docs/design/journal-snapshotting.md. + +The snapshot schema is EXPLICIT (`sessionSnapshot`, `engine/snapshot.go`), +never `json.Marshal` of a `*Session`: every field but `ID` is unexported, +and the config carries live callbacks, a `SessionManager` pointer, and open +file handles that must never be serialized. The rule for what belongs in it +is **exactly what the folds reconstruct** — a fold added without a matching +snapshot field is silently dropped, which is what +`TestSnapshotCarriesEveryFoldedField` and the replay-equivalence tests pin. +Two exclusions are deliberate: header-derived state (replayed instead), and +`turn`/`lastSystem`, which have NO durable source at all — capturing them +would make a snapshot-loaded session disagree with a full replay of the same +journal and make an observable field depend on whether a snapshot happened +to exist. + +The invariant is `state(snapshot@N) + replay(N+1..head) ≡ +full-replay(0..head)`. Every doubt falls back to a full replay: no snapshot, +a torn one, a checksum mismatch, a wrong version, another session's id, a +`seq` ahead of the journal head, or a journal whose first record is not a +session header. **Slower, never wrong.** The journal is never truncated; +snapshots are derived and can be deleted at any time. + +**The trigger is at the APPEND BOUNDARY, never inside `writeRecord`.** A +snapshot pairs a memory image with a journal position, and inside +`writeRecord` the two do not yet agree: some callers persist their record +BEFORE applying their own memory mutation (`EnqueuePromptDurable`, +deliberately), so a capture there would anchor past a record whose effect +memory has not applied — and the reload would skip that record and lose the +effect forever. `appendWithUsage` (after `persistMessage`, still under +`s.mu`) and the on-idle trigger (`runAgenticLoop`'s defer, and +`ReleaseFiles` on eviction) are the two boundaries where the caller has +completed both halves. + +The OPPOSITE direction is guarded by `snapshotSafeLocked`: +`SessionManager` splits some mutations into an in-memory half and a +DEFERRED durable half (`appendMemoryOnly`/`persistAppendedMessage`, +`enqueueTaskNotificationMemoryOnly*`/`persistQueuedTaskNotification`, +`queueRecordDeferredLocked`), so memory can be AHEAD of the journal. A +snapshot taken in that window carries the mutation AND leaves its record in +the tail, and the reload applies it twice — a duplicated message, or a +child-completion notification the parent renders to the model twice. +`Session.durableDebt` counts the outstanding halves; a capture is refused +while any is open, which merely postpones the snapshot to the next +boundary. The debt clamps at zero: a leaked increment stops this session +snapshotting (degrading to today's full replay), where a negative count +would ARM a capture in exactly the unsafe window. + +`Config.SnapshotEveryRecords` is the cadence. Zero — the engine zero value — +disables snapshot WRITING, so a bare embedder-built `engine.Config` keeps +the pre-snapshot behavior; the config/CLI layer supplies the product default +of 64 (`snapshot_every_records`), the same unset-versus-explicit-zero split +`prompt_retries` uses. READING a snapshot is never gated on it: recovery is +a property of the files on disk, not of the loading Config. A snapshot write +is background, coalesced (one in flight per session), and atomic (temp → +fsync → rename), and its failure lands in `lastSnapshotErr`, never +`lastPersistErr` — a snapshot is derived acceleration, not a durability +promise. + +A load that takes the snapshot path marks the metadata-index fold BROKEN +rather than building a partial one: the index summarizes EVERY record, and +this load deliberately did not see most of them. The index is a cache with +no repair path, so a reader that finds none refolds and `ensureLog` re-seeds +the fold from the journal on this session's next write. Snapshotting the +index fold itself is the obvious follow-up; nothing may guess at it. + ### Paginated message reads `GET /session/{id}/message?before_seq=N&limit=K` answers one bounded page of diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 07cabd7c..3d1737d3 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -753,6 +753,7 @@ func runCmd(args []string) error { StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), + SnapshotEveryRecords: cfg.SnapshotEveryRecordsValue(), CompactionThreshold: cfg.CompactionThreshold, CompactionKeepTurns: cfg.CompactionKeepTurns, // Tool-result retention (config keys tool_result_inline_bytes / @@ -1562,6 +1563,7 @@ func serveCmd(args []string) error { StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), + SnapshotEveryRecords: cfg.SnapshotEveryRecordsValue(), CompactionThreshold: cfg.CompactionThreshold, CompactionKeepTurns: cfg.CompactionKeepTurns, // Tool-result retention, same keys and defaults as runCmd diff --git a/config/config.go b/config/config.go index 4afbc922..9a6018c4 100644 --- a/config/config.go +++ b/config/config.go @@ -143,6 +143,20 @@ type Config struct { // like PromptRetries above. Resolve it with // MaxTokensContinuationsValue. MaxTokensContinuations *int `json:"max_tokens_continuations,omitempty"` + // SnapshotEveryRecords sets engine.Config.SnapshotEveryRecords: how + // many journal records a session appends before it checkpoints its + // state into a .snap file beside the journal, so a later + // load replays only the records after that checkpoint instead of the + // whole log (see package engine's snapshot.go and + // docs/design/journal-snapshotting.md). A nil value (the field + // omitted) leaves the product default of 64 in place; an explicit 0 + // disables snapshot WRITING entirely, reverting to a full replay on + // every load. Reading an existing snapshot is never gated on this + // key — recovery is a property of the files on disk. A *int + // distinguishes "unset" (64) from "0" (off) across the project-config + // merge, exactly like PromptRetries above. Resolve it with + // SnapshotEveryRecordsValue. + SnapshotEveryRecords *int `json:"snapshot_every_records,omitempty"` // StreamIdleTimeoutS sets engine.Config.StreamIdleTimeout (in seconds) // for every session this process creates: how long a streamed response // may go without a delta before the engine's idle-stream watchdog aborts @@ -1003,6 +1017,9 @@ func merge(base, over *Config) *Config { if over.MaxTokensContinuations != nil { out.MaxTokensContinuations = over.MaxTokensContinuations } + if over.SnapshotEveryRecords != nil { + out.SnapshotEveryRecords = over.SnapshotEveryRecords + } if over.StreamIdleTimeoutS != 0 { out.StreamIdleTimeoutS = over.StreamIdleTimeoutS } @@ -1262,6 +1279,25 @@ func (c *Config) MaxTokensContinuationsValue() int { return *c.MaxTokensContinuations } +// defaultSnapshotEveryRecords is the product default journal-snapshot +// cadence when `snapshot_every_records` is unset (see +// SnapshotEveryRecordsValue and engine.Config.SnapshotEveryRecords). It +// lives here, not in engine.Config's zero value, so an embedder building a +// bare engine.Config keeps the pre-snapshot behavior — the same split +// defaultPromptRetries uses. +const defaultSnapshotEveryRecords = 64 + +// SnapshotEveryRecordsValue reports the journal-snapshot cadence. The +// default is defaultSnapshotEveryRecords (64): only an explicit +// `snapshot_every_records: 0` turns snapshot writing off. A nil receiver +// (no config) uses the default too. +func (c *Config) SnapshotEveryRecordsValue() int { + if c == nil || c.SnapshotEveryRecords == nil { + return defaultSnapshotEveryRecords + } + return *c.SnapshotEveryRecords +} + // defaultToolResultInlineBytes / defaultToolResultRetainedBytes are the // product defaults for the tool-result retention keys (see the // ToolResultInlineBytes/ToolResultRetainedBytes fields and package engine's diff --git a/config/config_test.go b/config/config_test.go index a82a6cf1..e8bb48d6 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -1720,3 +1720,70 @@ func TestLoadProjectWithInfoSessionSync(t *testing.T) { t.Errorf("LoadInfo.SessionSync = %q, want %q", info.SessionSync, "volume") } } + +// TestSnapshotEveryRecords covers the snapshot_every_records config field: +// a *int on the same unset-versus-explicit-zero split PromptRetries uses, +// so unset means the product default (64) and an explicit 0 turns journal +// snapshot writing off. A project value overrides the user layer. +func TestSnapshotEveryRecords(t *testing.T) { + t.Run("unset uses default 64", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"model": "anthropic/claude-fable-5"}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.SnapshotEveryRecords != nil { + t.Errorf("SnapshotEveryRecords = %v, want nil (unset)", c.SnapshotEveryRecords) + } + if got := c.SnapshotEveryRecordsValue(); got != 64 { + t.Errorf("SnapshotEveryRecordsValue = %d, want 64 (default)", got) + } + }) + t.Run("explicit zero disables", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"snapshot_every_records": 0}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.SnapshotEveryRecords == nil || *c.SnapshotEveryRecords != 0 { + t.Fatalf("SnapshotEveryRecords = %v, want explicit 0", c.SnapshotEveryRecords) + } + if got := c.SnapshotEveryRecordsValue(); got != 0 { + t.Errorf("SnapshotEveryRecordsValue = %d, want 0 (disabled)", got) + } + }) + t.Run("explicit value", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"snapshot_every_records": 16}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if got := c.SnapshotEveryRecordsValue(); got != 16 { + t.Errorf("SnapshotEveryRecordsValue = %d, want 16", got) + } + }) + t.Run("nil receiver uses default", func(t *testing.T) { + var c *Config + if got := c.SnapshotEveryRecordsValue(); got != 64 { + t.Errorf("nil SnapshotEveryRecordsValue = %d, want 64", got) + } + }) + t.Run("project overrides user", func(t *testing.T) { + zero := 0 + base := &Config{SnapshotEveryRecords: intPtr(32)} + merged := merge(base, &Config{SnapshotEveryRecords: &zero}) + if merged.SnapshotEveryRecords == nil || *merged.SnapshotEveryRecords != 0 { + t.Errorf("merged = %v, want project override 0", merged.SnapshotEveryRecords) + } + }) + t.Run("unset project inherits user", func(t *testing.T) { + base := &Config{SnapshotEveryRecords: intPtr(32)} + merged := merge(base, &Config{}) + if merged.SnapshotEveryRecords == nil || *merged.SnapshotEveryRecords != 32 { + t.Errorf("merged = %v, want inherited 32", merged.SnapshotEveryRecords) + } + }) +} diff --git a/docs/design/journal-snapshotting.md b/docs/design/journal-snapshotting.md new file mode 100644 index 00000000..25e6eb91 --- /dev/null +++ b/docs/design/journal-snapshotting.md @@ -0,0 +1,279 @@ +# Journal Snapshotting — design + +**Status:** draft for review (2026-08-27). Author: coordinator, via brainstorm with Andy. +**Repos:** harness (`majorcontext/harness`, Layer B) + boxes (`meetneptune/boxes`, Layer A). +**Extends:** `boxes/docs/design/console-read-path.md` (this is the source-level fix its workstream 1 gestured at, and the seam its workstream 5 mirror plugs into). + +## 1. Problem + +Reading or writing a box session requires the in-memory `*engine.Session`, which +`engine.LoadSession` rebuilds by replaying the whole `.jsonl` journal from seq 0 +(`os.ReadFile` + record-by-record `scanLog`). Replay cost is **O(journal size)** +and grows for the life of the session. Measured on the deployed Webhooks box: + +- `GET /transcript` — **8.0s** (cold journal replay; `max_bytes` truncates + *after* the replay, so it saves wire bytes, not latency). +- `/model`, `/goal`, `/thinking` — **3–6s each**, tiny payloads: each read + independently forces a session load. +- `/prompt` — the same `LoadSession` replay fires synchronously in + `claimForPrompt` whenever the target session is not resident (first prompt + after harness start / wake-from-hibernation / LRU eviction), blocking the 202. + +The endpoint audit (2026-08-27) confirms this replay underlies the worst +findings (serial double-replay on `/transcript`, uncapped child transcript, +the goal/model/thinking reads). It is **not CPU** — the box sits at ~11m during +the reads; it is I/O + parse to rebuild the session, on the box's slow (gVisor) +fs. + +The write path already keeps the loaded session resident (LRU, 32), so it pays +the replay once per residency window; the read cold-path discards it and pays +every read. Either way the underlying cost is unbounded journal replay. + +Secondarily: control-plane handlers reach the journal in **ad-hoc, inconsistent** +ways — the audit found the correct pattern (parallel, byte-bounded reads) +independently invented in `console_bootstrap` and the MCP tools, but +`handleTranscript` regressed to a serial double-replay and `handleGetChildTranscript` +has no cap at all — because nothing shared enforces the discipline. + +## 2. Goals / non-goals + +**Goals** +- Bound session-load (replay) cost **independent of session age** — checkpoints. +- Fix reads *and* writes with one mechanism (both go through `LoadSession`). +- A single control-plane journal-access layer so no handler can reintroduce an + unbounded or serial-double read. + +**Non-goals (explicitly deferred)** +- Any residency/read cache in the control plane (deferred until the baseline is + proven — Andy: "no caching until we get the baseline architecture right"). +- The control-plane **mirror/projection** (read-path workstream 5). This design + is the *seam* it plugs into, not the projection itself. +- **Truncating** the journal. The journal stays the untruncated source of truth + (the mirror and any audit consume the full log). Snapshots are pure + acceleration. + +## 3. Architecture — two layers + +- **Layer B — harness journal discipline (the mechanism).** `engine` gains a + checkpoint: a snapshot is a *seq-anchored, rebuildable* materialization of the + session at seq N. `LoadSession` becomes "load newest valid snapshot ≤ head, + replay only records > N." Shared by both callers of `LoadSession` — the read + cold-path (`GET /session`) and the write path (`claimForPrompt`). +- **Layer A — control-plane shared journal access (the interface).** One + boxes-side set of methods every handler uses to touch the journal via harness: + resolve-once, read-bounded-by-default, multi-read-in-parallel. It makes access + uniform and is the seam the mirror later backs. + +Layer B is the root fix (bounds the cost everything pays). Layer A is what makes +that fix reachable uniformly and un-regressable. + +## 4. Layer B — the checkpoint + +### 4.1 Snapshot content — an explicit schema (verified) +`*engine.Session` (`engine/engine.go:638`) is **not directly serializable**: +every field but `ID` is unexported, so `json.Marshal` can't touch it, and +several fields are process-local and must never be snapshotted. So a snapshot +is an **explicitly-defined schema**, modeled on the existing `record` / +`JournalRecord` pattern (`engine/store.go:165`, `engine/journal.go:45`), +capturing exactly the fold-produced replayable state at seq N, plus the anchor +`seq` and a checksum. + +**Capture (the fold-produced state, all guarded by `s.mu`):** `history`, +`model`, `effort`, `usage`/`lastUsage`, `turn`, `lastSystem`, goal state +(`goalActive`, `goalCondition`), `compactCount`/`lastCompactedAt`, +`promptQueue`/`promptQueueNextID`/`enqueueSeq`, `toolResults`/`toolResultNextID`/ +`toolResultBytes`, `spawnedChildIDs`, the task-notification queues, and the +crash-recovery signals (`turnUnsettled`, `committedOutcome`). + +**Exclude (runtime-only / deliberately non-durable — re-created on load exactly +as a full replay does):** `mu`, `logFile`/`logStarted`/`lastPersistErr`, +`tools`, `cfg` (carries live callbacks + the `SessionManager` pointer + +`ProcessRegistry`), and the fields harness already documents as never-persisted: +`goalGen`, `goalParked*`, `compactHysteresis`. + +The rule of thumb: **snapshot exactly what `LoadSession`'s folds reconstruct, +nothing that a fresh process re-wires.** Keeping the schema in lock-step with the +fold logic is the one maintenance burden (a fold added without a matching +snapshot field would be silently dropped) — §7 pins this with a +snapshot-equals-replay round-trip test. + +### 4.2 Storage layout +- Snapshot file alongside the journal on the box disk, e.g. + `.snap` (single latest) or `..snap` (rolling, + keep last M). Start with a single latest snapshot; rolling is a later option. +- **The journal is never truncated.** Snapshots are derived and independently + deletable; deleting all snapshots reverts to today's full-replay behavior. + +### 4.3 Recovery (`LoadSession`) +1. Find the newest snapshot whose anchor `seq` ≤ current journal head and whose + checksum validates. +2. Deserialize it into the session. +3. Replay only journal records with `seq > N`. +4. On *any* missing/corrupt/mismatched snapshot → full replay from seq 0. + **Slower, never wrong** — the philosophy already in the codebase. + +Correctness invariant: state(snapshot@N) + replay(records N+1..head) ≡ +full-replay(0..head). A round-trip test pins this. + +**The seq anchor (verified).** Records carry **no persisted `seq`** today — seq +is the 1-based line number assigned at scan time by `scanLog` +(`engine/store.go:1528`), which the read-path already exposes as +`JournalRecord.Seq` and which is stable forever because the log is append-only +and never rewritten. Two ways to anchor a snapshot to it, a §9 decision: +- **(a) Live counter:** track a "records written" count in `Session`, bumped in + `writeRecord` — trivial given the single-writer lock, no on-disk format change. + Recovery counts scanned lines to know where the tail starts. +- **(b) Persisted `seq`:** add an explicit `seq` field to `record`. Stronger, + self-describing anchor that doesn't depend on recounting lines, at the cost of + a (backward-compatible, additive) journal-format change. + +Recommend starting with **(a)** — it's the smaller change and the line-number +seq is already durable; **(b)** is a clean follow-up if we want the anchor +independent of a full line count. + +### 4.4 Triggers (cadence) — decided +- **On-idle.** When a session goes quiescent (no active turn), snapshot at the + current head seq. This is the trivially-correct case (no concurrent append) + and also makes wake-from-hibernation and post-eviction reload fast. +- **Every-K-messages.** Bound steady-state replay to K records. Start K + conservative (≈50–100), tunable; a long-lived, continuously-active session + never accumulates more than ~K records of replay. + +The two triggers are coalesced (see 4.5): at most one snapshot in flight per +session. + +### 4.5 Concurrency discipline (the five rules) +1. **Seq-anchored.** A snapshot is "valid as of seq N"; recovery replays strictly + `> N`. A snapshot need not be the latest state — the tail replay closes the gap. +2. **Off the hot path.** Never block a turn/append on snapshot I/O. Capture a + consistent state + seq quickly, then serialize + fsync in a background goroutine. +3. **Atomic write.** temp file → fsync → atomic rename. A crash mid-write leaves + the previous snapshot (or none) intact — never a torn file. +4. **Single in-flight, coalesced.** One snapshot per session at a time; the idle + and every-K triggers cannot race to write the same file (a `snapshotting` + flag/mutex coalesces them). +5. **Rebuildable + validated.** Snapshot is derived; validate on load (seq + + checksum); on mismatch, discard and full-replay. A snapshot bug degrades to + *slow*, never *wrong*. + +**How rule 2's "capture a consistent state" is implemented — the clean branch +(verified: single-writer).** Harness serializes every append to one session +through `Session.mu` — `appendWithUsage` (`engine/engine.go:1546`) holds `s.mu`, +mutates `s.history`, and calls `persist*` → `writeRecord` (`engine/store.go:919`) +synchronously under the lock, and SessionManager's `StatusRunning` gating means +at most one turn drives a session at a time. So the snapshot is emitted **by the +append-owner at an append boundary, right after a `persist*` returns while `s.mu` +is already held** — grab a consistent shallow copy of the fold-state + the seq, +release, then serialize + write in a background goroutine. **No new +synchronization primitive and no copy-from-outside-the-lock is needed.** (One +wrinkle to honor: SessionManager's `deferPersist`/`unlockAndFlushPersist` +path — `engine/session_manager.go:318` — flushes some manager-level records +after `m.mu` releases; those re-take `s.mu` and stay per-session serialized, so +the same append-boundary rule applies to them.) + +### 4.6 Interaction with residency +`claimForPrompt` already inserts loaded sessions into the resident map (LRU 32). +Checkpointing bounds the *load* cost that both the read cold-path and the write +path pay when a session is not resident. We add **no** new residency cache +(deferred). Opportunistic snapshot-on-eviction (part of on-idle) keeps the next +reload fast. + +## 5. Layer A — shared control-plane journal access + +A single interface (Go, in `boxes/internal/api`) that every journal-touching +handler uses. Method set (finalized against the endpoint audit): + +- `ResolveSession(ctx, box) (sid, error)` — trust `box.CurrentSessionID`; on + empty, `scanRootSession` fallback **and persist the result** so the residual + `firstSession` scan (audit #7) fires at most once per box, ever. +- `ReadSessionState(ctx, sid, bounds)` — bounded by default. +- `ReadTranscript(ctx, sid, {tail|limit|before_seq})` — a **default cap** + applied when the caller omits bounds (fixes audit #2 uncapped box transcript, + #3 uncapped child transcript). +- `Bootstrap(ctx, sid)` — the console envelope, generalized: one resolve, the + multiple harness reads fired **concurrently** (fixes audit #1 serial + double-replay by making parallel the only way to multi-read). +- `AppendPrompt(ctx, sid, text)` — resolve-once, no transcript read. + +**Invariants the layer enforces** (each maps to an audit finding): +- No unbounded read — every read has a default bound (#2, #3). +- Multi-read is parallel, never serial (#1). +- Resolve once — sticky pointer, persisted fallback (#7). + +**The mirror seam:** `ReadSessionState`/`ReadTranscript`/`Bootstrap` read through +the mirror/projection when it exists and fall back to harness otherwise — so +workstream 5 lands behind this interface with **no handler changes**. + +## 6. Migration & sequencing + +1. **Quick wins first (independent, ship now):** the three audit fixes — + parallelize `handleTranscript`'s two reads (#1), default byte caps on the two + transcript routes (#2/#3). These are the first callers to move onto Layer A's + discipline and give immediate relief before Layer B lands. +2. **Layer B (harness):** additive in `engine`. `LoadSession` gains + snapshot-aware recovery; a snapshot writer + the two triggers are added. + Fully backward compatible — no snapshot present ⇒ full replay ⇒ today's + behavior. Ship dark, validate round-trip on real sessions, then rely on it. +3. **Layer A (boxes):** introduce the interface; migrate handlers + (transcript, goal/model/thinking, bootstrap, prompt) onto it. The mirror + (workstream 5) later backs the read methods behind the same interface. + +## 7. Testing + +**Layer B** +- Round-trip: snapshot@N restores identical state to full-replay@N. +- Recovery: state(snapshot@N) + replay(N+1..head) ≡ full-replay(0..head), for + N at several points. +- Fallback: missing / checksum-mismatch / seq-ahead-of-head snapshot ⇒ full + replay, correct result. +- Crash safety: a truncated/partial `.snap.tmp` never loads; prior snapshot + stands. +- **Bounded replay:** a synthetic long session's `LoadSession` time is ~constant + as journal length grows past K (the actual point of the feature) — assert the + *replayed-record count* is ≤ K, not a wall-clock number. + +**Layer A** +- `ResolveSession` issues no `firstSession` scan on a box with a sticky pointer; + fires (and persists) at most once when empty. +- Reads are bounded by default when the caller omits bounds. +- `Bootstrap`/multi-read fires its harness reads concurrently, not serially + (assert the concurrency, e.g. via overlapping timing or a fake harness that + records call ordering) — pins the #1 regression shut. + +House rule: assert the specific behavior (bounded, parallel, resolve-once, seq +count ≤ K), **not** raw aggregate request counts. + +## 8. Rollout & risk +- Snapshots are rebuildable and validated ⇒ safe to ship dark and fall back. +- Journal never truncated ⇒ the mirror and audit are unaffected. +- Backward compatible ⇒ old sessions with no snapshot behave as today until they + earn their first snapshot. + +## 9. Decisions (locked 2026-08-27) +Both harness facts are **verified** (§4.1 schema, §4.5 single-writer). Tuning/ +layout choices, decided: +1. **K cadence = 64 records**, exposed as **config** (not a magic constant) so it + can be tuned without a code change. On-idle snapshotting also always applies. +2. **Snapshot file layout = single-latest** (`.snap`). Rolling last-M is a + later option if a corrupt-newest fallback (to an older snapshot instead of a + full replay) proves worth it; single-latest already falls back to full replay + safely. +3. **Seq anchor = live counter** (§4.3 option a) — track records-written in + `Session`, bumped under `s.mu` in `writeRecord`. Persisted `record.seq` + (option b) is a clean follow-up, not needed for v1. + +## 10. Implementation touch points (verified against `/Users/andybons/dev/harness`) +- `engine/store.go`: `LoadSession` (**:968** — snapshot-aware recovery), + `writeRecord` (:919 — bump the seq counter / emit the append-boundary trigger), + `scanLog` (:1528 — tail replay from > N), the `record` type (:165 — if we take + seq-anchor option (b)), and the `persist*` helpers (~:440–670). +- `engine/engine.go`: the `Session` struct (:638 — the snapshot schema mirrors + its fold-state fields), `appendWithUsage` (:1546 — the append boundary), + `newSession`/`NewSession` (:956 — snapshot-aware construction). +- `engine/session_manager.go`: `deferPersist`/`unlockAndFlushPersist` (:318 — + the deferred manager-level writes that also snapshot per-session). +- `engine/journal.go`: reference for the existing curated-replayable-state + projection to model the snapshot schema on. +- Boxes (Layer A): `internal/api` — the shared journal-access interface and the + handler migrations (`harness_proxy.go` transcript, `harness_{goal,model,thinking}.go`, + `console_bootstrap.go`, `session_resolve.go`). diff --git a/engine/engine.go b/engine/engine.go index a5f848f7..19be4b72 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -14,6 +14,7 @@ import ( "sort" "strings" "sync" + "sync/atomic" "time" "github.com/majorcontext/harness/message" @@ -879,6 +880,29 @@ type Config struct { // bounds each of them, not their sum; a process-wide budget is the // natural follow-up if that proves insufficient. ToolReadBudgetBytes int64 + + // SnapshotEveryRecords is the journal-snapshot cadence: after this + // many records have been appended since the last snapshot, the session + // writes a new checkpoint beside its journal, so a later LoadSession + // replays only the records after it instead of the whole log. See + // snapshot.go and docs/design/journal-snapshotting.md. + // + // Zero or negative — the zero value — DISABLES snapshot WRITING + // entirely: an embedder building a bare engine.Config gets exactly the + // pre-snapshot behavior, with no .snap file ever created. The + // config/CLI layer supplies the product default of 64 (config key + // `snapshot_every_records`), the same unset-versus-explicit-zero split + // PromptRetries and MaxTokensContinuations use. + // + // READING a snapshot is never gated on this field. Recovery is a + // property of the files on disk, not of the loading Config: a session + // checkpointed by a process that had snapshotting on must load fast in + // a process that has it off, and a stale snapshot must be validated + // (and rejected) whatever this value is. + // + // Snapshot writing additionally requires SessionDir: with no session + // directory there is no journal to accelerate. + SnapshotEveryRecords int } // Session is one conversation: an in-memory history plus the agent loop. @@ -1007,6 +1031,53 @@ type Session struct { logStarted bool // the log file exists on disk lastPersistErr error + // recordsWritten is the journal's head SEQ: the count of records this + // session's journal holds, which — because the log is append-only and + // never rewritten — is also the 1-based LINE NUMBER of its last + // record. It is the anchor a snapshot is taken at (see snapshot.go and + // docs/design/journal-snapshotting.md §4.3, decision 3: a live counter + // rather than a persisted per-record seq). Bumped under s.mu by + // writeRecord for every record that actually lands, by ensureLog for + // the header records that bypass writeRecord, and set by LoadSession + // to the head of the journal it replayed. Guarded by mu. + recordsWritten int64 + // snapshotSeq is the anchor of the most recently SCHEDULED snapshot — + // see startSnapshotLocked for why it advances at scheduling time + // rather than on a successful write. snapshotting is the coalescing + // flag: at most one snapshot write is in flight per session. + // lastSnapshotErr holds the most recent snapshot write failure, and is + // deliberately NOT lastPersistErr: a snapshot is derived acceleration, + // never a durability promise. All three guarded by mu. + snapshotSeq int64 + snapshotting bool + lastSnapshotErr error + // snapshotWG tracks in-flight snapshot writes so a caller can wait for + // a settled disk (waitSnapshots). snapshotWrites/snapshotInFlight/ + // snapshotConcurrentPeak are the counters the coalescing invariant + // (rule 4) is asserted against; they are atomics because the + // background writer touches them while holding no lock. + snapshotWG sync.WaitGroup + snapshotWrites atomic.Int64 + snapshotInFlight atomic.Int64 + snapshotConcurrentPeak atomic.Int64 + // replayedRecords is how many journal records the LoadSession call + // that produced this session actually decoded and folded — the whole + // journal for a full replay, and only the header plus the post-anchor + // tail when a snapshot was used. It is the measurement the bounded- + // replay guarantee is stated over. + replayedRecords int64 + // durableDebt counts in-memory mutations whose durable record has been + // DEFERRED and has not landed yet — SessionManager's + // appendMemoryOnly/persistAppendedMessage and + // enqueueTaskNotificationMemoryOnly*/persistQueuedTaskNotification + // pairs, which split the two halves so the disk write happens after + // m.mu releases (see unlockAndFlushPersist). While it is non-zero, + // memory is AHEAD of the journal and no snapshot may be captured: one + // taken in that window would carry the mutation AND leave its record + // in the tail for a reload to apply a second time. See + // snapshotSafeLocked. Guarded by mu. + durableDebt int + // index is the running fold of every record this session has written // or replayed, and logSize the journal length that fold covers (see // index.go). Together they are what flushIndexLocked writes to the @@ -1936,6 +2007,12 @@ func (s *Session) appendWithUsage(m message.Message, usage *provider.Usage) { s.haveLastUsage = true } s.persistMessage(&m, usage) + // The append boundary (snapshot.go, rule 2): history, usage, and the + // journal all agree right here, and s.mu is already held, so this is + // where the every-K snapshot trigger belongs — not inside writeRecord, + // which runs before some callers have applied their own memory + // mutation. See maybeSnapshotLocked. + s.maybeSnapshotLocked() s.mu.Unlock() } @@ -1966,6 +2043,12 @@ func (s *Session) appendMemoryOnly(m message.Message) message.Message { } s.mu.Lock() s.history = append(s.history, m) + // This append is MEMORY AHEAD OF THE JOURNAL until + // persistAppendedMessage runs, so a snapshot taken in between would + // capture the message AND leave its record in the tail for a reload to + // append a second time. Record the debt; snapshotSafeLocked refuses to + // capture while any is outstanding. See snapshot.go. + s.durableDebt++ // See turnUnsettled's own doc comment — same as appendWithUsage. // recoverInterruptedTurnLocked's own closing-message append (this // method's one caller) relies on calling markTurnSettled AFTER this, @@ -1989,6 +2072,9 @@ func (s *Session) appendMemoryOnly(m message.Message) message.Message { func (s *Session) persistAppendedMessage(m message.Message) { s.mu.Lock() s.persistMessage(&m, nil) + // The journal has caught up with appendMemoryOnly's in-memory append + // — see the debt this settles there and in snapshotSafeLocked. + s.settleDurableDebtLocked() s.mu.Unlock() } @@ -2204,6 +2290,14 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) { s.emitStatus("busy") defer s.emitStatus("idle") + // The on-idle snapshot trigger (snapshot.go and docs/design/journal- + // snapshotting.md §4.4): a turn that has just finished is the + // quiescent moment — no append is in flight — and it is also the state + // a wake-from-hibernation or post-eviction reload starts from, so + // checkpointing here is what makes that next cold load cheap. It + // coalesces with the every-K trigger (one snapshot in flight per + // session) and is a no-op when nothing was written since the last one. + defer s.snapshotOnIdle() // maxTokensUsed counts every max_tokens auto-continuation issued in THIS // loop — i.e. across one Prompt call — never across a whole Session's diff --git a/engine/snapshot.go b/engine/snapshot.go new file mode 100644 index 00000000..c9aed2fa --- /dev/null +++ b/engine/snapshot.go @@ -0,0 +1,586 @@ +// Session journal snapshots: a seq-anchored checkpoint beside the journal +// that bounds what LoadSession has to replay. +// +// The problem it solves. A session's durable state is one append-only JSONL +// journal (store.go), and LoadSession rebuilds a session by decoding every +// record in it, building the whole history slice, and repairing it. That is +// O(journal size) and grows for the life of the session: on a deployed box +// a single transcript read cost 8 s, and every cold prompt paid the same +// replay before it could start. See docs/design/journal-snapshotting.md. +// +// The shape. A snapshot is an explicitly-defined schema (sessionSnapshot +// below), NOT json.Marshal of a *Session: every field of Session but ID is +// unexported, and several of them — the config's live callbacks, the +// SessionManager pointer, open file handles — must never be serialized at +// all. The schema captures exactly the state LoadSession's folds +// reconstruct, anchored to the 1-based journal LINE NUMBER of the last +// record it covers. Recovery restores it and replays only records after +// that line. +// +// Five rules make it safe. They are the design's §4.5 list, and every one +// of them is load-bearing: +// +// 1. Seq-anchored. A snapshot is valid AS OF seq N; it never claims to be +// current. The tail replay closes the gap, so a snapshot that lags the +// journal is correct, merely less of a saving. +// 2. Off the hot path. The capture happens under s.mu at an append +// boundary and copies a handful of slices and maps; the serialize and +// the write happen in a background goroutine that holds no lock. +// 3. Atomic write. temp file -> fsync -> rename. A crash mid-write leaves +// the PREVIOUS snapshot intact; a half-written .snap.tmp is never +// a file any reader looks at. +// 4. Single in-flight, coalesced. One snapshot per session at a time, so +// the every-K and on-idle triggers cannot race to write the same path. +// 5. Rebuildable and validated. A snapshot is derived state with a +// checksum and a version. ANY doubt — missing, torn, wrong version, +// wrong session, seq ahead of the journal head — discards it and falls +// back to a full replay. A snapshot bug degrades to slow, never wrong. +// +// The journal is never truncated. Snapshots are pure acceleration and can +// be deleted at any time; deleting them all restores exactly the behavior +// this package had before this file existed. +package engine + +import ( + "encoding/json" + "errors" + "hash/crc32" + "os" + "path/filepath" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// sessionSnapshotVersion is the snapshot format version. Recovery +// DISCARDS — never migrates — a snapshot carrying any other value, so a +// field added to the schema needs no migration path: bump this and every +// stored snapshot falls back to a full replay on its next load and is +// rewritten from the next trigger. +const sessionSnapshotVersion = 1 + +// sessionSnapshotSuffix names a session's snapshot file. Like the metadata +// index's own suffix it deliberately does not end in ".jsonl", so no +// journal scan can mistake a snapshot for a session. +const sessionSnapshotSuffix = ".snap" + +// sessionSnapshotTmpSuffix names the temp file the atomic write goes +// through. Nothing ever READS this path: that is what makes a crash +// mid-write invisible to recovery (rule 3). +const sessionSnapshotTmpSuffix = ".snap.tmp" + +// defaultSnapshotEveryRecords is the product default cadence supplied by +// the config/CLI layer (config.Config.SnapshotEveryRecordsValue), not by +// this package: engine.Config.SnapshotEveryRecords' own zero value +// disables snapshotting, so a bare embedder-built Config keeps exactly the +// pre-snapshot behavior. See that field's doc comment. +const defaultSnapshotEveryRecords = 64 + +// sessionSnapshot is the explicit schema: exactly the state LoadSession's +// folds reconstruct from records after the session header, plus the anchor +// and the identity a reader validates against. +// +// Two exclusions are deliberate and must stay that way. +// +// Header-derived state (created_at, workdir, parent/task lineage) is NOT +// here. The session header is line 1 of every journal and costs one record +// to decode, so recovery replays it unconditionally and the snapshot never +// has to reproduce its subtle "an absent field means keep the loading +// Config's value" restore rules (see LoadSession's recSession case). +// +// Session.turn and Session.lastSystem are NOT here either, though the +// design's §4.1 field list names them. They have no durable source: no +// record carries them, so a full replay reports turn=0 and no system +// segments. Capturing them would make a snapshot-loaded session disagree +// with a full replay of the same journal — breaking the §4.3 invariant this +// whole file is built around, and making an observable field +// (session_info's turn count) depend on whether a snapshot happened to +// exist. The invariant governs; the field list does not. +type sessionSnapshot struct { + Version int `json:"version"` + ID string `json:"id"` + // Seq is the 1-based journal line number of the LAST record this + // snapshot covers. Recovery replays strictly greater lines. + Seq int64 `json:"seq"` + // CreatedAt is when the snapshot itself was written — diagnostics + // only; nothing validates against it. + CreatedAt time.Time `json:"created_at"` + + History []message.Message `json:"history"` + + Model message.ModelRef `json:"model,omitzero"` + Effort message.Effort `json:"effort,omitempty"` + + Usage provider.Usage `json:"usage,omitzero"` + LastUsage provider.Usage `json:"last_usage,omitzero"` + HaveLastUsage bool `json:"have_last_usage,omitempty"` + + GoalActive bool `json:"goal_active,omitempty"` + GoalCondition string `json:"goal_condition,omitempty"` + + CompactCount int `json:"compact_count,omitempty"` + LastCompactedAt time.Time `json:"last_compacted_at,omitzero"` + + PromptQueue []QueuedPrompt `json:"prompt_queue,omitempty"` + PromptQueueNextID int64 `json:"prompt_queue_next_id,omitempty"` + EnqueueSeq int64 `json:"enqueue_seq,omitempty"` + + ToolResults map[string]toolResultMeta `json:"tool_results,omitempty"` + ToolResultNextID int64 `json:"tool_result_next_id,omitempty"` + ToolResultBytes int `json:"tool_result_bytes,omitempty"` + + MCPSelected []string `json:"mcp_selected,omitempty"` + + SpawnedChildIDs []string `json:"spawned_child_ids,omitempty"` + + // TaskNotifications is the UNDELIVERED set a full replay reconstructs: + // the in-flight entries first, then the still-pending ones, which is + // the same order requeueTaskNotifications restores them in and the + // same order the journal holds them. taskNotificationsInFlight has no + // record of its own — a checked-out notification is only "delivered" + // once commitTaskNotifications writes for it — so a snapshot that + // captured only Session.taskNotifications would silently drop the + // checked-out ones that a full replay keeps. + TaskNotifications []taskNotification `json:"task_notifications,omitempty"` + + TurnUnsettled bool `json:"turn_unsettled,omitempty"` + CommittedOutcome *taskNotification `json:"committed_outcome,omitempty"` +} + +// sessionSnapshotFile is the on-disk wrapper: the snapshot bytes plus a +// checksum over exactly those bytes. +// +// The checksum is what turns a torn or tampered file into a miss instead of +// a wrong session. CRC-32 detects corruption; it does not prove its +// absence, and it is not a trust boundary — this is a derived cache file +// written by this package alone, and a collision costs a load that would +// otherwise have been a full replay anyway. Same reasoning, same algorithm +// as the metadata index sidecar (index.go). +type sessionSnapshotFile struct { + CRC32 uint32 `json:"crc32"` + Snapshot json.RawMessage `json:"snapshot"` +} + +func sessionSnapshotPath(dir, id string) string { + return filepath.Join(dir, id+sessionSnapshotSuffix) +} + +func sessionSnapshotTmpPath(dir, id string) string { + return filepath.Join(dir, id+sessionSnapshotTmpSuffix) +} + +// marshalSessionSnapshot renders a snapshot as its on-disk bytes, checksum +// and all. +func marshalSessionSnapshot(snap *sessionSnapshot) ([]byte, error) { + inner, err := json.Marshal(snap) + if err != nil { + return nil, err + } + return json.Marshal(sessionSnapshotFile{CRC32: crc32.ChecksumIEEE(inner), Snapshot: inner}) +} + +// readSessionSnapshot loads a session's stored snapshot. It returns nil for +// every failure — absent, unreadable, malformed, checksum mismatch, wrong +// version, wrong session id — because a snapshot has no repair path: the +// caller full-replays instead. The seq-versus-head check is the caller's, +// since only it knows the journal head. +func readSessionSnapshot(dir, id string) *sessionSnapshot { + // The temp path is deliberately never consulted: a crash mid-write + // leaves a half-written .snap.tmp behind, and rule 3 is that no + // reader ever looks at it. + data, err := os.ReadFile(sessionSnapshotPath(dir, id)) + if err != nil { + return nil + } + var file sessionSnapshotFile + if err := json.Unmarshal(data, &file); err != nil { + return nil + } + if crc32.ChecksumIEEE(file.Snapshot) != file.CRC32 { + return nil + } + var snap sessionSnapshot + if err := json.Unmarshal(file.Snapshot, &snap); err != nil { + return nil + } + if snap.Version != sessionSnapshotVersion || snap.ID != id || snap.Seq <= 0 { + return nil + } + return &snap +} + +// writeSessionSnapshot replaces id's snapshot atomically: write a temp +// file, fsync it, rename it over the old one. A crash at any point leaves +// either the old snapshot or the new one, never a mix (rule 3). +// +// sync selects whether the temp file is fsynced before the rename. It is +// skipped in volume mode for the same reason ensureLog skips its directory +// fsync there — see Config.SessionSync. Losing an unsynced snapshot to a +// crash costs one full replay, which is the behavior this package had +// before snapshots existed. +func writeSessionSnapshot(dir, id string, snap *sessionSnapshot, sync bool) error { + b, err := marshalSessionSnapshot(snap) + if err != nil { + return err + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + tmp := sessionSnapshotTmpPath(dir, id) + f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) + if err != nil { + return err + } + if _, err := f.Write(b); err != nil { + f.Close() + os.Remove(tmp) + return err + } + if sync { + if err := f.Sync(); err != nil { + f.Close() + os.Remove(tmp) + return err + } + } + if err := f.Close(); err != nil { + os.Remove(tmp) + return err + } + if err := os.Rename(tmp, sessionSnapshotPath(dir, id)); err != nil { + os.Remove(tmp) + return err + } + return nil +} + +// snapshotEvery reports this session's every-K cadence, or 0 when +// snapshotting is off. See Config.SnapshotEveryRecords. +func (s *Session) snapshotEvery() int64 { + if s.cfg.SnapshotEveryRecords <= 0 { + return 0 + } + return int64(s.cfg.SnapshotEveryRecords) +} + +// snapshotEnabled reports whether this session can write a snapshot at all: +// it needs a cadence and somewhere to put the file. +func (s *Session) snapshotEnabled() bool { + return s.snapshotEvery() > 0 && s.cfg.SessionDir != "" +} + +// snapshotSafeLocked reports whether memory currently AGREES with the +// journal, which is the precondition every capture has: a snapshot pairs a +// memory image with a journal position, and recovery skips every record at +// or before that position. +// +// Two shapes break the agreement, and both are deliberate elsewhere in this +// package: +// +// - A mutation whose durable record is DEFERRED (Session.durableDebt): +// memory is ahead, and a snapshot would carry the mutation while +// leaving its record in the tail for the reload to apply again — +// duplicating a message or a task notification. +// - A prompt-queue record parked on the session +// (deferredQueueRecords, see queueRecordDeferredLocked in queue.go): +// the same shape, tracked by the parked record itself rather than by a +// counter. +// +// The opposite direction — a record written BEFORE its memory mutation, as +// EnqueuePromptDurable does deliberately — is handled by placing the +// trigger at the append boundary rather than inside writeRecord; see the +// comment there. +// +// Refusing to capture merely postpones a snapshot to the next boundary, so +// a guard that is too conservative costs a longer tail replay and nothing +// else. Caller holds s.mu. +func (s *Session) snapshotSafeLocked() bool { + return s.durableDebt == 0 && len(s.deferredQueueRecords) == 0 +} + +// settleDurableDebtLocked records that a deferred durable write has landed. +// It clamps at zero: a mispaired call site must not be able to push the +// count negative and permanently ARM a capture in an unsafe window. The +// other direction — a leaked increment — merely stops this session from +// snapshotting, which degrades to the full replay this package did before +// snapshots existed. Caller holds s.mu. +func (s *Session) settleDurableDebtLocked() { + if s.durableDebt > 0 { + s.durableDebt-- + } +} + +// maybeSnapshotLocked is the every-K trigger, called at an append boundary +// once the appending caller has applied BOTH its memory mutation and its +// durable record. Caller holds s.mu. +func (s *Session) maybeSnapshotLocked() { + if !s.snapshotEnabled() || !s.snapshotSafeLocked() { + return + } + if s.recordsWritten-s.snapshotSeq < s.snapshotEvery() { + return + } + s.startSnapshotLocked() +} + +// snapshotOnIdle is the on-idle trigger: a session that has just gone +// quiescent snapshots at the current journal head, so the next cold load — +// a wake from hibernation, a read after eviction — starts from there. It is +// a no-op when nothing has been written since the last snapshot. +func (s *Session) snapshotOnIdle() { + s.mu.Lock() + defer s.mu.Unlock() + s.snapshotIdleLocked() +} + +// snapshotIdleLocked is snapshotOnIdle for a caller that already holds +// s.mu (ReleaseFiles, which snapshots on eviction). +func (s *Session) snapshotIdleLocked() { + if !s.snapshotEnabled() || !s.snapshotSafeLocked() || s.recordsWritten <= s.snapshotSeq { + return + } + s.startSnapshotLocked() +} + +// startSnapshotLocked captures a consistent copy of the fold state and the +// seq it is anchored to, then serializes and writes it in a background +// goroutine. Caller holds s.mu. +// +// Coalescing (rule 4) is the snapshotting flag: while one write is in +// flight every other trigger returns immediately, so the two triggers can +// never race to write the same path and a burst of appends produces one +// write, not one per record. +// +// s.snapshotSeq advances at SCHEDULING time, not on success. A write that +// fails therefore does not re-arm the trigger on the very next record: the +// stored snapshot stays whatever it was, the next trigger fires K records +// later, and recovery replays a longer tail. Slower, never wrong. +func (s *Session) startSnapshotLocked() { + if s.snapshotting { + return + } + snap := s.captureSnapshotLocked() + s.snapshotting = true + s.snapshotSeq = snap.Seq + dir, id, sync := s.cfg.SessionDir, s.ID, !s.volumeSync() + s.snapshotWG.Add(1) + go func() { + defer s.snapshotWG.Done() + inFlight := s.snapshotInFlight.Add(1) + for { + peak := s.snapshotConcurrentPeak.Load() + if inFlight <= peak || s.snapshotConcurrentPeak.CompareAndSwap(peak, inFlight) { + break + } + } + err := writeSessionSnapshot(dir, id, snap, sync) + s.snapshotInFlight.Add(-1) + if err == nil { + s.snapshotWrites.Add(1) + } + s.mu.Lock() + s.snapshotting = false + if err != nil { + // Never lastPersistErr: a snapshot is derived acceleration, + // and its loss is not a durability failure any caller should + // be told about. The journal itself is untouched. + s.lastSnapshotErr = err + } + s.mu.Unlock() + }() +} + +// waitSnapshots blocks until every snapshot write this session started has +// finished. It is what a shutdown path (and a test that wants a settled +// disk) uses; it is NOT a barrier a new snapshot cannot start behind. +func (s *Session) waitSnapshots() { + s.snapshotWG.Wait() +} + +// captureSnapshotLocked builds the snapshot value for the state as it +// stands. Caller holds s.mu. +// +// Every slice and map is COPIED, so the background goroutine serializes a +// value nothing else can mutate. The messages themselves are copied by +// value and still share their Parts pointers with live history — the same +// sharing every other reader of s.history in this package accepts, and safe +// for the same reason: a message's parts are normalized before the append +// that publishes them and are not mutated in place afterwards. +func (s *Session) captureSnapshotLocked() *sessionSnapshot { + snap := &sessionSnapshot{ + Version: sessionSnapshotVersion, + ID: s.ID, + Seq: s.recordsWritten, + CreatedAt: time.Now().UTC(), + History: append([]message.Message(nil), s.history...), + Model: s.model, + Effort: s.effort, + Usage: s.usage, + LastUsage: s.lastUsage, + HaveLastUsage: s.haveLastUsage, + GoalActive: s.goalActive, + GoalCondition: s.goalCondition, + CompactCount: s.compactCount, + LastCompactedAt: s.lastCompactedAt, + PromptQueue: append([]QueuedPrompt(nil), s.promptQueue...), + PromptQueueNextID: s.promptQueueNextID, + EnqueueSeq: s.enqueueSeq, + ToolResultNextID: s.toolResultNextID, + ToolResultBytes: s.toolResultBytes, + SpawnedChildIDs: append([]string(nil), s.spawnedChildIDs...), + TurnUnsettled: s.turnUnsettled, + } + if len(s.toolResults) > 0 { + snap.ToolResults = make(map[string]toolResultMeta, len(s.toolResults)) + for k, v := range s.toolResults { + snap.ToolResults[k] = v + } + } + if len(s.mcpSelected) > 0 { + for name := range s.mcpSelected { + snap.MCPSelected = append(snap.MCPSelected, name) + } + } + // In-flight first, then pending — see TaskNotifications' doc comment. + if n := len(s.taskNotificationsInFlight) + len(s.taskNotifications); n > 0 { + snap.TaskNotifications = make([]taskNotification, 0, n) + snap.TaskNotifications = append(snap.TaskNotifications, s.taskNotificationsInFlight...) + snap.TaskNotifications = append(snap.TaskNotifications, s.taskNotifications...) + } + if s.committedOutcome != nil { + oc := *s.committedOutcome + snap.CommittedOutcome = &oc + } + return snap +} + +// restoreSnapshot writes a snapshot's state into a freshly-constructed +// session, in place of the records it covers. LoadSession calls it after +// applying the session header and before replaying the tail, so a later +// record still wins over anything here. +// +// Every message is Normalized, exactly as the record replay path +// normalizes each message it decodes: the snapshot was written from +// already-normalized history, so this is a no-op in practice, but making it +// unconditional is what keeps the two paths' output identical by +// construction rather than by assumption. +func (s *Session) restoreSnapshot(snap *sessionSnapshot) { + s.history = make([]message.Message, len(snap.History)) + for i, m := range snap.History { + m.Normalize() + s.history[i] = m + } + if !snap.Model.IsZero() { + s.model = snap.Model + } + s.effort = snap.Effort + s.usage = snap.Usage + s.lastUsage = snap.LastUsage + s.haveLastUsage = snap.HaveLastUsage + s.goalActive = snap.GoalActive + s.goalCondition = snap.GoalCondition + s.compactCount = snap.CompactCount + s.lastCompactedAt = snap.LastCompactedAt + s.promptQueue = append([]QueuedPrompt(nil), snap.PromptQueue...) + if snap.PromptQueueNextID > 0 { + s.promptQueueNextID = snap.PromptQueueNextID + } + s.enqueueSeq = snap.EnqueueSeq + if len(snap.ToolResults) > 0 { + s.toolResults = make(map[string]toolResultMeta, len(snap.ToolResults)) + for k, v := range snap.ToolResults { + s.toolResults[k] = v + } + } + if snap.ToolResultNextID > 0 { + s.toolResultNextID = snap.ToolResultNextID + } + s.toolResultBytes = snap.ToolResultBytes + for _, name := range snap.MCPSelected { + // Same defensive shape the recMCPToolsSelected fold applies: a + // name that is not mcp____ shaped is skipped, so one + // rule holds however the state arrives. + if _, _, ok := splitMCPToolName(name); !ok { + continue + } + if s.mcpSelected == nil { + s.mcpSelected = map[string]bool{} + } + s.mcpSelected[name] = true + } + s.spawnedChildIDs = append([]string(nil), snap.SpawnedChildIDs...) + s.taskNotifications = append([]taskNotification(nil), snap.TaskNotifications...) + s.turnUnsettled = snap.TurnUnsettled + if snap.CommittedOutcome != nil { + oc := *snap.CommittedOutcome + s.committedOutcome = &oc + } +} + +// snapshotStartAfter reports the journal line recovery may skip up to, and +// restores the snapshot's state into s when there is a usable one. +// +// It returns 0 — full replay — for every doubt: no snapshot, a snapshot for +// another session, a wrong version, a torn file, a seq that runs AHEAD of +// the journal head (a journal replaced or rolled back underneath a stale +// snapshot), or a journal whose first record is not a session header (in +// which case there is no header to apply before the restore, and the +// snapshot's own "the header is replayed separately" premise does not +// hold). +func (s *Session) snapshotStartAfter(dir, id string, data []byte, head int64) int64 { + snap := readSessionSnapshot(dir, id) + if snap == nil || snap.Seq > head { + return 0 + } + hdr, ok := firstJournalRecord(data) + if !ok || hdr.Type != recSession { + return 0 + } + // The header first, then the snapshot on top of it: the header carries + // the session's CREATE-time effort, which a later recEffort record (and + // so the snapshot) supersedes. + s.applySessionHeader(hdr) + s.restoreSnapshot(snap) + return snap.Seq +} + +// firstJournalRecord decodes a journal's first non-empty line. It exists so +// recovery can apply the session header without decoding the records the +// snapshot already covers. +func firstJournalRecord(data []byte) (record, bool) { + var out record + found := false + err := scanLogRaw(data, func(raw []byte, line int, isLast bool) error { + if err := json.Unmarshal(raw, &out); err != nil { + return errStopScan + } + found = true + return errStopScan + }) + if err != nil && !errors.Is(err, errStopScan) { + return record{}, false + } + return out, found +} + +// errStopScan ends a scanLogRaw walk early. scanLogRaw propagates every +// error but errTruncatedFinalRecord, so the caller matches on this +// sentinel rather than treating an early stop as a failure. +var errStopScan = errors.New("engine: stop scan") + +// countJournalRecords counts a journal's records without decoding any of +// them — the journal head, for validating a snapshot's anchor against. +// +// It counts LINES, which can exceed the number of records a fold applies by +// at most one: a crash mid-write leaves a torn final line scanLog drops. +// That cannot admit a bad snapshot, because a torn write never advanced the +// writer's own record counter, so no snapshot's seq can name it. +func countJournalRecords(data []byte) int64 { + var n int64 + _ = scanLogRaw(data, func(raw []byte, line int, isLast bool) error { + n = int64(line) + return nil + }) + return n +} diff --git a/engine/snapshot_test.go b/engine/snapshot_test.go new file mode 100644 index 00000000..7faf541a --- /dev/null +++ b/engine/snapshot_test.go @@ -0,0 +1,652 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// foldState is every piece of session state a journal replay reconstructs — +// the exact set a snapshot must carry, and the exact set the +// snapshot-equals-replay invariant (docs/design/journal-snapshotting.md +// §4.3) is stated over. +// +// It is deliberately built by READING the loaded session rather than by +// re-marshaling the snapshot: a test that compared snapshots to snapshots +// would pass for a field the snapshot drops on the floor. +type foldState struct { + History json.RawMessage + Model message.ModelRef + Effort message.Effort + Usage provider.Usage + LastUsage provider.Usage + HaveLastUsage bool + GoalActive bool + GoalCondition string + CompactCount int + LastCompactedAt string + Queue []QueuedPrompt + QueueNextID int64 + EnqueueSeq int64 + ToolResults map[string]toolResultMeta + ToolResultNext int64 + ToolResultBytes int + MCPSelected map[string]bool + SpawnedChildren []string + Notifications []taskNotification + TurnUnsettled bool + Committed *taskNotification + CreatedAt string + WorkDir string + ParentSession string + TaskParentID string + TaskAgentType string + TaskDepth int +} + +func foldStateOf(t *testing.T, s *Session) string { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + h, err := json.Marshal(s.history) + if err != nil { + t.Fatalf("marshal history: %v", err) + } + st := foldState{ + History: h, + Model: s.model, + Effort: s.effort, + Usage: s.usage, + LastUsage: s.lastUsage, + HaveLastUsage: s.haveLastUsage, + GoalActive: s.goalActive, + GoalCondition: s.goalCondition, + CompactCount: s.compactCount, + LastCompactedAt: s.lastCompactedAt.UTC().String(), + Queue: s.promptQueue, + QueueNextID: s.promptQueueNextID, + EnqueueSeq: s.enqueueSeq, + ToolResults: s.toolResults, + ToolResultNext: s.toolResultNextID, + ToolResultBytes: s.toolResultBytes, + MCPSelected: s.mcpSelected, + SpawnedChildren: s.spawnedChildIDs, + Notifications: s.taskNotifications, + TurnUnsettled: s.turnUnsettled, + Committed: s.committedOutcome, + CreatedAt: s.createdAt.UTC().String(), + WorkDir: s.cfg.WorkDir, + ParentSession: s.cfg.ParentSession, + TaskParentID: s.cfg.TaskParentID, + TaskAgentType: s.cfg.TaskAgentType, + TaskDepth: s.cfg.TaskDepth, + } + b, err := json.Marshal(st) + if err != nil { + t.Fatalf("marshal fold state: %v", err) + } + return string(b) +} + +// snapshotTestProvider returns a scripted provider with n plain assistant +// turns, so a test can drive an arbitrarily long session. +func snapshotTestProvider(n int) *scriptedProvider { + turns := make([][]provider.Event, 0, n) + for i := range n { + turns = append(turns, asstTurn(provider.StopEndTurn, &message.Text{Text: fmt.Sprintf("reply %d", i)})) + } + return &scriptedProvider{name: "test", turns: turns} +} + +// idleOnly is a cadence so large the every-K trigger never fires in a +// test, leaving the on-idle trigger as the only source of snapshots. It is +// not "off": zero disables snapshot writing entirely (see +// Config.SnapshotEveryRecords), which is what TestSnapshotDisabledWritesNothing +// exercises. +const idleOnly = 1 << 20 + +// snapshotCfg is persistCfg with an explicit snapshot cadence. +func snapshotCfg(dir string, prov *scriptedProvider, every int) Config { + cfg := persistCfg(dir, prov) + cfg.SnapshotEveryRecords = every + return cfg +} + +// buildSession drives turns prompts through s and returns it, draining any +// in-flight snapshot write so the test sees a settled disk. +func drive(t *testing.T, s *Session, turns int) { + t.Helper() + for i := range turns { + if _, err := s.Prompt(context.Background(), fmt.Sprintf("prompt %d", i)); err != nil { + t.Fatalf("Prompt %d: %v", i, err) + } + } + if err := s.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + s.waitSnapshots() +} + +// TestSnapshotRoundTrip pins the first half of the §4.3 invariant: a +// session restored from a snapshot taken at seq N holds exactly the state a +// full replay of the same journal at seq N produces. It compares against a +// load with snapshotting switched OFF, so the two paths are compared on the +// same journal bytes rather than against each other's assumptions. +func TestSnapshotRoundTrip(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(8), 4)) + drive(t, s, 8) + + // The journal head is exactly where the last snapshot landed only by + // luck, so take an explicit idle snapshot at head first. + s.snapshotOnIdle() + s.waitSnapshots() + + snapLoaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 4), s.ID) + if err != nil { + t.Fatalf("LoadSession (snapshot path): %v", err) + } + if snapLoaded.replayedRecords >= snapLoaded.recordsWritten { + t.Fatalf("replayed %d of %d records — the snapshot was not used", snapLoaded.replayedRecords, snapLoaded.recordsWritten) + } + + // Full replay of the identical journal: same dir, snapshot removed. + full := t.TempDir() + copyFile(t, sessionPath(dir, s.ID), sessionPath(full, s.ID)) + fullLoaded, err := LoadSession(snapshotCfg(full, snapshotTestProvider(0), 4), s.ID) + if err != nil { + t.Fatalf("LoadSession (full replay): %v", err) + } + + if got, want := foldStateOf(t, snapLoaded), foldStateOf(t, fullLoaded); got != want { + t.Errorf("snapshot-loaded state != full-replay state\n got: %s\nwant: %s", got, want) + } +} + +// TestSnapshotPlusTailEqualsFullReplay pins the whole §4.3 invariant: +// state(snapshot@N) + replay(N+1..head) ≡ full-replay(0..head), at several +// N. Each iteration takes a snapshot at the current head and then appends +// more records, so the snapshot is genuinely BEHIND the journal when it is +// loaded. +func TestSnapshotPlusTailEqualsFullReplay(t *testing.T) { + for _, tail := range []int{1, 2, 5, 9} { + t.Run(fmt.Sprintf("tail=%d", tail), func(t *testing.T) { + dir := t.TempDir() + // Snapshotting off during the build: this test drives the + // anchor explicitly so the snapshot sits at a known N. + s := NewSession(snapshotCfg(dir, snapshotTestProvider(4+tail), idleOnly)) + drive(t, s, 4) + s.snapshotOnIdle() + s.waitSnapshots() + drive(t, s, tail) + + snapLoaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 0), s.ID) + if err != nil { + t.Fatalf("LoadSession (snapshot path): %v", err) + } + if snapLoaded.replayedRecords >= snapLoaded.recordsWritten { + t.Fatalf("replayed %d of %d records — the snapshot was not used", snapLoaded.replayedRecords, snapLoaded.recordsWritten) + } + + full := t.TempDir() + copyFile(t, sessionPath(dir, s.ID), sessionPath(full, s.ID)) + fullLoaded, err := LoadSession(snapshotCfg(full, snapshotTestProvider(0), idleOnly), s.ID) + if err != nil { + t.Fatalf("LoadSession (full replay): %v", err) + } + if got, want := foldStateOf(t, snapLoaded), foldStateOf(t, fullLoaded); got != want { + t.Errorf("snapshot+tail state != full-replay state\n got: %s\nwant: %s", got, want) + } + }) + } +} + +// TestSnapshotCarriesEveryFoldedField drives every fold LoadSession runs — +// model, effort, goal, prompt queue, usage — into one session, snapshots +// it, and proves the reload through the snapshot agrees with the full +// replay field for field. It is the guard the design calls for against a +// fold added without a matching snapshot field. +func TestSnapshotCarriesEveryFoldedField(t *testing.T) { + dir := t.TempDir() + cfg := snapshotCfg(dir, snapshotTestProvider(3), idleOnly) + cfg.WorkDir = t.TempDir() + cfg.ParentSession = "ses_1111111111111111" + s := NewSession(cfg) + + drive(t, s, 1) + s.SetModel(message.ModelRef{Provider: "test", Model: "m2"}) + s.SetEffort(message.EffortHigh) + if err := s.RegisterGoal("ship it"); err != nil { + t.Fatalf("RegisterGoal: %v", err) + } + if _, err := s.EnqueuePrompt("queued one"); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + if _, _, err := s.EnqueuePromptDurable("queued two", 7); err != nil { + t.Fatalf("EnqueuePromptDurable: %v", err) + } + drive(t, s, 2) + + s.snapshotOnIdle() + s.waitSnapshots() + + snapLoaded, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatalf("LoadSession (snapshot path): %v", err) + } + full := t.TempDir() + copyFile(t, sessionPath(dir, s.ID), sessionPath(full, s.ID)) + fullCfg := cfg + fullCfg.SessionDir = full + fullLoaded, err := LoadSession(fullCfg, s.ID) + if err != nil { + t.Fatalf("LoadSession (full replay): %v", err) + } + if got, want := foldStateOf(t, snapLoaded), foldStateOf(t, fullLoaded); got != want { + t.Errorf("snapshot-loaded state != full-replay state\n got: %s\nwant: %s", got, want) + } + // Spot-check a couple of fields directly, so a bug that drops BOTH + // paths' state identically still fails. + if !snapLoaded.goalActive || snapLoaded.goalCondition != "ship it" { + t.Errorf("goal = (%v, %q), want (true, %q)", snapLoaded.goalActive, snapLoaded.goalCondition, "ship it") + } + if got := snapLoaded.Model(); got != (message.ModelRef{Provider: "test", Model: "m2"}) { + t.Errorf("model = %v, want test/m2", got) + } + if len(snapLoaded.promptQueue) != 2 { + t.Errorf("queue = %d entries, want 2", len(snapLoaded.promptQueue)) + } +} + +// TestSnapshotFallbacks covers every way a snapshot can fail validation. +// Each one must degrade to a full replay that produces the correct state — +// "slower, never wrong". +func TestSnapshotFallbacks(t *testing.T) { + build := func(t *testing.T) (string, string, string) { + t.Helper() + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(6), idleOnly)) + drive(t, s, 6) + s.snapshotOnIdle() + s.waitSnapshots() + // The reference state, from a full replay of the same journal. + full := t.TempDir() + copyFile(t, sessionPath(dir, s.ID), sessionPath(full, s.ID)) + ref, err := LoadSession(snapshotCfg(full, snapshotTestProvider(0), idleOnly), s.ID) + if err != nil { + t.Fatalf("reference LoadSession: %v", err) + } + return dir, s.ID, foldStateOf(t, ref) + } + + cases := []struct { + name string + corrupt func(t *testing.T, dir, id string) + }{ + {"missing", func(t *testing.T, dir, id string) { + if err := os.Remove(sessionSnapshotPath(dir, id)); err != nil { + t.Fatal(err) + } + }}, + {"checksum mismatch", func(t *testing.T, dir, id string) { + p := sessionSnapshotPath(dir, id) + data, err := os.ReadFile(p) + if err != nil { + t.Fatal(err) + } + var f sessionSnapshotFile + if err := json.Unmarshal(data, &f); err != nil { + t.Fatal(err) + } + f.CRC32++ + b, err := json.Marshal(f) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, b, 0o644); err != nil { + t.Fatal(err) + } + }}, + {"seq ahead of head", func(t *testing.T, dir, id string) { + rewriteSnapshot(t, dir, id, func(snap *sessionSnapshot) { + snap.Seq += 1000 + }) + }}, + {"wrong version", func(t *testing.T, dir, id string) { + rewriteSnapshot(t, dir, id, func(snap *sessionSnapshot) { + snap.Version = sessionSnapshotVersion + 1 + }) + }}, + {"wrong session id", func(t *testing.T, dir, id string) { + rewriteSnapshot(t, dir, id, func(snap *sessionSnapshot) { + snap.ID = "ses_9999999999999999" + }) + }}, + {"garbage", func(t *testing.T, dir, id string) { + if err := os.WriteFile(sessionSnapshotPath(dir, id), []byte("{not json"), 0o644); err != nil { + t.Fatal(err) + } + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir, id, want := build(t) + tc.corrupt(t, dir, id) + + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 0), id) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if loaded.replayedRecords != loaded.recordsWritten { + t.Errorf("replayed %d of %d records, want a FULL replay", loaded.replayedRecords, loaded.recordsWritten) + } + if got := foldStateOf(t, loaded); got != want { + t.Errorf("fallback state != full-replay state\n got: %s\nwant: %s", got, want) + } + }) + } +} + +// TestSnapshotPartialTempFileNeverLoads is the crash-safety case: a +// half-written .snap.tmp is not a snapshot, and the previous .snap +// still stands. +func TestSnapshotPartialTempFileNeverLoads(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(4), idleOnly)) + drive(t, s, 4) + s.snapshotOnIdle() + s.waitSnapshots() + + good, err := os.ReadFile(sessionSnapshotPath(dir, s.ID)) + if err != nil { + t.Fatalf("no snapshot written: %v", err) + } + // A crash mid-write leaves the temp file behind, truncated. + if err := os.WriteFile(sessionSnapshotTmpPath(dir, s.ID), good[:len(good)/2], 0o644); err != nil { + t.Fatal(err) + } + + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 0), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if loaded.replayedRecords >= loaded.recordsWritten { + t.Errorf("replayed %d of %d records — the surviving snapshot was ignored", loaded.replayedRecords, loaded.recordsWritten) + } + if got, want := len(loaded.History()), len(s.History()); got != want { + t.Errorf("history = %d messages, want %d", got, want) + } + // And a temp file with NO snapshot beside it loads nothing at all. + if err := os.Remove(sessionSnapshotPath(dir, s.ID)); err != nil { + t.Fatal(err) + } + loaded, err = LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 0), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if loaded.replayedRecords != loaded.recordsWritten { + t.Errorf("replayed %d of %d records, want a FULL replay", loaded.replayedRecords, loaded.recordsWritten) + } +} + +// TestSnapshotBoundsReplayedRecords is the point of the whole feature: a +// long session's load replays at most K records however long the journal +// grows. It asserts the REPLAYED-RECORD COUNT, never a wall-clock number +// and never a raw total. +func TestSnapshotBoundsReplayedRecords(t *testing.T) { + const every = 8 + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(60), every)) + + var maxReplayed int64 + for turn := range 60 { + if _, err := s.Prompt(context.Background(), fmt.Sprintf("p%d", turn)); err != nil { + t.Fatalf("Prompt %d: %v", turn, err) + } + s.waitSnapshots() + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), every), s.ID) + if err != nil { + t.Fatalf("LoadSession after turn %d: %v", turn, err) + } + if loaded.replayedRecords > maxReplayed { + maxReplayed = loaded.replayedRecords + } + if loaded.replayedRecords > every { + t.Fatalf("after turn %d: replayed %d records (journal head %d), want <= K=%d", + turn, loaded.replayedRecords, loaded.recordsWritten, every) + } + } + // The journal is far longer than K, so a bound that held only because + // nothing was ever written would be no evidence at all. + if s.recordsWritten <= every { + t.Fatalf("journal head = %d records, want well past K=%d", s.recordsWritten, every) + } + if maxReplayed == 0 { + t.Fatal("no load ever replayed a record — the test measured nothing") + } +} + +// TestSnapshotDisabledWritesNothing pins the config seam: a session with +// snapshotting off writes no .snap at all and loads exactly as it always +// has. +func TestSnapshotDisabledWritesNothing(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(20), 0)) + drive(t, s, 20) + + if _, err := os.Stat(sessionSnapshotPath(dir, s.ID)); !os.IsNotExist(err) { + t.Errorf("stat .snap = %v, want not-exist with snapshotting disabled", err) + } + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), 0), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if loaded.replayedRecords != loaded.recordsWritten { + t.Errorf("replayed %d of %d records, want a FULL replay", loaded.replayedRecords, loaded.recordsWritten) + } + if got, want := len(loaded.History()), len(s.History()); got != want { + t.Errorf("history = %d messages, want %d", got, want) + } +} + +// TestSnapshotWritesAreCoalesced pins rule 4: only one snapshot per session +// is ever in flight, so the every-K and on-idle triggers cannot race to +// write the same file. +func TestSnapshotWritesAreCoalesced(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(4), 2)) + drive(t, s, 4) + for range 10 { + s.snapshotOnIdle() + } + s.mu.Lock() + inFlight := s.snapshotting + s.mu.Unlock() + if inFlight { + // One may legitimately still be running; what must never happen is + // two. The counter below is the real assertion. + _ = inFlight + } + s.waitSnapshots() + if got := s.snapshotWrites.Load(); got == 0 { + t.Fatal("no snapshot written") + } + if got := s.snapshotConcurrentPeak.Load(); got > 1 { + t.Errorf("peak concurrent snapshot writers = %d, want 1", got) + } +} + +func copyFile(t *testing.T, from, to string) { + t.Helper() + data, err := os.ReadFile(from) + if err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Dir(to), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(to, data, 0o644); err != nil { + t.Fatal(err) + } +} + +// rewriteSnapshot mutates a stored snapshot and rewrites it WITH a valid +// checksum, so the test exercises the field's own validation rather than +// the checksum's. +func rewriteSnapshot(t *testing.T, dir, id string, fn func(*sessionSnapshot)) { + t.Helper() + data, err := os.ReadFile(sessionSnapshotPath(dir, id)) + if err != nil { + t.Fatal(err) + } + var f sessionSnapshotFile + if err := json.Unmarshal(data, &f); err != nil { + t.Fatal(err) + } + var snap sessionSnapshot + if err := json.Unmarshal(f.Snapshot, &snap); err != nil { + t.Fatal(err) + } + fn(&snap) + b, err := marshalSessionSnapshot(&snap) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(sessionSnapshotPath(dir, id), b, 0o644); err != nil { + t.Fatal(err) + } +} + +// TestSnapshotRefusesWhileMemoryIsAheadOfJournal pins the capture +// precondition (snapshotSafeLocked): SessionManager splits some mutations +// into an in-memory half and a DEFERRED durable half, and a snapshot taken +// between the two would carry the mutation while leaving its record in the +// tail — so the reload would apply it twice. +// +// It drives the production pair directly, in the order SessionManager runs +// it, and asserts on the reloaded state rather than on the guard's own +// boolean: a duplicate message is the defect, and the reload is where it +// shows. +func TestSnapshotRefusesWhileMemoryIsAheadOfJournal(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(1), idleOnly)) + drive(t, s, 1) + + closing := s.appendMemoryOnly(message.Message{ + ID: "msg_deferred", + Role: message.RoleAssistant, + Parts: message.Parts{&message.Text{Text: "lost to restart"}}, + }) + // The window: memory holds the message, the journal does not. Advance + // the journal with an unrelated record so a snapshot is due, then try + // to take one — this is exactly the state the guard must refuse. + s.SetModel(message.ModelRef{Provider: "test", Model: "m2"}) + s.snapshotOnIdle() + s.waitSnapshots() + s.persistAppendedMessage(closing) + s.waitSnapshots() + + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), idleOnly), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + var n int + for _, m := range loaded.History() { + if m.ID == "msg_deferred" { + n++ + } + } + if n != 1 { + t.Errorf("reloaded history holds %d copies of the deferred message, want 1", n) + } + if got, want := len(loaded.History()), len(s.History()); got != want { + t.Errorf("history = %d messages, want %d", got, want) + } +} + +// TestSnapshotRefusesWhileNotificationIsAheadOfJournal is the same +// precondition for the OTHER split pair: a child-completion notification +// appended to memory under m.mu, with its record written after m.mu +// releases. A snapshot in that window duplicates the notification on +// reload, which the parent renders to the model as two completed children. +func TestSnapshotRefusesWhileNotificationIsAheadOfJournal(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(1), idleOnly)) + drive(t, s, 1) + + n := taskNotification{ChildID: "ses_2222222222222222", Agent: "explore", Status: StatusDone, Result: "done"} + s.enqueueTaskNotificationMemoryOnly(n) + // The window: memory holds the notification, the journal does not. + // Advance the journal with an unrelated record so a snapshot is due, + // then try to take one — exactly the state the guard must refuse. + s.SetModel(message.ModelRef{Provider: "test", Model: "m2"}) + s.snapshotOnIdle() + s.waitSnapshots() + s.persistQueuedTaskNotification(n) + s.waitSnapshots() + + loaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), idleOnly), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + loaded.mu.Lock() + got := len(loaded.taskNotifications) + loaded.mu.Unlock() + if got != 1 { + t.Errorf("reloaded session holds %d pending notifications, want 1", got) + } +} + +// TestSnapshotAnchorSurvivesTornTail pins the anchor's domain: seq is a +// journal LINE NUMBER, and a crash mid-write leaves a torn final line that +// the load tolerates (scanLog drops it) and the next write repairs away +// (ensureLog truncates it). A resumed session that counted that dropped +// line would take every later anchor one line too high — permanently ahead +// of the journal head, so every one of its snapshots would be rejected as +// seq-ahead and the session would never load fast again. +func TestSnapshotAnchorSurvivesTornTail(t *testing.T) { + dir := t.TempDir() + s := NewSession(snapshotCfg(dir, snapshotTestProvider(2), idleOnly)) + drive(t, s, 2) + + // A crash mid-write: a partial record with no trailing newline. + f, err := os.OpenFile(sessionPath(dir, s.ID), os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatal(err) + } + if _, err := f.WriteString(`{"type":"message","message":{"id":"msg_torn","ro`); err != nil { + t.Fatal(err) + } + f.Close() + + cfg := snapshotCfg(dir, snapshotTestProvider(1), idleOnly) + resumed, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatalf("LoadSession over torn tail: %v", err) + } + if _, err := resumed.Prompt(context.Background(), "after the crash"); err != nil { + t.Fatalf("Prompt: %v", err) + } + if err := resumed.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + resumed.waitSnapshots() + + reloaded, err := LoadSession(snapshotCfg(dir, snapshotTestProvider(0), idleOnly), s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if reloaded.replayedRecords >= reloaded.recordsWritten { + t.Errorf("replayed %d of %d records — the anchor taken after a torn tail was rejected", + reloaded.replayedRecords, reloaded.recordsWritten) + } + if got, want := len(reloaded.History()), len(resumed.History()); got != want { + t.Errorf("history = %d messages, want %d", got, want) + } +} diff --git a/engine/store.go b/engine/store.go index b62511d8..3fcd9c4e 100644 --- a/engine/store.go +++ b/engine/store.go @@ -1038,6 +1038,10 @@ func (s *Session) ensureLog() error { for _, rec := range headerRecs { s.index.applyIndexRecordBestEffort(indexRecordOf(rec), false) } + // Same reason the fold is applied here: these records bypass + // writeRecord, so the snapshot anchor must count them here or every + // seq this session ever takes is short by the header's length. + s.recordsWritten += int64(len(headerRecs)) // A file fsync (as EnqueuePromptDurable does before attesting // durability — see queue.go) commits the file's *contents* but not // its directory entry: POSIX leaves the entry itself up to the @@ -1148,6 +1152,13 @@ func (s *Session) ensureLog() error { func (s *Session) ReleaseFiles() { s.mu.Lock() defer s.mu.Unlock() + // Eviction is the on-idle trigger's other half (docs/design/journal- + // snapshotting.md §4.6): the caller has already decided this session + // is idle and will be reloaded from disk, so checkpointing now is + // exactly what makes that reload cheap. Background, coalesced, and a + // no-op when nothing has been written since the last snapshot — see + // snapshot.go. + s.snapshotIdleLocked() if s.logFile != nil { s.logFile.Close() s.logFile = nil @@ -1193,8 +1204,24 @@ func (s *Session) writeRecord(rec record) error { return err } s.logSize += int64(n) + // The journal head advanced by exactly one record, so the snapshot + // anchor does too (see Session.recordsWritten). Only a record that + // actually landed counts: the failed-write branch above returns before + // this line, so a torn line can never be named by a snapshot's seq. + s.recordsWritten++ s.index.applyIndexRecordBestEffort(indexRecordOf(rec), false) s.flushIndexLocked() + // Deliberately NO snapshot trigger here, though this is the single + // record choke point and so the obvious place for one. A snapshot + // captures MEMORY and anchors it to a JOURNAL POSITION, and inside + // this function the two do not yet agree: several callers persist + // their record BEFORE they apply their own in-memory mutation + // (EnqueuePromptDurable, deliberately — see queue.go), so a capture + // taken here would anchor past a record whose effect memory has not + // applied, and the reload would skip that record and lose the effect + // permanently. The trigger lives at the append boundary instead (see + // Session.maybeSnapshotLocked call sites), where the caller has + // completed both halves. return nil } @@ -1279,13 +1306,45 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.ID = id s.logStarted = true + // The journal head, in LINES. It is the domain the snapshot anchor + // lives in (see Session.recordsWritten) and it costs a byte scan, no + // decoding. It can exceed the number of records the scan below + // actually applies by one, when a crash mid-write left a torn final + // line; the loop corrects s.recordsWritten to the last line genuinely + // applied, so a session resumed over a torn tail keeps taking anchors + // that agree with the line numbers a later load will see. + head := countJournalRecords(data) + + // Snapshot-aware recovery (snapshot.go and docs/design/journal- + // snapshotting.md §4.3). startAfter is the last journal line a valid + // snapshot covers — 0 when there is no usable one, which is a full + // replay and exactly the behavior this function had before snapshots + // existed. snapshotStartAfter has already applied the session header + // and restored the snapshot's state by the time it returns non-zero. + startAfter := s.snapshotStartAfter(cfg.SessionDir, id, data, head) + if startAfter > 0 { + s.snapshotSeq = startAfter + // The metadata index (index.go) is a fold of EVERY record, and + // this load is deliberately not going to see most of them. Mark it + // broken rather than build a partial fold that would flush a + // sidecar claiming to summarize the whole journal while describing + // its tail. The index is a cache with no repair path: a reader + // that finds none refolds, and this session's next write re-seeds + // the fold from the journal in ensureLog. Snapshotting the index + // fold itself is a possible follow-up; nothing here may guess at + // it. + s.index.broken = true + } + // The prompt queue folds through promptQueueFold (queue.go), seeded - // from this fresh session's own counters and written back after the - // scan — the same fold the metadata index uses, so the two can never - // drift on the torn-write and ID-burn rules it holds. - qf := promptQueueFold{nextID: s.promptQueueNextID, seq: s.enqueueSeq} + // from this fresh session's own counters — or from the snapshot's + // restored queue, so the tail's prompt.queued/prompt.dequeued records + // fold onto the state the snapshot already holds — and written back + // after the scan. It is the same fold the metadata index uses, so the + // two can never drift on the torn-write and ID-burn rules it holds. + qf := promptQueueFold{queue: s.promptQueue, nextID: s.promptQueueNextID, seq: s.enqueueSeq} - err = scanLog(data, func(rec record, line int, isLast bool) error { + apply := func(rec record, line int, isLast bool) error { // Seed the session's metadata-index fold from the same records // (index.go). A resumed session keeps writing that index through // on every later record, so it must start from the state this @@ -1295,67 +1354,7 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.index.applyIndexRecordBestEffort(indexRecordOf(rec), isLast) switch rec.Type { case recSession: - s.createdAt = rec.CreatedAt - // A restored WorkDir wins over the loading Config.WorkDir: the - // header is the durable truth for a resumed session. A legacy - // header (written before this field existed) omits it, so an - // empty value here means "nothing to restore" — the loading - // Config.WorkDir is kept unchanged. - if rec.WorkDir != "" { - s.cfg.WorkDir = rec.WorkDir - } - // Same restore rule as WorkDir above: the header is the durable - // truth for a resumed session, and an empty value here means - // nothing to restore (legacy header, or no lineage recorded), - // never "clear the loading Config's ParentSession". - if rec.ParentSession != "" { - s.cfg.ParentSession = rec.ParentSession - } - // Same restore rule, but see Config.TaskParentID's doc comment - // for why this is a different field entirely from - // ParentSession above. - if rec.TaskParentID != "" { - s.cfg.TaskParentID = rec.TaskParentID - } - // Same restore rule again — see Config.TaskAgentType/ - // TaskToolNames's own doc comment. - if rec.TaskAgentType != "" { - s.cfg.TaskAgentType = rec.TaskAgentType - } - if rec.TaskToolNames != nil { - s.cfg.TaskToolNames = *rec.TaskToolNames - } - // Same restore rule again — see Config.TaskDepth's own doc - // comment. 0 means "this header genuinely predates the field" - // (a real depth is always >= 1) — but unlike ParentSession/ - // TaskAgentType above, the loading Config's OWN TaskDepth is - // NOT always safe to leave untouched on that branch: it is not - // guaranteed unpopulated the way this restore rule assumes - // elsewhere. SessionManager's crash-recovery sweep - // (recoverCrashedChildrenLocked, session_manager.go) calls - // LoadSession with a Config built from configSnapshot() of the - // PARENT node currently being adopted — which, since - // configSnapshot copies Config by value, carries THAT PARENT's - // own live TaskDepth. A legacy child (this header predates the - // field) loaded under that Config would otherwise silently - // inherit its parent's depth instead of correctly falling back - // to adoptReloadedLocked's own m.maxDepth refusal sentinel. - // Reset to 0 unconditionally whenever s.cfg.TaskParentID is - // non-empty (this IS a task-tool child, restored above either - // from this record or the loading Config) but this specific - // header recorded no depth, so the sentinel fallback always - // applies for a genuinely legacy child regardless of what the - // loading Config happened to carry in for an unrelated reason. - // A genuine root (TaskParentID empty either way) is unaffected - // either branch — TaskDepth is never read for one. - if rec.TaskDepth > 0 { - s.cfg.TaskDepth = rec.TaskDepth - } else if s.cfg.TaskParentID != "" { - s.cfg.TaskDepth = 0 - } - // The effort at create time. Omitted (EffortUnset) on a legacy - // header, which restores as the provider default — unchanged. - s.effort = rec.Effort + s.applySessionHeader(rec) case recMessage: if rec.Message == nil { if isLast { @@ -1670,10 +1669,49 @@ func LoadSession(cfg Config, id string) (*Session, error) { } } return nil + } + + // The tail scan. scanLogRaw, not scanLog, so a record the snapshot + // already covers is never DECODED — skipping the decode is where the + // saving is, since decoding a message record builds its whole part + // tree. The decode below reproduces scanLog's corruption discipline + // verbatim (a corrupt or truncated FINAL line ends the scan silently; + // corruption anywhere else is an error, with the same message text) so + // the two paths cannot drift. + // + // A corrupt record at or before the anchor is not detected on the + // snapshot path. That is the accepted consequence of not reading it: + // the snapshot was DERIVED from those exact records by the process + // that wrote them, so its state already reflects them. + err = scanLogRaw(data, func(raw []byte, line int, isLast bool) error { + if int64(line) <= startAfter { + s.recordsWritten = int64(line) + return nil // covered by the snapshot; the header is already applied + } + var rec record + if err := json.Unmarshal(raw, &rec); err != nil { + if isLast { + return errTruncatedFinalRecord // crash mid-write, ignore + } + return fmt.Errorf("corrupt record at line %d: %v", line, err) + } + s.replayedRecords++ + // Only a line that DECODED counts toward the head: a torn final + // line returns above, and ensureLog's own tail repair removes it + // from the file before this session appends again. + s.recordsWritten = int64(line) + return apply(rec, line, isLast) }) if err != nil { return nil, fmt.Errorf("engine: session %s: %w", id, err) } + if startAfter > 0 { + // The header record this load applied without folding it into the + // tail scan (see snapshotStartAfter) is still a record this load + // decoded, and the bounded-replay guarantee is stated over records + // decoded, not over records folded in one particular place. + s.replayedRecords++ + } s.promptQueue, s.promptQueueNextID, s.enqueueSeq = qf.queue, qf.nextID, qf.seq // A log from an older binary or an external writer can carry an // assistant tool_call whose turn died before a result was recorded. @@ -1745,6 +1783,78 @@ func LoadSession(cfg Config, id string) (*Session, error) { return s, nil } +// applySessionHeader restores the state a session's header record carries +// into s: its creation time and the Config fields the header is the durable +// truth for. Factored out of LoadSession's own recSession fold case because +// snapshot recovery (snapshot.go) applies the header WITHOUT replaying any +// other record — the header is line 1 of every journal, so replaying it +// unconditionally is cheaper than reproducing these restore rules, each of +// which turns on the difference between "this header omitted the field" and +// "the loading Config already has a value", in a second place. +func (s *Session) applySessionHeader(rec record) { + s.createdAt = rec.CreatedAt + // A restored WorkDir wins over the loading Config.WorkDir: the + // header is the durable truth for a resumed session. A legacy + // header (written before this field existed) omits it, so an + // empty value here means "nothing to restore" — the loading + // Config.WorkDir is kept unchanged. + if rec.WorkDir != "" { + s.cfg.WorkDir = rec.WorkDir + } + // Same restore rule as WorkDir above: the header is the durable + // truth for a resumed session, and an empty value here means + // nothing to restore (legacy header, or no lineage recorded), + // never "clear the loading Config's ParentSession". + if rec.ParentSession != "" { + s.cfg.ParentSession = rec.ParentSession + } + // Same restore rule, but see Config.TaskParentID's doc comment + // for why this is a different field entirely from + // ParentSession above. + if rec.TaskParentID != "" { + s.cfg.TaskParentID = rec.TaskParentID + } + // Same restore rule again — see Config.TaskAgentType/ + // TaskToolNames's own doc comment. + if rec.TaskAgentType != "" { + s.cfg.TaskAgentType = rec.TaskAgentType + } + if rec.TaskToolNames != nil { + s.cfg.TaskToolNames = *rec.TaskToolNames + } + // Same restore rule again — see Config.TaskDepth's own doc + // comment. 0 means "this header genuinely predates the field" + // (a real depth is always >= 1) — but unlike ParentSession/ + // TaskAgentType above, the loading Config's OWN TaskDepth is + // NOT always safe to leave untouched on that branch: it is not + // guaranteed unpopulated the way this restore rule assumes + // elsewhere. SessionManager's crash-recovery sweep + // (recoverCrashedChildrenLocked, session_manager.go) calls + // LoadSession with a Config built from configSnapshot() of the + // PARENT node currently being adopted — which, since + // configSnapshot copies Config by value, carries THAT PARENT's + // own live TaskDepth. A legacy child (this header predates the + // field) loaded under that Config would otherwise silently + // inherit its parent's depth instead of correctly falling back + // to adoptReloadedLocked's own m.maxDepth refusal sentinel. + // Reset to 0 unconditionally whenever s.cfg.TaskParentID is + // non-empty (this IS a task-tool child, restored above either + // from this record or the loading Config) but this specific + // header recorded no depth, so the sentinel fallback always + // applies for a genuinely legacy child regardless of what the + // loading Config happened to carry in for an unrelated reason. + // A genuine root (TaskParentID empty either way) is unaffected + // either branch — TaskDepth is never read for one. + if rec.TaskDepth > 0 { + s.cfg.TaskDepth = rec.TaskDepth + } else if s.cfg.TaskParentID != "" { + s.cfg.TaskDepth = 0 + } + // The effort at create time. Omitted (EffortUnset) on a legacy + // header, which restores as the provider default — unchanged. + s.effort = rec.Effort +} + // toolResultHandleInTextPattern matches a canonical trh_N handle token // (digits only, no leading zero, no sign — the exact grammar // parseToolResultHandle enforces) anywhere inside a larger string, with NO diff --git a/engine/taskdelivery.go b/engine/taskdelivery.go index 84722dfd..59c69ae7 100644 --- a/engine/taskdelivery.go +++ b/engine/taskdelivery.go @@ -139,6 +139,10 @@ func (s *Session) enqueueTaskNotification(n taskNotification) { func (s *Session) enqueueTaskNotificationMemoryOnly(n taskNotification) { s.mu.Lock() s.taskNotifications = append(s.taskNotifications, n) + // Memory ahead of the journal until persistQueuedTaskNotification runs + // — see snapshotSafeLocked (snapshot.go) for why a snapshot must not be + // captured in this window. + s.durableDebt++ s.mu.Unlock() } @@ -172,6 +176,9 @@ func (s *Session) enqueueTaskNotificationMemoryOnlyDeduped(n taskNotification) b } } s.taskNotifications = append(s.taskNotifications, n) + // Same debt as enqueueTaskNotificationMemoryOnly, and only on the + // branch whose caller goes on to persist. + s.durableDebt++ return true } @@ -188,6 +195,9 @@ func (s *Session) enqueueTaskNotificationMemoryOnlyDeduped(n taskNotification) b func (s *Session) persistQueuedTaskNotification(n taskNotification) { s.mu.Lock() s.persistTaskNotifyLocked(recTaskNotifyQueued, n) + // The journal has caught up with the memory-only enqueue this pairs + // with — see snapshotSafeLocked (snapshot.go). + s.settleDurableDebtLocked() s.mu.Unlock() } From 69a6b8b4baaaa610b0f0f37372729b07eac9e958 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Thu, 27 Aug 2026 20:25:13 -0400 Subject: [PATCH 21/95] server: stop cold-replaying sessions on every list; engine: refuse a model with no known context window (#210) * server: stop re-replaying a non-resident session on every cold read GET /session renders a non-resident session from its metadata index, but when that index cannot answer -- a legacy header with no workdir, or a crash that tore away the initial model record, so SessionIndex.Complete is false -- it falls back to the authoritative engine.LoadSession. That fallback threw the loaded session away, so the next read replayed the same journal from byte 0 again. GET /session is polled: a control-plane activity probe hits it every ~20s for the life of the process. A box's finished sub-agent sessions were therefore cold-replayed on that cadence forever, which is the repeating `reason=start` context-window log line an operator sees -- logContextWindowArmed fires once per LoadSession. Keep the loaded session instead of discarding it. retainLoaded makes it resident under exactly the MaxResident budget every other loader lives under, using claimForPrompt's and handleSetModel's shape rather than a third variant: the load already ran outside s.mu, so it re-acquires the lock, defers to any resident that appeared meanwhile, evicts, and releases evicted handles after unlocking. The retained session is idle, so it is immediately eviction-eligible; a listing can displace a warm idle session, never a running one. A read that mutates residency deserves its justification stated. The alternative is not "no mutation": it is an unbounded full journal replay on every poll of an endpoint built to be polled, which is worse for that session and for everything else sharing the process's disk. Rendering is unchanged. The index path still answers every session whose index is complete, without loading anything, and the fallback still renders through the same authoritative load and the same omit-what-cannot-render rule. * engine: refuse an unknown model instead of silently disabling compaction modelmeta.ContextWindow answers "how big is this model's context window". When it did not recognize a ref, resolveContextWindow folded that into the SAME answer a deliberate opt-out produces: source "disabled", compaction_armed=false, and a session that started anyway and ran with no context management at all -- until it died with "context exhausted" instead of compacting. An unrecognized model is not a state to degrade into; it is a configuration an operator has to fix, and the silence is what stops them finding out until the session is already dead. resolveContextWindow now REPORTS the miss (an error wrapping ErrUnknownContextWindow, whose text always names the offending ref) rather than swallowing it, and does not decide what to do about it. Config.RequireContextWindow does, through the single policy point requiredContextWindowErr -- so the definition of a miss lives in one place, the policy lives with the session that has to honor it, and one ERROR log line fires per miss however it was reached. Only a registry MISS refuses. Four ways to have no window stay legitimate and silent: an explicit positive ContextWindowTokens (naming the window IS the missing information, so it satisfies the requirement for any model), an explicit NEGATIVE one (a stated opt-out, now told apart from "disabled" precisely because "disabled" could mean "unrecognized"), a zero model ref (nothing to look up), and a model the registry knows whose window is below the auto-arm floor (a known model, not a gap). The refusal is recorded at the earliest point of use and surfaced wherever a session starts using a model: newSession, SetModel, and LoadSession's post-replay re-derive set it; ContextWindowErr lets POST /session refuse before the session is durable or resident; CheckModel is ModelSupported's sibling gate, called at the same three SetModel routes BEFORE the swap so a rejected ref never reaches the durable model record; and every Prompt returns it before touching history, the provider, or the instructions read. A resume is deliberately not fatal -- a session that cannot load cannot be listed, read, or exported either, and an operator would lose the transcript along with the ability to fix the config. Config.RequireContextWindow false, the engine zero value, keeps the pre-fix behavior, so a bare embedder-built engine.Config and every test in the package are unaffected. The config/CLI layer supplies the product default of true (context_window_required), the same unset-versus-explicit split prompt_retries uses; an operator who wants a model the registry cannot know either names its window with context_window_tokens (the better answer -- compaction needs that value anyway) or sets context_window_required=false. * test(monitor/e2e): drop the one guessed deadline in the real e2e TestRealEndToEnd failed in CI on "the idle-send's own pending indicator appears (its turn is gated, so it cannot have streamed yet)". The turn was gated and the indicator was correct; the wait simply gave up after 3s. That wait was the only one in real_e2e.mjs with a budget of its own -- the other 32 all use 20000. The gate is what makes the indicator's PRESENCE reliable, as the comment right above it says, so the timeout is purely a failure bound: a shorter one catches nothing a longer one misses, because the indicator stays up until the scenario releases the gate a few lines later. What it did catch was a slow machine. On a loaded 2-core runner the SSE round trip plus the jsdom re-render can exceed 3s, which is the short arbitrary failsafe AGENTS.md's "no guessed deadlines" rule exists to keep out. Align it with the rest of the file. Not a product change, and unrelated to the two fixes on this branch -- this test flakes on main too, where it failed once in eleven local runs with none of those changes present. --------- Co-authored-by: andybons --- AGENTS.md | 45 ++++++ cmd/harness/main.go | 2 + config/config.go | 30 ++++ config/config_test.go | 59 +++++++ engine/context_window.go | 90 ++++++++++- engine/context_window_required_test.go | 207 +++++++++++++++++++++++++ engine/context_window_test.go | 12 +- engine/engine.go | 91 ++++++++++- engine/model_tool.go | 8 + engine/store.go | 9 +- server/context_window_required_test.go | 116 ++++++++++++++ server/handlers.go | 67 ++++++++ server/list_cold_load_test.go | 96 ++++++++++++ tools/monitor/e2e/real_e2e.mjs | 12 +- 14 files changed, 829 insertions(+), 15 deletions(-) create mode 100644 engine/context_window_required_test.go create mode 100644 server/context_window_required_test.go create mode 100644 server/list_cold_load_test.go diff --git a/AGENTS.md b/AGENTS.md index 85614319..df64b28d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1528,6 +1528,51 @@ session (404), or an empty model (400) — the same validation as the tool — t calls `SetModel`. Aliases are not resolved at this endpoint; resolve them client-side, as the CLI does. +### An unknown model's context window is a refusal, not a shrug + +`modelmeta.ContextWindow` answers "how big is this model's context window". +When it does not recognize a ref, `resolveContextWindow` +(`engine/context_window.go`) used to fold that into the SAME answer a +deliberate opt-out produces: source `disabled`, `compaction_armed=false`, and +a session that started anyway and ran with **no context management at all** — +until it died with "context exhausted" instead of compacting. An unrecognized +model is not a state to degrade into; it is a configuration an operator has to +fix. + +`resolveContextWindow` now REPORTS the miss (an error wrapping +`ErrUnknownContextWindow`, whose text always names the offending ref) instead +of swallowing it, and does not decide what to do about it. +`Config.RequireContextWindow` does, through the single policy point +`requiredContextWindowErr` — so the definition of a miss lives in one place +and the policy lives with the session that must honor it, and the ERROR log +line fires once per miss however it was reached. + +Only a REGISTRY MISS refuses. Four ways to have no window stay legitimate and +silent: an explicit positive `ContextWindowTokens` (naming the window IS the +missing information, so it satisfies the requirement for any model), an +explicit NEGATIVE one (`contextWindowSourceOptOut` — a stated choice, told +apart from `disabled` precisely because `disabled` can mean "unrecognized"), a +ZERO model ref (nothing to look up; the refusal belongs to whatever later +names a model), and a model the registry KNOWS whose window is below +`minAutoContextWindowTokens` (a known model, not a gap). + +The refusal is recorded at the earliest point of use and surfaced everywhere a +model starts being used: `newSession`, `SetModel`, and `LoadSession`'s +post-replay re-derive set `Session.contextWindowErr`; `ContextWindowErr()` lets +a create route refuse before the session is durable or resident; `CheckModel` +is `ModelSupported`'s sibling gate, called at the same three `SetModel` routes +BEFORE the swap so a rejected ref never reaches the durable `recModel` record; +and every `Prompt` returns it before touching history, the provider, or the +instructions read. A RESUME is deliberately not fatal: a session that cannot +load cannot be listed, read, or exported either, and an operator would lose the +transcript along with the ability to fix the config. + +`Config.RequireContextWindow` false — the engine zero value — keeps the +pre-fix behavior, so a bare embedder-built `engine.Config` and every test in +the package are unaffected; the config/CLI layer supplies the product default +of TRUE (`context_window_required`), the same unset-versus-explicit split +`prompt_retries` uses. + ### Reasoning effort `message.Effort` is the unified, provider-agnostic reasoning-effort level: diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 3d1737d3..fd7b5071 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -750,6 +750,7 @@ func runCmd(args []string) error { MCPToolLoadingByServer: mcpToolLoadingByServer(cfg.MCPServers), Processes: processRegistry(procMgr), ContextWindowTokens: cfg.ContextWindowTokens, + RequireContextWindow: cfg.ContextWindowRequiredValue(), StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), @@ -1560,6 +1561,7 @@ func serveCmd(args []string) error { MCPToolLoadingByServer: mcpToolLoadingByServer(cfg.MCPServers), Processes: processRegistry(procMgr), ContextWindowTokens: cfg.ContextWindowTokens, + RequireContextWindow: cfg.ContextWindowRequiredValue(), StreamIdleTimeout: time.Duration(cfg.StreamIdleTimeoutS) * time.Second, PromptRetries: cfg.PromptRetriesValue(), MaxTokensContinuations: cfg.MaxTokensContinuationsValue(), diff --git a/config/config.go b/config/config.go index 9a6018c4..f5799f91 100644 --- a/config/config.go +++ b/config/config.go @@ -121,6 +121,21 @@ type Config struct { // derivation was added to close: ContextWindowTokens was opt-in and set // nowhere on the boxes platform, so compaction never armed on any box. ContextWindowTokens int `json:"context_window_tokens,omitempty"` + // ContextWindowRequired sets engine.Config.RequireContextWindow: a model + // the context-window registry does not recognize is a hard refusal at + // session creation, model set, and every Prompt, instead of a session + // that silently runs with no context management and later dies with + // "context exhausted". A nil value (the field omitted) leaves the + // product default of TRUE in place; an explicit false allows the old + // silent-degradation behavior for an operator running a model the + // registry cannot know (a local or gateway-fronted one) who does not + // want to name its window. Naming it with `context_window_tokens` + // satisfies the requirement for any model and is the better answer, + // since that value is what automatic compaction needs anyway. A *bool + // distinguishes "unset" (true) from "false" (off) across the + // project-config merge, like PromptRetries' *int. Resolve it with + // ContextWindowRequiredValue. + ContextWindowRequired *bool `json:"context_window_required,omitempty"` // PromptRetries sets engine.Config.PromptRetries: how many ADDITIONAL // attempts the base interactive Prompt loop makes when a model call fails // with a transient, retryable provider error (an HTTP 5xx/429/529 or a @@ -1011,6 +1026,9 @@ func merge(base, over *Config) *Config { if over.ContextWindowTokens != 0 { out.ContextWindowTokens = over.ContextWindowTokens } + if over.ContextWindowRequired != nil { + out.ContextWindowRequired = over.ContextWindowRequired + } if over.PromptRetries != nil { out.PromptRetries = over.PromptRetries } @@ -1261,6 +1279,18 @@ func (c *Config) PromptRetriesValue() int { return *c.PromptRetries } +// ContextWindowRequiredValue reports whether an unrecognized model is a +// hard refusal. The default is TRUE: only an explicit +// `context_window_required: false` allows a model with no known context +// window to run with compaction silently disabled. A nil receiver (no +// config) uses the default too. +func (c *Config) ContextWindowRequiredValue() bool { + if c == nil || c.ContextWindowRequired == nil { + return true + } + return *c.ContextWindowRequired +} + // defaultMaxTokensContinuations is the product default for how many // consecutive max_tokens stops the base interactive Prompt loop // auto-continues when `max_tokens_continuations` is unset (see diff --git a/config/config_test.go b/config/config_test.go index e8bb48d6..19cfef55 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -1787,3 +1787,62 @@ func TestSnapshotEveryRecords(t *testing.T) { } }) } + +// TestContextWindowRequired covers the context_window_required config +// field: a *bool on the same unset-versus-explicit split PromptRetries +// uses, so unset means the product default (TRUE — an unrecognized model is +// a hard refusal) and an explicit false restores the old silent +// compaction-disabled behavior. +func TestContextWindowRequired(t *testing.T) { + t.Run("unset requires a known context window", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"model": "anthropic/claude-fable-5"}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.ContextWindowRequired != nil { + t.Errorf("ContextWindowRequired = %v, want nil (unset)", c.ContextWindowRequired) + } + if !c.ContextWindowRequiredValue() { + t.Error("ContextWindowRequiredValue = false, want true (default)") + } + }) + t.Run("explicit false allows an unknown model", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "config.json") + writeFile(t, p, `{"context_window_required": false}`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + if c.ContextWindowRequired == nil || *c.ContextWindowRequired { + t.Fatalf("ContextWindowRequired = %v, want explicit false", c.ContextWindowRequired) + } + if c.ContextWindowRequiredValue() { + t.Error("ContextWindowRequiredValue = true, want false (explicitly off)") + } + }) + t.Run("nil receiver uses default", func(t *testing.T) { + var c *Config + if !c.ContextWindowRequiredValue() { + t.Error("nil ContextWindowRequiredValue = false, want true") + } + }) + t.Run("project overrides user", func(t *testing.T) { + no := false + yes := true + base := &Config{ContextWindowRequired: &yes} + merged := merge(base, &Config{ContextWindowRequired: &no}) + if merged.ContextWindowRequired == nil || *merged.ContextWindowRequired { + t.Errorf("merged = %v, want project override false", merged.ContextWindowRequired) + } + }) + t.Run("unset project inherits user", func(t *testing.T) { + no := false + base := &Config{ContextWindowRequired: &no} + merged := merge(base, &Config{}) + if merged.ContextWindowRequired == nil || *merged.ContextWindowRequired { + t.Errorf("merged = %v, want inherited false", merged.ContextWindowRequired) + } + }) +} diff --git a/engine/context_window.go b/engine/context_window.go index c62b0492..fc0462ba 100644 --- a/engine/context_window.go +++ b/engine/context_window.go @@ -9,6 +9,8 @@ package engine import ( + "errors" + "fmt" "log/slog" "github.com/majorcontext/harness/message" @@ -31,8 +33,40 @@ const ( // no usable model metadata was found — automatic compaction is disarmed, // identical to today's behavior before this file existed. contextWindowSourceDisabled = "disabled" + // contextWindowSourceOptOut: the operator DELIBERATELY disabled + // automatic compaction with a negative Config.ContextWindowTokens. It + // is told apart from contextWindowSourceDisabled on purpose: "disabled" + // can mean the registry did not recognize the model, which + // Config.RequireContextWindow turns into a hard refusal, while this is + // a stated choice that never refuses. See resolveContextWindow. + contextWindowSourceOptOut = "disabled-by-config" ) +// ErrUnknownContextWindow marks a model ref the context-window registry +// (package modelmeta) does not recognize, so no context window can be +// derived for it. +// +// It exists because the old answer to that lookup was SILENCE: source +// "disabled", compaction_armed=false, and a session that ran anyway with +// no context management at all — until it died with "context exhausted" +// instead of compacting. An unrecognized model is not a state to degrade +// into, it is a configuration the operator has to fix, and +// Config.RequireContextWindow turns it into a loud refusal at the earliest +// point of use. Errors returned for it wrap this sentinel, and their text +// always names the offending ref. +var ErrUnknownContextWindow = errors.New("engine: no context window configured for model") + +// unknownContextWindowError builds the operator-facing refusal for ref. The +// text names the ref, says plainly what is missing, and names both ways out +// — an explicit window, or turning the requirement off — because an error +// that only reports a problem makes an operator go read source to act on +// it. +func unknownContextWindowError(ref message.ModelRef) error { + return fmt.Errorf("%w: unknown model %q: refusing to run without a context window "+ + "(set context_window_tokens for this model, or context_window_required=false to allow it)", + ErrUnknownContextWindow, ref.String()) +} + // minAutoContextWindowTokens is the sanity floor a model-derived context // window must clear to arm automatic compaction. A metadata value below // this is far more likely a bug — a zeroed, truncated, or misparsed field @@ -55,13 +89,33 @@ var modelContextWindowLookup = modelmeta.ContextWindow // never the already-resolved value from a previous call); model is the ref // to derive from when explicitTokens is 0. Returns the effective window (0 // when disabled) and which source produced it. -func resolveContextWindow(explicitTokens int, model message.ModelRef) (tokens int, source string) { +// A registry MISS is reported through miss (wrapping +// ErrUnknownContextWindow) INSTEAD of being folded into a silent +// "disabled" answer. resolveContextWindow does not decide what to do about +// it: the caller does, from Config.RequireContextWindow, so the definition +// of a miss lives in exactly one place and the policy lives with the +// session that has to honor it. miss is nil for every legitimate way to +// end up without a window — an explicit operator window, an explicit +// opt-out, no model at all, or a model the registry knows whose window is +// simply below the auto-arm floor. +func resolveContextWindow(explicitTokens int, model message.ModelRef) (tokens int, source string, miss error) { if explicitTokens > 0 { - return explicitTokens, contextWindowSourceConfig + return explicitTokens, contextWindowSourceConfig, nil + } + if explicitTokens < 0 { + // A stated choice, not a gap: the operator asked for no automatic + // compaction. Never a miss, whatever the model is. + return 0, contextWindowSourceOptOut, nil + } + if model.IsZero() { + // No model to look up. An embedder that has not chosen one yet is + // not running anything against it either, so there is nothing to + // refuse — the refusal belongs to whatever later names a model. + return 0, contextWindowSourceDisabled, nil } got, ok := modelContextWindowLookup(model) if !ok { - return 0, contextWindowSourceDisabled + return 0, contextWindowSourceDisabled, unknownContextWindowError(model) } if got < minAutoContextWindowTokens { // INFO, not WARN: the table legitimately keeps some genuinely @@ -72,9 +126,35 @@ func resolveContextWindow(explicitTokens int, model message.ModelRef) (tokens in // starts on or switches to such a model. slog.Info("engine: model-derived context window below auto-compaction floor; compaction disabled", "model", model.String(), "tokens", got, "floor", minAutoContextWindowTokens) - return 0, contextWindowSourceDisabled + return 0, contextWindowSourceDisabled, nil } - return got, contextWindowSourceModelDerived + return got, contextWindowSourceModelDerived, nil +} + +// requiredContextWindowErr turns a resolveContextWindow miss into this +// session's refusal, or into nothing at all. +// +// It is the ONE place Config.RequireContextWindow is consulted, so the +// policy cannot drift between session start, a model switch, and a resume. +// The ERROR log line fires here rather than at each call site, for the same +// reason: an operator gets the same message with the same fields however +// the miss was reached, and gets it even if the caller ignores the returned +// error entirely. reason names which of those the caller was, mirroring +// logContextWindowArmed's own reason field. +// +// A miss with the requirement OFF is not silent either — it is the state +// logContextWindowArmed already reports as source=disabled, +// compaction_armed=false — so nothing is logged here for it. +func requiredContextWindowErr(cfg Config, ref message.ModelRef, miss error, reason string) error { + if miss == nil || !cfg.RequireContextWindow { + return nil + } + slog.Error("engine: refusing to run: model has no known context window", + "model", ref.String(), + "reason", reason, + "error", miss.Error(), + ) + return miss } // logContextWindowArmed emits the one operator-facing INFO line stating diff --git a/engine/context_window_required_test.go b/engine/context_window_required_test.go new file mode 100644 index 00000000..73768c45 --- /dev/null +++ b/engine/context_window_required_test.go @@ -0,0 +1,207 @@ +package engine + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// A registry MISS and a registry HIT, both through the REAL modelmeta +// table: these tests are about what the shipped registry does or does not +// know, so stubbing the lookup would test nothing. +var ( + unknownRef = message.ModelRef{Provider: "openrouter", Model: "anthropic/claude-opus-4.1"} + knownRef = message.ModelRef{Provider: "openai", Model: "gpt-5.6-sol"} +) + +// TestResolveContextWindowReportsRegistryMiss is the core of the fix: an +// unrecognized model must be REPORTED as a miss, not folded silently into +// the same "disabled" answer a deliberate opt-out produces. Silently +// running an unknown model with no context management is how a session +// dies with "context exhausted" instead of compacting. +func TestResolveContextWindowReportsRegistryMiss(t *testing.T) { + if _, _, err := resolveContextWindow(0, knownRef); err != nil { + t.Errorf("known model %s reported a miss: %v", knownRef, err) + } + tokens, source, err := resolveContextWindow(0, unknownRef) + if err == nil { + t.Fatalf("unknown model %s resolved silently to %d/%q, want a reported miss", unknownRef, tokens, source) + } + if !errors.Is(err, ErrUnknownContextWindow) { + t.Errorf("error = %v, want it to wrap ErrUnknownContextWindow", err) + } + if !strings.Contains(err.Error(), unknownRef.String()) { + t.Errorf("error = %q, want it to name the offending model ref %q", err, unknownRef.String()) + } +} + +// TestResolveContextWindowLegitimateDisabledCases pins what must stay +// allowed. Only a registry miss on a real model ref is a failure. +func TestResolveContextWindowLegitimateDisabledCases(t *testing.T) { + t.Run("explicit operator window wins over an unknown model", func(t *testing.T) { + tokens, source, err := resolveContextWindow(400_000, unknownRef) + if err != nil { + t.Errorf("explicit window still reported a miss: %v", err) + } + if tokens != 400_000 || source != contextWindowSourceConfig { + t.Errorf("= %d, %q; want 400000, %q", tokens, source, contextWindowSourceConfig) + } + }) + t.Run("explicit negative is an opt-out", func(t *testing.T) { + tokens, source, err := resolveContextWindow(-1, unknownRef) + if err != nil { + t.Errorf("explicit opt-out reported a miss: %v", err) + } + if tokens != 0 || source != contextWindowSourceOptOut { + t.Errorf("= %d, %q; want 0, %q", tokens, source, contextWindowSourceOptOut) + } + }) + t.Run("no model at all is nothing to look up", func(t *testing.T) { + if _, _, err := resolveContextWindow(0, message.ModelRef{}); err != nil { + t.Errorf("zero model ref reported a miss: %v", err) + } + }) + t.Run("a known model below the auto floor stays allowed", func(t *testing.T) { + stubContextWindowLookup(t, testContextWindowTable()) + tokens, source, err := resolveContextWindow(0, modelBogusTiny) + if err != nil { + t.Errorf("a known-but-small model reported a miss: %v", err) + } + if tokens != 0 || source != contextWindowSourceDisabled { + t.Errorf("= %d, %q; want 0, %q", tokens, source, contextWindowSourceDisabled) + } + }) +} + +func requireCfg(prov *scriptedProvider, ref message.ModelRef) Config { + return Config{ + Providers: provider.Registry{prov.name: prov}, + Model: ref, + RequireContextWindow: true, + } +} + +// TestPromptRefusesUnknownModelWhenRequired is the loud failure itself: a +// session whose model has no known context window must not run. Before +// this, it started happily with source=disabled and no context management +// at all. +func TestPromptRefusesUnknownModelWhenRequired(t *testing.T) { + prov := &scriptedProvider{name: "openrouter", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "should never run"}), + }} + s := NewSession(requireCfg(prov, unknownRef)) + if err := s.ContextWindowErr(); err == nil { + t.Fatal("NewSession reported no context-window error for an unknown model") + } + + _, err := s.Prompt(context.Background(), "go") + if err == nil { + t.Fatal("Prompt succeeded for a model with no known context window, want a loud failure") + } + if !errors.Is(err, ErrUnknownContextWindow) { + t.Errorf("error = %v, want it to wrap ErrUnknownContextWindow", err) + } + if !strings.Contains(err.Error(), unknownRef.String()) { + t.Errorf("error = %q, want it to name %q", err, unknownRef.String()) + } + // It must fail BEFORE touching history or the provider. + if n := len(s.History()); n != 0 { + t.Errorf("history = %d messages, want 0: the refusal must precede any append", n) + } + if prov.call != 0 { + t.Errorf("provider called %d times, want 0", prov.call) + } +} + +// TestPromptAcceptsKnownModelWhenRequired is the other half: the check must +// not disturb a model the registry knows. +func TestPromptAcceptsKnownModelWhenRequired(t *testing.T) { + prov := &scriptedProvider{name: "openai", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + s := NewSession(requireCfg(prov, knownRef)) + if err := s.ContextWindowErr(); err != nil { + t.Fatalf("known model reported a context-window error: %v", err) + } + if s.contextWindowSource != contextWindowSourceModelDerived { + t.Errorf("source = %q, want %q", s.contextWindowSource, contextWindowSourceModelDerived) + } + if got := s.cfg.ContextWindowTokens; got != 1_050_000 { + t.Errorf("context window = %d, want 1050000", got) + } + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt: %v", err) + } +} + +// TestUnknownModelRunsWhenNotRequired pins the engine's zero value: an +// embedder building a bare engine.Config (and every test in this package) +// keeps the pre-fix behavior. The config/CLI layer is what turns the +// requirement on. +func TestUnknownModelRunsWhenNotRequired(t *testing.T) { + prov := &scriptedProvider{name: "openrouter", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg := requireCfg(prov, unknownRef) + cfg.RequireContextWindow = false + s := NewSession(cfg) + if err := s.ContextWindowErr(); err != nil { + t.Fatalf("ContextWindowErr = %v, want nil when the requirement is off", err) + } + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt: %v", err) + } +} + +// TestExplicitWindowSatisfiesTheRequirement pins the operator escape hatch: +// naming the window explicitly is exactly the missing information, so it +// must satisfy the requirement for any model. +func TestExplicitWindowSatisfiesTheRequirement(t *testing.T) { + prov := &scriptedProvider{name: "openrouter", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg := requireCfg(prov, unknownRef) + cfg.ContextWindowTokens = 200_000 + s := NewSession(cfg) + if err := s.ContextWindowErr(); err != nil { + t.Fatalf("ContextWindowErr = %v, want nil with an explicit window", err) + } + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatalf("Prompt: %v", err) + } +} + +// TestSetModelToUnknownModelIsRefused covers the model-set route: a switch +// is the other point where a session starts calling a model, and +// CheckModel is the pre-set gate every SetModel route consults (the same +// shape as ModelSupported). +func TestSetModelToUnknownModelIsRefused(t *testing.T) { + prov := &scriptedProvider{name: "openai", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg := requireCfg(prov, knownRef) + cfg.Providers["openrouter"] = &scriptedProvider{name: "openrouter"} + s := NewSession(cfg) + + if err := s.CheckModel(knownRef); err != nil { + t.Errorf("CheckModel(known) = %v, want nil", err) + } + err := s.CheckModel(unknownRef) + if err == nil { + t.Fatalf("CheckModel(%s) = nil, want a loud refusal", unknownRef) + } + if !errors.Is(err, ErrUnknownContextWindow) || !strings.Contains(err.Error(), unknownRef.String()) { + t.Errorf("error = %v, want ErrUnknownContextWindow naming %q", err, unknownRef.String()) + } + + // A route that sets it anyway must not leave the session silently + // running with no context management: the next Prompt fails loudly. + s.SetModel(unknownRef) + if _, err := s.Prompt(context.Background(), "go"); !errors.Is(err, ErrUnknownContextWindow) { + t.Errorf("Prompt after switching to an unknown model = %v, want ErrUnknownContextWindow", err) + } +} diff --git a/engine/context_window_test.go b/engine/context_window_test.go index 04d6510a..9d761b80 100644 --- a/engine/context_window_test.go +++ b/engine/context_window_test.go @@ -46,7 +46,7 @@ func testContextWindowTable() map[message.ModelRef]int { func TestResolveContextWindowExplicitConfigWinsOverModel(t *testing.T) { stubContextWindowLookup(t, testContextWindowTable()) - tokens, source := resolveContextWindow(50_000, modelKnownBig) + tokens, source, _ := resolveContextWindow(50_000, modelKnownBig) if tokens != 50_000 || source != contextWindowSourceConfig { t.Fatalf("resolveContextWindow(50000, big) = %d, %q; want 50000, %q", tokens, source, contextWindowSourceConfig) } @@ -55,7 +55,7 @@ func TestResolveContextWindowExplicitConfigWinsOverModel(t *testing.T) { func TestResolveContextWindowModelDerivedWhenUnset(t *testing.T) { stubContextWindowLookup(t, testContextWindowTable()) - tokens, source := resolveContextWindow(0, modelKnownBig) + tokens, source, _ := resolveContextWindow(0, modelKnownBig) if tokens != 500_000 || source != contextWindowSourceModelDerived { t.Fatalf("resolveContextWindow(0, big) = %d, %q; want 500000, %q", tokens, source, contextWindowSourceModelDerived) } @@ -64,7 +64,7 @@ func TestResolveContextWindowModelDerivedWhenUnset(t *testing.T) { func TestResolveContextWindowUnknownModelDisabled(t *testing.T) { stubContextWindowLookup(t, testContextWindowTable()) - tokens, source := resolveContextWindow(0, modelUnknown) + tokens, source, _ := resolveContextWindow(0, modelUnknown) if tokens != 0 || source != contextWindowSourceDisabled { t.Fatalf("resolveContextWindow(0, unknown) = %d, %q; want 0, %q", tokens, source, contextWindowSourceDisabled) } @@ -79,7 +79,7 @@ func TestResolveContextWindowUnknownModelDisabled(t *testing.T) { func TestResolveContextWindowFloorRejectsBogusValue(t *testing.T) { stubContextWindowLookup(t, testContextWindowTable()) - tokens, source := resolveContextWindow(0, modelBogusTiny) + tokens, source, _ := resolveContextWindow(0, modelBogusTiny) if tokens != 0 || source != contextWindowSourceDisabled { t.Fatalf("resolveContextWindow(0, bogus-tiny) = %d, %q; want 0, %q (floor must reject it)", tokens, source, contextWindowSourceDisabled) } @@ -89,7 +89,7 @@ func TestResolveContextWindowFloorBoundary(t *testing.T) { stubContextWindowLookup(t, map[message.ModelRef]int{ modelKnownSmall: minAutoContextWindowTokens, // exactly at the floor }) - tokens, source := resolveContextWindow(0, modelKnownSmall) + tokens, source, _ := resolveContextWindow(0, modelKnownSmall) if tokens != minAutoContextWindowTokens || source != contextWindowSourceModelDerived { t.Fatalf("resolveContextWindow(0, exactly-floor) = %d, %q; want %d, %q (floor is inclusive)", tokens, source, minAutoContextWindowTokens, contextWindowSourceModelDerived) @@ -98,7 +98,7 @@ func TestResolveContextWindowFloorBoundary(t *testing.T) { stubContextWindowLookup(t, map[message.ModelRef]int{ modelKnownSmall: minAutoContextWindowTokens - 1, }) - tokens, source = resolveContextWindow(0, modelKnownSmall) + tokens, source, _ = resolveContextWindow(0, modelKnownSmall) if tokens != 0 || source != contextWindowSourceDisabled { t.Fatalf("resolveContextWindow(0, one-under-floor) = %d, %q; want 0, %q", tokens, source, contextWindowSourceDisabled) } diff --git a/engine/engine.go b/engine/engine.go index 19be4b72..59eecc94 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -785,6 +785,32 @@ type Config struct { // context-compaction.md and, for the derivation itself, // context_window.go's package doc comment. ContextWindowTokens int + // RequireContextWindow makes an UNRECOGNIZED model a hard refusal + // instead of a silent degradation. + // + // Without it, a model the context-window registry (package modelmeta) + // does not know resolves to source "disabled": the session starts, runs + // with NO context management at all, and dies later with "context + // exhausted" instead of compacting. That silence is the bug. With it, + // the miss is recorded at the earliest point of use — NewSession, + // LoadSession, SetModel — logged at ERROR naming the ref, surfaced by + // ContextWindowErr/CheckModel so a create or model-set route can refuse + // up front, and returned by every Prompt before it touches history or + // the provider. + // + // False — the zero value — keeps the pre-fix behavior, so an embedder + // building a bare engine.Config (and every test in this package) is + // unaffected. The config/CLI layer supplies the product default of true + // (config key `context_window_required`), the same unset-versus-explicit + // split PromptRetries uses. + // + // Only a REGISTRY MISS refuses. An explicit positive + // ContextWindowTokens satisfies it for any model (naming the window IS + // the missing information), an explicit NEGATIVE one is a deliberate + // opt-out, a zero model ref has nothing to look up, and a model the + // registry knows whose window is below the auto-arm floor is a known + // model, not a gap. + RequireContextWindow bool // CompactionThreshold is the fraction of ContextWindowTokens at which // automatic compaction triggers. Zero defaults to 0.8, mirroring // newSession's existing zero-fills-a-default pattern for BashTimeout. @@ -1237,6 +1263,13 @@ type Session struct { contextWindowExplicit bool contextWindowSource string + // contextWindowErr is the refusal a registry MISS produces when + // Config.RequireContextWindow is set — see that field's doc comment. + // Set by newSession, by LoadSession's post-replay re-derive, and by + // SetModel; cleared by a switch back to a model the registry knows. + // Every Prompt returns it before appending anything. Guarded by mu. + contextWindowErr error + // toolConcurrency is the resolved (never-zero, never-negative) cap on // how many of one batch's tool calls run at once — see // Config.ToolConcurrency and resolveToolConcurrency (toolexec.go). Set @@ -1394,7 +1427,9 @@ func newSession(cfg Config) *Session { // comment and resolveContextWindow's precedence. contextWindowExplicit := cfg.ContextWindowTokens > 0 var contextWindowSource string - cfg.ContextWindowTokens, contextWindowSource = resolveContextWindow(cfg.ContextWindowTokens, cfg.Model) + var contextWindowMiss error + cfg.ContextWindowTokens, contextWindowSource, contextWindowMiss = resolveContextWindow(cfg.ContextWindowTokens, cfg.Model) + contextWindowErr := requiredContextWindowErr(cfg, cfg.Model, contextWindowMiss, "session_start") s := &Session{ cfg: cfg, model: cfg.Model, @@ -1404,6 +1439,7 @@ func newSession(cfg Config) *Session { promptQueueNextID: 1, contextWindowExplicit: contextWindowExplicit, contextWindowSource: contextWindowSource, + contextWindowErr: contextWindowErr, toolResultNextID: 1, toolResults: make(map[string]toolResultMeta), toolConcurrency: resolveToolConcurrency(cfg.ToolConcurrency), @@ -1489,7 +1525,13 @@ func (s *Session) SetModel(ref message.ModelRef) { s.model = ref s.persistModel(ref) if !s.contextWindowExplicit { - nextTokens, nextSource := resolveContextWindow(0, ref) + nextTokens, nextSource, miss := resolveContextWindow(0, ref) + // Re-derived, so it REPLACES whatever the previous model left: + // switching to a model the registry knows clears an earlier + // refusal, and switching away to one it does not arms a new one. + // A config-pinned window (the branch this sits in) never depends + // on the registry at all, so it can never miss. + s.contextWindowErr = requiredContextWindowErr(s.cfg, ref, miss, "model_switch") if nextTokens != s.cfg.ContextWindowTokens || nextSource != s.contextWindowSource { s.cfg.ContextWindowTokens, s.contextWindowSource = nextTokens, nextSource s.compactHysteresis = false @@ -1509,6 +1551,40 @@ func (s *Session) ModelSupported(ref message.ModelRef) bool { return err == nil } +// ContextWindowErr reports this session's context-window refusal, or nil. +// It is non-nil only when Config.RequireContextWindow is set AND the +// session's current model is one the registry does not recognize — see that +// field's doc comment. +// +// A create route calls it right after NewSession, and a resume route after +// LoadSession, to refuse up front instead of handing back a session whose +// every Prompt will fail. Prompt returns the same error, so a caller that +// does not check still cannot run the model silently. +func (s *Session) ContextWindowErr() error { + s.mu.Lock() + defer s.mu.Unlock() + return s.contextWindowErr +} + +// CheckModel reports whether this session may switch to ref: nil when it +// may, a refusal wrapping ErrUnknownContextWindow when ref has no known +// context window and Config.RequireContextWindow is set. +// +// It is ModelSupported's sibling and is called at the same three SetModel +// routes, for the same reason: validate BEFORE the swap, so a rejected ref +// never reaches the durable recModel record. A session whose window is +// config-pinned accepts any ref — an explicit window does not depend on the +// registry. +func (s *Session) CheckModel(ref message.ModelRef) error { + s.mu.Lock() + defer s.mu.Unlock() + if s.contextWindowExplicit { + return nil + } + _, _, miss := resolveContextWindow(0, ref) + return requiredContextWindowErr(s.cfg, ref, miss, "model_check") +} + // Model returns the session's current model. func (s *Session) Model() message.ModelRef { s.mu.Lock() @@ -2236,6 +2312,17 @@ func (s *Session) PromptEngineResume(ctx context.Context, text string) (*message // the caller already made. One parameterized entry point for the value // means a future third Origin value is handled in exactly one place. func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string) (*message.Message, error) { + // Refuse a model with no known context window, before anything else + // happens: no history append, no provider call, no instructions read. + // Running one anyway is running with NO context management at all, + // which ends in "context exhausted" rather than a compaction — see + // Config.RequireContextWindow. Same shape as the instructions check + // below: a present-but-unusable configuration fails every Prompt + // identically, loudly, and without recording a user message. + if err := s.ContextWindowErr(); err != nil { + s.emitSessionError(err) + return nil, err + } // Load project instructions once, before mutating history: a // present-but-unusable AGENTS.md fails the prompt without recording a // user message or calling the provider. diff --git a/engine/model_tool.go b/engine/model_tool.go index 0215b907..8f37bc83 100644 --- a/engine/model_tool.go +++ b/engine/model_tool.go @@ -116,6 +116,14 @@ func runModelTool(s *Session, raw json.RawMessage) (message.Parts, error) { if !s.ModelSupported(ref) { return nil, fmt.Errorf("model: provider %q is not configured (%s)", ref.Provider, s.modelChoicesHint()) } + // And that a context window is known for it, for the same + // before-the-swap reason: CheckModel is the sibling gate every + // SetModel route shares (see Config.RequireContextWindow), so a + // model with no known window is refused here instead of silently + // leaving the session with no context management. + if err := s.CheckModel(ref); err != nil { + return nil, fmt.Errorf("model: %w", err) + } s.SetModel(ref) return jsonResult(s.modelToolStatus()) diff --git a/engine/store.go b/engine/store.go index 3fcd9c4e..a56f9597 100644 --- a/engine/store.go +++ b/engine/store.go @@ -1740,7 +1740,14 @@ func LoadSession(cfg Config, id string) (*Session, error) { // no recModel record, or explicit config), so this never double-logs the // sanity-floor warning for the unchanged case. if !s.contextWindowExplicit && s.model != cfg.Model { - s.cfg.ContextWindowTokens, s.contextWindowSource = resolveContextWindow(0, s.model) + var miss error + s.cfg.ContextWindowTokens, s.contextWindowSource, miss = resolveContextWindow(0, s.model) + // A resume must not be FATAL for an unrecognized model: a session + // that cannot load cannot be listed, read, or exported either, and + // the operator would lose the transcript along with the ability to + // fix the config. Record the refusal instead — every Prompt against + // this session returns it, so it still cannot silently run. + s.contextWindowErr = requiredContextWindowErr(s.cfg, s.model, miss, "session_resume") } logContextWindowArmed(s.ID, s.model, s.cfg.ContextWindowTokens, s.contextWindowSource, "start") // Review finding (round 5): advance toolResultNextID past every trh_N diff --git a/server/context_window_required_test.go b/server/context_window_required_test.go new file mode 100644 index 00000000..c6196366 --- /dev/null +++ b/server/context_window_required_test.go @@ -0,0 +1,116 @@ +package server + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// requireWindowHarness builds a server whose sessions run with +// engine.Config.RequireContextWindow — the product default `harness serve` +// sets (config key context_window_required) — over two providers: one whose +// models the context-window registry knows, one whose models it cannot. +func requireWindowHarness(t *testing.T) *harness { + t.Helper() + const token = "secret-run-token" + dir := t.TempDir() + known := &scriptedProvider{name: "openai"} + unknown := &scriptedProvider{name: "openrouter"} + srv := newServer(t, dir, known, 0, func(o *Options) { + o.NewSession = func(m message.ModelRef, workDir, parentSession string) (*engine.Session, error) { + if m.IsZero() { + m = message.ModelRef{Provider: "openai", Model: "gpt-5.6-sol"} + } + return engine.NewSession(engine.Config{ + Providers: provider.Registry{"openai": known, "openrouter": unknown}, + Model: m, + SessionDir: dir, + WorkDir: workDir, + ParentSession: parentSession, + RequireContextWindow: true, + }), nil + } + }) + ts := httptest.NewServer(srv) + t.Cleanup(ts.Close) + return &harness{t: t, dir: dir, token: token, srv: srv, ts: ts} +} + +// TestCreateRefusesModelWithNoKnownContextWindow: a session that can never +// run one prompt must not look created. Before this, POST /session happily +// returned 201 for a model the registry does not know, and that session ran +// with no context management at all until it died of context exhaustion. +func TestCreateRefusesModelWithNoKnownContextWindow(t *testing.T) { + h := requireWindowHarness(t) + + resp, data := h.do("POST", "/session", map[string]any{"model": "openrouter/anthropic/claude-opus-4.1"}) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("POST /session = %d, want 400: %s", resp.StatusCode, data) + } + if !strings.Contains(string(data), "openrouter/anthropic/claude-opus-4.1") { + t.Errorf("error body = %s, want it to name the offending model ref", data) + } + if !strings.Contains(string(data), "context window") { + t.Errorf("error body = %s, want it to say what is missing", data) + } + + // Nothing was created. + resp, data = h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session = %d: %s", resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatal(err) + } + if len(list) != 0 { + t.Errorf("listing = %s, want no session created by the refused request", data) + } + + // A model the registry knows is unaffected. + resp, data = h.do("POST", "/session", map[string]any{"model": "openai/gpt-5.6-sol"}) + if resp.StatusCode != http.StatusCreated { + t.Fatalf("POST /session (known model) = %d, want 201: %s", resp.StatusCode, data) + } +} + +// TestSetModelRefusesModelWithNoKnownContextWindow covers the other point +// where a session starts calling a model. The refusal must precede the +// swap: SetModel persists a durable model record, so a rejected ref must +// never reach it. +func TestSetModelRefusesModelWithNoKnownContextWindow(t *testing.T) { + h := requireWindowHarness(t) + resp, data := h.do("POST", "/session", map[string]any{"model": "openai/gpt-5.6-sol"}) + if resp.StatusCode != http.StatusCreated { + t.Fatalf("POST /session = %d: %s", resp.StatusCode, data) + } + var created struct { + ID string `json:"id"` + } + if err := json.Unmarshal(data, &created); err != nil { + t.Fatal(err) + } + + resp, data = h.do("POST", "/session/"+created.ID+"/model", map[string]any{"model": "openrouter/anthropic/claude-opus-4.1"}) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("POST /session/{id}/model = %d, want 400: %s", resp.StatusCode, data) + } + if !strings.Contains(string(data), "openrouter/anthropic/claude-opus-4.1") { + t.Errorf("error body = %s, want it to name the offending model ref", data) + } + + // The swap did not happen. + resp, data = h.do("GET", "/session/"+created.ID, nil) + if resp.StatusCode != 200 { + t.Fatalf("GET /session/{id} = %d: %s", resp.StatusCode, data) + } + if got := decodeSession(t, data).Model.String(); got != "openai/gpt-5.6-sol" { + t.Errorf("model = %q, want the refused swap to have changed nothing", got) + } +} diff --git a/server/handlers.go b/server/handlers.go index f13d31b2..85a5457a 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -782,6 +782,18 @@ func (s *Server) handleCreate(w http.ResponseWriter, r *http.Request) { return } s.reportCreatePhase(sess.ID, "new_session", time.Since(phaseStart)) + // Refuse a model with no known context window at CREATE time, before + // the session becomes durable or resident: a session that can never + // run one Prompt is worse than a 400, because it looks created. The + // session object is simply dropped — nothing has been journaled or + // registered for it yet. See engine.Config.RequireContextWindow. + if err := sess.ContextWindowErr(); err != nil { + if wt != nil { + s.discardWorktree(wt) + } + writeErr(w, http.StatusBadRequest, err.Error()) + return + } // Report "total" on every return past this point — success or error — // not just the success tail below. Without this, a failure after // new_session (recordWorktreeOwner, Persist) never reports "total", and @@ -1097,6 +1109,16 @@ func (s *Server) coldSessionJSON(id string, ix engine.SessionIndex, usable bool) if err != nil { return sessionJSON{}, false } + // Keep it. Before this, the fallback threw the loaded session + // away, so the NEXT read of the same session replayed the same + // journal from byte 0 again — and GET /session is polled by a + // control-plane activity probe every ~20s, forever. A box's + // finished sub-agent sessions were therefore cold-replayed on that + // cadence for the life of the process, which is the repeating + // `reason=start` context-window line an operator sees + // (logContextWindowArmed fires once per LoadSession). Retaining + // makes it at most one replay per session per residency window. + s.retainLoaded(id, sess) body = s.buildSession(liveSession{id: id}.withLoaded(sess)) } if lv := s.resolveLive(id); lv.session() != nil { @@ -1105,6 +1127,35 @@ func (s *Server) coldSessionJSON(id string, ix engine.SessionIndex, usable bool) return body, true } +// retainLoaded makes a session a cold READ just loaded resident, so the +// next read of it does not replay the journal again. +// +// A read that mutates residency deserves its justification stated. The +// alternative is not "no mutation": it is an unbounded full journal replay +// on every poll of an endpoint built to be polled, which is strictly worse +// for the same session and for every other session sharing the process's +// disk. What is retained is bounded by exactly the same MaxResident budget +// every other loader lives under, and the retained session is idle +// (running/goalLoop both false), so it is immediately eviction-eligible on +// the very next evictResidentLocked sweep — a listing can displace a warm +// idle session, never a running one. +// +// The shape is claimForPrompt's and handleSetModel's, deliberately not a +// third variant: LoadSession has already run OUTSIDE s.mu (it hits disk), +// and this re-acquires the lock and defers to any resident that appeared +// while it was loading, so two *engine.Session instances for one log are +// never both retained. releaseEvicted runs after the unlock, as it must. +func (s *Server) retainLoaded(id string, sess *engine.Session) { + s.mu.Lock() + var evicted []*engine.Session + if s.sessions[id] == nil { + s.sessions[id] = &sessionState{sess: sess, lastUsed: time.Now()} + evicted = s.evictResidentLocked() + } + s.mu.Unlock() + releaseEvicted(evicted) +} + // messagePlaceholder substitutes for a resident message that fails to // marshal (see handleMessages): it carries just enough to identify which // message broke and why, without ever risking a second marshal failure @@ -1660,6 +1711,13 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusBadRequest, fmt.Sprintf("provider %q is not configured", body.Model.Provider)) return } + // Same gate, same place, for the third SetModel route — see + // handleSetModel's own call and engine.Session.CheckModel. + if err := st.sess.CheckModel(body.Model); err != nil { + s.releasePromptClaim(st) + writeErr(w, http.StatusBadRequest, err.Error()) + return + } // SetModel emits EventModelChanged on a real change, which Publish // journals as the durable "model" record (see server/journal.go). No // explicit emitDurable here: that would double-journal one swap. @@ -2895,6 +2953,15 @@ func (s *Server) handleSetModel(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusBadRequest, fmt.Sprintf("provider %q is not configured", body.Model.Provider)) return } + // The context-window gate, checked BEFORE the swap for the same reason + // as the provider gate above: a model with no known context window runs + // with no context management at all, and SetModel would already have + // persisted the durable recModel record by the time the first Prompt + // failed. See engine.Session.CheckModel. + if err := st.sess.CheckModel(body.Model); err != nil { + writeErr(w, http.StatusBadRequest, err.Error()) + return + } st.sess.SetModel(body.Model) writeJSON(w, http.StatusOK, setModelResponseJSON{Model: st.sess.Model()}) } diff --git a/server/list_cold_load_test.go b/server/list_cold_load_test.go new file mode 100644 index 00000000..7444714f --- /dev/null +++ b/server/list_cold_load_test.go @@ -0,0 +1,96 @@ +package server + +import ( + "encoding/json" + "os" + "path/filepath" + "sync" + "testing" + + "github.com/majorcontext/harness/engine" +) + +// TestListDoesNotReplayNonResidentSessionsRepeatedly is the churn guard. +// +// A control-plane activity probe polls GET /session every ~20s forever. The +// listing renders a non-resident session from its metadata index, but when +// that index cannot answer — a journal whose header predates the workdir +// field, so SessionIndex.Complete is false — it falls back to the +// authoritative engine.LoadSession. That fallback used to throw the loaded +// session away, so every poll re-replayed the same journal cold, forever: +// the `reason=start` context-window churn observed on a box whose finished +// sub-agent sessions were re-loaded on a 20s cadence for the life of the +// process. +// +// The assertion is the specific behavior — at most ONE cold load per +// session across N list calls — not a raw total. +func TestListDoesNotReplayNonResidentSessionsRepeatedly(t *testing.T) { + dir := t.TempDir() + // A legacy-shaped journal: a session header with no workdir, so its + // metadata index is usable but INCOMPLETE, which is exactly the case + // the listing answers with a full load. + const legacy = "ses_0123456789abcdef" + journal := `{"type":"session","id":"` + legacy + `","created_at":"2026-01-02T03:04:05Z"} +{"type":"model","model":"test/m1"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"hi"}]}} +{"type":"message","message":{"id":"msg_2","role":"assistant","parts":[{"type":"text","text":"yo"}]}} +` + if err := os.WriteFile(filepath.Join(dir, legacy+".jsonl"), []byte(journal), 0o644); err != nil { + t.Fatal(err) + } + // A second session whose index CAN answer: the listing must render it + // from metadata alone and never load it at all, not even once. + indexed := coldSession(t, dir, nil).ID + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + // Count cold loads per session id, through the same seam every handler + // loads by. + var mu sync.Mutex + loads := map[string]int{} + inner := h.srv.opts.LoadSession + h.srv.opts.LoadSession = func(id string) (*engine.Session, error) { + mu.Lock() + loads[id]++ + mu.Unlock() + return inner(id) + } + + const polls = 4 + for i := range polls { + resp, data := h.do("GET", "/session", nil) + if resp.StatusCode != 200 { + t.Fatalf("poll %d: GET /session = %d: %s", i, resp.StatusCode, data) + } + var list []sessionJSON + if err := json.Unmarshal(data, &list); err != nil { + t.Fatalf("poll %d: decode list: %v (%s)", i, err, data) + } + var found *sessionJSON + for j := range list { + if list[j].ID == legacy { + found = &list[j] + } + } + if found == nil { + t.Fatalf("poll %d: session %s missing from listing %s", i, legacy, data) + } + // The entry must stay renderable on every poll, not just the first. + if found.Model.String() != "test/m1" { + t.Errorf("poll %d: model = %q, want test/m1", i, found.Model.String()) + } + if found.Messages != 2 { + t.Errorf("poll %d: messages = %d, want 2", i, found.Messages) + } + } + + mu.Lock() + defer mu.Unlock() + for id, n := range loads { + if n > 1 { + t.Errorf("GET /session cold-loaded %s %d times across %d polls, want at most 1", id, n, polls) + } + } + if n := loads[indexed]; n != 0 { + t.Errorf("GET /session cold-loaded %s %d times, want 0: its index can answer", indexed, n) + } +} diff --git a/tools/monitor/e2e/real_e2e.mjs b/tools/monitor/e2e/real_e2e.mjs index 9c59873e..5a664bc0 100644 --- a/tools/monitor/e2e/real_e2e.mjs +++ b/tools/monitor/e2e/real_e2e.mjs @@ -565,7 +565,17 @@ async function main() { ); const pendingIndicatorEl = await waitFor( () => doc.querySelector("#transcript .pending-indicator"), - { timeoutMs: 3000, label: "the idle-send's own pending indicator appears (its turn is gated, so it cannot have streamed yet)" } + // 20000 like every other wait in this file, not a shorter budget of its + // own. The gate is what makes the indicator's PRESENCE reliable (see the + // comment above); the timeout is only a failure bound, so a short one + // adds no strictness — it cannot make this assertion catch anything the + // long one misses, because the indicator stays up until this scenario + // releases the gate below. What it did add was a false failure: on a + // loaded 2-core runner the SSE round trip plus the jsdom re-render can + // take longer than 3s, and CI failed here with the turn correctly gated + // and the indicator correctly on its way. That is the guessed deadline + // AGENTS.md's "no guessed deadlines" rule is about. + { timeoutMs: 20000, label: "the idle-send's own pending indicator appears (its turn is gated, so it cannot have streamed yet)" } ); { const posVsPending = orderOperatorEl.compareDocumentPosition(pendingIndicatorEl); From c4387165006d5cdbf7238bf82fc8d29c1073b8af Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Fri, 28 Aug 2026 11:02:02 -0400 Subject: [PATCH 22/95] server: document task_depth in the journal record schema (#211) projectJournalRecord has projected TaskDepth into the session-header records of the GET /session/{id}/journal debug endpoint since PR #157, alongside the sibling lineage fields parent_session, task_parent_id, and task_agent_type. server/openapi.yaml's JournalRecord schema never gained a matching task_depth entry, though, so a client generated from the schema stayed blind to the field: a reader of the journal could see a child's parent and task-parent but had no documented way to see its tree depth without separately walking the parent chain, even though the endpoint already carried the answer on the wire. Add the task_depth entry, following the style task_fail_kind's entry established (f1a9fdb): type integer, on session records only, recorded at spawn time (mirroring Config.TaskDepth's own doc comment), and omitted (0) on a session predating the field rather than a real child's true depth, which is always >= 1. This is a pure documentation addition: it touches only server/openapi.yaml, not the Go types or projectJournalRecord, so task_depth's wire behavior on session records is unchanged. Verified: go build ./..., go vet ./..., and go test ./engine/... -run 'TaskDepth|ProjectsAllRecordTypes|Journal' all pass. The repo has no test that validates openapi.yaml against the handler/type shapes programmatically (no openapi3/kin-openapi usage under any *_test.go), so this stays a manually kept-in-sync fix, same as f1a9fdb before it. --- server/openapi.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/server/openapi.yaml b/server/openapi.yaml index 393dc830..c08214f4 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -1356,6 +1356,13 @@ components: task_agent_type: type: string description: Only on `session` records. + task_depth: + type: integer + description: >- + Only on `session` records. The child's durable task-tree depth, + recorded at spawn time (see Config.TaskDepth). Omitted (0) on a + session predating this field — never a real child's true depth, + which is always >= 1. message_id: type: string description: Only on `message` records. Identity only — never content. From 276923ef387fd5b6b83686062bf8a44679f34df7 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Fri, 28 Aug 2026 13:09:17 -0400 Subject: [PATCH 23/95] docs(agents): split repository guidance by scope (#212) The root AGENTS.md mixed repository-wide rules with detailed subsystem contracts and implementation history, forcing agents to load more than 3,000 lines before ordinary work. Keep universal rules at the root, add concise scoped files for each subsystem, mirror those scopes through CLAUDE.md imports, and move technical contracts into indexed subject documents under docs/. Agents now load the root guidance plus the scope they change. Runtime behavior is unchanged; TestHooksIntegration disables ambient project-instruction discovery so its prompt assertion remains hermetic. Verified with go build ./..., go test -race ./..., go vet ./..., all 290 JavaScript tests, gofmt, Markdown link and instruction-chain audits, git diff --check, and two independent zero-finding review passes. --- .github/workflows/claude-code-review.yml | 10 +- AGENTS.md | 3178 +---------------- README.md | 4 +- cmd/harness/AGENTS.md | 76 + cmd/harness/CLAUDE.md | 1 + cmd/harness/main.go | 8 +- cmd/harness/plugins.go | 7 +- config/AGENTS.md | 49 + config/CLAUDE.md | 1 + config/config.go | 2 +- docs/README.md | 39 + docs/design/fleet-model.md | 19 +- docs/design/journal-snapshotting.md | 2 +- docs/design/mcp-lazy-tools.md | 18 +- docs/design/nested-instruction-loading.md | 34 +- docs/development-interfaces.md | 227 ++ docs/engine-request-cycle.md | 671 ++++ docs/fleet-and-serve.md | 281 ++ docs/goal-loop.md | 1072 +++--- docs/history/goal-loop-resilience.md | 672 ++++ docs/mcp-tool-loading.md | 142 + docs/models-and-providers.md | 444 +++ docs/plans/2026-07-20-goal-eval-resilience.md | 4 +- docs/plans/2026-07-21-goal-worker-park.md | 4 +- docs/plugins-and-protocols.md | 109 + docs/session-storage-and-queue.md | 411 +++ e2e/AGENTS.md | 25 + e2e/CLAUDE.md | 1 + engine/AGENTS.md | 175 + engine/CLAUDE.md | 1 + engine/bash.go | 2 +- engine/compact.go | 14 +- engine/context_window_test.go | 4 +- engine/engine.go | 17 +- engine/engine_test.go | 9 +- engine/filetools.go | 4 +- engine/filetools_test.go | 10 +- engine/goal.go | 13 +- engine/instructions_outline.go | 13 +- engine/mcp_lazy.go | 4 +- engine/mcp_lazy_test.go | 4 +- engine/prompt_retry_test.go | 5 +- engine/searchtools.go | 2 +- engine/searchtools_test.go | 3 +- engine/session_info_test.go | 3 +- engine/session_key_test.go | 4 +- engine/session_manager.go | 3 +- engine/session_manager_test.go | 4 +- engine/stream_watchdog.go | 4 +- engine/toolexec.go | 9 +- engine/toolexec_test.go | 4 +- engine/toolresult.go | 4 +- imageclamp/AGENTS.md | 30 + imageclamp/CLAUDE.md | 1 + mcp/AGENTS.md | 37 + mcp/CLAUDE.md | 1 + message/AGENTS.md | 77 + message/CLAUDE.md | 1 + modelmeta/AGENTS.md | 38 + modelmeta/CLAUDE.md | 1 + plugin/AGENTS.md | 66 + plugin/CLAUDE.md | 1 + plugin/PROTOCOL.md | 36 +- plugin/hooks.go | 6 +- plugin/host.go | 10 +- process/AGENTS.md | 30 + process/CLAUDE.md | 1 + provider/AGENTS.md | 116 + provider/CLAUDE.md | 1 + provider/openai/session_affinity_test.go | 4 +- provider/openai/transcode.go | 3 +- .../openaicompat/session_affinity_test.go | 4 +- provider/openaicompat/transcode.go | 3 +- sdk/AGENTS.md | 32 + sdk/CLAUDE.md | 1 + server/AGENTS.md | 137 + server/CLAUDE.md | 1 + server/goal_worker_park_test.go | 2 +- server/handlers.go | 6 +- server/journal.go | 4 +- server/openapi.yaml | 2 +- server/queue_test.go | 4 +- server/server.go | 5 +- server/session_journal.go | 4 +- server/wait.go | 3 +- server/wait_test.go | 3 +- skill/AGENTS.md | 42 + skill/CLAUDE.md | 1 + tools/AGENTS.md | 97 + tools/CLAUDE.md | 1 + tools/hub/hub.go | 8 +- tools/hub/hub_test.go | 2 +- tools/hub/index.html | 16 +- tools/hub/spawn.go | 6 +- tools/hub/spawn_test.go | 4 +- 95 files changed, 4816 insertions(+), 3858 deletions(-) create mode 100644 cmd/harness/AGENTS.md create mode 100644 cmd/harness/CLAUDE.md create mode 100644 config/AGENTS.md create mode 100644 config/CLAUDE.md create mode 100644 docs/README.md create mode 100644 docs/development-interfaces.md create mode 100644 docs/engine-request-cycle.md create mode 100644 docs/fleet-and-serve.md create mode 100644 docs/history/goal-loop-resilience.md create mode 100644 docs/mcp-tool-loading.md create mode 100644 docs/models-and-providers.md create mode 100644 docs/plugins-and-protocols.md create mode 100644 docs/session-storage-and-queue.md create mode 100644 e2e/AGENTS.md create mode 100644 e2e/CLAUDE.md create mode 100644 engine/AGENTS.md create mode 100644 engine/CLAUDE.md create mode 100644 imageclamp/AGENTS.md create mode 100644 imageclamp/CLAUDE.md create mode 100644 mcp/AGENTS.md create mode 100644 mcp/CLAUDE.md create mode 100644 message/AGENTS.md create mode 100644 message/CLAUDE.md create mode 100644 modelmeta/AGENTS.md create mode 100644 modelmeta/CLAUDE.md create mode 100644 plugin/AGENTS.md create mode 100644 plugin/CLAUDE.md create mode 100644 process/AGENTS.md create mode 100644 process/CLAUDE.md create mode 100644 provider/AGENTS.md create mode 100644 provider/CLAUDE.md create mode 100644 sdk/AGENTS.md create mode 100644 sdk/CLAUDE.md create mode 100644 server/AGENTS.md create mode 100644 server/CLAUDE.md create mode 100644 skill/AGENTS.md create mode 100644 skill/CLAUDE.md create mode 100644 tools/AGENTS.md create mode 100644 tools/CLAUDE.md diff --git a/.github/workflows/claude-code-review.yml b/.github/workflows/claude-code-review.yml index ad077879..7ce432b4 100644 --- a/.github/workflows/claude-code-review.yml +++ b/.github/workflows/claude-code-review.yml @@ -39,16 +39,18 @@ jobs: Use the /code-review skill to review this pull request. This is a Go agent harness (engine, provider transcoders, plugin - protocol). Read AGENTS.md first and review with these priorities: + protocol). Read the root AGENTS.md first. Then read every scoped + AGENTS.md that governs a changed path. Review with these priorities: - Race conditions, deadlocks, missing synchronization — the engine, plugin host, and streams are heavily concurrent - Protocol correctness (plugin JSON-RPC framing, hook chaining semantics, manifest/lazy-spawn invariants) - Transcoding correctness (canonical <-> provider wire formats, ProviderData replay/drop rules, tool-call ID stability) - - Startup-speed regressions: any init-time network, disk, or - subprocess work is a bug per AGENTS.md - - Testing rules from AGENTS.md: TDD, no raw sleeps, synctest for + - Startup-speed regressions: init-time network or subprocess work, + and disk reads beyond the permitted user/project config path, + are bugs per the root and cmd/harness scoped AGENTS.md files + - Testing rules from the root AGENTS.md: TDD, no raw sleeps, synctest for timer logic - Bugs, logic errors, nil/zero-value pitfalls, unchecked errors diff --git a/AGENTS.md b/AGENTS.md index df64b28d..159bb526 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,2709 +1,104 @@ # AGENTS.md -Instructions for AI coding agents working in this repository. - -## Project Overview - -Harness is a Go agent harness (in the spirit of pi and opencode) built around four priorities, in order: - -1. **Speed** — especially startup speed. `harness --version` under ~5ms, TUI first frame under ~30ms. These are CI-enforced budgets, not aspirations. -2. **Extensibility** — a language-agnostic plugin protocol with a first-class Go SDK. -3. **Composability** — headless engine, event streams on stdout, client/server split, MCP in both directions. -4. **Dynamic model choice** — swap providers/models mid-session or per-subagent with zero migration cost. +Repository-wide instructions for AI coding agents. + +## Instruction scope and reading order + +Read this file before you change the repository. Then read the scoped file for +each subtree that you will change. A scoped file adds local rules. If a local +rule conflicts with a root rule, the local rule wins for that subtree. + +Harness currently injects only the closest `AGENTS.md` to its working +directory. It does not merge ancestor files. Each scoped file therefore tells +a Harness agent to read this root file. A root-started agent must use this +table to load scoped instructions before it edits a subsystem. + +| Path | Scoped instructions | +|---|---| +| `cmd/harness/` | `cmd/harness/AGENTS.md` | +| `config/` | `config/AGENTS.md` | +| `engine/` | `engine/AGENTS.md` | +| `imageclamp/` | `imageclamp/AGENTS.md` | +| `mcp/` | `mcp/AGENTS.md` | +| `message/` | `message/AGENTS.md` | +| `modelmeta/` | `modelmeta/AGENTS.md` | +| `provider/` | `provider/AGENTS.md` | +| `server/` | `server/AGENTS.md` | +| `skill/` | `skill/AGENTS.md` | +| `sdk/` | `sdk/AGENTS.md` | +| `plugin/` | `plugin/AGENTS.md` | +| `process/` | `process/AGENTS.md` | +| `tools/` | `tools/AGENTS.md` | +| `e2e/` | `e2e/AGENTS.md` | + +Keep detailed technical material in `docs/`. Keep design decisions in +`docs/design/`. Use `docs/README.md` to find the document for a change. + +## Project overview + +Harness is a Go agent harness with four priorities, in order: + +1. **Speed.** Startup budgets are CI-enforced product requirements. +2. **Extensibility.** Plugins use a language-neutral process protocol. +3. **Composability.** The engine is headless. Frontends consume one event stream. +4. **Dynamic model choice.** A session can change providers without history migration. ## Architecture -The engine is a headless Go library; every frontend (CLI, TUI, server API) is a client. - -``` -cmd/harness thin CLI: flags → engine or client -engine/ session loop, tool registry, event log -provider/ one adapter per API family (anthropic, openai-responses, gemini, openai-compat, bedrock) -message/ canonical message/part types + per-provider transcoders -plugin/ hook bus, JSON-RPC stdio protocol, plugin SDK -server/ HTTP+SSE / unix socket exposing the engine -tui/ a client, nothing more -``` - -### Core invariants - -- **A session is an append-only log of typed events.** User messages, model deltas, tool calls, results, model switches — all events. UIs, JSON output, and plugins are subscribers to the same stream. -- **The session log stores the canonical message format, never a provider's.** Every request, the provider adapter transcodes canonical history → provider wire format from scratch (stateless transcoding). Mid-session model swap = next request uses a different transcoder. No migration step. -- **Provider-specific opaque data (reasoning/thinking blocks, encrypted reasoning items) is stored as provider-tagged attachments** on canonical messages: replayed verbatim to the same provider, dropped when crossing providers. Tool-call IDs are internal; each transcoder maps deterministically to provider-compliant IDs. Prompt-cache markers are injected at transcode time, never stored. -- **Model refs are `provider/model`** plus user-defined aliases (`fast`, `smart`) from config. The models.dev catalog snapshot is embedded at build time and refreshed async — never on the startup path. -- **A history repair that runs on live or persisted state is additive-only.** `LoadSession` writes the repaired slice back into live history, so a repair that deletes loses data permanently — not for one request, but for the life of the session. Add synthetic parts; never drop, reorder, or relocate a part another producer wrote. A transcode-time repair MAY be destructive, because it builds one throwaway request and never touches the record. Put every destructive rule on that side of the line. (Incident: a `ResolveOrphanToolCalls` rewrite deleted genuine tool output in three shapes and was reverted; see NEP-5293.) The concrete split is in "Wire normalization" below. -- **An empty tool result must never serialize as `null`.** The provider reads a null-content `tool_result` as ABSENT and rejects the whole request with "tool_use ids were found without tool_result blocks immediately after" — naming a block that IS in the payload. A tool that produces no output (a `grep` that matches nothing) is enough to wedge a session forever. `message.NoToolOutputText`, `ToolResult.SafeContent`, and `ToolResult.MarshalJSON` hold this line; every transcoder reads through `SafeContent`, never `Content`. (Incident: NEP-5272.) - -### Wire normalization - -Two functions repair `tool_use`/`tool_result` pairing. They sit on opposite -sides of the live-versus-transcode line in the invariant above. - -`message.ResolveOrphanToolCalls` is the LIVE-path repair. `LoadSession` -applies it and writes the result back into history, so it stays purely -additive. It deliberately leaves several shapes wire-invalid. Do not "fix" -it — that is the whole point of the split. - -`message.NormalizeForWire` (`message/wire_normalize.go`) is the -transcode-only sibling. Every transcoder calls it instead. It builds one -throwaway request, so it may relocate a part. It must still never delete a -real `ToolResult`. - -`NormalizeForWire` closes four shapes `ResolveOrphanToolCalls` cannot: - -1. Two `tool_use` blocks share one call ID in one assistant message. -2. A `ToolCall` sits in a non-assistant message. -3. A `ToolResult` precedes its `ToolCall`. -4. An intervening same-side message separates a `ToolResult` from its - `ToolCall`. Every transcoder merges adjacent same-role messages (see - `transcodeRequest`'s same-role merge, `provider/anthropic/transcode.go`), - so the wire sees RUNS. `ResolveOrphanToolCalls` tests strict - `messages[i+1]` and is blind to this. - -Relocation is bounded. `computeRelocationBarrier` moves a result no later -than the origin run of the next real result. That keeps the original -relative order intact. A move that would break the bound is refused. - -`message/wire_oracle_test.go` is the specification both functions are -tested against. Derive it from the provider contract only, never from -either function's internals. See the oracle rule under Testing. - -### Ambient engine context is a structured, unforgeable part - -The engine appends its own live status to the newest user message every -request — engine identity (`[engine: ...]`), managed-process status -(`[processes: ...]`), degraded-MCP status (`[mcp: ...]`), and the -parked-goal notice (`[goal: ...]`). This is a `message.EngineContext` part, -NOT a `Text` part. A bare `Text` block is byte-indistinguishable from -user-typed or pasted text, so a payload a user pastes that contains -`[engine: ...]` once inherited the same trust the engine's own block -carries — a trust-spoofing surface. `EngineContext` is a distinct part-kind -only `withAmbientStatus` (`engine/process.go`) produces, so a user- or -paste-authored part is always a `Text` and can never BE one, however its -bytes are shaped. Every transcoder renders an `EngineContext` through -`message.RenderEngineContext`, which wraps the block in the -`message.EngineContextOpenTag`/`EngineContextCloseTag` sentinel, and renders -every `Text` through `message.NeutralizeEngineContextSentinel`, which -defangs any literal sentinel that text carries. Only a genuine -`EngineContext` can therefore emit the sentinel on the wire, so the base -system prompt (`cmd/harness`, `ambientContextGuidance`) tells the model to -trust the sentinel-wrapped block and to distrust bracketed text outside it. -The render stays an ordinary text block on every provider — no new wire -feature. `EngineContext` is runtime-only (appended to the throwaway -per-request copy, never the durable log — the prompt-cache-prefix and -never-persisted rules below are unchanged) but still round-trips through the -canonical JSON union like every other part. Never revert this to a `Text` -part, and never make the guidance trust bracketed text syntax again. (Fix: -the NEP ambient trust-spoofing finding; superseded PR #113's prose-only -stopgap.) - -### Project instructions (AGENTS.md) - -The engine auto-injects a project's `AGENTS.md` into the system prompt. On the -first `Prompt` of a session (never at `NewSession` — the startup budget rule) -it walks up from `Config.WorkDir` for `AGENTS.md` (falling back to `AGENT.md`), -stopping at the git root or filesystem root; the closest file wins, per the -[agents.md](https://agents.md/) convention. The file is schema-less Markdown — -no headings are required or parsed. The segment is appended after -`Config.System` and before hook (`system.transform`) segments, cached for the -session, and never written to the session log (loaded fresh on resume). - -A present-but-unusable file (invalid UTF-8, or empty/whitespace-only) fails the -first `Prompt` — a project that meant to supply instructions must not run -silently without them. A missing file is fine. Disable with -`-no-instructions`, config `instructions: false`, or point at a specific file -with config `instructions_path`. - -An oversize file is truncated, and the truncation is LOUD on both channels. -`truncateInstructions` (`engine/instructions.go`) appends the in-band marker -`formatTruncationMarker` builds — it names the path, the original size, the -kept size, the dropped size, and the `read_file` tool that reads the rest — -and writes one WARN log line with the same counts. A silent cut was the -earlier behavior: a 408 KiB `AGENTS.md` reached the model as 64 KiB with no -sign of the missing 344 KiB, so the model followed a half specification and -believed it read the whole one. A truncated instruction file must always -announce itself; never make this cut quiet again. - -`InstructionsConfig.MaxBytes` sets the cap: 0 (the zero value) takes -`defaultMaxInstructionsBytes` (64 KiB), a positive value sets it, a NEGATIVE -value disables the cap so the whole file is injected. Config key -`instructions_max_bytes` (bytes) and the operator seam -`HARNESS_INSTRUCTIONS_MAX_KB` (kilobytes; negative disables) resolve it in -`cmd/harness`, the environment variable winning — the engine never reads an -environment variable itself. - -An oversize file is not merely marked, it is SPLIT. -`renderInstructions` (`engine/instructions_outline.go`) injects a HEAD plus an -OUTLINE: the head is every section that fits whole under the cap, and the -outline lists every section the head does not carry — one line each with the -heading, the exact `read_file` range that reads it -(`read_file(path=, offset=, limit=)`), and a short teaser -from the section body. The model pulls a section with `read_file`, whose -`offset`/`limit` are already 1-based line numbers, so this adds NO tool and no -schema to any request. The shape is the Agent Skills stage-1/stage-2 split -below, applied to one file: the outline is an index the model MUST read -through before it relies on a section. This file is itself over the cap, so -this file is itself split: the head carries the sections that fit, and every -later section — including this one — reaches the model only when it reads the -advertised range. Nothing is out of reach, where the marker alone left the -whole tail unreachable. - -`scanSections` tracks fenced code blocks by FENCE CHARACTER AND RUN LENGTH, -not with a boolean. A `#` comment inside a ```` ```bash ```` block read as a -heading would advertise a range that points at a shell comment. The character -and run-length rules cover the next shape up: an instruction file that -documents Markdown wraps a three-backtick example in a four-backtick fence, -which a naive toggle closes early, turning the rest of the document into -false sections. - -The split composes with the cap in ONE direction: the cap decides what is -EAGER, the outline makes everything else REACHABLE, and nothing is dropped in -silence either way. Three shapes keep the marker. A file with fewer than two -sections (no heading, or one giant section) has nothing to outline and takes -the `truncateInstructions` path unchanged. `InstructionsConfig.Mode` -`InstructionsModeFull` selects that path for every file. And a file whose -FIRST section alone exceeds the cap has no boundary to cut on, so the head is -that truncated first section — marker and WARN line both firing, through -`truncateInstructionsOf`, which reports the WHOLE file's size and not the -first section's — with the outline still listing every later section. Never -let the head lose its marker in that shape: it is the one place where an -outline could hide a cut. Config key `instructions_mode` and the operator seam -`HARNESS_INSTRUCTIONS_MODE` resolve the mode in `cmd/harness`; only the value -`full` (case-insensitive) turns the outline off, because an unreadable knob -must not quietly drop an outline nobody asked to lose. Design: -docs/design/nested-instruction-loading.md. - -### Agent Skills - -The engine advertises [Agent Skills](https://agentskills.io) in the system -prompt following the spec's progressive-disclosure model. On the first `Prompt` -(alongside instructions loading, same load-once-cache-error pattern) it runs -`skill.Discover` over each configured directory, merges the results sorted by -name, and injects one system segment **after** the instructions segment and -before hook (`system.transform`) segments. That segment is stage 1 only: a -header telling the model it MUST read a skill's `SKILL.md` with the `read_file` -tool before relying on it, then one line per skill — `name — description (path: -)`. Stage 2 (the body) is deferred to that read. - -`Config.SkillsDirs` selects the directories: nil (the default) uses -`/.agents/skills` when it exists; an explicit empty slice disables -discovery. A malformed `SKILL.md` or a duplicate skill name across dirs fails -the first `Prompt` loudly (same semantics as a malformed AGENTS.md). Skills are -never written to the session log — a resumed session rediscovers them. Config -`skills_dirs` (array; a non-empty project value overrides the user value -entirely) and the repeatable `-skills-dir` run/serve flag drive it. - -### Tool-batching guidance - -The engine executes one assistant message's tool calls concurrently -(`engine/toolexec.go`, capped by `Config.ToolConcurrency`), but a model -that emits one call per turn never produces a batch wider than one. The -executor is only as useful as the model's willingness to batch, so the -engine asks for it: `toolBatchingSegment` (`engine/toolexec.go`) injects -one system segment telling the model to put independent calls in the same -message, and to wait when a call's arguments depend on an earlier call's -result. Both halves matter — the second is what stops a model -parallelizing genuinely dependent work, which no amount of executor -correctness can repair. - -The segment sits immediately after `Config.System` and before the -instructions segment (`engine/engine.go`): it describes how this engine -runs tools, not anything about the project. It is **gated on the -session's resolved concurrency and is empty at 1** — an operator who set -`HARNESS_SEQUENTIAL_TOOLS=1`, or an embedder who set `ToolConcurrency: 1`, -must not be told calls run concurrently when for that session they do not. -The cap in the text is rendered from `s.toolConcurrency`, so the number -the model reads is the number the executor enforces. Like every other -engine-injected segment it is never written to the session log. - -Adding a base segment shifts every later segment's index, so the -segment-layout assertions across `engine/*_test.go` pin it explicitly via -`isBatchingSegment` (`engine/toolbatching_test.go`); only that file pins -the wording itself. - -### read_file image support - -The built-in `read_file` tool (`engine/filetools.go`) can return an image -file as real visual content, not mangled text. `readPathContent` opens the -target path exactly once and classifies it by its magic bytes -(`http.DetectContentType` over at most the first 512 bytes) — never by its -extension: a `.txt` file that is actually a PNG is still recognized as an -image, and a `.png` file that is actually text stays a text read. On a -recognized image (`image/png`, `image/jpeg`, `image/gif`, `image/webp`), -`read_file` returns a `message.ToolResult` whose Content is `[Text, Blob]`: -a one-line Text summary (format, byte size, and pixel dimensions) followed -by a `message.Blob` carrying the real file bytes. This is the same -`Text`+`Blob` shape MCP's `mcpContentToParts` already produces -(`engine/mcp.go`) — `read_file` is a second producer of it — so every -transcoder's existing Blob handling and the imageclamp dimension/byte-size -pass (`imageclamp.Clamp`, called from every transcoder's -`transcodeRequest`) apply with no new wiring. `read_file` never bypasses -that clamp: it does not resize, re-encode, or otherwise touch pixels -itself. Because `imageclamp.Clamp` runs later, at transcode time, an image -it downscales or re-encodes can end up described by dimensions or a byte -size that no longer match the summary `read_file` reported when it read -the file; this is a known, accepted mismatch, not a defect to fix in -`read_file` itself. - -**Only the Anthropic route puts a tool-result image on the wire.** -`imageclamp.Limits.RecurseToolResults` is true for `provider/anthropic` -only; `provider/openai` and `provider/openaicompat` set it false and -instead replace a tool-result Blob with a text note, -`"[N image attachment(s) omitted]"` (`toolResultOutput`, -`provider/openai/transcode.go` and `provider/openaicompat/transcode.go`). -This is pre-existing wire-format behavior `read_file` inherits, not -something this feature introduces, but it means a `read_file` image reaches -the model as pixels only on the Anthropic route; on the other two the model -sees only the one-line Text summary. - -`readPathContent` applies three guards on the image path, in order: - -1. The sniff read uses `io.ReadFull`, not a single `Read`, so a short - `read(2)` — realistic on a pipe or FUSE mount — never misclassifies a - real image as plain text. -2. The read is bounded at `readFileMaxImageBytes` (20MB), checked - against an `io.LimitReader` over the same open handle, never against a - separately captured `os.Stat` size a concurrently growing file could - outrun. This cap is separate from and smaller than any provider's own - wire limit, which `imageclamp.Clamp` enforces at transcode time; it - exists only so `read_file` itself never loads an unbounded file into - memory. An over-cap image returns a plain text error and no Blob. -3. The body must decode with `image.DecodeConfig` before `read_file` - commits to the image outcome. A corrupt or truncated file that merely - opens with a matching magic-byte prefix fails this check; `read_file` - then reads the true remainder of the file (unbounded, same handle) and - returns it as ordinary text instead of shipping a Blob the model cannot - use. This guard is not airtight for GIF: the `GIF87a`/`GIF89a` header - carries no checksum, so text that happens to start with those exact six - bytes still "decodes" with fabricated dimensions. A real file colliding - with that prefix is vanishingly unlikely; this is a documented, accepted - residual. - -A non-image binary file (sniffed as `application/octet-stream` or similar) -keeps `read_file`'s existing (unbounded) text-read behavior; `readPathContent` -still reads it exactly once, through the same handle its sniff already -opened. - -**Known gap, filed as issue #129**: a transcode-time degrade of an image -Blob to a text placeholder for a model with no vision capability is not -implemented. No per-model vision-capability signal exists anywhere in the -codebase to gate it on — the embedded models.dev catalog this file's own -"Architecture" section describes as a design goal is not yet built, and -`provider.Request` carries no capability flag comparable to `Effort` or -`SessionKey` that a caller could set from one. Building this now would mean -inventing an ad hoc, likely-wrong static model list, so it is deferred to -issue #129. Until it lands, a model with no vision support receives the -image Blob exactly as any vision-capable model does; how it handles that -block is between the model and its provider. - -### Bounding concurrent-read memory - -`read_file`'s text path is deliberately unbounded (`readPathContent`, -`engine/filetools.go`) — a coding agent legitimately reads whole files, and -no byte cap is right for every file. Only the IMAGE path is capped -(`readFileMaxImageBytes`), and `bash` caps its own output -(`defaultBashOutputCap`). - -That was safe while tool calls ran strictly one at a time: peak heap held -at most ONE file's raw bytes plus the line-numbered copy built from them. -The concurrent executor (`engine/toolexec.go`) removed that implicit bound -without replacing it, so a batch of N large reads holds N working sets at -once. Measured with eight 16MB files, retention swallowing the finals so -only the transient term shows: **~325MB peak parallel against ~73MB -sequential — ~4.3x, bounded only by `ToolConcurrency`**, with the model -choosing both the batch width and the file sizes. - -`toolReadBudget` (`engine/toolmem.go`) is the replacement bound. Each -`read_file` reserves its file's Stat size against a per-session byte -budget before touching the file, and holds it until the call returns — -spanning the line-numbering too, since for a large file `strings.Split` -is the single biggest allocation and releasing after the read would leave -the expansion outside the bound. With the budget set to one file's size -the same batch peaks at the SEQUENTIAL figure: ~73MB, **1.0x**. It bounds the **product** of read size and -concurrency, which is the actual hazard: a count limit still admits two -500MB reads, and a size limit breaks the legitimate large read. Ordinary -work never contends — a full-width batch of kilobyte reads reserves a -rounding error against the default and stays fully parallel -(`TestReadBudgetKeepsSmallReadsFullyParallel`). - -It bounds the TRANSIENT working set, not the ACCUMULATED results: -`runToolBatch` holds every call's output until the join in BOTH execution -modes, so N results occupy the same memory however they were produced. -That term is not a concurrency regression, and retention already collapses -oversized results where configured. - -Three properties keep it safe. A worker takes its pool slot first and then -reserves, which is head-of-line blocking but cannot deadlock: only a slot -holder ever holds budget, so someone is always making progress, and a -reservation is never held while acquiring another. A read larger than the -whole budget is CLAMPED to it, not refused, so it waits for the budget to -drain and runs alone — a batch always progresses. Waiters are served -strictly FIFO, because a retry-when-there-is-room loop lets a stream of -small reads starve one large read forever. - -`Config.ToolReadBudgetBytes`: 0 (the zero value) takes -`defaultToolReadBudgetBytes` (64 MiB), a negative value DISABLES the bound, -a positive value sets it. `HARNESS_TOOL_READ_BUDGET_MB` is the operator -seam (megabytes; negative disables), resolved in `cmd/harness` like -`HARNESS_TOOL_CONCURRENCY`. The budget is per SESSION: a process running -many sessions bounds each, not their sum — a process-wide budget is the -natural follow-up if that proves insufficient. - -### write_file read-before-overwrite guard - -`write_file` (`engine/filetools.go`) refuses to overwrite an EXISTING file -the session has not read. Before this guard, `write_file` overwrote any -existing file unconditionally — a model could destroy a file it never -opened, with no recovery path. `edit_file` never had this hole: its -`old_string` match is exact-content-required by construction, so it cannot -blindly clobber unseen content. Claude Code and opencode both close the -same gap on their own write tools (opencode's is a literal `"You must read -file X before overwriting it"` error); this guard gives `write_file` the -same property. - -`Session.recordRead`/`readHashFor` (`engine/filetools.go`) track, per live -session and in memory only, every path `read_file` has read or -`write_file`/`edit_file` has written, keyed on the RESOLVED absolute path -(`s.resolvePath`'s output, never the raw tool argument — two different -relative arguments that resolve to the same file must not be tracked as two -separate paths), mapped to the sha256 hash of that path's raw on-disk bytes -at the moment of that read or write. `read_file` hashes the complete raw -file bytes it already read off disk for its own content classification -(`readPathContent`'s `TextData`/`ImageData`) — never the offset/limit-sliced -text it returns to the model. This matches the reference guards (Claude -Code, opencode): the guard authorizes per successful OPEN, not per byte -displayed — a windowed read of a large file authorizes replacing the whole -file. The hash is recorded only at a `read_file` return that hands the -model content; a read that errors (offset past end-of-file) records -nothing. - -`write_file` on a path that `os.Stat` resolves to an existing regular file -requires, in order: (1) the path is present in this session's read set — -absent means `write_file: %s exists and has not been read this session; -read it first (or use edit_file)`; (2) hashing the path's CURRENT on-disk -bytes fresh (never trusting the recorded hash's age) matches the recorded -hash — a mismatch means `write_file: %s changed on disk since it was read; -read it again before overwriting`. A path that does not exist -(`fs.ErrNotExist`) is unguarded — creation is `write_file`'s main job, and -there is no prior content to protect. Any OTHER stat failure (permission, -transient metadata error) refuses the write with `write_file: cannot stat -%s to check the read-before-overwrite guard` — a failed stat cannot prove -no protected file exists there. A -successful `write_file` or `edit_file` records/updates the written path's -hash to the new content's hash, so a write immediately followed by another -write to the SAME path (the assistant overwriting its own just-written -content, or an `edit_file` followed by a `write_file` on the same path) -never spuriously re-triggers the guard — the session already knows exactly -what is on disk because it just put it there. - -The read set is runtime-only: never persisted, never folded by -`LoadSession`. A reloaded session starts with an EMPTY read set, so a -resumed session must `read_file` a path again before `write_file` can -overwrite it — even a path the session genuinely read in a prior process -life. This is deliberately conservative and matches the guard's purpose: -the guard exists to stop an overwrite of content the model never actually -saw THIS session, and a resumed model has no live memory of raw file bytes -from a prior process either, only whatever text happened to land in the -persisted transcript. The read set lives on `Session` state, not `Config`, -so `configSnapshot` (used to seed a spawned child's config) never copies -it — a spawned child starts with its own empty read set, correctly, since -it is a different session that has read nothing yet. - -Tool calls execute strictly sequentially today -(`Session.runToolCalls`/`runToolCall`, `engine/engine.go`), so the read -set's own `sync.Mutex`-guarded map access is sufficient protection now. -A future parallel tool executor must serialize concurrent -`write_file`/`edit_file` calls against the SAME resolved path — matching -`edit_file`'s existing same-path safety requirement — rather than relying -on the map's per-operation lock alone, which does not cover the -check-current-hash-then-write sequence as one atomic unit against a -concurrent writer to the same path. - -`bash` writes (a model redirecting output to an existing file via a shell -command) are explicitly OUT of scope: harness cannot classify an arbitrary -shell command as a file write versus anything else it might do, so this -guard covers only the two built-in tools that make a structured, typed -claim to write a file. - -### Base loop retry - -The base interactive `Prompt` loop retries a transient provider error itself, -so a plain box prompt never surfaces a one-off HTTP 500. `streamTurnWithRetry` -(`engine/prompt_retry.go`) wraps `streamTurn` at its single call site in -`runAgenticLoop` (`engine/engine.go`). It retries only when the error is -classified retryable through `provider.AsRetryable` — `server_error`, -`overloaded`, `rate_limited`, or `stream_truncated`, never by matching error -text — AND the budget has an attempt left. Every other error returns on the -first attempt with ZERO retries: a `context.Canceled` abort, an -`*interruptedTurnError` (whose partial `runAgenticLoop` must still append — -retrying would duplicate the model's already-emitted tool intent), a -`provider.AsPermanent` malformed-request shape, or any deterministic failure. -The final surfaced error still emits one `session.error` and drops the usage -exactly as before; an intermediate masked attempt emits neither. A masked -attempt is still a full `streamTurn`, so it DOES bump the per-session turn -counter (`s.turn`, reported by `session_info`) and re-fire the per-request -hooks (`chat.params`, `system.transform`) and `OnRequest` — one bump and one -hook pass per attempt, exactly like the goal loop's per-attempt behavior. Only -the `session.error` and usage are suppressed for a masked attempt. - -One class is retry-eligible WITHOUT `provider.AsRetryable`: a completed but -EMPTY turn (no non-empty text, no tool call — e.g. thinking consumed the -whole `max_tokens` ceiling; see `emptyTurnError`). Two deliberate deviations -from the masked-attempt rules above. First, a discarded empty attempt's -usage IS accumulated into cumulative `Usage()` (it was a fully billed -completion, unlike a transport failure — same principle as the empty -compaction summary), while `lastUsage` is left alone. Second, the nesting -math: an empty turn that survives all `PromptRetries+1` attempts surfaces a -deterministic error, which goal mode's worker tier retries -`goalWorkerRetries` more times — worst case `(PromptRetries+1) * -(goalWorkerRetries+1)` = 9 fully-billed calls — and then STOPS the goal -with the empty-turn reason. Before this class existed the same turn was a -silent success and a goal limped on with nothing appended; halting with a -legible reason is the intended trade. The fail-fast for the deterministic -`max_tokens`-exhaustion shape (cutting the 9 to 3) is a filed follow-up on -the PR that introduced this. - -Retrying `streamTurn` is idempotent for history and tool side effects: -`streamTurn` makes ONE model call and never executes a tool (`runAgenticLoop` -runs tools only AFTER `streamTurn` returns a `StopToolUse` message), so a -failed attempt ran no side effect to redo. The one shape that DID emit tool -intent before failing arrives as `*interruptedTurnError` and is excluded. - -The emit stream is NOT idempotent, so `streamTurnWithRetry` closes that gap. -A failed attempt can emit `EventTextDelta`/`EventReasoningDelta` for partial -text before its stream dies, and the retry re-streams that text from scratch. -`streamTurnWithRetry` emits one `EventTurnRestart` (`engine.go`) before each -retry, so a subscriber that renders deltas incrementally drops the stale -partial and rebuilds it from the retry — never the two runs concatenated -(`Hello wor` then `Hello world` shown as `Hello worHello world`). The server -forwards `EventTurnRestart` live over SSE (`server/journal.go`'s `Publish`); -the turn's final `EventMessage` still reconciles history regardless. - -`Config.PromptRetries` bounds it: additional attempts, zero (the engine zero -value) DISABLES retry, config/CLI default 2 via `config.Config`'s `*int` -`prompt_retries` key (`PromptRetriesValue`). The backoff -(`basePromptRetryDelay`: 1s, then 2s, `time.NewTimer`) is deliberately SMALLER -and SHORTER than the goal loop's tiers below — an interactive user waits on -the turn, so this smooths a blip in a second or two, never the goal loop's -~30min weather schedule (`promptTurnWithRetry`, `goal.go`). The two are -distinct: the base loop wraps ONE model call and is inherently idempotent; the -goal loop's `promptTurnWithRetry` wraps a whole worker turn (with the -tool-executed non-idempotency gate) and parks on exhaustion. - -The two also NEST. A goal worker turn runs through `s.Prompt`/ -`s.runAgenticLoop` (`goal.go`), so every one of `promptTurnWithRetry`'s outer -attempts now issues up to `1+PromptRetries` inner `streamTurn` calls. For a -persistent retryable condition the worst case is `goalRetryableMaxAttempts` -(12) times `1+PromptRetries` (3) — about 36 full-input-price model calls, -where the goal-loop tiers alone assume ~12. This is deliberate: the fast inner -budget (1s, then 2s) smooths a one-off blip inside a single worker turn before -the outer weather tier ever counts it, so a goal loop rides a brief provider -blip without spending an outer attempt. `PromptRetries` 0 disables the inner -budget for a host that wants the outer tiers to be the only retry. - -### Max-tokens auto-continue - -A turn whose stop reason is `provider.StopMaxTokens` means the provider cut -the model off mid-emission — it did not choose to stop. Before this existed, -`runAgenticLoop`'s `if stop != provider.StopToolUse` branch -(`engine/engine.go`) treated every non-`tool_use` stop reason alike: append -`asst`, synthesize an is_error result for any orphaned `ToolCall` part via -`appendUnexecutedToolCallResults` (NEP-5272, see "Wire normalization" -above), and return. For `max_tokens` specifically that return settles the -session idle with nothing further ever prompting it. Incident: box -harness-parallel-tools — the model emitted a large tool call, the provider -stopped mid-emission with `max_tokens`, the engine synthesized the -unexecuted-call result exactly as designed, and the session then sat idle -until a human noticed and re-prompted it. On an autonomous fleet a silent -work stoppage is as bad as a crash. - -`runAgenticLoop` now branches on `stop == provider.StopMaxTokens` inside -that same `if`: `maybeAutoContinueMaxTokens` decides whether to `continue` -the loop (issuing a real follow-up model call in this SAME `Prompt` call) -instead of returning. This applies identically whether or not `asst` carried -a `ToolCall` part — a pure-text `max_tokens` truncation (Claude Code's own -behavior is to let the turn end and rely on the user re-prompting) gets the -same auto-continue an autonomous harness session needs, not a human-facing -half-answer. - -**A genuinely mid-emission tool call keeps its identity; only its Arguments -are cleared.** A cross-model adversarial review of this PR raised a -CRITICAL finding: a `StopMaxTokens` turn's trailing `ToolCall` can carry -non-empty but syntactically invalid `Arguments` — the raw, truncated -`partial_json` Anthropic's `assembledBlock.toolCall` -(`provider/anthropic/anthropic.go`) leaves behind when `max_tokens` lands -before the block's own `content_block_stop`, e.g. `{"comm` — and claimed -replaying that into the continuation request fails `json.Marshal` before it -reaches the provider. That premise does NOT hold: `message.Message.Normalize` -(`Session.append`'s `appendWithUsage`, run on every append) already coerces -the identical invalid-Arguments shape to nil in place, through the SAME -`*ToolCall` pointer `asst.Parts` already holds — so by the time the -continuation request is built, `Arguments` is already safe. This is not -incidental; it is the deliberate, incident-tested fix for a real production -defect (two goal sessions dead at "json: error calling MarshalJSON... "; see -`TestPersistTruncatedToolCallArguments`, `engine/tool_call_poison_test.go`). -The finding is REBUTTED WITH EVIDENCE, not implemented: no code change. See -`TestMaxTokensPartialJSONMarshalsThroughRealTranscoder` -(`engine/max_tokens_wire_test.go`), which drives a genuinely truncated -`partial_json` tool call through a REAL `anthropic.Client` (`provider/anthropic`, -via an `httptest` server, not a hand-rolled stand-in) and proves the -continuation request the client actually sends decodes cleanly server-side, -with the truncated call's identity preserved and its `Arguments` cleared — -this is the test that pins the rebuttal and must go red before any future -"drop the call entirely" change lands unchallenged. - -**The continuation nudge is a genuine new turn, never assistant prefill.** -The follow-up call carries the synthetic unexecuted-tool-call result (for -any COMPLETE call the engine chose not to execute) plus a one-shot nudge — -`s.pendingContinuationNudge`/`continuationNudgeSegment`. Unlike every other -ambient status segment (process, MCP, goal-parked, identity, task -notifications — see "Ambient engine context" above), this one is NOT glued -onto an existing message via `withAmbientStatus`: -`appendContinuationNudgeMessage` appends a genuine NEW `message.RoleUser` -message, carrying the nudge as its own `*message.EngineContext` part, to the -END of `streamTurn`'s own throwaway per-request message copy. -`withAmbientStatus` scans backward for the newest EXISTING `RoleUser` -message, which by the time a continuation request is built is an EARLIER -message than the just-truncated assistant turn (and its synthetic tool -result, if any) — leaving the canonical request ending in `RoleAssistant` or -`RoleTool`. Anthropic serializes that as assistant PREFILL: some models -reject it with a permanent 400, and even an accepting model sees a -"continue" instruction that chronologically precedes the output it refers -to. `appendContinuationNudgeMessage` never touches `s.history` — same as -every ambient segment — so a session reload, or any later unrelated -request, never sees the nudge. It still rides every attempt a -transient-error retry makes for that ONE follow-up call (mirroring -`checkoutTaskNotificationsSegment`'s idempotent-reread shape, -`taskdelivery.go`) and is cleared by `runAgenticLoop` the instant that whole -`streamTurnWithRetry` call returns, so it never bleeds into a later, -unrelated turn. - -**Queued operator input is drained before every continuation, not only at -tool-call boundaries.** `drainQueuedPromptsIntoHistory` (`engine/engine.go`) -is the shared implementation behind the tool-call-boundary drain (after a -`StopToolUse` round actually runs a tool) and the max_tokens continuation -branch (right before looping back for another follow-up call) — both are -points where `runAgenticLoop` is about to issue another provider request in -the SAME `Prompt` call, so both are valid mid-turn steering opportunities. -An operator prompt queued while a long truncated response streams is -delivered on the very next continuation request, not left undelivered for -the whole continuation chain. - -Loop safety is the critical part: `runAgenticLoop`'s local `maxTokensUsed` -is a PER-PROMPT BUDGET, spent by every continuation issued in the loop and -NEVER reset — including across an intervening `StopToolUse` round. (An -earlier version of this counter, `maxTokensStreak`, DID reset on any -non-`max_tokens` stop, which let a model alternate `max_tokens` and -`tool_use` — including denied, unknown, or failing tool calls, none of -which touch `toolExecCount` — indefinitely inside one `Prompt` call, -spending an unbounded number of continuations without ever tripping -`Config.MaxTokensContinuations`.) `maxTokensUsed` is bounded by -`Config.MaxTokensContinuations`: `Config.MaxTokensContinuations+1` -max_tokens stops used within the loop trips the bound. -`maybeAutoContinueMaxTokens` then returns a -`*maxTokensContinuationExhaustedError` naming the bound, wrapped -`provider.MarkPermanent`, instead of arming yet another doomed attempt; -`runAgenticLoop` emits `session.error` and returns it, the same "honest -terminal, never a silent success" shape `emptyTurnError`'s own budget -exhaustion uses (see "Base loop retry" above). - -**A goal-loop retry must never re-run an already-exhausted continuation -chain.** With the default bound of 3, one worker attempt that exhausts the -budget already makes 4 completed, fully billed `max_tokens` calls before -`*maxTokensContinuationExhaustedError` is even returned. -`maybeAutoContinueMaxTokens` wraps every value of that type -`provider.MarkPermanent` at its one construction site, so -`promptTurnWithRetry`'s existing `provider.AsPermanent` fail-fast branch -(`engine/goal.go`) stops after that ONE attempt instead of re-running the -whole exhausted chain up to `goalWorkerRetries` (2) additional times — which -would otherwise multiply 4 calls into 12 for one goal boundary. Like every -other permanent-classified worker error, this PARKS the goal (stays -resumable) rather than clearing it: the condition that produced the -exhaustion might not recur on a later resume. - -`Config.MaxTokensContinuations` follows `PromptRetries`'s own -unset-vs-zero config idiom: the engine field's zero value DISABLES -auto-continue entirely (a bare embedder-built `engine.Config`, and every -test that constructs one directly, keeps the exact pre-fix behavior — the -turn ends immediately on the first `max_tokens` stop). The config/CLI layer -(`config.Config.MaxTokensContinuations *int`, key -`max_tokens_continuations`, resolved via `MaxTokensContinuationsValue`) -supplies the product default of 3; an explicit `0` disables it the same way -`prompt_retries: 0` disables base-loop retry. - -A task child runs its turn through this exact same `runAgenticLoop` — a -child `Session` is a full `NewSession(childCfg)` -(`SessionManager.Spawn`, `engine/session_manager.go`), and `childCfg` comes -from `configSnapshot`, a whole-struct copy of the parent's `engine.Config` -— so `MaxTokensContinuations` (and every other engine.Config field) reaches -a child with no separate wiring. A child that hits `max_tokens` -auto-continues under the identical bound the root does. - -### Per-turn metrics - -`streamTurn` (`engine/engine.go`) emits one structured `turn_metrics` line per -COMPLETED model call — a stream that reached `EventDone`; a turn that errors -or is interrupted mid-stream (see `interruptedTurnError` above) reports -nothing, since there is no finished call to summarize. This is the box-fleet -answer to "why does this session feel slow": TTFT and stream duration, token -and prompt-cache accounting, and request shape, greppable straight off a -process's stderr. - -Fields: `session_id`, `model` (full `provider/model` ref), `ttft_ms` (elapsed -from just before `prov.Stream` to the first non-`EventActivity` stream event -— `EventActivity` carries no content, so a keep-alive ping or an in-progress -tool-argument chunk must never be mistaken for "first byte"; if `EventDone` -itself is the first event, `ttft_ms` covers the whole call and `stream_ms` is -0), `stream_ms` (first delta to `EventDone`), `input_tokens`/`output_tokens`/ -`cache_read_tokens`/`cache_write_tokens` (passed through from -`provider.Usage` verbatim), `system_len`/`tools_count`, and `retry` (the -1-indexed attempt number `streamTurnWithRetry` — `engine/prompt_retry.go` — -was on when this call completed; 1 for a turn that succeeded on its first -try). `system_len` is computed identically to the server's `request.meta` -record (`len(strings.Join(req.System, "\n"))`, see `server/journal.go`'s -`OnRequest`) — deliberately, not coincidentally: `session_id` + `model` + -`system_len` together are a natural join key between a `turn_metrics` stderr -line and the durable `request.meta` record for the same request, with no new -ID threaded through the provider boundary. - -`Config.OnTurnMetrics func(TurnMetrics)` is the seam. Unlike every other -`On*` callback in `Config` (`OnEvent`, `OnRequest`, `OnStorePhase`), nil is -NOT "disabled": `emitTurnMetrics` substitutes `defaultTurnMetricsLog` -(`engine/turn_metrics.go`), a `slog.NewJSONHandler` line written to -`os.Stderr` — the same stream every other structured log line in this repo -uses (see `cmd/harness/main.go`'s "Structured logging: JSON to stderr" -comment). Stderr keeps the line out of `harness run`'s stdout, which is -the model's answer channel, while a deployment's log pipeline (Kubernetes -captures both streams) scrapes it identically. A plain `harness run`/`harness serve` process -with no embedder wiring therefore still emits this line by default; an -embedder that wants a different sink (an OTel exporter, an in-memory test -recorder) sets `OnTurnMetrics` and never needs to suppress the default -first. - -`Config.Now func() time.Time` is the clock this measurement reads (nil -resolves to `time.Now` in `newSession`), scoped to this one seam rather than -a general engine clock — every other timestamp in the package still reads -`time.Now` directly. It exists so a test can script an exact instant sequence -instead of depending on real elapsed wall-clock time between two in-process -calls with nothing to wait on between them, per the Testing rule against real -sleeps. - -This was built for a deployment that ships a served process's stderr to a -log pipeline (a fleet of boxes running `harness serve`, each pod's stderr -collected by a Vector-style agent into BetterStack or an equivalent log -store). The intended query there filters `msg: "turn_metrics"` and groups by -`model`/`session_id` to compare TTFT and stream-duration distributions across -sessions — quantifying, with real numbers instead of a feeling, whether a -session "feels slow" because of provider latency, prompt-cache misses, or -something else entirely. - -### Goal loop - -`Session.PursueGoal(ctx, condition, GoalOptions)` drives the ordinary `Prompt` -loop toward a natural-language completion condition. Turn 1 prompts the raw -condition; after **every** turn an independent, TOOL-LESS evaluator model -(`GoalOptions.Evaluator`, resolved through the same `Config.Providers` registry, -`MaxTokens` 256) is asked to answer `MET: ` / `NOT MET: ` -(parsed leniently). The evaluator request always pins `message.EffortOff` -(`runEvaluator`, `engine/goal.go`) — it is a classifier, not a reasoning task, -and it never inherits the session's own effort level. On openaicompat, -`EffortOff` sends the literal `"off"`; on anthropic, it emits no thinking -block — both routes now spend none of the evaluator's 256-token budget on -reasoning. (Issue #124.) The openai Responses route is a known residual: -`reasoningEffort` (`provider/openai/transcode.go`) omits the `reasoning` -object for `EffortOff` exactly as it does for `EffortUnset`, and a -gpt-5-class model reasons by default with no adapter-level way to disable -it — so an evaluator on that route can still spend its budget on reasoning. -A NOT MET verdict re-prompts -with a fixed-template guidance message carrying the reason; MET returns -`Achieved`. `MaxTurns` (0 = unlimited) bounds it. Evaluation is advisory: a -retryable-class provider error from the -evaluator call rides the matching in-boundary backoff before the boundary -counts as failed — the long weather-tier schedule -(`goalRetryableMaxAttempts`, ~30min) for `overloaded`/`rate_limited`/ -`server_error`, or the short stream-truncation tier -(`goalStreamTruncatedMaxAttempts`, 3 attempts, ~5s) for a stream cut before -its terminal event — `runEvaluatorWithRetry` mirrors `promptTurnWithRetry`'s -own per-class split exactly (see below); two unparseable replies in a row -(the second re-asked with a stricter prompt) or a non-retryable provider -error also fail the boundary immediately. A failed boundary no longer -clears the goal — it journals a durable `goal.eval_failed` record (carrying the consecutive -failure count), substitutes a fixed evaluation-unavailable notice for the next -turn's guidance in place of the evaluator's text, and `continue`s: the worker -keeps working. Any later boundary that DOES parse a verdict (MET or NOT MET) -resets the consecutive count to zero — the horizon is a streak, not a -lifetime total. Only after `goalEvalFailureLimit` (5) consecutive failed -boundaries does the loop treat the evaluator as durably broken: it clears the -goal with a dedicated reason, and the server maps that terminal to a -`session.error` plus a distinct `turn.end outcome=evaluator_exhausted` — loud -and machine-distinguishable, since every failure below the horizon is -deliberately silent apart from the journaled record. -Durable `goal.set` / `goal.eval` / `goal.eval_failed` / `goal.parked` / -`goal.achieved` / `goal.cleared` records land in the session log, so -`LoadSession` restores an active goal (condition only; counters reset) via -`Session.ActiveGoal()` — resume never auto-runs it, the caller decides. The -loop also emits `goal.*` engine events so the server journals them. Config -`goal_evaluator_model` supplies the evaluator for `harness run -goal` and -`POST /session/{id}/goal`. - -The evaluator's own `CONVERSATION TRANSCRIPT` field is BOUNDED, independent -of whatever context window the MAIN session model has and independent of -whether automatic compaction (below) has fired at all. -`renderConversationBounded` (`engine/goal.go`, called from `runEvaluator`) -replaced the old unconditional `renderConversation(s.History())`: box -bx-01m0x8996, a real long session, died with "engine: goal evaluator failed -at 5 consecutive turn boundaries: context exhausted: prompt 245332 tokens > -limit ..." because the evaluator's prompt grows with the WHOLE session -transcript forever — unlike the main session, which automatic compaction -protects, the evaluator had no bound of its own at all. The budget comes -from `goalEvaluatorTranscriptBudgetBytes`, which resolves the EVALUATOR -model's own context window via `modelContextWindowLookup` -(`modelmeta.ContextWindow`) — the same table automatic compaction's -`resolveContextWindow` (`engine/context_window.go`) reads, called -DIRECTLY rather than through `resolveContextWindow` itself: that -function's `minAutoContextWindowTokens` floor answers "should automatic -compaction ARM for this window," so a real, small, KNOWN window (gpt-4's -documented 8,192 tokens) reports identically to a genuinely UNRECOGNIZED -model (0, disabled) — conflating them would give a real small-window -evaluator a budget roughly double its actual limit, the exact overflow -class this fix closes. `goalEvaluatorTranscriptBudgetBytes` trusts ANY -positive, known window from the table, however small, and falls back to -`goalEvaluatorFallbackContextWindowTokens` (mirroring -`minAutoContextWindowTokens`'s value) only when the model has NO entry at -all. It also reserves headroom for the system prompt and MaxTokens' output -budget, and applies a conservative fraction (`goalEvaluatorContextBudgetFraction`, -0.5) on top of the same crude ~4-bytes-per-token estimate -(`bytesPerTokenEstimate`) automatic compaction's own resilience fallback -uses — reused, not reinvented. -`renderConversationBounded` walks history from the NEWEST message backward, -accumulating rendered blocks until the budget would be exceeded, and -prefers "summary + tail" for free rather than summarizing a second time: -Compact (`engine/compact.go`) already splices its own summary message in -place of whatever range it folded, tagged with the `compactionSummaryIDTag` -prefix, so the backward walk simply STOPS the instant it includes such a -message — everything before it is already captured there. The newest -message is always kept regardless of budget (an empty transcript can never -be assessed); a truncated transcript is prefixed with -`goalEvaluatorTruncationNotice` so the evaluator, and an operator reading a -later `goal.eval` record, never mistakes a bounded view for the whole -session. - -A worker-turn error (`s.Prompt` failing) is retried by `promptTurnWithRetry` -on one of FOUR independent budgets, chosen by classification via -`provider.AsRetryable` — never by matching error text. - -One class skips every budget. Before it selects a budget, -`promptTurnWithRetry` tests `provider.AsPermanent` — its fail-fast check -(`engine/goal.go`) — and fails fast: a permanent error gets ONE attempt and -no retry. -`provider.MarkPermanent` marks a malformed request shape. The anthropic -adapter applies it to an HTTP 400 `invalid_request_error`, and to the same -error type mid-stream (`provider/anthropic/anthropic.go:114` and `:484`), -only after `parseContextOverflow` rules overflow out — the two are disjoint. -A retry never repairs a malformed request, and each attempt costs a full -turn at full input price. A permanent error still PARKS, exactly like every -budget exhaustion; it never clears. `permanent` is threaded through only to -select a more accurate classified reason and tier name -(`classifyGoalWorkerError`, `goalWorkerParkedError`), so an operator can -tell a single-attempt park from `goalWorkerRetries`+1 identical attempts. - -One shape is wrapped `provider.MarkPermanent` by the adapter but is -DELIBERATELY EXCLUDED from this fail-fast branch: `provider.AsProviderExhausted` -— an ACCOUNT-level usage/quota wall (PR #174's `provider.ErrKindProviderExhausted`, -originally added for task-child resumability; see `engine/session_manager.go`'s -`FailKindProviderExhausted`). An adapter marks it permanent for ordinary -HTTP-retry purposes (no short backoff schedule outlives a monthly quota), -but a wall lifts on its own, unchanged, so treating it as a doomed malformed -request silently kills goal supervision on the very first usage-limit -rejection. Live evidence: box bx-01m0x8996 parked after "1 permanent-tier -attempt(s)" on "You have reached your specified API usage limits" and never -resumed without an operator `DELETE` + re-register. `promptTurnWithRetry` -and `PursueGoal`'s worker-turn handling both check -`provider.AsProviderExhausted` explicitly and fold a positive result into -their local `retryable`/`class` bookkeeping (`class` set to the dedicated -marker `goalClassProviderExhausted`, never one of `provider.RetryableClass`'s -real values) — reusing the existing stall/park recording machinery rather -than adding a fourth field throughout. - -A deterministic -failure (not classified retryable, not permanent) gets `goalWorkerRetries` (2) additional -attempts on the short schedule (~5s total: 1s, then 4s). A provider error -classified `overloaded`/`rate_limited`/`server_error` gets a separately -budgeted `goalRetryableMaxAttempts` (12) backoff (~30min total, jittered, 5s -doubling to a 5min cap) that never spends the deterministic budget. A -provider error classified `provider.RetryableStreamTruncated` — a response -stream that died before its terminal event, with no HTTP status or inline -error to classify from (see the idle-stream watchdog below) — gets its own -`goalStreamTruncatedMaxAttempts` (3) budget on the SAME short schedule the -deterministic tier uses (~5s total): truncation is retryable, but it is not -weather — waiting longer never raises a stream ceiling, and every retry -re-prompts a full turn at full input cost — so it must ride neither the fast -deterministic budget nor the long weather-tier one. A `provider.AsProviderExhausted` -failure gets its OWN `goalProviderExhaustedMaxAttempts` budget (equal in -size to `goalRetryableMaxAttempts`, on the identical jittered schedule) — -never the ordinary weather counter, so a concurrent overload spell and an -account wall in the same turn can never silently share or steal from one -another's budget. It deliberately rides the SAME schedule ordinary weather -uses rather than computing a wait from the provider's own `RecoverHint` -("you regain access on "): `RecoverHint`'s format varies by provider -and by plan (see `provider.Error.RecoverHint`'s doc comment), so it is -never parsed into a duration, only ever quoted verbatim to a model-visible -caller. A wall that clears within the budget (a burst rate limit that -reached this classification, or a short-lived cap) resumes the worker turn -— and the whole goal — with NO operator action; a wall measured in hours or -days still exhausts the budget and parks, but honestly classified via -`classifyGoalWorkerError`'s dedicated branch ("provider account usage limit -exhausted the retry budget"), never as a permanent, unretriable request. -Every attempt records a -`goal.stalled` record regardless of tier, so the loop is never silent. -Exhausting ANY of the four budgets — or the non-idempotency gate stopping -retries early once a tool has already executed this attempt — PARKS the goal -instead of clearing it: `PursueGoal` exits, journals a durable, CLASSIFIED -`goal.parked` record (never raw provider error text — the same leak rule -`goal.eval_failed` follows), and returns a distinct `*goalWorkerParkedError` -sentinel (`engine.IsGoalWorkerParked`) WITHOUT calling `clearGoal` — -`goalActive` stays true, the condition is untouched, generation-gated exactly -like `goal.stalled`/`goal.eval_failed` so a park racing a concurrent -`UpdateGoal` is silently discarded rather than attributed to a condition the -model never saw. This supersedes both this package's earlier -deterministic-tier clear and GitHub issue #61's in-loop retryable-tier -self-re-arming `continue` — the latter pinned the run slot to the parked loop -for the whole outage; exiting instead frees the slot, so a queued prompt -dispatches as an ordinary turn during a long outage instead of only ever -being injected mid-turn into a doomed attempt. Context overflow (issue #62) -is the one deliberate exception and still clears immediately, never parks: -no amount of waiting fixes an oversized request, so parking it would just be -a slower-burning zombie instead of a fix. Parking has no streak horizon -(unlike the evaluator's 5-boundary terminal above) — every exhaustion parks -immediately, and `DELETE /session/{id}/goal` remains the only clear path for -a parked goal. - -Each retry re-issues the SAME directive, and `Prompt` appends whatever text -it gets as a brand-new user message — it has no notion of "this is a retry, -do not duplicate." Left alone, N failed attempts leave N unanswered copies of -one directive, and every LATER request pays for all of them. `Prompt` -persists each copy before the provider call that fails, so the duplicates -reach the durable log, not just live history. - -`promptTurnWithRetry` therefore never appends a second copy for the common -case. It tracks one `anchorID`, naming the point right before this turn's -CURRENT, still-unanswered directive — starting as `lastMessageID`, captured -once before attempt 1 — then dispatches each retry one of three ways -(`engine/goal.go`, `tailAfterAnchor` shares the anchor-to-tail lookup; see -docs/design/goal-retry-directive-reuse.md): - -- Attempt 1 calls `Prompt`, which appends the directive. -- A retry whose tail after `anchorID` is EXACTLY the previous attempt's - unanswered directive (`directiveReuseEligible`) calls `runAgenticLoop` - instead. That runs the turn loop against history as it stands and appends - nothing, so the existing message is answered rather than duplicated. -- Any other tail falls back to `dropUnansweredDirective` plus `Prompt`, then - re-anchors: `anchorID` moves to `lastMessageID`, the point right before - the fresh directive `Prompt` is about to append. A later attempt's reuse - check then measures from that new directive, never from the turn's - original start. - -`runAgenticLoop` is `Prompt`'s own loop body, split out unchanged -(`engine/engine.go`). `Prompt` still appends and then calls it, so `Prompt`'s -observable behavior is identical: same events, same `emitStatus`, same usage -accounting. Note that `maybeAutoCompact` stays in `Prompt` and does NOT run -on the reuse path. That is deliberate, and the reason is that history did -not grow: the reuse path is reachable only when the tail is exactly one -message, so no new completed turn appeared to fold since attempt 1 already -ran the check. (`maybeAutoCompact` folds only COMPLETED turns, so it would -never have folded the unanswered tail directive itself.) One narrow -residual: history sitting right at the threshold, where appending a -directive would tip it over, no longer triggers a mid-outage fold. That is -accepted — the outage that piles up retries is also when the summarizer's -own provider call fails, and compaction is best-effort anyway. - -`dropUnansweredDirective` remains the fallback for the interrupted-turn tail -(the directive plus a partial assistant message and its synthetic -tool-result message), and for any tail a denied tool call or delivered mail -makes undroppable. It anchors on a message ID, never on a history length, -and `isSafeToDropDirectiveTail` approves only that interrupted-turn shape -and the bare directive. Any other tail is left untouched — a denied tool's -result, or an already-delivered "OPERATOR MESSAGES" block, must never be -discarded. It mutates only live history and can never retract a journaled -record, which is why the reuse path above, not a retraction, is what keeps -the log clean. `promptTurnWithRetry`'s re-anchor above bounds an undroppable -residue's cost to ONE extra duplicate directive for the rest of the turn, -never one per remaining attempt: re-anchoring past it lets reuse resume on -the very next attempt instead of re-appending against a tail that can never -shrink back to a droppable shape again. - -An idle provider stream — one that goes silent with no bytes, no -`EventDone`, no error, ever — is bounded by a per-request idle-stream -watchdog (`engine/stream_watchdog.go`, `Config.StreamIdleTimeout`, config key -`stream_idle_timeout_s`): every stream event resets its timer, and on expiry -it cancels the request's child context and converts the resulting -cancellation into a classified `provider.RetryableStreamTruncated` error -instead of an anonymous "context canceled" — this is what feeds the -stream-truncation tier above. It defaults to 5 minutes (mirroring Codex's -`stream_idle_timeout_ms`), a negative value disables it, and it guards the -worker turn, the goal evaluator, and the compaction summarizer's streams -alike (`armIdleWatchdog` wraps all three, so a silent stream at any of them -can no longer wedge the session forever while holding the run slot). - -Automatic compaction's over-threshold check -(`maybeAutoCompact`/`estimatePromptTokensFromHistory`, `engine/compact.go`) -has its own resilience fallback: a provider route that reports all-zero -input usage on a turn that DID complete is treated as missing data, never as -"0 tokens, never over" — the check falls back to a crude ~4-bytes-per-token -estimate walked from the actual session history so the overflow-prevention -layer keeps functioning instead of going permanently dark on that route, -which otherwise runs to a hard context overflow that clears (never parks) an -active goal. - -`Config.ContextWindowTokens` — the size that gates automatic compaction at -all — is resolved by `newSession` (`engine/context_window.go`, -`resolveContextWindow`), not just read verbatim from whatever the embedder -passed in. Precedence: an explicit, positive `Config.ContextWindowTokens` -always wins and is pinned for the session's lifetime (`contextWindowExplicit` -on `Session`, set once at construction); otherwise the session's MODEL is -looked up in package `modelmeta` — a curated, static table of -`provider/model` -> context-window tokens sourced from models.dev's -`limit.context` field (bifrost's `/v1/models` was investigated first and -ruled out: it returns the bare OpenAI listing shape with no context-length -data at all). A model-derived value under `minAutoContextWindowTokens` -(16k) is treated as bogus metadata and ignored — logged, never armed. An -unrecognized model (or no metadata at all) leaves compaction disabled, -identical to the field's original zero-value behavior. `SetModel` re-runs -the same derivation against the new model whenever the window wasn't -explicitly pinned, so a mid-session model switch keeps the window matched -to whichever model is actually running — switching FROM a recognized model -TO an unrecognized one disarms compaction again, not just leaves the old -window in place. One INFO log line (`"engine: context window"`) fires at -session start and on every switch that changes the effective window, naming -the resolved tokens and source (`config`/`model-derived`/`disabled`) — the -operator signal for "is compaction armed and why," added after a box -(`jumpy-pizza`) died with a raw `context exhausted: prompt N tokens > limit -M` provider error because `ContextWindowTokens` was opt-in and the boxes -platform set it nowhere. See docs/design/context-compaction.md's "Where -`ContextWindowTokens` comes from" addendum for the full incident writeup. - -On the server, a worker-parked sentinel maps to `session.error` plus a -distinct `turn.end outcome=worker_parked`, and `goalTracker` folds the -durable `goal.parked` record into a third `paused` arm (`pause_reason: -"worker_failure"`, alongside the existing boot-only `"restart"` and live -`"provider-backoff"`) — `compositeState` forces `idle` for it, and for a -restart pause, unless a turn is actually running, which reads `busy`: forced -idle must never mask a live turn (an ordinary prompt, or the resume prompt -that eventually re-arms the goal, can be streaming while the goal itself -sits parked), whereas provider-backoff's loop is merely waiting and keeps -reading `goal-running` regardless of whether a turn happens to be running. -Resume needs no new machinery: the existing activity-driven -`maybeAutoArmGoal` re-arms any active goal — parked or not — the next time an -ordinary prompt turn completes, resetting the `worker_failure` presentation; -`runGoal`'s own tail deliberately never auto-arms (the same anti-churn -property that already stops a freshly-parked goal from immediately -respawning a loop against an empty queue). - -`harness serve` can also make this turn/goal lifecycle visible on stderr: -`server.Options.Logger`, when set (`cmd/harness/main.go` wires a -`slog.NewJSONHandler(os.Stderr, nil)` logger into it for `serveCmd`), emits a -structured line at every `recordTurnEnd` call (INFO for outcome "completed", -WARN otherwise) and at the `goal.*`/`session.error` durable-record choke -points — a heartbeat for the life of the box instead of logging only at -boot/config/MCP wiring, matching Codex's own structured stream-retry -logging. Nil (the default) disables all of it; every call site nil-guards -first, so an unset Logger is exactly the prior silent behavior. - -A worker-parked goal is also surfaced in-session, model-facing: -`Session.goalParked` (set when a park lands, cleared at every `PursueGoal` -entry) drives a third ambient status segment — alongside the process and MCP -segments — appended to the newest user message of any turn that is NOT -itself one of this loop's own worker turns, naming the classified reason and -stating the goal resumes automatically. It is runtime-only and never -persisted; after a process restart, visibility reverts entirely to the -boot-only `goal.paused`/`pause_reason: "restart"` presentation instead — a -deliberate, documented asymmetry. - -The condition itself is adjustable mid-loop. `Session.UpdateGoal` rewrites an -active goal's condition, journals a durable `goal.updated` record, and emits -`EventGoalUpdated` — same lock-and-emit-under-`s.mu` shape as `RegisterGoal`; -a same-condition update is a silent no-op, updating an inactive goal errors. -`PursueGoal` takes a per-turn snapshot (condition, a runtime-only generation -counter, active) instead of closing over the original parameter, so a live -loop picks up new text at its very next turn boundary — both the worker -directive and the evaluator call. The generation counter guards stale -verdicts: if `UpdateGoal` lands while an evaluator call for generation N is -in flight, a MET (or stalled) verdict for N is discarded on return — no -`goal.achieved`, no `goal.eval`, the loop just continues against the new -condition, never a false-positive completion against text the model never -saw. `ClearGoal` is unaffected — it keys on `goalActive`, not condition -equality, so it still stops the loop at every point it does today. - -A built-in `goal` session tool (gated on `Config.GoalTool`) lets the model -inspect or drive its own goal in-process: no HTTP round-trip, no run-slot -claim. `status` reports `{active, condition}`; `set` arms a new goal via -`RegisterGoal` (errors telling the model to use `adjust` if a goal is already -active); `adjust` rewrites an active goal's condition via `UpdateGoal`. There -is deliberately **no `clear` action** — see below. - -`Config.GoalTool` is on whenever `goal_evaluator_model` is configured, in -`harness run` and `harness serve` alike, entirely independent of the `-goal` -flag — a plain `harness run -p ...` with that config set still registers the -tool. But what happens after `set`/`adjust` differs by host: `harness serve` -auto-arms (see `maybeAutoArmGoal` below) — the loop actually starts running -once the current turn ends. Plain `harness run` (no `-goal`) has no such -auto-arm step: a tool-driven `set` call registers and journals the goal -(`goal.active` becomes true) but nothing ever calls `PursueGoal` for it, so -it never actually starts evaluating — the process runs its one `Prompt` call -and exits with the goal armed but inert. Only `harness run -goal ` -itself drives `PursueGoal` to completion. - -`POST /session/{id}/goal` on a busy session no longer flatly 409s. A running -goal loop updates its condition in place (`status: "updated"`, 200 — no -second loop, no run-slot claim; the loop picks it up at its next turn -boundary). A plain prompt holding the slot with no goal yet active registers -the goal (`RegisterGoal` needs no run slot) and then retries the claim once, -closing the race against that same prompt's own `runPrompt` tail: if the -retry wins the now-freed slot, the loop spawns immediately and the response -reports `status: "started"` (202); otherwise the prompt's tail is still -ahead of us, its own auto-arm check (`maybeAutoArmGoal`) will claim the slot -and spawn the loop itself once that tail finishes, and the response reports -`status: "armed"` (202) — either way the loop starts exactly once, never -zero times, never twice, no further client action needed. This is also how -the `goal` tool's own `set` action takes effect: arming a goal mid-turn, the -same auto-arm path starts the loop the instant the current turn ends. A -workdir held by a genuinely different session still 409s, -unchanged. - -No self-clear is deliberate: a goal-supervised agent must never be able to -cancel its own supervision from inside a running turn, so the `goal` tool -has no `clear` action — `DELETE /session/{id}/goal` remains the only clear -path, and it is operator-only. - -The goal loop is a **plan-artifact-free, gate-free** control loop: it is -`Prompt` plus a read-only evaluator call, with no plan document, no edit/plan -mode, and no permission gate. It does not violate the no-plan-mode decision -below. - -### Session metadata index - -`GET /session` and `GET /session/{id}` do not replay a session journal. Each -session log has a sidecar `.index.json` holding one -`engine.SessionIndex`. The index carries every wire `Session` field with a -durable source: timestamps, model, effort, workdir, parent session, task -lineage, message count, usage, durable goal state, queue depth, and -compaction counters. - -Before the index, a read of a non-live session called `LoadSession`. That -call decodes every message body and rebuilds the whole history. The handler -then reported a dozen scalars and dropped the rest. The list endpoint paid -that cost once per non-live session (meetneptune/boxes -`docs/design/console-read-path.md`, workstream 1). - -The index is a fold of the journal (`engine/index.go`). Three rules keep it -honest. - -**It folds records, not memory.** `Session.writeRecord` folds every record -it appends. `ensureLog` folds the header records it writes directly, which -is the one write that does not pass `writeRecord`. Memory is never the -source: `EnqueuePromptDurable` writes its record before it mutates the -queue, so a fold of memory at that instant would disagree with the log it -claims to summarize. - -**It is a cache, never an authority.** `ReadSessionIndex` serves a stored -index only when three checks pass: a checksum over the stored bytes, the -journal's byte length, and the journal's modification time. Anything else -refolds from byte 0. There is no repair path, so no repair path can be -wrong. The checksum catches a torn read — both writers replace the sidecar -in place, and a reader in another process can otherwise mix old bytes with -new ones that parse. Length and modification time are a staleness key, not a -proof: they rest on the journal's own contract of one writer and append-only -writes. - -**It reports what a full load reports.** `Messages` and `LastActivityAt` run -through `message.ResolveOrphanToolCalls`, over a skeleton of ids, roles, and -tool-call ids. A crash between a tool call and its result therefore counts -the same on both paths. `DurableMessages` counts the records alone, for a -reader that must map a message to a byte offset; paginated message reads are -numbered against it. - -Some journals cannot be answered by a fold at all. A legacy header records -no workdir, and a crash can tear away the initial model record. `LoadSession` -answers those from the loading `Config`; a fold has no `Config`. -`SessionIndex.Complete` reports the difference, and -`Server.coldSessionJSON` uses the authoritative load path for such a -session. - -Three folds have real state machines. Each has one implementation, shared by -`LoadSession` and the index: `applyCompactRecord` (`compact.go`), -`applyGoalRecord` (`store.go`), and `promptQueueFold` (`queue.go`). -`engine/index_test.go`'s oracle test pins every index field against a full -`LoadSession`. - -A refold is far cheaper than a full `LoadSession`, and a current index is -cheaper again. Write-through costs one marshal and one rewrite per record. -The sidecar never gets an `fsync`: losing it in a crash costs one refold. - -`server.Options.Plugins` supplies a session's plugins to a cold read. -Plugins are process configuration, not durable session state, and a cold -read has no `Session` to ask. - -### Journal snapshotting - -`LoadSession` does not always replay a whole journal. Beside the log sits -`.snap`, a **seq-anchored checkpoint**: the fold-produced state as of -journal line N, plus a CRC-32 and a format version. Recovery loads the -snapshot, applies the session HEADER record (line 1, always — it is what -carries `created_at`, workdir, and task lineage, whose restore rules turn on -absent-versus-present and are not worth reproducing twice), and then replays -only lines `> N`. The tail scan reads through `scanLogRaw`, so a covered -record is never DECODED: skipping the decode of a message record's whole -part tree is where the saving is. Design: -docs/design/journal-snapshotting.md. - -The snapshot schema is EXPLICIT (`sessionSnapshot`, `engine/snapshot.go`), -never `json.Marshal` of a `*Session`: every field but `ID` is unexported, -and the config carries live callbacks, a `SessionManager` pointer, and open -file handles that must never be serialized. The rule for what belongs in it -is **exactly what the folds reconstruct** — a fold added without a matching -snapshot field is silently dropped, which is what -`TestSnapshotCarriesEveryFoldedField` and the replay-equivalence tests pin. -Two exclusions are deliberate: header-derived state (replayed instead), and -`turn`/`lastSystem`, which have NO durable source at all — capturing them -would make a snapshot-loaded session disagree with a full replay of the same -journal and make an observable field depend on whether a snapshot happened -to exist. - -The invariant is `state(snapshot@N) + replay(N+1..head) ≡ -full-replay(0..head)`. Every doubt falls back to a full replay: no snapshot, -a torn one, a checksum mismatch, a wrong version, another session's id, a -`seq` ahead of the journal head, or a journal whose first record is not a -session header. **Slower, never wrong.** The journal is never truncated; -snapshots are derived and can be deleted at any time. - -**The trigger is at the APPEND BOUNDARY, never inside `writeRecord`.** A -snapshot pairs a memory image with a journal position, and inside -`writeRecord` the two do not yet agree: some callers persist their record -BEFORE applying their own memory mutation (`EnqueuePromptDurable`, -deliberately), so a capture there would anchor past a record whose effect -memory has not applied — and the reload would skip that record and lose the -effect forever. `appendWithUsage` (after `persistMessage`, still under -`s.mu`) and the on-idle trigger (`runAgenticLoop`'s defer, and -`ReleaseFiles` on eviction) are the two boundaries where the caller has -completed both halves. - -The OPPOSITE direction is guarded by `snapshotSafeLocked`: -`SessionManager` splits some mutations into an in-memory half and a -DEFERRED durable half (`appendMemoryOnly`/`persistAppendedMessage`, -`enqueueTaskNotificationMemoryOnly*`/`persistQueuedTaskNotification`, -`queueRecordDeferredLocked`), so memory can be AHEAD of the journal. A -snapshot taken in that window carries the mutation AND leaves its record in -the tail, and the reload applies it twice — a duplicated message, or a -child-completion notification the parent renders to the model twice. -`Session.durableDebt` counts the outstanding halves; a capture is refused -while any is open, which merely postpones the snapshot to the next -boundary. The debt clamps at zero: a leaked increment stops this session -snapshotting (degrading to today's full replay), where a negative count -would ARM a capture in exactly the unsafe window. - -`Config.SnapshotEveryRecords` is the cadence. Zero — the engine zero value — -disables snapshot WRITING, so a bare embedder-built `engine.Config` keeps -the pre-snapshot behavior; the config/CLI layer supplies the product default -of 64 (`snapshot_every_records`), the same unset-versus-explicit-zero split -`prompt_retries` uses. READING a snapshot is never gated on it: recovery is -a property of the files on disk, not of the loading Config. A snapshot write -is background, coalesced (one in flight per session), and atomic (temp → -fsync → rename), and its failure lands in `lastSnapshotErr`, never -`lastPersistErr` — a snapshot is derived acceleration, not a durability -promise. - -A load that takes the snapshot path marks the metadata-index fold BROKEN -rather than building a partial one: the index summarizes EVERY record, and -this load deliberately did not see most of them. The index is a cache with -no repair path, so a reader that finds none refolds and `ensureLog` re-seeds -the fold from the journal on this session's next write. Snapshotting the -index fold itself is the obvious follow-up; nothing may guess at it. - -### Paginated message reads - -`GET /session/{id}/message?before_seq=N&limit=K` answers one bounded page of -a session's messages, read from the journal's tail. The unparameterized call -is unchanged, byte for byte: no `before_seq` and no `limit` still returns the -bare array of the whole history every existing caller expects. A request that -names either parameter gets a `MessagePage` envelope instead — `messages`, -`first_seq`, `last_seq`, `total`, `has_more` — because a client that pages -needs the page's position and a client that does not must not have to learn a -new shape. A console loads the tail and pages older messages in on scroll -(meetneptune/boxes `docs/design/console-read-path.md`, workstream 2 and -directive 1). Before this, every console open transferred the whole -transcript. - -**Seq is an ordinal over the DURABLE message sequence**: message records in -log order, with each compact record's fold applied (the folded range replaced -by that record's summary). `SessionIndex.DurableMessages` counts that same -sequence, so the newest message's seq equals it. `SessionIndex.Messages` can -be larger — it also counts the synthetic tool results -`message.ResolveOrphanToolCalls` derives — and a derived message has no -record, so it has no seq and no page carries it. This definition is what -makes a bounded read possible: an ordinal over durable records can be counted -backwards from the end of a file, while an ordinal over a materialized -history cannot be known without materializing it. - -`engine.ReadMessagePage` (`engine/messagepage.go`) serves a page two ways, -and both are numbered by the same index: - -- The **tail walk** reads `revChunkBytes` blocks backwards from - `SessionIndex.LogSize` and numbers message records down from the total. It - touches only the tail however long the journal is. It gives up the moment - it meets a compact record, because the messages a fold KEPT sit in the log - between the folded range and the compact record itself — undoing that - backwards would be a second, subtly different implementation of a fold. -- The **fold path** then reuses `indexFold` — the same forward fold and the - same compact-range occurrence selection — to learn which messages occupy - the requested seqs. `indexFold` carries each surviving message's journal - record ordinal beside its skeleton; `foldedPage` reads back by that ordinal, - never by message ID alone. This matters for hand-written or externally - assembled journals that repeat an ID: one occurrence can be compacted away - while another survives, and the surviving sequence entry must decode the - surviving record. The path costs one slim pass (ids, roles, and ordinals, - never message bodies), which is still three orders of magnitude below - materializing the history. - -The scan is bounded by `SessionIndex.LogSize`, never the file's current size, -so a turn appending records while a page is read cannot renumber that page -under it. - -Two properties are deliberate. A page carries durable messages **verbatim**: -it never runs `message.ResolveOrphanToolCalls`. That repair exists to keep a -provider REQUEST valid, this endpoint builds no request, and fabricating a -tool failure in a read view has real production history (see `Server.lookup`'s -doc comment: a healthy child's in-flight tool call rendered as failed in the -console for as long as it kept running). And compaction **renumbers** — a fold -replaces N messages with one summary, so every later seq shifts down by N-1. -A client paging across a compaction can see one page overlap another; message -ids are stable and are the way to de-duplicate. - -A page is read from the durable records even when the session is resident, so -one seq definition covers both cases: a resident history can carry messages -the log does not (load-time repairs, recovery's memory-only closers), and -numbering those would give one message two different seqs depending on -residency. A session with no journal at all — created through the API and -never prompted — falls back to its resident history, which for such a session -IS the durable sequence. - -### Prompt queue - -`POST /session/{id}/prompt_async` against a session already busy (another -prompt, or a running goal loop) no longer 409s — it queues. The prompt is -enqueued durably (`engine.Session.EnqueuePrompt`, persisting a `prompt.queued` -record and assigning a session-monotonic ID) synchronously, before any -response is written — the same enqueue-durable-before-202 shape `RegisterGoal` -already uses for goals, closing the accept-vs-lose race structurally. The -response is 202 either way: `status: "started"` when a turn is now running for -this request's own prompt (an idle claim against an EMPTY queue, or a -freed-slot retry that happens to win and dispatch this same prompt), or -`status: "queued"` (carrying the current depth) when it is durably waiting — -including the idle-claim case where the queue is already non-empty (a -restart refold, or any other drain gap that ever left a prompt stranded): -`handlePrompt` enqueues the incoming text behind whatever is already waiting, -then dispatches the queue's HEAD — not necessarily this request's own text — -into the run slot it just claimed, so a fresh arrival can never cut the -line ahead of prompts already queued. The workdir-held-by-another-session 409 -is unchanged — only same-session busy gets queue semantics. - -The queue drains FIFO, by queue ID, at every run-slot release, with no -exceptions: `runPrompt`'s, `runGoal`'s, and `handleCompact`'s tails all call -`maybeDispatchQueued`, which claims the freed slot, dequeues the head -(`reason: "delivered"`), and spawns it as a normal prompt turn — whose own -tail repeats the check, so the whole queue drains one turn at a time before -anything else gets a look. `handlePrompt`'s own claim-success path (previous -paragraph) is the one non-tail drain site: an admission-time head-dispatch -for the idle-with-non-empty-queue case, closing the gap a tail-only drain -would otherwise leave open between "session goes idle with a queue still -non-empty" and "the next prompt/goal/compact activity happens to touch it." -This is also where -**queue beats goal auto-arm**: `runPrompt`'s and `handleCompact`'s tails call -`maybeDispatchQueued` *before* `maybeAutoArmGoal` (see above), so a prompt -sitting in the queue when a turn or a compact call ends is dispatched first — -direct user input outranks the background objective — and the goal only -auto-arms once the queue is empty. - -**Delivery granularity is per tool-call boundary, not per turn.** Inside -`Session.Prompt`'s agentic loop (`engine/engine.go`), the instant a -tool-result message is appended — after the model made one or more tool -calls and before the next provider request in that SAME turn — the loop -drains the ENTIRE queue, FIFO, in one locked op (`DequeueAllPrompts -("injected")`) and appends the drained batch as a single, durable user -message: the same labeled "OPERATOR MESSAGES" block template -(`operatorMessagesBlock`, `engine/queue.go`, shared by every drain site so a -batch renders identically apart from one parameterized word — this -call site passes `operatorContextTask`, so its header says "continue the -task", never "continue the goal", even when this drain happens to fire -inside a goal loop's worker turn; only goal.go's own turn-boundary drain -below passes `operatorContextGoal`). This only ever -APPENDS — never rewrites an earlier message — so a provider's prompt-cache -prefix stays intact, the same principle the managed-processes ephemeral -status block below relies on, except this message is a REAL, durable -delivery, not a disposable status line. A turn that ends WITHOUT any tool -call never reaches this drain point at all (the model's own end-of-turn -return precedes it), so that path — and anything still queued when it -happens — is left entirely to the mechanisms below. Because `PursueGoal`'s -worker turns run through this exact same `Prompt` loop -(`promptTurnWithRetry`), goal loops inherit tool-call-boundary injection -automatically, with no separate wiring: a prompt queued while a goal's -worker turn is mid-tool-call is delivered inside that SAME worker turn — -matching Claude Code's mid-turn steering granularity — rather than waiting -for the goal's own turn boundary described next. - -`PursueGoal` keeps a second, complementary drain at its own turn boundary: -at the top of every turn (the same `snapshotGoal` boundary #77's -condition-update snapshot uses, and before that turn's own tool-call-boundary -drain above has any chance to run) it drains the *entire* queue, FIFO — -catching anything still queued from a turn that made no tool calls at all, or -that arrived in the gap between one turn ending and the next one's snapshot — -and prepends it to that turn's directive as the same labeled "OPERATOR -MESSAGES" block (`operatorMessagesBlock`, `operatorContextGoal` — so its -header says "continue the goal"), ahead of — never replacing — the -ordinary condition/guidance text. The evaluator's condition string is -unchanged by this — it is built from the condition alone, never from the -block or the turn's rendered directive — so goal injection judges only the -goal there; the evaluator's separate transcript field does render the full -history, so it does see the block once the worker turn that received it has -run. Every drained prompt journals its own `prompt.dequeued(injected)` record -before the turn's directive is even built, so it counts as delivered at that -point even if the turn's outcome later turns out stale and gets discarded — -an injected prompt is never re-queued, at either drain site. This means an -abort (`POST /abort`) or a goal clear (`DELETE /session/{id}/goal`) racing a -goal turn boundary consumes an entire just-injected batch at once: every -prompt the boundary drained is already journaled `dequeued(injected)` before -the worker turn even starts, so a turn that gets cancelled or whose outcome is -later discarded as stale still loses all of them together — several operator -messages, not just one — the same exposure class an ordinary in-flight prompt -already has, just multiplied across the whole drained batch. The two drain -sites can never double-deliver the same prompt: `DequeueAllPrompts` is one -atomic, locked pop of the whole queue, so whichever site runs first against a -given prompt is the only one that ever sees it. - -Every enqueue/dequeue is a durable record — `prompt.queued` and -`prompt.dequeued`, the latter carrying a `reason` of `"delivered"` (idle -drain), `"injected"` (tool-call-boundary or goal-turn-boundary injection — -both drain sites share the reason, see above), or `"cleared"` (see below) — -journaled and emitted (`EventPromptQueued`/`EventPromptDequeued`) under -`s.mu` in the same critical section, mirroring `RegisterGoal`/`ClearGoal` -exactly. Dequeue always journals *before* the text enters any turn, so a crash -between that journal write and the dispatched turn's completion cannot -double-deliver — the prompt is simply gone from the queue on replay, the same -exposure any in-flight prompt already has today. **Boot never auto-dispatches -a resumed queue**: `LoadSession` folds `prompt.queued`/`prompt.dequeued` -records back into the exact undelivered set, `GET /session`'s `queued` count -reflects it immediately, and it sits there until the next natural drain -trigger (an idle prompt, the next tool-call boundary inside a running turn, or -a goal loop's next turn boundary) — the same settled boot rule goals follow. -`DELETE /session/{id}/queue` is the one explicit clear surface: it journals -`prompt.dequeued(cleared)` for every pending item then 204, idempotent on an -empty queue, and never touches a currently running turn — `POST /abort` is -unrelated and does not touch the queue either way (it only cancels the -in-flight turn's context). - -Two v1 limits are deliberate, not gaps: **text-only** (queued prompts carry a -plain string — `QueuedPrompt{ID, Text}` — no attachment machinery, matching -the plain-prompt contract's `parts` being text-only already), and **a -per-request `model` override is silently dropped when the prompt is queued** -— there is no slot in `QueuedPrompt` to carry it through to a future drain, so -a caller that needs a model swap to take effect must re-issue the request once -it is confirmed `started`. - -`POST /session/{id}/enqueue` (docs/plans/2026-07-21-durable-enqueue.md) is -`prompt_async`'s durable, idempotent sibling for a caller whose own upstream -ack rides on this call succeeding — an inbox poller or coordinator relay, -not an interactive client. `Session.EnqueuePromptDurable` extends -`EnqueuePrompt` with three properties the plain path deliberately lacks: -write-ahead durability (the `prompt.queued` record is written and, in the -default `session_sync: "fsync"` mode, *fsynced* before any in-memory -mutation or response, so a 2xx is an honest attestation rather than a -best-effort ack — a write/fsync failure returns 500 "enqueue not durable" -instead of the swallowed `lastPersistErr` every other persist path uses), a -caller-issued session-monotonic `seq` deduplicated against a durable -high-water mark (`Session.EnqueueSeq()`, journaled on the record and -rebuilt by `LoadSession` — a seq at or below the mark is a clean 200 -`duplicate` no-op, so retries are always safe, including across a process -restart), and torn-write healing (a burned-but-failed queue ID is never -reused, and replay folds same-seq records last-writer-wins). Delivery is the -exact same FIFO/tool-boundary/goal-boundary machinery described above — this -is a new *acceptance* contract, not a new delivery path: durable means -accepted into the queue, and delivery-out is still the queue's normal -at-most-once-per-dequeue machinery, so a crash between dequeue and turn -completion loses that delivery once rather than redelivering it, exactly -like any in-flight prompt (`maybeDispatchQueued`'s "No-double-delivery -equivalence", invariant 7, in server/handlers.go). `GET -/session/{id}/queue` is the paired reconciliation read: the watermark plus -the pending queue (FIFO, `seq` present only on durable-enqueue entries), for -an upstream recovering from its own crash to check what's already inside the -durability domain instead of re-sending blind. `prompt_async` remains the -right choice for an interactive client that has no upstream ack to protect — -it is not going away, and `POST /session/{id}/enqueue` adds no new limits -beyond what queued prompts already have (text-only, no model override). - -The `fsync` in "write-ahead durability" above is itself mode-selectable: -config's `session_sync` ("fsync", the default, or "volume") gates both this -durable-enqueue fsync and the one-time session-create directory fsync -(`ensureLog`'s fresh-file `syncDir` call, store.go) — nothing else changes. -"volume" is for a session store on a continuously-synced network volume -whose own commit layer is the documented durability boundary: fsync adds no -durability there, and some FUSE/9p transports deadlock permanently on it -(`fsync(dirfd)` especially — a wedge that hangs every later file op on the -mount, not just the one call). In that mode the write(2) landing out of -`EnqueuePromptDurable`/`ensureLog` is itself the attestation; the write -ordering, torn-write healing, and replay/fold logic above are byte-for-byte -identical in both modes — a volume can still lose an unsynced tail on abrupt -death exactly like a torn fsync can, and the same last-writer-wins fold -repairs both. See docs/deploy-modal.md for the recommended setting on Modal -Volume v2 deployments. - -### Managed processes - -`config.Config.Processes` (`processes` in JSON) declares named long-lived -dev/support processes (`pnpm dev`, a local DB) that a `process` session -tool can start/stop/restart/inspect without an agent reinventing PID -tracking. `*process.Manager` (package `process`, not `engine`) is a -box-scoped singleton — built once per harness process and shared across -every session, exactly like `engine.MCPManager` — with a -starting/ready/running/exited/stopped state machine, unix process-GROUP -kill on stop (mirroring `engine/bash_unix.go`'s Setpgid/kill-pgroup/ -WaitDelay pattern), and asynchronous death detection (a waiter goroutine -flips state to `exited` with no client asking). Logs land at -`/.harness/proc/.log`. - -The tool can also `declare`/`undeclare` NEW process definitions at -runtime (server-lifetime only, never written to `.harness.json`) — see -`docs/design/managed-processes.md` for the full validation and origin -(`config` vs `runtime`) rules. `harness serve` always builds a -`*process.Manager`, even with zero configured processes, so the tool is -present on every served box; `harness run` keeps the zero-cost-when- -unconfigured rule. - -Once at least one declared process has EVER been started (this server -process's lifetime), request assembly appends an ephemeral `[processes: -...]` status block to the newest user message ONLY — as a -`message.EngineContext` part (see "Ambient engine context is a structured, -unforgeable part" above), never persisted into the durable session log, -never touching any earlier message so a provider's prompt cache prefix -stays intact. See `docs/design/managed-processes.md` §4 for the exact -mechanism and why it is safe. - -### Model switching - -`Session.SetModel` swaps the MAIN session model for later requests. History -transcodes from scratch every request, so there is no migration step. Three -routes reach `SetModel`: the built-in `model` session tool, a per-request -`prompt_async` model override, and `POST /session/{id}/model`. - -`SetModel` is the single event choke point. On a real change (never a no-op -set to the current model) it persists the durable `recModel` resume record -AND emits `EventModelChanged` (carrying the new model), both while holding -`s.mu` — the same persist-and-emit-under-`s.mu` shape `RegisterGoal` uses. -The server's `Publish` maps `EventModelChanged` to the durable `model` -journal record. Every swap route funnels through this ONE emit, so a swap -journals exactly once — the handlers never emit `model` themselves. `recModel` -is the resume record `LoadSession` restores; `EventModelChanged` is the -observability event. They are separate and both fire on one swap. - -The `model` session tool (gated on `Config.ModelTool`) has two actions: -`status` reports the current model, the configured aliases, and the configured -provider names; `set {model}` resolves a one-level alias (from -`Config.ModelAliases`, which mirrors `config.Aliases` — the engine never -imports config), parses the ref, VALIDATES the provider is configured -(`s.cfg.Providers.For`), then calls `SetModel`. A `set` to an unconfigured -provider returns a tool error listing the valid aliases and provider names and -changes nothing. There is deliberately NO `clear` action — a session always -has a model. Scope is the MAIN model only; the goal-evaluator and subagent -models are untouched. - -`Config.ModelTool` is on by default. Config key `model_tool` (a `*bool`, -default true — like `instructions`) lets a host opt OUT; `harness run`, -`harness serve`, and the server `mkCfg` all set it from -`config.ModelToolEnabled()`. This differs from `GoalTool`, which opts IN only -when an evaluator is configured. - -`POST /session/{id}/model` is the network counterpart: a client/dashboard swap -decoupled from prompting, so it never claims the run slot (it applies even -while a turn is running, taking effect on the next request). It validates a -non-empty `{model}` and rejects an unconfigured provider (400), an unknown -session (404), or an empty model (400) — the same validation as the tool — then -calls `SetModel`. Aliases are not resolved at this endpoint; resolve them -client-side, as the CLI does. - -### An unknown model's context window is a refusal, not a shrug - -`modelmeta.ContextWindow` answers "how big is this model's context window". -When it does not recognize a ref, `resolveContextWindow` -(`engine/context_window.go`) used to fold that into the SAME answer a -deliberate opt-out produces: source `disabled`, `compaction_armed=false`, and -a session that started anyway and ran with **no context management at all** — -until it died with "context exhausted" instead of compacting. An unrecognized -model is not a state to degrade into; it is a configuration an operator has to -fix. - -`resolveContextWindow` now REPORTS the miss (an error wrapping -`ErrUnknownContextWindow`, whose text always names the offending ref) instead -of swallowing it, and does not decide what to do about it. -`Config.RequireContextWindow` does, through the single policy point -`requiredContextWindowErr` — so the definition of a miss lives in one place -and the policy lives with the session that must honor it, and the ERROR log -line fires once per miss however it was reached. - -Only a REGISTRY MISS refuses. Four ways to have no window stay legitimate and -silent: an explicit positive `ContextWindowTokens` (naming the window IS the -missing information, so it satisfies the requirement for any model), an -explicit NEGATIVE one (`contextWindowSourceOptOut` — a stated choice, told -apart from `disabled` precisely because `disabled` can mean "unrecognized"), a -ZERO model ref (nothing to look up; the refusal belongs to whatever later -names a model), and a model the registry KNOWS whose window is below -`minAutoContextWindowTokens` (a known model, not a gap). - -The refusal is recorded at the earliest point of use and surfaced everywhere a -model starts being used: `newSession`, `SetModel`, and `LoadSession`'s -post-replay re-derive set `Session.contextWindowErr`; `ContextWindowErr()` lets -a create route refuse before the session is durable or resident; `CheckModel` -is `ModelSupported`'s sibling gate, called at the same three `SetModel` routes -BEFORE the swap so a rejected ref never reaches the durable `recModel` record; -and every `Prompt` returns it before touching history, the provider, or the -instructions read. A RESUME is deliberately not fatal: a session that cannot -load cannot be listed, read, or exported either, and an operator would lose the -transcript along with the ability to fix the config. - -`Config.RequireContextWindow` false — the engine zero value — keeps the -pre-fix behavior, so a bare embedder-built `engine.Config` and every test in -the package are unaffected; the config/CLI layer supplies the product default -of TRUE (`context_window_required`), the same unset-versus-explicit split -`prompt_retries` uses. - -### Reasoning effort - -`message.Effort` is the unified, provider-agnostic reasoning-effort level: -`off`, `minimal`, `low`, `medium`, `high`, plus the zero value `EffortUnset` -(empty string) that sends NO control at all. It rides one `provider.Request` -field (`Request.Effort`), the same way `MaxTokens` does, and each adapter maps -it to that provider's own wire shape at transcode time — so an effort swap, -like a model swap, needs no migration step: - -- `provider/anthropic` enables extended thinking with a `thinking.budget_tokens` - budget (minimal 1024, low 4096, medium 8192, high 16384). The API requires - `max_tokens > budget_tokens` and rejects an explicit `temperature`/`top_p` - while thinking is on, so `transcodeRequest` bumps `max_tokens` above the - budget and drops both. `off`/unset emit no `thinking` block. -- `provider/openai` (Responses) sets `reasoning.effort` to the level string - (minimal/low/medium/high). `off`/unset omit the `reasoning` object. -- `provider/openaicompat` sets the top-level `reasoning_effort` string; a - gateway (Bifrost) maps it to the upstream provider's own knob. A non-off - level sends the level string; `EffortOff` sends the literal string `"off"`, - not an omitted field — several gateway upstreams reason BY DEFAULT when - the field is absent, so omitting it cannot express "disabled." Measured - (2026-08-12): Fireworks kimi-k3 through Bifrost streamed a full reasoning - block (266 chars) with the field absent, and zero reasoning content (0 - chars, 8 vs 133 completion tokens) with the literal `"off"` sent. Only - `EffortUnset` omits the field, leaving the gateway/model default in force. - It surfaces returned reasoning from EITHER wire field — Bifrost/DeepSeek - `reasoning_content` or OpenRouter `reasoning` — as a `Reasoning` part; a - gateway sends one field, never both. - -`Effort` does NOT police which model accepts which level — that is a -provider-and-model fact the engine cannot know from the ref alone. The adapter -sends the requested level and the provider is the final judge. A caller that -must gate levels per model (a dashboard picker) holds its OWN mapping. - -**Downgrade strip — DELIBERATELY asymmetric between the two reasoning -adapters.** A stored thinking block (anthropic) or reasoning item (openai -Responses) can be a transcode-time destructive drop (throwaway request, intact -record); a later reasoning-ON turn replays the part from the unchanged history. -A strip is ever needed because a stored block shipped while the request omits -the reasoning control can be rejected, and durable in history it 400s every -later turn — a permanent wedge. But WHEN each adapter strips differs, because -the two providers default differently: - -- `provider/anthropic` strips whenever the request enables no reasoning - (`off`/unset, or a swap to a non-reasoning model). This is safe: Claude emits - NO thinking block unless the control is sent, so an unset turn carries none - to preserve. -- `provider/openai` (Responses) strips ONLY on an EXPLICIT `off` (a genuine - "reasoning disabled" intent), NEVER on `EffortUnset`. OpenAI reasoning models - (gpt-5) reason BY DEFAULT, so an unset turn — the default of every `harness - run`/`serve` session, since nothing sets `Config.Effort` — still produces - encrypted reasoning items, and those items are REQUIRED for stateless - (`Store:false`) multi-turn tool use. Stripping them on unset wedged every - turn-2+ gpt-5 tool continuation; an unset session now replays them exactly as - every pre-effort-control build did (`stripReasoning` in - `provider/openai/transcode.go`, gated on `req.Effort == EffortOff`). So - `unset != off` here — do NOT re-fold the openai strip back onto - `!Reasoning()`. (Regression: NEP-5272 review of PR #117.) One residual the - off-only strip cannot enforce: a `SetModel` swap to a NON-reasoning openai - model (gpt-5 -> gpt-4o) at unset effort still replays the stored items — the - same per-model gating punt the enable direction has, so the caller (a - dashboard picker) clears/re-validates effort on a model swap, NOT this - transcoder. - -The reverse (ENABLE) direction — turning reasoning ON over a prior tool_use -that lacks a thinking block — stays a documented limitation, since a signed -thinking block cannot be synthesized (see `provider/anthropic/ -transcode.go`). - -`Session.SetEffort` is the single event choke point, mirroring `SetModel` -exactly. On a real change (never a no-op set to the current level) it persists -the durable `recEffort` resume record AND emits `EventEffortChanged`, both under -`s.mu`. The server's `Publish` maps `EventEffortChanged` to the durable `effort` -journal record. That record ALWAYS carries the `effort` field, even on a clear: -`server/journal.go`'s `Event.Effort` is a `*message.Effort` (the same -explicit-zero-vs-absent pattern `QueueLen` uses), so a clear to `EffortUnset` -renders as an explicit `"effort":""`, never a dropped key — "cleared to the -provider default" stays byte-distinguishable from a malformed record. -`LoadSession` restores the level: the create-time level rides -the session header record, and every later `SetEffort` writes a `recEffort` -record. `Session.Effort()` reads it back. - -`POST /session/{id}/thinking` is the network counterpart: a client/dashboard -swap decoupled from prompting, so it never claims the run slot. It validates the -`{effort}` value with `message.ParseEffort` (400 on an unknown level), accepts -an empty string as "clear to provider default", and rejects an unknown session -(404). Unlike the model endpoint it has NO provider gate (see above). The -current level is read back on `GET /session/{id}` (`effort`), the same way the -current model is. - -**Effort at the three request-build sites is NOT uniform, by design.** The -main turn (`streamTurn`, `engine/engine.go`) sends `s.Effort()` — the -session's current level, read fresh every request. The two internal -tool-less calls diverge from that and from each other (issue #124): the -goal-loop evaluator (`runEvaluator`, `engine/goal.go`) always pins -`EffortOff` — see "Goal loop" above — because it is a classifier the model -must answer in one line, and reasoning-by-default gateway models can burn -its 256-token budget before ever emitting a verdict. The compaction -summarizer (`runCompactionSummary`, `engine/compact.go`) instead inherits -`s.Effort()`, the same as the main turn, because summarization is a real -writing task that benefits from the session's own quality setting; -`EffortUnset` stays `EffortUnset` there. Do not fold these two internal -sites onto one shared rule — one is a classifier, the other is prose. -Known residual (not addressed by issue #124, filed as issue #126): a -non-off session effort can raise the summarizer's effective output cap -above `compactionMaxTokens` (the anthropic and openai adapters both bump -the cap for reasoning — up to ~20480 tokens at `EffortHigh`, versus the -documented 1024 cap), and openaicompat applies no cap floor at all, so a -reasoning-heavy summary can truncate silently — `runCompactionSummary` has -no `StopReason` guard to catch it. A raised cap also delivers less context -reduction from this call, at the layer whose own failure runs to a hard -overflow that clears an active goal. A second, related residual (issue -#127): the summarizer sends folded history containing `ToolCall` parts -from turns that ran with no thinking block, and a non-off level here -enables thinking over that same history — the documented ENABLE-direction -"thinking blocks expected before tool_use" reject case, just reached from -compaction instead of a live turn. - -**The summarization request always ends in a trailing `RoleUser` message, -never the folded range's own last message verbatim** (2026-08-19 incident, -session `ses_jumpy-pizza`). `foldEnd` (`Session.Compact`) is the last -message before the next KEPT turn's leading `RoleUser` message — ordinarily -that folded turn's own final assistant reply, `RoleAssistant` — so sending -`folded` as `req.Messages` verbatim ordinarily ends the wire request in an -assistant-role message, which the Anthropic Messages API treats as -assistant message prefill; some models reject prefill outright (400 -`invalid_request_error`, "This model does not support assistant message -prefill. The conversation must end with a user message."). `runCompactionSummary` -builds its request via `compactionRequestMessages`, which appends one -trailing `RoleUser` instruction message (`compactionInstructionText`) after -`folded`, unconditionally — never a conditional check on the folded range's -last role, since a `RoleTool` message (a `message.ResolveOrphanToolCalls` -synthetic repair, or an ordinary tool result) also wire-transcodes to -Anthropic's `"user"` role and would otherwise mask the same bug depending on -where a fold boundary happens to land, exactly as it did live (`keep_turns=8` -happened to succeed on the same session where `keep_turns=20` failed). - -**An empty summary is a graceful no-op, never an error surfaced to the -caller.** A summarization call that completes without a transport/stream -error but returns no usable text (`errEmptyCompactionSummary`) is reported -by `Session.Compact` as the same `TurnsFolded == 0` "nothing worth folding" -shape the too-few-turns case above already uses — no history mutation, no -journal write, no error returned — though `EventCompactionFailed` still -fires so the attempt stays visible to anything tailing events, and the -call's real usage is still accumulated into cumulative `Usage()` (it was a -billed call even though it produced nothing — this accumulation is -live-only, not journaled, since no compact record exists for a skipped -fold). Before ever calling the provider, `Compact` also skips a fold range -whose entire content is a single earlier compaction's own summary message -(`isLoneExistingSummary`): re-summarizing an already-compressed summary with -nothing new alongside it has nothing to gain, and was the live incident's -concrete trigger (a small `keep_turns` landed a fold range dominated by a -prior summary). Do not conflate this with a REAL summarization failure -(rate limit, transient 5xx, a truncated stream, a range too large to -summarize) — those still abort with an error, per §2 "Failure handling" in -`docs/design/context-compaction.md`. - -`CompactResult.SkipReason` names WHICH of the three `TurnsFolded == 0` -shapes occurred (`SkipReasonNotEnoughTurns`, `SkipReasonLoneExistingSummary`, -`SkipReasonSummarizerEmpty`) — they used to be wire-identical, which hid two -real defects (review follow-up on PR #136, Findings A/B/C, fixed before -merge): - -- **Hysteresis must latch on `SkipReasonSummarizerEmpty`, never on the two - free skip reasons.** `maybeAutoCompact` only armed its churn-guard - hysteresis when `TurnsFolded > 0`. A summarizer that always returns empty - therefore never latched it: every subsequent over-threshold turn - re-triggered a full, billed summarization call, indefinitely, at full - input price — the "free" no-op was actually a recurring-spend bug - (Finding A). It now also latches when `SkipReason == - SkipReasonSummarizerEmpty`, since that reason DID cost a call; it must - still NOT latch on `SkipReasonNotEnoughTurns`/`SkipReasonLoneExisting - Summary` — both are free, and latching there would permanently disarm - compaction for an over-threshold session that simply lacks enough turns - yet, since the guard only clears once `LastUsage()` dips back under - threshold. -- **`isLoneExistingSummary` gates on the summary message's `ID`, never on - `CompactionSummaryBanner`'s text.** The banner is a display convention; a - user-typed or pasted message that happens to start with the exact banner - string is a genuine turn with real content, not a lone existing summary — - matching on text alone false-positived on it, skipped it forever without - ever calling the provider, and under the automatic trigger the session - never compacted again (Finding B). Every compaction summary's `ID` is now - minted with the `cmpsum_` prefix (`compactionSummaryIDTag`) instead of the - ordinary `msg_` prefix every other message gets, and `isCompactionSummaryID` - tests exactly that prefix — a structural, unforgeable marker of - compaction origin, the same pattern `message.IsSyntheticOrphanID` already - establishes for a different synthetic-message kind. No text-based - fallback exists for a summary minted by an earlier pre-fix build of this - same PR (still `msg_`-prefixed): the miss is bounded and self-healing — - `Compact` just re-summarizes that one old-style range like any other real - content, and the fresh summary it produces carries the new ID tag from - then on. -- **The `skip_reason` field on `POST /session/{id}/compact`'s response** - (`compactResponseJSON`, `server/handlers.go`) surfaces - `CompactResult.SkipReason` directly, `omitempty` (absent on a real fold) — - see `docs/design/context-compaction.md` §1 for the wire shape (Finding - C). - -### Session affinity (prompt-cache routing hint) - -`provider.Request.SessionKey` carries a stable, opaque session identifier on -every request the engine builds — one field on the same per-request struct -`Effort` rides, though unlike `Effort` (set at one call site), the engine -sets `SessionKey` to `Session.ID` at all three request-build sites: -`streamTurn` (`engine/engine.go`, the main turn), `runEvaluator` -(`engine/goal.go`, the goal-loop evaluator), and `runCompactionSummary` -(`engine/compact.go`, the compaction summarizer). The field itself is never -persisted; the value it carries (`Session.ID`) already is, as the session's -own identity. - -Two adapters forward it, each to its own field, because each provider -documents its own affinity hint: - -- `provider/openaicompat` sets the wire top-level `user` field. This is a - generic chat-completions gateway adapter (fronting Bifrost, OpenRouter, - and similar); `user` is the field a Fireworks-style backend behind such a - gateway reads for routing. OpenAI itself has deprecated `user` on its own - API in favor of `prompt_cache_key`/`safety_identifier` (see the next - bullet), but that deprecation is OpenAI's, not the gateway's: the - openaicompat route keeps sending `user` because `user` is the field the - measured Bifrost/Fireworks path above actually reads. Do not "fix" this - adapter by swapping in `prompt_cache_key` — that field is specific to - OpenAI's own API, and the openaicompat adapter targets non-OpenAI - backends behind a gateway, whose measured path reads `user`. Swapping it - would silently drop the measured cache-affinity win. The adapter now sends - `prompt_cache_key` ALONGSIDE `user`, set from the same `SessionKey`: a - gateway fronts several upstream shapes, and an OpenAI-shaped upstream - behind it reads `prompt_cache_key` while the measured Fireworks path reads - `user`. Both fields carry the identical value, one extra field costs - nothing, and an upstream that knows neither ignores both. Add, never swap - — the rule above still binds. Config key `no_prompt_cache_key` on an - `openai-compat` providers entry suppresses that ONE field for a strict - self-hosted upstream that rejects an unknown top-level parameter; `user` - keeps carrying the session key, so the opt-out never costs the measured - affinity win. It is rejected on any entry that is not `openai-compat` — - the native openai adapter always sends `prompt_cache_key`, its own - documented field. -- `provider/openai` (Responses API) sets the wire top-level - `prompt_cache_key` field — the Responses API's own documented routing/ - cache-affinity hint, distinct from `user`. OpenAI combines it with the - request's prefix hash to raise the chance repeat requests land on the - same cache-holding backend. - -Both follow the same omit-on-empty rule: a non-empty `SessionKey` sets the -field; an empty key omits it entirely, never an empty string. -`provider/anthropic` ignores `SessionKey` — it already uses explicit -`cache_control` markers, so a routing hint would add nothing; a live probe -through Bifrost (2026-08-12) confirmed a 41k-token cache write followed by a -41k-token cache read on the very next turn with no `SessionKey` involved. - -The reason `SessionKey` exists at all is measured, not theoretical: Fireworks -serverless prompt caching is prefix-based, automatic, and PER-REPLICA. -Without a routing hint, a re-sent request can land on a different replica -and miss its own prefix cache. A live probe through Bifrost (2026-08-12) -sent a byte-identical 150k-token prompt twice: with no `user` field, the -second call still read `cached_tokens=0` at 10.8s time-to-first-token; with -a stable `user` field, the second call read `cached_tokens=150,300` at 2.8s -time-to-first-token, through the same gateway. Harness sessions re-send the -whole history every request (stateless transcoding), so a long session on -the openaicompat route (a gateway to Fireworks kimi-k3 and similar models) -pays full prefill on nearly every turn without this hint. - -### Anthropic cache TTL (default 1 hour) - -`provider/anthropic` marks two prompt-cache breakpoints on every request — -the last system block and the last content block of the final message — and -never stores a marker in the session log (`transcodeRequest`, -`provider/anthropic/transcode.go`). The marker's TTL defaults to the -EXTENDED 1-hour cache, not the API's own 5-minute default. - -This is an opt-OUT default, and it changes the wire for an operator who -configures nothing: every anthropic request carries the beta header and -writes 1h entries. Two deployments must know it. A proxy that rejects an -unknown `anthropic-beta` value fails every request, and a workload of short -one-shot sessions pays the 2x incremental write premium with no later turn -to read the entry back. Both set `cache_ttl: "5m"`, which restores the -previous bytes exactly. - -`Client.CacheTTL` selects it: `"5m"`, `"1h"`, or empty for -`DefaultCacheTTL` (`"1h"`). Config key `cache_ttl` on the NATIVE `anthropic` -providers entry sets it, and `cmd/harness`'s `registry` passes it to the -client. The value is validated twice, and both checks fail loudly rather -than fall back: `config.validateCacheTTL` rejects an unknown value, and -rejects `cache_ttl` on any entry that is not the native anthropic adapter — -matching on IDENTITY, the map key `anthropic` with no `type`, never on the -key alone, since an entry keyed `anthropic` but typed `openai-compat` builds -an openaicompat client that would never read the value. `anthropic. -resolveCacheTTL` then rejects an unknown value again at the first `Stream` -call, like a missing API key. A typo must never silently ship different -cache economics. - -Wire shapes, by TTL: - -- `"1h"` sends `cache_control: {"type":"ephemeral","ttl":"1h"}` on both - breakpoints, plus the request header `anthropic-beta: - extended-cache-ttl-2025-04-11`. That header is the documented gate for the - extended TTL. Some endpoints no longer enforce the gate and accept the TTL - without it. Harness sends it regardless, because an endpoint that DOES - enforce it must not fail. -- `"5m"` sends `cache_control: {"type":"ephemeral"}` and NO beta header — - byte-identical to a build with no TTL support at all. This is the escape - hatch for a gateway that rejects an unknown beta. - -The default is 1h because of cost. Cache READS price the same at both TTLs. -A 1h WRITE costs 2x base input where a 5m write costs 1.25x, and that -premium applies only to the INCREMENTAL tokens each turn adds to the prefix. -A 5m expiry on a mature session, by contrast, rewrites the WHOLE prefix — -the entire history, at full input price. One such miss costs more than the -1h write premium over hundreds of turns. Agentic sessions exceed 5 minutes -by construction: one build, one live probe, or one subagent runs longer than -the window, and a user reads an answer before sending the next turn. The -commit that introduced this default carries the measured evidence. - -### A second native Responses provider - -`provider/openai` speaks the OpenAI Responses API. Other vendors speak the -same wire at their own host, under their own request path. Two config -fields let a deployment reach one without new adapter code. - -`Provider.Type` accepts `"openai"` (`config.TypeOpenAI`) under ANY providers -map key. The key becomes the provider family, routed by the first segment of -a `provider/model` ref, exactly like an `"openai-compat"` entry. -`cmd/harness`'s `registerOpenAIProviders` builds one `openai.Client` per such -entry. `base_url` is required: an arbitrary endpoint under a caller-chosen -key has no sensible built-in default, the same rule `"openai-compat"` -follows. The bare `openai` key with an empty type keeps its own built-in -default and is unchanged. - -`Provider.ResponsesPath` (`responses_path`) sets `Client.ResponsesPath`, the -path appended to the base URL. Empty means `/v1/responses`, the path the -Responses API documents and the only path this adapter could reach before. -The field is valid ONLY on an entry that builds this adapter — the native -`openai` key with an empty type, or any key with type `"openai"`. -`config.validateResponsesPath` rejects it elsewhere, matching on IDENTITY -(`buildsResponsesAdapter`) rather than on the map key alone, for the reason -`validateCacheTTL` already documents: the key `openai` with type -`"openai-compat"` builds an openaicompat client that would never read the -value. - -`openai.Client.Family` overrides the family key `Name()` reports AND the -`ProviderData` tag the transcoder reads and the stream writes. Empty means -the package `Family` constant, so every existing caller is unchanged; -`registerOpenAIProviders` sets it to the providers map key. The tag matters -beyond routing. A Responses reasoning item is opaque, usually ENCRYPTED, and -scoped to the endpoint that minted it, and history replays it verbatim on -every later request. One shared `"openai"` tag across two endpoints would -make the canonical family match succeed between them, so a session that -swapped models would replay one endpoint's ciphertext to the other. A -per-client family makes that a cross-family DROP instead — the canonical -crossing rule — which costs one turn of reasoning continuity and nothing -else. - -### Lazy MCP tools (deferred schemas) - -An MCP server's tools reach the model as full JSON Schemas in the tools -array. A box that wires several large servers therefore pays for hundreds -of schemas on every turn, at the FRONT of the cached prefix. The MCP -CONNECTION was already lazy; the schema cost was not. - -`engine/mcp_lazy.go` defers that cost, opt-in. A DEFERRED server's tools -leave the tools array and appear instead as a name-only catalog — one -`name — one-line description` line each — in a system segment placed after -the Agent Skills catalog and before hook (`system.transform`) segments. It -is the same progressive-disclosure staging Agent Skills already use. The -model loads a schema with the `mcp` tool's `select` action, and the loaded -def is back in the tools array on the next request, so a selected tool is -called exactly like a statically registered one. `runAgenticLoop` rebuilds -the request per tool round, so a `select` takes effect inside the same -turn. - -Config: `mcp_tool_loading` is `eager` (the default, and today's behaviour -byte for byte), `auto` (defer once the live catalog exceeds -`mcp_tool_loading_threshold`, default 20 tools), or `lazy` (always). -`mcp_servers..tool_loading` pins one server `eager` or `lazy`; -`auto` is global-only, because the threshold measures whole-catalog -pressure. Any non-positive threshold resolves to the DEFAULT, never to a -floor of 1 — that value would defer every catalog. - -Four rules are load-bearing. Do not relax them: - -- **A session that does not hold the `mcp` tool defers nothing.** Never - defer what the session cannot select. A subagent restricted by an agent - definition that omits `"mcp"` (`restrictTools`) would otherwise lose - every MCP schema AND the only path to load one back. -- **The `auto` threshold counts the WHOLE catalog**, including a server - pinned `eager`. A pin says "always keep these loaded", never "ignore - their cost". -- **The catalog listing sorts by full tool name, in the engine**, not by - the registry's server-then-tool slice order (the two differ for servers - `a` and `a0`). The tools array stays byte-stable because the partition - preserves the registry's order and changes only when a selection does. -- **`streamTurn` resolves the provider BEFORE it computes the tool plan.** - The plan's `Tools(ctx)` call is what dials a server for the first time - and spawns a child process for every stdio server. A turn naming an - unconfigured provider must return before any of that. The plan still - runs before `mcpStatusSegment`, which is the pre-existing rule that a - first-attempt failure is reported in its own turn. - -The same reorder moved one hook. `chat.params` still runs first and still -fires on every turn. `system.transform` now runs AFTER provider -resolution, so a turn naming an unconfigured provider returns without -firing it — it used to fire, then fail. Building a system prompt for a -request that is never sent buys nothing, and a plugin that counts -`system.transform` calls now counts sent requests. `chat.params` is -unaffected because provider resolution needs the model it returns. - -A stale selection is reaped at plan time: a selected name whose server is -CONNECTED and whose catalog lacks it is dropped. That is what keeps an -invented name — accepted while a server was unconnected, where a real name -and an invented one are indistinguishable — out of the effective set. A -selection whose server is still unconnected is KEPT, so it arms itself on -reconnect. The reap is memory-only; replay re-unions the log and prunes -again. - -The `mcp` session tool carries two extra actions when the session can -defer, and only then — a session that defers nothing must not advertise an -action with nothing to act on. `search(query)` ranks the live catalog by -keyword: substring matching over lowercased text, scored once per DISTINCT -query token per field (remote name 50, description 10, server name 5, plus -100 once when the whole query equals a name), sorted by score then name. -Tokens split on Unicode letter/digit classes, never the ASCII ranges — an -ASCII split truncates `café` to `caf` and reduces a CJK query to nothing. A -blank query errors rather than dumping the catalog. Both actions are -refused at DISPATCH, not only omitted from the advertised enum, on a -session that can defer nothing. `select(tools)` loads -schemas, and every name lands in exactly one bucket, tested TOP TO BOTTOM: -`already`, `selected`, `pending` (its server is configured but not -connected — it arms on reconnect), `missing` (no connected server holds it, -or the name is malformed). Its `note` is conditional on that outcome: a -`pending`-only batch must not claim its tools are callable next request. -`select` returns NO schemas: the tools array is the one authoritative copy, -and echoing them would write every schema a second time into durable -history. - -**Use implies selection.** An MCP tool call that ROUTES records its own -name. Without it, a tool of an eager server — which needs no `select`, and -which the model is told not to select — would lose its schema the moment an -`auto` flip deferred its server mid-task. The gate is per SERVER, not per -session: a server pinned `eager` can never flip, so a record for its tools -could never pay for itself, even in a session that defers a different -server. A plain `eager` config therefore records nothing at all. - -**Both writers of the record apply that same gate.** `select` records a -name only when its server could ever defer, exactly as a routed call does. -A record exists only to survive a flip, so "can this server ever flip" has -one answer whichever writer asks. A pinned-`eager` server's tool is still -reported `selected` — it is loaded and callable — and simply records -nothing. - -A selection is durable. `mcp.tools_selected` (`recMCPToolsSelected`, -`engine/store.go`) records the names that ENTER the set, and `LoadSession` -unions every record back. It follows `recToolResultRetained`: -engine-internal state, journaled and folded, with no engine event and no -server journal mapping. **Two writers produce it** — `select`, and a routed -MCP call through use-implies-selection. Wiring only the first silently -loses a tool the model used but never selected. - -Recovery degrades in one direction. A restored name whose server is absent -or parked is KEPT, so it arms on reconnect. One whose server connects -WITHOUT it is reaped. A malformed name is skipped on replay, exactly as -`select` refuses to record one — one rule at both ends of the record's -life. - -Full design, including the durable record: `docs/design/mcp-lazy-tools.md`. - -### The tool array is byte-stable across requests - -`Session.toolDefs` (`engine/engine.go`) sorts the BUILT-IN tool group by -name. That sort is a prompt-cache requirement, not cosmetics. `Session.tools` -is a map, Go randomizes map iteration on every range, and tools sit at the -FRONT of the cached prefix on every provider — Anthropic caches tools, then -system, then messages. An unsorted build therefore emitted a different tools -array on every request and invalidated the WHOLE prefix each turn, which no -TTL can help. - -The defect is invisible to a unit test that checks the tool SET, and it -appears only in live traffic: consecutive turns of one session each report a -full cache write and no cache read, for a byte-identical system prompt. A -new test must therefore assert the byte-stability of the array, not its -membership. The commit that introduced the sort carries the measured -before/after evidence. - -Group order stays built-ins, then MCP, then plugins. The other two groups were -already deterministic — `MCPManager.rebuildToolsLocked` sorts by server then -tool, and `plugin.Host.Tools` walks the configured instance slice — so the -sort applies WITHIN the built-in group only. Adding an MCP server must never -reshuffle the built-in block ahead of it. Any new tool source must be -deterministic before it joins this list. - -### Deliberately absent — do not add - -- **No permission system.** Tool calls are never gated. There is no `permission.ask` hook, no approval UI, no pre-flight rule evaluation. -- **No plan mode.** No edit-mode/plan-mode distinction anywhere in the engine. (The goal loop above is not plan mode — it produces no plan artifact and gates nothing.) -- **No JS runtime and no opencode plugin compatibility shim.** Plugins are native processes. -- **No auth hooks.** Credential injection happens at the network layer (gatekeeper) in deployed environments. - -These are settled decisions. Do not propose or implement them. - -## Dispatching goal-supervised sessions - -- **Completion conditions must demand world-state evidence, never transcript - claims.** Require branch-verified-on-origin (`git fetch && git status -sb` - output shown), pasted test output, etc. — not a model's assertion that it - did the work. Why: an evaluator once declared files created while the disk - was empty. -- **Push is the durability mechanism.** Commit as soon as the first test file - exists; push after every green milestone. Why: lease death and loop death - have each destroyed unpushed work. -- **Write conditions as timeless end-state predicates, never turn-relative - phrasing.** The condition string is re-sent verbatim in every guidance - message (`goalGuidance` embeds it in full on each NOT MET re-prompt, not - just turn 1), so wording like "on the first turn..." or "don't do X yet" - keeps re-asserting a stale instruction turn after turn instead of describing - the state the evaluator should actually check for. Why: live-run evidence - — such phrasing looped 32 turns chasing an instruction that only ever made - sense once. - -## Plugin System - -Plugins are separate processes (any language; Go SDK provided) speaking a versioned JSON-RPC protocol over stdio. - -- **Manifest cache**: `harness plugin install` runs the binary once and caches its manifest (name, protocol version, hooks subscribed, tool definitions) keyed by binary hash. Startup reads cached manifests only — nothing spawns at boot. -- **Lazy spawn**: a plugin process starts on first hook dispatch or tool call, then stays warm for the session (module-level caches in plugins are expected and fine). -- Sync hooks chain across plugins in config order — each sees the previous plugin's mutations — and every sync dispatch carries a deadline so a hung plugin can't wedge a session. -- **Plugin visibility**: `Host.Plugins()` reports every CONFIGURED plugin — name, spawn state (`not-spawned`/`running`/`errored`/`stopped`), registered tools, subscribed hooks — from the cached manifest plus live spawn state. The `session_info` tool (field `plugins`) and `GET /session/{id}` (field `plugins`) both surface it, so a not-yet-spawned plugin still appears. The engine reads it through the `Hooks.Plugins()` interface method, nil-guarded exactly like the other `s.cfg.Hooks` dispatch sites. The state read is lock-free (`instance.liveState`, `plugin/host.go`): `instance.start` holds `inst.mu` for the whole dial-plus-handshake, and `Host` is a box-scoped singleton shared by every session on the box, so a read gated on `inst.mu` would let one session's plugin spawn stall `GET /session`/`session_info` for every other session too — the same "a hung plugin can't wedge a session" rule above, applied to a status read instead of a hook dispatch. `errored` also covers a plugin that died AFTER a successful spawn (its connection closed, detected via the existing `conn.closed` signal), not only a failed start. - -### Hook protocol v1 - -| Hook | Mode | Purpose | -|---|---|---| -| `event` | async, fire-and-forget | full event stream (batched) | -| `chat.params` | sync, mutating | model, temperature, etc. per request | -| `chat.message` | sync, mutating | messages before they enter the log | -| `system.transform` | sync, additive | append segments to the system prompt (runs after `chat.params`) | -| `shell.env` | sync, mutating | inject env vars into shell/tool commands | -| `tool.execute.before` | sync, mutating/blocking | rewrite args or block with `{deny: "message"}` | -| `tool.execute.after` | sync, mutating | rewrite/annotate tool results | - -Plugins may also register **custom tools** (defs in manifest, execution via RPC). - -### Plugin client API - -Plugins are API clients over the same channel: `Session.Messages`, `MCP.Call`, `Generate` (LLM calls through the harness provider layer — plugins never carry their own API keys), and `plugin.HTTPClient()` (outbound HTTP with harness-configured headers, e.g. workspace attribution). - -Events v1: `session.status`, `question.asked`, `file.edited`, -`tool.execute.start`, `tool.execute.end`, `session.error`. Message-delta -events are deliberately deferred (see plugin/PROTOCOL.md) pending a -throttling design. - -Capability parity bar: the protocol must be able to express the plugin -patterns common in opencode setups — event-driven activity tracking, token -refresh via `shell.env`, tool-call rewriting/vetoing and result guards via -`tool.execute.*`, path-scoped system prompt injection, and custom tools that -call back into the platform. - -## External Protocol Surfaces - -Standards we conform to at the edges. The internal model (event log, canonical -messages, hook protocol) is ours; these are adapters, never the internal -representation. - -- **ACP (Agent Client Protocol, agentclientprotocol.com)** — the editor ↔ agent - standard (Zed, JetBrains, Neovim, Emacs). Implemented as a thin adapter in - `server/` mapping the event log to `session/update` notifications. Where our - event vocabulary has arbitrary naming choices, prefer ACP's names to keep the - adapter mechanical. We never send `session/request_permission` (no permission - system) — an agent that never asks is fully conformant. Note: this is Zed's - Agent *Client* Protocol, not IBM's dead Agent Communication Protocol. -- **MCP** — client (consume tool servers) and server (expose sessions/tools) - modes. ACP forwards editor MCP config to us, so the two compose. - - A server's first connect (Initialize+ListAllTools) stays lazy — - triggered by a session's first `Tools()`/`CallTool()`, bounded by a - per-server `connect_timeout_s` config field (`MCPServerSpec`, integer - seconds, <= 0/absent defaults to the engine's own 15s). A server whose - first attempt fails is never dropped for the process's life: it gets a - detached background retry on a capped exponential backoff (~1s doubling - to a 5min cap, jittered) — but bounded to `mcpRetryMaxAttempts` (3) - further attempts (under ~10s of background effort total). Once those - are exhausted the entry is marked Parked and the retry goroutine exits - for good — no further attempt ever fires spontaneously; only an - explicit re-trigger (the `mcp` tool's `connect` action, below) can move - it again. A HEALTHY server, by contrast, connects exactly once and is - never re-probed. `Tools()` always reads live state, so a server that - recovers mid-session — background retry or explicit reconnect — - contributes tools on the very next turn automatically, no new session - required. `CallTool`/`CallServerTool` split the old combined error into - two: a server name absent from config errors "not configured" (never - recoverable); a configured-but-unconnected server (still retrying, or - parked) errors naming that state explicitly (recoverable — retrying may - still self-heal, parked needs the `mcp` tool). While at least one - server is degraded, request assembly appends an ambient `[mcp: - unavailable — (; retrying), ...]` block to the newest - user message only — computed fresh every turn, never persisted, - self-correcting as retries succeed; a Parked server's clause instead - reads ` (; use the mcp tool action "connect" to retry)` — - sharing its append-only-to-the-newest-message mechanism - (`withAmbientStatus`) with the managed-processes status block above. - - A built-in `mcp` session tool is registered in `newSession` whenever - the session's MCP registry reports at least one configured server (no - config flag, unlike `GoalTool`). `status` reports every configured - server's live state — `{name, connected, attempts, parked, reason}`; - `connect {server}` makes ONE bounded, synchronous attempt for a named - server — the only path back for a Parked server, though it works - against a still-retrying or never-yet-attempted one too. An - already-connected server is a friendly no-op; an unknown name errors - listing the configured names. A per-server in-flight guard (under the - manager's own lock) serializes a tool-triggered connect against both a - concurrent `connect` call and `retryServer`'s own background attempt - for the same server — whichever gets there first dials, the other - reports "attempt already in progress." Every model-visible string on - this surface — the ambient block, `status`'s `reason`, `connect`'s - failure result — is `classifyMCPConnectError`'s output, never a raw - error (which can embed the server's endpoint URL and any secret it - carries). -- **OpenTelemetry GenAI semantic conventions** — for span/metric naming when - observability lands. Configuration via standard `OTEL_*` env vars only. -- **A2A** — deliberately not implemented. Cross-org agent meshes are a - different layer; revisit only if a concrete need appears. - -## Development hub - -`harness hub` is a local, single-operator control surface over a FLEET of -`harness serve` boxes — a fleet dashboard for "what are my agents -doing right now" and for dispatching new goal-supervised sessions, not a -deployed product. It serves one embedded, single-file page -(`tools/hub/index.html`, `go:embed`) on -`localhost:7777` by default (`-addr` to change it). - -- **No server-side state.** The hub keeps no registry and reads no config - file: every box (name, base URL, run token) and the current selection - live only in that browser tab's URL fragment, base64-encoded JSON - (`#s=...`), kept in sync via `history.replaceState`. That makes a hub URL - bookmarkable and shareable between local tabs with zero persistence code - — and means **run tokens ride the URL by design**; treat a hub link like - a secret. -- **The page talks to boxes directly** from the browser, over each box's - normal HTTP+SSE API (`server/openapi.yaml`) — never proxied through the - hub's own server. Every box must therefore be started with `-cors-origin` - set to the hub's origin (or `*` for local hacking), e.g. `harness serve - -cors-origin http://localhost:7777`; a box without it will look - permanently unreachable from the hub. -- **The Go side is minimal on purpose**, exactly one API: `POST /spawn`. - It execs the command given by `-spawn-command` (or `$HARNESS_HUB_SPAWN`) - via `sh -c` and streams its combined stdout+stderr live to the page over - SSE. The **spawn-command contract** — the only coupling between this repo - and any deployment-specific provisioning tool — is plain lines anywhere - in that output: `TUNNEL_URL=` and `RUN_TOKEN=` (required to - add the box), and any number of `PORT_URL_=` lines (optional — - one per exposed port's own tunnel/preview URL, collected into a - `port_urls` map; see the process strip in `tools/hub/index.html`'s header - comment). Once the command exits, the stream ends with a summary carrying - those values (if found) and the exit code; the page adds the new box to - its own URL state itself. Nothing box-provisioning-specific lives in this - repo. - - **Box name passthrough.** `POST /spawn`'s JSON body optionally carries - `{"name": "..."}` — the page's generated (or, on a Respawn/ADOPT, reused) - box name. The Go handler sets it as `HARNESS_HUB_BOX_NAME` in the spawn - command's own environment (`tools/hub/spawn.go`'s `runSpawn`), exactly - the deployment-environment contract `docs/design/fleet-model.md` §8 - specifies: deployment tooling invoked by `-spawn-command` reads this - variable to derive per-name storage (typically setting - `HARNESS_SESSION_DIR` from it before `harness serve` starts) — harness's - own code never reads `HARNESS_HUB_BOX_NAME` at all. A request with no - body, or no `name` field, spawns exactly as before (no env var set). -- The hub binds loopback-only by default (`resolveAddr` in `tools/hub/hub.go`). -- **Browser-security hardening** (both in `tools/hub/hub.go`, tested in - `tools/hub/hub_test.go`). `POST /spawn` execs a real, costly provision - command, so `handleSpawn` rejects a browser cross-origin request before any - exec: if an `Origin` header is present it must match the request's `Host` - (OWASP verify-origin). Loopback binding alone does not stop this — any page - the operator visits can `fetch("http://localhost:7777/spawn",{method: - "POST"})` as a no-preflight CORS simple request — but the page's own - same-origin `fetch("/spawn")` (Origin == Host) and non-browser clients (no - Origin, so not a CSRF vector) pass unchanged. The served page also carries - a strict `Content-Security-Policy` (`default-src 'none'` + `'unsafe-inline'` - script/style — the page is a single no-build `go:embed`'d file with no - external resources and no per-response nonce hook — + `connect-src *`, - required because it fetches/streams from arbitrary operator-added box - origins the stateless hub cannot enumerate, + `frame-ancestors`/`base-uri`/ - `form-action` pinned to `'none'`): defense-in-depth for a page holding run - tokens in its URL fragment. -- **Pure hub logic is unit-tested** by `tools/hub/hub_test.mjs` (run: - `node --test tools/hub/*_test.mjs`). **End-to-end, against a real backend** - is `tools/hub/e2e` (see its README): a `go test -race ./...` subtree that - starts an actual `server.Server` + `hub.NewHandler` and drives the real, - served `index.html` with Node + jsdom and an unmocked `fetch` — no manual - setup step; it installs its own `npm` dependency on first run. - -### UI design language - -The hub is styled as **tactical telemetry** — a committed dark-only -brutalist archetype (derived from the public -[taste-skill](https://github.com/Leonxlnx/taste-skill) brutalist + -anti-slop skills). Any new hub UI — and future passes on the inspector, -which still wears the older soft theme — follows these rules: - -- **One substrate, no theme toggle**: `#0a0a0a` background, `#eaeaea` - phosphor foreground, `#2a2a2a` hairline borders. Never reintroduce a - light mode here; pick-one-and-commit is the point. -- **Two semantic colors only.** Hazard red (`--accent`, `#ff2a2a`) means - trouble or destructive action, nothing else. Terminal green (`--ok`, - `#4af626`) is reserved for exactly one semantic: live or succeeded goal - execution. Everything else is monochrome. -- **Monospace dominance**: body text is the `ui-monospace` stack; - headers are heavy uppercase system-ui. Micro-labels are uppercase with - `.06–.1em` tracking. No webfonts — the page is CSP-self-contained. -- **Geometry**: `border-radius: 0` absolutely everywhere; square status - markers; 1px compartment borders; inverted-video hover - (foreground/background swap). No gradients, soft shadows, or - translucency. The scanline overlay is static — motion requires a - stated purpose. -- **Copy discipline**: no emoji in UI strings, no em-dashes anywhere, and - every piece of "telemetry" displayed must be real data (vcs revisions, - seqs, PIDs, token counts) — never decorative or fabricated metadata. -- **Selectors are load-bearing**: the renderers create elements by class - name (`.sess`, `.box-card`, `.dot`, `.goalnarr`, …). Restyle classes; - never rename them in a styling pass. - -## Session monitor - -`tools/monitor` (`tools/monitor/index.html`) is the single-instance -counterpart to the hub above: where the hub answers "what are my boxes doing" -across a FLEET, the monitor answers "what is THIS `harness serve` instance -doing right now" — a live board of every session on that one box (phase, -current tool, staleness), a per-session detail view with a scrolling -transcript, and a composer to speak into a session (`prompt_async`). Like the -hub and the inspector, it is a build-free, dependency-free single HTML file -with no Go-side handler of its own. - -- **How to run it**: open the file directly (`file://`) or serve it from any - static host — nothing box-specific is baked in. The target box must be - started with `-cors-origin` covering the monitor's origin (or `*` for local - hacking), exactly like the hub's requirement above; a box without it looks - permanently unreachable. The base URL and run token are entered in the - page itself and persisted to `localStorage` in plaintext (same documented - tradeoff as the inspector — a dev tool, not for a shared origin with a - long-lived token) so a reload can reconnect automatically. Routing lives in - the URL fragment as small explicit params, not the hub's base64 blob: - `#b=` (box base URL) and `#s=` (open detail view) — both - bookmarkable, encoded/decoded by a tested pure helper. -- **Embedded serving, frictionless local**: every `harness serve` box also - offers its own copy same-origin, at `GET /monitor` — by default - `http://localhost:4096/monitor` (the port follows `-addr`, default - `localhost:4096`; the exact URL, with any `#t=` capability suffix, is - printed to the terminal on startup — `monitorTerminalHint`). The bare root - is a convenience redirect: `GET /` 302s to `/monitor` (via the `GET /{$}` - route — `{$}`-anchored to the root path only, never a catch-all — registered - under the same `MonitorPage` guard, so a pure-API box keeps `/` a clean 404). - `tools/monitor` - (package `monitor`, `embed.go`) `//go:embed`s the exact committed - `index.html`; `cmd/harness`'s `serveCmd` wires it into - `server.Options.MonitorPage`, which the server serves unauthenticated - (like `/health` — the page itself carries no secrets) with a - same-origin-scoped `Content-Security-Policy` (`connect-src 'self'`). - `server` itself never imports `tools/monitor` (layering: `server` must - not import `tools/*`); only `cmd/harness` does, the same pattern - `harness hub` already uses. The `file://`/static-host path is unaffected - — this is additive, and stays the only option for monitoring a box from a - different origin (the embedded route's CSP deliberately does not permit - that). - - **Unauthenticated-on-loopback** (`server.Options.Unauthenticated`, an - EXPLICIT opt-in never inferred from an empty `RunToken` on its own — - `New` still fails closed otherwise): `serveCmd` classifies `-addr` - (`isLoopbackAddr` — `localhost`, `127.0.0.1`/`::1`, any - `net.IP.IsLoopback()` address; a bare `:port`, `0.0.0.0`, `::`, or any - other routable address is NOT loopback). `HARNESS_RUN_TOKEN` unset + - loopback bind runs the box fully unauthenticated (every route, not just - `/health`/`/monitor`) and logs a clear warning; unset + non-loopback - still hard-errors `HARNESS_RUN_TOKEN is required` exactly as before. The - token guards network reachability, and loopback is a server-verifiable - proxy for that (unlike, say, `Origin`, which a client controls). - - **Unauthenticated on a non-loopback bind is also possible, but ONLY via - a SECOND, separate, EXPLICIT opt-in** — the `-unauthenticated` serve - flag, or `HARNESS_UNAUTHENTICATED=1` (`envUnauthenticated`, parsed with - `strconv.ParseBool`; an unset or unparsable value is false, fail-closed - on a malformed setting). `resolveUnauthenticated` (`cmd/harness/main.go`) - is the single decision point both serve flags feed: a non-empty token - always wins (token path unaffected, any bind); an empty token on a - loopback bind is unchanged from above; an empty token on a non-loopback - bind needs this opt-in or it still hard-errors exactly as before — the - opt-in is never inferred from the empty token alone, only from the flag - or env var. This is for a deployment where a trusted external gate - already restricts reachability (e.g. a Cloudflare Access-gated tunnel, - or a sandboxed network boundary) so the token is redundant with that - gate — `server.Options.Unauthenticated` itself is bind-address-agnostic - (see its own doc comment); the safety property lives entirely in - `cmd/harness` deciding WHEN to set it. Landing this on a non-loopback - bind logs a SEPARATE, distinctly worded loud warning ("serving - unauthenticated on a non-loopback bind") from the loopback one above, so - the two are distinguishable in a log search. - - **Same-origin auto-connect** (`embeddedConnectPlan`, index.html): opening - a box's own `/monitor` attempts the connection immediately against - `location.origin`, using whatever token is available (`#t=` fragment, - then a stored one, then none) — success lands straight on the board, no - panel, nothing typed (covers both the unauthenticated-loopback case and - a valid capability URL/returning operator with the SAME call). Only a - failed attempt (a real token is required and none was available) falls - back to a minimal token-only panel — host is already known, so the base - field never reappears. A `#t=` fragment (mirroring `tools/hub`'s - "run tokens ride the URL by design" precedent, `extractFragmentToken`) - is adopted into the SAME `localStorage` key manual entry uses and - immediately scrubbed from the visible URL via `history.replaceState`; a - fragment `#b=` naming a DIFFERENT origin than this box's own — which the - embedded route's CSP would block outright if tried — surfaces a - plain-text notice instead of attempting it, composing with (never - blocking) the own-origin auto-connect. None of this weakens the auth - model itself: a token is still required and still checked exactly as a - hand-typed one would be, on every box that hasn't explicitly opted into - `Unauthenticated`. - - On a TTY, `serveCmd` also prints a click-ready line to stderr after - "serve start" — `monitor: http:///monitor#t=` (or, when - running unauthenticated-loopback, the plain URL with no `#t=` at all, - since there's no credential to carry) — gated on `stderrIsTerminal` - (`os.ModeCharDevice`, stdlib-only) so a tokenized URL never lands in - piped/production stderr, only an interactive operator's own terminal. -- **Test layers.** Pure helpers (SSE parser, activity reducer, transcript - fold, route codec, formatters) live inside index.html's `/* TESTABLE-BEGIN - */ … /* TESTABLE-END */` region and are covered by - `tools/monitor/monitor_test.mjs` (run: `node --test tools/monitor/*_test.mjs`), - using the same extraction-and-`vm`-evaluate pattern as the inspector's - `inspector_test.mjs` (and now the hub's, above) — no build step, so the - tests read the region straight out of the committed HTML. End-to-end, - against a real backend, is `tools/monitor/e2e` (see its README): a `go - test` subtree that starts a real `server.Server` plus a plain static file - server for the actual committed `index.html`, and drives it with Node + - jsdom and an unmocked `fetch` — mirroring `tools/hub/e2e`'s structure and - conventions. A `window.__monitorTuning = {QUIET_MS, STALL_MS}` seam (set - via jsdom's `beforeParse` before the page's own script runs, a no-op in - production since nothing else ever sets it) lets both the unit and e2e - suites shrink the staleness thresholds so `quiet`/`stalled` transitions are - observable in test time instead of real minutes. -- **UI design language.** The monitor deliberately does NOT inherit the - hub's committed dark brutalist archetype above. It carries its own - "instrument sheet" language: light-first with a dark variant, both driven - by one OKLCH token set (`--surface`, `--text-1..3`, `--separator`, etc.); - semantic green/amber/red (`--ok`/`--warn`/`--critical`) are reserved - strictly for session/staleness state, never decoration; a single accent - color is owned by interaction (the composer's send affordance is the - page's only filled-accent control). `docs/design/monitor-mockup.html` is - the user-approved visual spec — its tokens, grid template, and markup - shapes are binding; restyle within that spec rather than importing the - hub's theme onto it. - -## Fleet model (the deploy story) - -The full build spec lives in `docs/design/fleet-model.md` — read it before -touching anything box-identity, session-lineage, or goal-pause related. The -short version this repo's code assumes: identity is an operator-chosen box -**NAME**; storage is one volume/directory per name (`HARNESS_SESSION_DIR` -points at it), never shared between concurrently-live servers; a box is -ephemeral compute serving one name (cattle), the name and its volume are -durable (pets). Respawning the same name over the same volume is **ADOPT** -— history restores, and any goal that was armed when the box died surfaces -as `paused`/`pause_reason: "restart"` (see the goal loop's paused -presentation, `engine/goal.go` and `server/journal.go`'s `goal.paused` -record) rather than a false "still running" reading. `parent_session` -(`POST /session`, see `engine/store.go`) is the lineage thread connecting a -re-dispatch to the task it continues from, so a fleet UI can group a box's -history by task across boxes. - -Subagent lineage is durable. `SessionManager.Spawn` records -`task_parent_id`, `task_agent_type`, and `task_depth` on the child's -session header (`engine/store.go`), and appends each child id to the -parent's own log. `LoadSession` restores all of them with no -SessionManager adoption needed. `GET /session/{id}.lineage` prefers the -durable `task_depth` over the live tree's derived depth, and merges live -children with the durable spawn list (`childIDsUnion`, -`server/handlers.go`) — so `lineage.depth` and `lineage.children` survive -`Reap` and a process restart. `childIDsUnion` merges both sides through -ONE de-duplicating loop and trusts neither side to be duplicate-free: an -id appears exactly once, whichever side carried it. Never re-add a -per-side fast path that skips the merge — an earlier one copied `live` -verbatim when `durable` was empty, so one repeated id survived or -collapsed depending only on whether the OTHER argument had anything in -it. A legacy header without `task_depth` -restores 0; `adoptReloadedLocked` then falls back to the `m.maxDepth` -refusal sentinel, exactly as before the field existed. - -A failed child's `fail_reason` carries the CAUSE, not only a class. -`classifySpawnFailure` (`engine/session_manager.go`) builds it as a fixed -classified prefix, then the underlying error message — masked with -`maskSecrets` and capped at `spawnErrorDetailCap` (500) runes. One prefix -covers a whole family of causes (a permanent 400 is a malformed request -AND a quota rejection AND a policy refusal), so a parent that reads only -the prefix must guess: a live incident measured that guess as "respawn a -sibling straight into the same fleet-wide provider wall". The #82 leak -rule still holds in its narrower form — never surface a provider error -RAW — through masking plus the cap, the same best-effort trade a retained -tool result already makes. `context.Canceled`/`context.DeadlineExceeded` -keep their short fixed `canceled`/`timed out` strings, with no cause -appended. The reason reaches the parent through the `[tasks: ...]` -notification, `SessionNode.FailReason` (so `task status` and -`GET /session/{id}.lineage.fail_reason`), and the journal's -`task_fail_reason`. - -Server-side session resolution has ONE entry point: `Server.resolveLive` -returns a `liveSession` snapshot (`server/live.go`) that holds the -residency half (`Server.sessions`, one `s.mu` hold) and the SessionManager -half (one `SessionAndInfo` hold) together. Read a session, its status, or -its lineage from that snapshot — never from `s.sessions` or `sessMgr` -directly, and never take a second manager read later in the same request. -The two halves are separate holds on purpose: `server.mu` is a leaf lock -with respect to `SessionManager.mu`, so one atomic hold over both would -build the cycle that rule forbids. Residency wins whenever it has an -answer, because a resident session's own `running` flag is authoritative -for itself (`freeRunSlotAndEmitIdle` clears it before `ReportTurnEnd` -flips the node). The manager half answers only what residency cannot: a -Spawn-driven child, which is never a residency key. - -**Provider exhaustion is not a child failure.** An ACCOUNT-level supply -wall — the API key's usage limit, quota, credit balance, or spend cap — is -FLEET-WIDE (every sibling on the same key hits the identical wall at the -identical moment) and TEMPORAL (the child's session and work are intact and -re-runnable once the provider's clock rolls over). A parent that reads it as -an ordinary failure respawns a replacement into the same wall, which a live -incident measured. Three layers carry it: - -- The ADAPTER classifies, never the engine. `provider/anthropic`'s - `parseUsageExhaustion` gates on the HTTP status (400/402/403/429, or none - at all for a mid-stream `error` event) and then matches - `usageExhaustionPatterns` — a deliberately flat, extensible list of - observed wall wordings, one regexp per shape, each of which must name a - spent SUPPLY, never a per-minute THROTTLE. It returns a - `provider.Error{Kind: ErrKindProviderExhausted, RecoverHint}` wrapped - permanent (no backoff outlives a spent quota). This is the second place - message matching is tolerated, under `parseContextOverflow`'s rules. Other - adapters opt in by producing the same kind; only anthropic does today. -- The ENGINE reads the typed classification, never text. - `classifySpawnFailure` (`engine/session_manager.go`) maps - `provider.AsProviderExhausted` — or a `RetryableRateLimited` class that - outlived the retry budget — to `FailKindProviderExhausted` - (`"provider_exhausted"`). Overload and 5xx weather deliberately do NOT - qualify: those clear in seconds and a sibling may well succeed. -- The STATUS VOCABULARY is unchanged. An exhausted child is `StatusFailed`, - with the kind in a SEPARATE `FailKind` field (`SessionNode`, - `taskNotification`, the durable `taskNotifyRecord`, `task status`'s - `fail_kind`, `GET /session/{id}.lineage.fail_kind`, the journal's - `task_fail_kind`). A sixth `SessionStatus` value would have forced every - cancellation/Reap/delivery/restore switch to grow an arm that behaves - exactly like `StatusFailed`; only the PARENT's next move differs. - -The rate-limit arm conflates a spent quota with a per-minute throttle that -outlived the child's small `PromptRetries` budget, one-directionally and on -purpose: a missed wall makes a parent respawn into it (the incident), while -a false wall costs one deferred resume of an intact child, and a hintless -guidance names no waiting period. An adapter that classifies its own quota -shape never reaches that arm. Both the cause and the recover-at hint go through -`boundedProviderText` (mask, then cap), so model-visible provider text on -this surface has one rule, not one per field. The hint is stated in ONE -engine-authored place — `taskFailureGuidance`'s "after " — never in -the reason prefix as well: the hint is extracted FROM the provider message -the reason already quotes, so naming it there made one rendered line -repeat the same time three times. It rides the durable record -(`taskNotifyRecord.FailHint`) because that guidance clause is now the only -carrier of the fact. - -`taskFailureGuidance` (`engine/taskdelivery.go`) appends the parent's -instructions to that child's own notification line — child preserved, do not -spawn a replacement, resume with `task send` on this session id, after the -recover-at hint when the provider gave one. Resuming is the existing -send-to-a-settled-descendant re-run path, unchanged. A turn that then -succeeds clears `failReason`/`failKind` on the node, so a resumed child -stops reporting a wall it already got past; `finalizeTurn`'s -`alreadyCanceled` branch clears them too, since a CANCELED re-run must not -keep snapshotting a classification no live cancellation sets and -`restoreKnownStatusLocked` restores as empty. - -A REAPED descendant is still resolvable, not "no such session." `Reap` -collects a done/failed/canceled LEAF the instant it settles (its own doc -comment), which a caller that spawned it has no way to observe before -asking about it again — a live incident hit exactly this: `task send` to -a settled child answered `no such session` depending on internal reap -timing the parent could not see. `resolveOrReviveDescendantLocked` -(`engine/session_manager.go`), the shared first step of all four verbs, -falls back to disk when a live-tree lookup misses: `LoadSession` the -target, then confirm ancestry from its own DURABLE `task_parent_id` -chain (`durableAncestorChainHas`) — never from live state alone, which is -exactly what `Reap` already erased. Only a target with no session log on -disk either, or whose durable lineage does not reach the caller, still -answers `ErrUnknownSession`/`ErrNotDescendant`. The four verbs then -diverge on what a successful disk resolution does: `send` RE-ADOPTS the -revived child into the tree (`adoptReloadedLocked`, the same -adopt-on-first-sight path `AdoptReloaded`/`handleSpawnChild`'s -parent-lookup fallback already use) and re-runs it exactly like a -settled-but-unreaped child — `budgetedByChild` surviving `Reap` by design -is what stops that re-adopt from double-crediting its already-spent -usage. `status`/`log` serve the disk-loaded state directly -(`durableSnapshot`/`deriveSettledStatus`) WITHOUT re-adopting: a -read-only, poll-shaped verb must not have the side effect of pinning a -reaped descendant back into memory. `cancel` on a reaped target is a -no-op success (nothing left to interrupt) reporting its real terminal -status, never `StatusCanceled`. The disk-bound half of this resolution -runs with `m.mu` released — one slow disk read must never stall every -other session's `Info`/`Reap`/`Spawn`/`Send` call — and re-validates -`m.nodes[targetID]` on reacquiring the lock, so a concurrent adopt of the -same id (another `Spawn`, `AdoptReloaded`, or a second racing revival) -always wins single-handedly: whichever adoption reaches `m.nodes` first -is authoritative, and the loser's own "already managed" is ignored, the -same rule `AdoptReloaded`'s existing callers already follow for that -race. - -A parent can read a dead child's tail. The `task` tool's `log` verb -(`runTaskLog`, `engine/task_tool.go`, over -`SessionManager.DescendantTranscript`) returns the last N transcript -entries of a descendant, LIVING OR DEAD, under the same ancestor gate -(`isDescendantLocked`, or the disk-backed lineage check above once -`Reap` has removed the live node) cancel/status/send use — a terminal -node keeps its `*Session`, history included, until `Reap`, so no reload -and no disk read is involved for a still-tracked descendant. It is -bounded on three axes, because its output lands in the -PARENT's context and replays on every later turn: `tail` (default 20, -clamped at 100, a negative value is an error), a per-entry rune cap, and a -total rune budget filled NEWEST-first so the messages nearest a death -always survive. The reply reports the descendant's whole message count -next to how many entries came back, so a model knows it is reading a -window, and it carries `fail_kind` alongside `fail_reason` — the same -structured half `task status` reports, so a reader with the tail in front -of it never needs a second call to learn a death was an account wall -rather than the child. Every non-text part is rendered rather than dropped — a tool call -with capped arguments, a tool result, a reasoning summary, and an -attachment COUNT that includes blobs nested inside a tool result, which -`Parts.Text()` itself drops. Content is deliberately NOT masked: parent -and child are the same operator's sessions in one process, and a child's -final text already reaches the parent verbatim in its completion -notification. - -`Config.OnRequest` receives the firing session's own id as its first -parameter (`engine/engine.go`). Never wire it as a closure over a -captured session variable: `configSnapshot` copies the func value into -every spawned child, which misattributes the child's `request.meta` -records to the closed-over session's id. - -**Hub spawn contract:** the hub that spawns boxes — `harness hub`, now -implemented in `tools/hub/` (see the Development hub above) — passes the -generated box NAME to the spawn command's environment as -`HARNESS_HUB_BOX_NAME`, so deployment scripts can derive per-name storage -(e.g. mount/create a volume named after it) without the hub and the box -needing any other side channel to agree on identity. Harness itself never -reads this variable — it is a contract between the hub and deployment -tooling, documented in `docs/design/fleet-model.md` §8. - -## Serve-mode latency diagnostics - -A caller that waits seconds for a reply cannot tell, from outside the -process, whether `harness serve` was slow, the network in front of it was -slow, or the whole process was stopped by garbage collection. Three -threshold-gated WARN lines answer that, and nothing runs always-on. - -- **`slow request`** (`server/timing.go`). `serveTimed` wraps the mux - dispatch in `Server.ServeHTTP` and warns when this process took longer - than `slowRequestThreshold` (500ms) to answer, with `method`, `route`, - `status`, `duration_ms`, and the caller's `X-Request-Id` as - `request_id`. The route is `http.Request.Pattern`, which the mux sets - during the dispatch, so a session id never reaches a log line; a request - that matched no route logs the fixed `unmatched` label, because the path - is caller-controlled. `requestID` drops a header value over 64 bytes or - carrying anything outside one printable ASCII token — it is untrusted - input that lands in a log line. `longLivedRoutes` exempts `GET /event` - and `GET /session/{id}/wait`: both run for as long as their caller - wants, so timing them would warn for every healthy client. Keep that map - in step with any new streaming or long-poll route. -- **`long gc pause`** (`cmd/harness/gcwatch.go`). A stop-the-world pause - stops every goroutine, so the process logs nothing at all while it lasts - and looks exactly like a wedged handler. `gcWatcher` samples the - runtime's `/gc/pauses:seconds` histogram every 5 seconds and warns about - a new pause at or past 200ms. It reads `runtime/metrics`, never - `runtime.ReadMemStats` — `ReadMemStats` itself stops the world, so - sampling it would add the pause this watcher exists to find. - `longest_pause_ms` is the LOWER bound of the highest bucket that gained - a pause, since a histogram records a range. The first sample reports - nothing: the counts are cumulative for the whole process life. Same - lifecycle as `inFlightWatchdog` — one cancelable context, cancelled when - `serveCmd` returns. -- **`/debug/pprof/`** (`server/pprof.go`, `Options.PProf`, `harness serve - -pprof`). OFF by default, and authed like every other route when on. It - is the third step, not the first: `GET /debug/goroutines` needs no flag - and already answers "what is this process blocked on". Turn `-pprof` on - for a process under investigation when the CPU, heap, block, or mutex - profile is what is missing. - - **Never import `net/http/pprof` in this repository.** That package's - `init` registers `/debug/pprof/*` on `http.DefaultServeMux` for the - whole linked binary, so importing it — even to borrow its handler - functions behind this flag — exposes profiling in ANY program that - links `server` and serves the default mux - (`http.ListenAndServe(addr, nil)`), with no opt-in and no way for - `Options.PProf` to prevent it. Go runs a package's `init` on import; - there is no way to take the handlers without the side effect. So - `server/pprof.go` implements them on `runtime/pprof` and - `runtime/trace` directly. - `TestPProf_NotRegisteredOnDefaultServeMux` asserts the absence and is - the regression guard: a 404 test through this server's own mux passes - WITH the bad import and proves nothing. It exists TWICE, in `server/` - and in `cmd/harness/`, because each only covers its own package's - import graph — the binary links the engine, providers, plugins, MCP and - the tools, and any one of them pulling in `net/http/pprof` would - publish the endpoints for the whole process. - - `?seconds=N` on `profile`/`trace` is clamped to 1-60 (default 30); a - malformed or repeated value is a 400, through the same `intParam` every - other integer parameter uses. A second concurrent CPU profile or trace - is a 409, not a 500 — only one of each can run in a process — and the - refusal removes the download headers it had to set before starting, so - the error is JSON rather than a file a browser saves. A client - disconnect ends the profile early rather than holding a runtime-wide - lock for an abandoned request. - - `GET /debug/pprof/{name}` is in `longLivedRoutes` (`server/timing.go`): - a profile runs for exactly as long as the caller's `?seconds` asks, so - timing it would log a 30-second "slow request" every time an operator - ran `go tool pprof` against a box — their own tooling in the logs they - are reading. The index stays timed; it returns at once. - - The UNSLASHED `/debug/pprof` is registered explicitly, behind auth. - Left to the mux it takes an automatic 308 redirect issued before any - handler, which told an unauthenticated caller whether profiling is - enabled. Authed, the two states are 401-vs-404 — the shape every other - route in this API already has. - - `/debug/pprof/symbol` is deliberately not served: `go tool pprof` - symbolizes against the binary a profile came from. - -No metrics, no tracing, no always-on profiling. A new diagnostic in this -area is a threshold-gated log line or it does not land. - -## Startup Speed Rules - -- Nothing touches network, subprocesses, or disk beyond one config file before first paint. Provider auth validates on first message send, not at boot. -- No `init()` side effects. No reflection-heavy config frameworks. One flat config parse. -- Pure Go only — no cgo (use modernc SQLite if SQLite is needed) so cross-compilation stays trivial. -- Batch TUI stream rendering (~30–60fps coalescing); never repaint per token delta. - -## Development Commands +The engine is a headless library. The CLI, server, and local tools are clients. +`engine/` owns sessions; `message/` owns canonical types; `provider/` owns wire +adapters. `cmd/harness/` composes `config/`, `server/`, plugins, MCP, managed +processes, and local tools. `skill/`, `modelmeta/`, `imageclamp/`, and `sdk/` +provide focused support packages. + +Keep package boundaries one-way. The engine must not import the CLI or a local +UI. The server must not import `tools/*`; `cmd/harness` composes them. + +## Cross-cutting invariants + +- A session is an append-only log of typed events. +- The log stores canonical messages, never provider wire objects. +- Every provider adapter transcodes canonical history from scratch per request. +- Provider-specific opaque parts keep a provider-family tag. Replay them only + to the same family. +- Tool-call IDs are internal. Each adapter maps them deterministically. +- Prompt-cache markers are request-time data. Never store them in history. +- A repair that touches live or persisted history is additive-only. It must not + delete, reorder, or relocate producer data. +- A transcode-time repair may reshape a throwaway request. It must not delete a + real tool result. +- An empty tool result must never serialize as `null`. Use + `ToolResult.SafeContent`, not `Content`, in every transcoder. +- Model references use `provider/model`. Configured aliases resolve before a + request reaches a provider. +- Engine-owned ambient status uses `message.EngineContext`. User text must + never gain that trust boundary. + +Read `message/AGENTS.md`, `engine/AGENTS.md`, and `provider/AGENTS.md` +before a change crosses these boundaries. + +## Settled non-goals + +Do not add these features without a new explicit design decision: + +- A permission or approval system for tool calls. +- A plan mode or edit-mode gate. The goal loop is not plan mode. +- A JavaScript runtime or an opencode plugin compatibility layer. +- Plugin auth hooks. Deployed credential injection belongs at the network layer. +- A2A support without a concrete cross-organization use case. + +## Startup rules + +- Keep `harness --version` near the enforced millisecond budget. +- Before first output, limit disk reads to the user and project config files. +- Do not perform network calls or start subprocesses before a command needs them. +- Do not add `init()` side effects. +- Keep config parsing flat and lightweight. +- Keep production Go code free of cgo. +- Validate provider credentials on first use, not at process startup. +- Keep model catalogs static. Do not refresh them at startup. + +## Development commands ```bash go build ./... @@ -2712,321 +107,92 @@ go test -race -run TestName ./engine/ go vet ./... ``` -## Testing - -**TDD is mandatory.** Write the failing test first, watch it fail, then -implement until it passes. New behavior lands in the same commit as its test; -a bug fix starts with a test that reproduces the bug. - -Rules: - -- **Timer-dependent and concurrency-timeout logic is tested inside a - `testing/synctest` bubble** (Go 1.25+): time is fake and advances only when - every goroutine in the bubble is durably blocked, so timeouts fire - deterministically and instantly. `net.Pipe` and channel-based plumbing work - in bubbles; real network and file I/O do not. Note fake time stops - advancing once the test function returns — a goroutine parked in - `time.Sleep` at bubble end is reported as a deadlock, which is the bubble's - goroutine-leak detection working for you. -- **For concurrency-sensitive code (locks, queues, backpressure), write the - invariants down in the brief/design before implementation** and test - against them. Deriving the design from review findings one round at a time - took four rounds on a recent PR. -- **Exception — cross-process observation** (`e2e/`, and the packages whose - own subprocess machinery is under test: `process/`, `engine/bash_pipe_test.go`, - the `live`-tagged tests that call a real remote model): a test may observe - out-of-process state with deadline-bounded poll loops, because no - in-process channel crosses an OS process boundary. Every such wait goes - through `internal/testpoll` — never an inline sleep loop. Its timeout is a - FAILURE bound, never a synchronization delay: the happy path returns on - the first successful check. Anything observable in-process still uses - channels or synctest. -- **In-process state gets a seam, never a poll loop.** A wait on a manager's - status, a server's session state, or a queue depth blocks on a signal. - Three production seams exist for exactly this, and a new wait extends one - rather than sampling: `engine.SessionManager.Changed` (a node's status, - finalized flag, or tree membership settled — arm it BEFORE the read, so a - transition landing between read and wait is still delivered), - `process.Manager.WaitExit` (blocks until a managed OS process has exited, - and returns that instance's own terminal state — never a later - restart's), and `GET /session/{id}/wait?until=idle` (the production - long-poll, which also spans a queue drain). Sampling one of these on an - interval is a guessed deadline: it flakes under load, and it turns a real - hang into a slow pass. -- **`time.Sleep` is banned in test code. Absolute — not "for - synchronization," not "just 10ms," not behind a helper. There are exactly - two sanctioned time mechanisms in tests: a `testing/synctest` bubble, or - an injected fake clock/timer seam.** If code under test reads real time - and cannot run in a bubble, the fix is to add the seam to the production - code, not to sleep in the test. Reviewers treat any `time.Sleep` in a - test diff as an automatic blocker; do not push one expecting discussion. - The single carve-out is cross-process observation (the exception bullet - above), and only through `internal/testpoll` — never a bare sleep loop - written inline. To simulate a hung component, block on a channel closed - in `t.Cleanup`; in a bubble the hang deterministically outlasts any - timeout with zero wall-clock cost, and the cleanup release lets the - goroutine exit before bubble end. -- **No guessed deadlines.** Block directly on channels for expected events - and let the test binary timeout catch hangs; don't wrap waits in short - arbitrary `time.After` failsafes that flake under load. The rule binds the - Node/jsdom end-to-end scripts too (`tools/hub/e2e/real_e2e.mjs`, - `tools/monitor/e2e/real_e2e.mjs`): each waits on a CONDITION through its - own `waitFor` helper, never `await sleep(N)` followed by an assertion. A - fixed sleep before an assertion fails two ways at once — it flakes when - the real round trip runs long, and it passes VACUOUSLY when the state it - checks is trivially still the pre-action value. A wait for the state a - negative assertion needs (the render that must not move the viewport) - makes the assertion mean what its message claims. -- Always run with `-race`; CI runs `go test -race ./...`. -- `t.Helper()` in every test helper; `t.Cleanup` over `defer` in helpers so - cleanup composes. -- `httptest` for HTTP surfaces; in-process pipes (`net.Pipe`) for protocol - tests — never spawn real subprocess fixtures unless the subprocess - machinery itself is under test. -- Table tests where cases multiply; golden JSON comparisons for transcoders - (struct field order makes marshaled output deterministic). -- Production timers use `time.NewTimer` + `defer Stop()`, not `time.After`, - when the surrounding function can return before the timer fires. -- **Regression tests must be red-verified.** Prove the test fails against the - pre-fix code — revert the fix, observe red, re-apply it — and show that - evidence. A regression guard that never ran red is unverified. -- **Red-verify the NAMED mechanism, not just some failure.** A test name is a - claim. Revert the exact mechanism the name asserts, then confirm THAT test - fails for THAT reason. A test that passes from birth, or that goes red for - an unrelated reason, is not a guard. (Incident: three tests on one branch - were green against the exact defect they were named for.) -- **Verification drives the production entry point.** Call the same function - production calls. A check that builds, normalizes, or repairs its input by - hand proves nothing about the path a user takes — it verifies the - preparation. (Incident: a fix was reported "verified end-to-end" from a - test that called `Normalize` by hand. That skipped the `LoadSession` resume - path, which was the only path that mattered.) -- **An oracle never imports the implementation.** Derive a property-test - oracle from the external contract — the provider's wire rules, the API - spec. A predicate that calls a production symbol, or copies its logic, - cannot fail on a wrong definition, which is the defect class an oracle - exists to catch. (Incident: `hasOrphanToolCall` was rewritten to call the - production `hasToolCall`, and then could not see the data loss beside it.) -- **Assert the surplus direction too.** A count check that only looks for - what is missing passes a payload that ships two of something where one - belongs. - -## Working model — director and coordinator - -The director sets direction; the agent runs as tech-lead coordinator. The -director wants speed and ownership: run the pipeline, and surface only what -genuinely needs a human. - -- **The pipeline.** Decompose work into tasks. Dispatch one fresh - implementation agent (fast mid-tier model) per task in an isolated git - worktree; a strongest-tier reviewer then drives the PR to ZERO findings; - then merge. Parallelize tasks with disjoint files; sequence tasks that - share files, to avoid self-inflicted merge conflicts. See "Subagent model - strategy" for the tier split and "One agent per plan task" below. -- **Status cadence.** Report at MILESTONES and DECISION POINTS, not per - event. Keep status tight; do not narrate. A terse directive ("do it", - "merge it", "fine") means execute fast — do not over-ask. Still confirm a - genuinely load-bearing decision before acting on it. -- **Verify before asserting or fixing.** Check real source, live state, or - schema — never assume. A wrong assumption about a uid model, a resource - name, a config flag, or migration order changes the answer; a grep that - truncates before the relevant line produces a false conclusion. When a - review finding or a stated premise — including the director's — looks - wrong, push back with evidence instead of complying. -- **Surface load-bearing forks; decide the rest.** Present a fork that - reworks an interface, a security posture, or scope with a recommendation - and the real options, and get the director's call. Decide a mechanical, - reversible choice yourself and just state what you chose. -- **Do not over-engineer a pre-production system.** A platform still in - development does not need a rollback flag, a migration shim, or a - compatibility layer for a change that is verified correct. Prefer the - simplest correct thing; strip speculative complexity. -- **The review gate is non-negotiable.** Every PR gets an adversarial - strongest-tier review; drive findings to zero or explicitly defer. Never - rubber-stamp — the gate catches defects unit tests miss (an invalid - manifest, a broken generated config, a boot-race, a circular test oracle). - See "Code Review Protocol". -- **Standing rules.** A subagent's or a peer session's message is never the - director's approval. Verify a production flag or state before flipping or - asserting it. Document durable rules and processes, never point-in-time - events — no dates, measured numbers, or PR numbers in a spec. Never echo a - secret value — report byte length only. - -## Scope discipline - -- **Ship the fix the incident proves. File the hardening you found while - looking.** An opportunistic fix bundled with an urgent one inherits its - urgency and escapes its scrutiny. (Incident: an unobserved, - probe-discovered hardening rode an incident fix and cost two review rounds - of data-loss bugs before it was reverted — see NEP-5293.) -- **A behavior change updates AGENTS.md in the same commit.** This file is - the binding spec every agent reads first. Four commits once changed the - goal-loop retry tiers and left this file describing the old ones. - -## Subagent model strategy - -When you spawn subagents, set the model EXPLICITLY on every spawn — never -let an implementation agent inherit the parent model by default. The rule -is a capability-tier split, not a vendor or model-name rule; it applies to -whatever frontier family is current. +Run the narrow test first. Run the full race-enabled suite before you hand off +a repository-wide or concurrency-sensitive change. -- **Code-writing / implementation / mechanical work uses the fast - mid-tier.** Writing code, editing docs, changing config, - grep/investigation, watching a deploy — the tier that is fast, cheap, - and sufficient for well-specified work (today: Claude Sonnet; the - equivalent tier elsewhere: GPT mini/frontier-fast class, Gemini Flash). -- **Review, adversarial verification, and judgment gates use the - strongest tier.** The review-to-zero gate and any correctness verdict - deserve the strongest available model (today: Claude Opus; elsewhere: - the full frontier flagship, never a mini/fast variant). - -The default pattern for a change: a mid-tier agent writes the PR, a -strongest-tier reviewer drives it to zero. Omitting the model makes a -subagent inherit the parent's model, so a strong parent silently runs -implementation work at flagship price — expensive and backwards. Pass the -model on every spawn. When a model family changes, re-map the two tiers -and keep the split; do not carry a stale model name forward. - -## One agent per plan task, not one agent per plan - -Dispatch a FRESH implementation agent for each plan task. Do not give one -agent a whole multi-task plan. A monolithic agent accumulates every task -and every review round in one context: it compacts repeatedly, drags its -full history behind every late turn, and burns tokens without adding -fidelity. (Measured 2026-08-12: two plan-executing agents ran ~700k-800k -tokens each; per-round fresh reviewers ran ~120k-300k and stayed sharp.) - -- Plans are written for a zero-context engineer (exact files, signatures, - test code — see the plan format), so a fresh agent per task loses - nothing. Give each task-agent the plan file path and its ONE task. -- Reviewers stay fresh per round, as they already are. -- Keep one agent across tasks only when the tasks share heavy state that - a plan file cannot carry (a live debugging session, an unreproducible - environment). -- Give every dispatch exact file:line pointers instead of letting the - agent re-derive repo context by grep. - -## Never end a turn to wait on an external event - -An agent that ends its turn "waiting" on an external event can wait -forever. An external event is any completion signal from outside the -agent's own tool calls: a GitHub workflow run, a deploy, a remote queue. -No notification arrives for an external event. Six agents stalled this -way on 2026-08-12 alone. - -- Watch a workflow run with `gh run watch --exit-status` in the - foreground. Do not poll once and yield. -- A spawned subagent (a reviewer) DOES notify on completion — but its - reply can misroute when it cannot resolve your address. If a verdict - is overdue, message the reviewer and ask; do not keep waiting. -- To wait on any other external event, use a blocking command in the - foreground, or the Monitor tool with an until-loop when the harness - blocks foreground sleep. End your turn only when the task is complete - or you are blocked on input only a human can provide. - -## Debugging invariants - -Rules learned from production incidents (2026-07-09), written so they apply -without knowing the incidents: - -- **Cleansing marshals hide poison.** Persisted session logs are scrubbed by - the guarded marshal paths (`ToolCall.safeArguments` normalizes, - `ProviderData.MarshalJSON` drops empty entries), so on-disk state can be - provably clean while resident in-memory state is unmarshalable. When a - resident session misbehaves but its journal round-trips cleanly through - `engine.LoadSession` + `json.Marshal`, the defect lives in memory between - ingest and persist — do not conclude from a clean log that no defect - exists. (Incident: truncated `ToolCall.Arguments`, fixed in the commit - titled "fix(message,engine): truncated ToolCall.Arguments must never - poison history"; see also the tests in `engine/tool_call_poison_test.go`.) -- **Error text names the rejection, not the cause.** Treat error strings as - the symptom surface — enumerate which layer actually produced the - credential/config/input being rejected before acting. (Incident: a git 403 - citing SAML SSO was actually a system-level gitconfig credential helper - serving a rotated-stale token; the SSO re-auth it demanded was - irrelevant.) -- **Verify binary identity before blaming staleness.** A deployed binary's - exact commit is embedded — `go version -m ` shows - `vcs.revision`/`vcs.time` — check that before hypothesizing that a fix is - missing from a running process. - -## Commit messages and PR descriptions - -Model: https://go.dev/wiki/CommitMessage. A commit message is documentation -for a future reader who has none of your context; the diff shows what -changed, the message must carry everything the diff cannot. - -Subject line: [Conventional Commits](https://www.conventionalcommits.org/), -`type(scope): description` (e.g. `fix(server): stop false-idle wake -between back-to-back turns`) — the repo's existing convention. Lowercase -description, no trailing period, under ~72 chars; scope names the primary -package. - -Body — required for every non-trivial change, written as prose, wrapped -~76 columns, in this shape: - -1. **The problem, as a story a reader can follow.** What was observably - wrong, who hits it, how it was found. Not "fix race in waitSnapshot" — - say what the caller experienced ("a waiter on until=idle could wake in - the gap between a turn marking idle and its tail dispatching the next - queued prompt, and read a transcript that was about to change"). -2. **Why this design.** The mechanism chosen and the reasoning — including - alternatives considered and rejected, and why. If review or a design - fork shaped the outcome, say so ("a suppression design was abandoned - because collectUntilIdle depends on unconditional idle emission"). -3. **The semantic change.** What is now true that was not, stated - precisely, including deliberate non-changes ("status reporting is - unchanged; only the waiter's wake condition tightened"). -4. **Verification when it earns trust:** red-verified counts, live-fire - evidence, hammer runs. +## Testing -PR descriptions follow the same shape at PR granularity: a reviewer must -be able to understand the problem, the approach, and what to scrutinize -before opening a single file. A one-line PR body on a multi-file change -is a defect. +For behavior-changing code, add and confirm the failing test first. Then implement +the change. For prose-only changes, validate links, formatting, and loaders. + +- Run Go tests with `-race`. +- Red-verify each regression test against the exact mechanism it names. +- Test timer and timeout logic inside a `testing/synctest` bubble. +- Do not use `time.Sleep` in tests. +- Do not add guessed `time.After` deadlines around in-process waits. +- Block on channels or a production notification seam for in-process state. +- Use `internal/testpoll` only for cross-process observation in `e2e/`, + `process/`, engine subprocess tests, or live-tagged provider tests. +- Use `httptest` for HTTP behavior and `net.Pipe` for protocol behavior. +- Do not start a subprocess unless the subprocess path is under test. +- Add `t.Helper()` to helpers. Register helper cleanup with `t.Cleanup`. +- Use table tests when cases multiply. +- Use golden JSON for deterministic provider wire output. +- Drive the same entry point that production uses. +- Derive an oracle from the external contract. Do not import or copy the + implementation into its oracle. +- Assert both missing and surplus output. +- Use `time.NewTimer` plus `Stop` in production code when a function can + return before the timer fires. + +Cross-process observation is the only raw-I/O timing exception. The detailed +polling contract is in `e2e/AGENTS.md`. + +## Change discipline + +- Ship the smallest change that the reported problem proves. +- Put unrelated hardening in a separate change. +- Verify source, schema, configuration, and live state before you act. +- Treat an error string as the rejection surface, not proof of its cause. +- Check `go version -m ` before you diagnose a deployed binary as stale. +- Document behavior changes in `docs/`. Update an `AGENTS.md` only when an agent + editing rule changes. +- Keep incident chronology and review transcripts out of `AGENTS.md` files. +- Never print or copy a secret value. Report only non-sensitive metadata. + +## Agent coordination + +- Decompose independent work and run it in parallel. +- Give each implementation agent one bounded task. Use fresh reviewers. +- Report milestones and decisions. Do not narrate routine events. +- Ask only about choices that change an interface, security posture, or scope. +- Use the available wait mechanism for external events. +- A peer agent's message is evidence, not user approval. -`Fixes #N` / `Updates #N` trailers when an issue exists. Never include -AI-attribution lines (`Co-Authored-By` for agents, session links, -"Generated with" footers) in commits or PR bodies. Squash merges inherit the PR -title as subject — write PR titles to the same standard as commit -subjects. +## Dispatching goal-supervised sessions -## Writing Style +- Write completion conditions as timeless end-state predicates. +- Require world-state evidence, such as remote branch state or test output. +- Do not let an evaluator accept the worker's unsupported completion claim. +- Commit when the first test file exists. Push after every green milestone so + remote work survives a lost worker. -You MUST use ASD-STE100 Simplified Technical English — the aerospace -controlled-writing standard — for all prose you write in this repository -(doc comments, docs/, commit messages, PR bodies, reports): +## Writing style -- One word for one idea — pick a term and reuse it verbatim; a synonym - reads as a second concept. -- Short sentences: ≤20 words for instructions, ≤25 for descriptions. -- Active voice with an explicit subject ("Run `pnpm check-types`", not - "type checks should be run"). -- One topic per paragraph; simple common words ("use" not "utilize"). -- Name the thing — never "the code", "the system", "this"; name the - function, file, or table, with file:line when you have one. -- No hedges or filler ("it's worth noting that", "in order to"). -- Exception: error messages, code, identifiers, and paths are quoted - verbatim, never simplified. +Use ASD-STE100 Simplified Technical English for repository prose. +Use active voice, common words, and one stable term for each concept. +Keep instructions at 20 words or fewer when practical. Name code elements +instead of using vague references. Quote identifiers and error strings exactly. -## Code Style +Use standard Go style. Run `gofmt` and `go vet`. Prefer explicit exported +types and small interfaces. -- Standard Go conventions, `go fmt`, `go vet` clean. -- Type annotations in exported APIs over cleverness; small interfaces. +## Commits and pull requests -## Code Review Protocol +Use `type(scope): description` Conventional Commit subjects. Keep the +description lowercase, without a final period, and about 72 characters or less. +For a non-trivial change, explain the problem, design, semantic change, and +verification in the commit or pull request body. -PRs merge only after the latest automated review round has been read **in -full — including the summary comment**. Inline-thread count is not a merge -gate: the reviewer files findings both as inline threads and as items in the -top-level summary, and both must be addressed (or explicitly acknowledged as -deferred) before merge. Iterate until a round produces zero findings. +Use `Fixes #N` or `Updates #N` when an issue exists. Do not add +AI-attribution footers. -A green check is not a review. The reviewer has failed silently before: an -instant API error produces a placeholder comment and zero findings, which -reads as mergeable. Before merging, verify the review summary is substantive. +## Code review -Read and act on every review thread individually — never batch-resolve. One -explicit resolve command per thread id. A batch operation once resolved -unread findings. +Read the latest automated review in full, including its top-level summary. +Treat a placeholder or failed review as a failed gate. Address each finding or +record an explicit deferral. Iterate until a substantive round reports zero +findings. +Read and resolve each review thread individually. Do not batch-resolve threads. +A green check alone is not a substantive review. diff --git a/README.md b/README.md index cb40d0d9..3175e2d8 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,9 @@ A fast, extensible, composable agent harness in Go. - **Composable** — headless engine, event streams, client/server, MCP both directions - **Model-fluid** — swap providers/models mid-session or per-subagent with no migration -See [AGENTS.md](AGENTS.md) for architecture and design decisions. +See [AGENTS.md](AGENTS.md) for repository-wide rules and the scoped +instruction index. Each major subsystem has its own concise `AGENTS.md`. +See [docs/README.md](docs/README.md) for technical documentation. ## Configuration diff --git a/cmd/harness/AGENTS.md b/cmd/harness/AGENTS.md new file mode 100644 index 00000000..95205216 --- /dev/null +++ b/cmd/harness/AGENTS.md @@ -0,0 +1,76 @@ +# Harness command instructions + +These rules apply to `cmd/harness/`. Harness does not merge ancestor files. +If root guidance is not active, locate the Git root and read +`/AGENTS.md`. Resolve repository paths from that root. + +## Composition root + +Keep the command package thin. It resolves config, constructs providers and +managers, and composes the engine, server, and embedded local tools. + +Do not move engine behavior into command handlers. Do not make `server` import +`tools/*`; inject pages and dependencies through options. + +## Startup budget + +Keep `harness --version` on the millisecond startup path. + +- Do not add network access at startup. +- Before first output, limit disk reads to the user and project config files. +- Do not start a long-lived plugin process before its first hook or tool call. + A bounded manifest probe can run for a missing or stale cache entry. +- Do not start MCP servers or provider clients before first use. +- Do not scan skills or project instructions at `NewSession`. +- Do not add `init()` side effects. +- Keep plugin manifests and model metadata local and static on the hot path. +- Keep production dependencies pure Go. + +Run the startup budget tests after a change to command initialization. + +## Config and environment resolution + +The command layer resolves environment variables. The engine must not read +operator environment variables directly. + +Keep one decision point for each config precedence rule. Preserve explicit +zero, negative opt-out, and unset distinctions. + +Validate adapter-only fields against the adapter that an entry builds, not the +provider map key alone. Fail loudly on an unreadable or unknown value. + +Keep run and serve wiring in parity for shared engine settings. + +## Provider construction + +A provider-map key is the model-reference family. Pass that family into native +Responses clients so opaque data cannot cross endpoints. + +Do not validate credentials during registry construction. The first provider +request owns credential validation. + +## Monitor and hub composition + +`cmd/harness` may import `tools/hub` and `tools/monitor`. The server may +not. + +Only allow empty-token unauthenticated service when `resolveUnauthenticated` +proves loopback or receives the explicit non-loopback opt-in. Keep the two +warning messages distinct. + +Print tokenized monitor URLs only to an interactive terminal. Do not write a +tokenized URL to piped production logs. + +## GC and pprof diagnostics + +Use `runtime/metrics` for GC pause observation. Do not use +`runtime.ReadMemStats` in the watcher. + +Never import `net/http/pprof`. Keep the command-level default-mux regression +test because the binary import graph is wider than the server package. + +## Tests + +Test flag and environment precedence as tables. Cover malformed values and +explicit zero values. Do not use a live provider or a real deployment command +in ordinary command tests. diff --git a/cmd/harness/CLAUDE.md b/cmd/harness/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/cmd/harness/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/cmd/harness/main.go b/cmd/harness/main.go index fd7b5071..68d95c11 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -1,7 +1,7 @@ // Command harness is the CLI for the harness agent engine. // -// Startup speed is a budget (see AGENTS.md): nothing here touches the -// network, spawns processes, or reads more than flags before first output. +// Startup speed is a budget (see cmd/harness/AGENTS.md): nothing here touches +// the network or spawns processes before the selected command needs it. // Provider auth is validated on first message send, not at boot. Session // persistence is lazy too: the engine creates the session directory and log // file on first message append, and the CLI reads the directory only when @@ -235,7 +235,7 @@ func usage() { harness sessions [--json] list persisted sessions harness hub [-addr host:port] [-spawn-command cmd] serve the local fleet hub UI (see - AGENTS.md's "Development hub" section) + tools/AGENTS.md's "Development hub" section) harness version print version run flags: @@ -1622,7 +1622,7 @@ func serveCmd(args []string) error { Plugins: pluginInfoFn(pluginHost), // MonitorPage: every `harness serve` box offers its own same-origin // monitor at GET /monitor — no CORS/-cors-origin dance, no - // separately hosted copy required (see AGENTS.md's "Session + // separately hosted copy required (see tools/AGENTS.md's "Session // monitor" section). tools/monitor.Page embeds the exact committed // tools/monitor/index.html; the static/file:// hosting path it // documents keeps working unchanged alongside this. diff --git a/cmd/harness/plugins.go b/cmd/harness/plugins.go index 0e960fb9..d8ec8a94 100644 --- a/cmd/harness/plugins.go +++ b/cmd/harness/plugins.go @@ -43,10 +43,9 @@ func pluginCachePath() string { return filepath.Join(filepath.Dir(config.Path()), "plugin_cache.json") } -// pluginManifestCache is the on-disk manifest cache named in AGENTS.md: -// "harness plugin install runs the binary once and caches its manifest ... -// keyed by binary hash". Entries are keyed by plugin name *and* a digest of -// the plugin's Config/Env/Dir (see pluginCacheKey/pluginSpecDigest) so a +// pluginManifestCache is the on-disk probe cache described in +// plugin/AGENTS.md. Entries are keyed by plugin name *and* a digest of the +// plugin's Config/Env/Dir (see pluginCacheKey/pluginSpecDigest) so a // renamed config entry, or a plugin whose config/env/dir changed since it // was last probed, both re-probe rather than silently reusing an unrelated // or stale manifest. The binary's own identity (content hash, size, mtime) diff --git a/config/AGENTS.md b/config/AGENTS.md new file mode 100644 index 00000000..7eaf6732 --- /dev/null +++ b/config/AGENTS.md @@ -0,0 +1,49 @@ +# Config instructions + +These rules apply to `config/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. +Read `cmd/harness/AGENTS.md` for environment resolution and provider +construction. + +## Config model + +Keep configuration flat and cheap to parse. The config package must not perform +network access, start a subprocess, or initialize a provider. + +Use pointer fields when zero and unset have different meanings. When a field's +documented contract distinguishes zero, false, empty, or a negative opt-out +from unset, preserve that distinction through merging. Slice fields may use a +different documented rule, such as an empty project value inheriting the user +value. + +## Layering and merge + +Load user config first and project config second. A project value overrides the +user value according to the field's documented merge rule. + +Validate providers after merge and native-default application. A partial +project entry can become valid through its inherited fields. + +Keep `LoadInfo` observational. It reports the effective source and summary; it +must not affect behavior or trigger a second read. + +## Provider fields + +Validate an adapter-specific field against the adapter that the entry builds. +Do not infer adapter identity from the map key alone. + +Reject unreadable or unsupported values. Do not silently select a different +cache policy, request path, tool-loading mode, or durability mode. + +## Environment boundary + +The config package can define config values and merge semantics. The command +package owns operator environment variables and precedence. The engine must not +read those variables directly. + +## Tests + +Use table tests for merge and validation matrices. Cover absent, explicit zero, +negative, and malformed values. Assert the final merged config, not one layer +before defaults apply. diff --git a/config/CLAUDE.md b/config/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/config/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/config/config.go b/config/config.go index f5799f91..e10e5bf3 100644 --- a/config/config.go +++ b/config/config.go @@ -901,7 +901,7 @@ func LoadProject(dir string) (*Config, error) { // LoadInfo describes which config file LoadProjectWithInfo actually found // (if any) and summarizes the resulting merged config, for the one boot- // time observability log line `harness serve`/`harness run` emit (see -// AGENTS.md's startup-config-observability rule). It carries no +// config/AGENTS.md's "Layering and merge" section). It carries no // behavior — Path is purely which file to report to an operator, never // re-parsed or re-read. type LoadInfo struct { diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..ed4837a7 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,39 @@ +# Harness documentation + +Use this index to find technical documentation. The current source and tests +remain authoritative when historical material describes an earlier behavior. + +## Runtime behavior + +| Document | Subject | +|---|---| +| [engine-request-cycle.md](engine-request-cycle.md) | Request assembly, file tools, retries, and metrics | +| [goal-loop.md](goal-loop.md) | Goal supervision and evaluator behavior | +| [session-storage-and-queue.md](session-storage-and-queue.md) | Indexes, snapshots, paging, queues, and processes | +| [models-and-providers.md](models-and-providers.md) | Model state, effort, cache affinity, and adapters | +| [mcp-tool-loading.md](mcp-tool-loading.md) | Deferred MCP schemas and stable tool ordering | +| [plugins-and-protocols.md](plugins-and-protocols.md) | Plugin lifecycle and external protocol boundaries | +| [development-interfaces.md](development-interfaces.md) | Hub and monitor behavior | +| [fleet-and-serve.md](fleet-and-serve.md) | Fleet state, lineage, exhaustion, and diagnostics | +| [deploy-modal.md](deploy-modal.md) | Deployment modal behavior | + +The plugin wire contract is in [plugin/PROTOCOL.md](../plugin/PROTOCOL.md). +The goal-loop implementation history is in +[history/goal-loop-resilience.md](history/goal-loop-resilience.md). + +## Designs and plans + +`design/` contains architectural designs and durable decisions. `plans/` +contains dated implementation plans. Keep current behavior in the runtime +documents above and keep superseded chronology in `history/` or `plans/`. + +| Design | Subject | +|---|---| +| [context-compaction.md](design/context-compaction.md) | Automatic and manual context compaction | +| [fleet-model.md](design/fleet-model.md) | Task lineage, fleet state, and provider exhaustion | +| [goal-retry-directive-reuse.md](design/goal-retry-directive-reuse.md) | Durable directive reuse across goal retries | +| [journal-snapshotting.md](design/journal-snapshotting.md) | Journal snapshot format and recovery | +| [managed-processes.md](design/managed-processes.md) | Box-scoped managed process lifecycle | +| [mcp-lazy-tools.md](design/mcp-lazy-tools.md) | Deferred MCP schema design | +| [nested-instruction-loading.md](design/nested-instruction-loading.md) | Project instruction discovery and truncation | +| [monitor-mockup.html](design/monitor-mockup.html) | Monitor visual reference | diff --git a/docs/design/fleet-model.md b/docs/design/fleet-model.md index b01fef78..63df0e27 100644 --- a/docs/design/fleet-model.md +++ b/docs/design/fleet-model.md @@ -14,10 +14,10 @@ identifies a box, what survives its death, what a client can rely on across a restart, and the one environment-variable contract a hub and a box's deployment tooling share. It is a build spec in the same register as `docs/design/context-compaction.md` — states, wire fields, invariants, a -test list — except most of what it specifies is **already implemented** -(session lineage and goal pausing shipped alongside this document); only -§8's hub-side half is deliberately deferred. `docs/design/managed- -processes.md` is a later doc in the same register: dev/support processes +test list — except most of what it specifies is **already implemented**. +Session lineage, goal pausing, and the §8 hub-side contract have shipped. +`docs/design/managed-processes.md` is a later doc in the same register: +dev/support processes (`pnpm dev` and the like) are explicitly box-scoped state, exactly like everything else this document describes — a managed process does not survive its box's death, and nothing in that design tries to make it. @@ -231,10 +231,9 @@ retryable && waiting) — see §9 and `server/openapi.yaml`. ## 8. The hub spawn contract: `HARNESS_HUB_BOX_NAME` -This section documents a contract; **it is not implemented in this repo** -(the hub itself is out of scope here and lands on another branch). It is -recorded now so the deployment side of the model in §1–§2 has one concrete -handle to build against. +`tools/hub` implements this contract. Deployment-specific provisioning stays +outside Harness: the operator supplies the spawn command, and Harness passes +the selected box name through the environment described below. When a hub spawns a box, it generates or selects the box's NAME (§1) and passes it to the spawn command's environment as `HARNESS_HUB_BOX_NAME`. @@ -250,8 +249,8 @@ the latter (see `cmd/harness/main.go`'s `sessionDir`) and has no notion of run unmodified whether it was spawned by a hub, a human running a script by hand (who sets `HARNESS_SESSION_DIR` directly), or a test harness. -See `AGENTS.md`'s "Fleet model (the deploy story)" section for the -short-form cross-reference back to this document. +Scoped implementation rules live in `engine/AGENTS.md`, `server/AGENTS.md`, +and `tools/AGENTS.md`. ## 9. Wire fields (summary) diff --git a/docs/design/journal-snapshotting.md b/docs/design/journal-snapshotting.md index 25e6eb91..3ce34238 100644 --- a/docs/design/journal-snapshotting.md +++ b/docs/design/journal-snapshotting.md @@ -1,6 +1,6 @@ # Journal Snapshotting — design -**Status:** draft for review (2026-08-27). Author: coordinator, via brainstorm with Andy. +**Status:** Harness Layer B implemented (2026-08-27); Boxes Layer A remains proposed here. **Repos:** harness (`majorcontext/harness`, Layer B) + boxes (`meetneptune/boxes`, Layer A). **Extends:** `boxes/docs/design/console-read-path.md` (this is the source-level fix its workstream 1 gestured at, and the seam its workstream 5 mirror plugs into). diff --git a/docs/design/mcp-lazy-tools.md b/docs/design/mcp-lazy-tools.md index a1d3c227..02c23e85 100644 --- a/docs/design/mcp-lazy-tools.md +++ b/docs/design/mcp-lazy-tools.md @@ -14,8 +14,9 @@ turn, before the model has read one word of the user's request. The cost is structural, not incidental: - Tool schemas sit at the FRONT of the cached prefix on every provider - (Anthropic caches tools, then system, then messages — see AGENTS.md, - "The tool array is byte-stable across requests"). A large catalog + (Anthropic caches tools, then system, then messages — see + `docs/mcp-tool-loading.md`, "The tool array is byte-stable across requests"). + A large catalog inflates every cache write and every cache read for the life of the session. - A catalog the model never uses still competes for attention with the @@ -312,8 +313,9 @@ plan to after it, so two things change for that hook: `system.transform` is handed the session id and the model, never the tools array or the system slice. -Both are behaviour changes outside the opt-in path, so they land documented -in AGENTS.md and pinned by a test, not silently. +Both are behavior changes outside the opt-in path. Record them in +`docs/mcp-tool-loading.md` and `docs/engine-request-cycle.md`, and pin them +with regression tests. Shape: @@ -812,10 +814,10 @@ The action gate, in both directions: a global-`eager` session ## 10. Non-goals -- **No provider-side tool search.** Anthropic's server-side tool-search - beta solves this inside one vendor's API. Harness is provider-agnostic, - and a per-provider mechanism would not serve the openai, openaicompat, - or gemini routes. +- **Do not make provider-side tool search the portable contract.** The + Anthropic adapter delegates deferred schemas to supported first-party + models through server-side tool search. Other provider routes still use + Harness's `mcp` catalog, `search`, and `select` flow. - **No deselect, no eviction, no TTL** on a selected tool (§4). - **No cached catalog for an unconnected server.** `search` ranks over live tools only (§4). diff --git a/docs/design/nested-instruction-loading.md b/docs/design/nested-instruction-loading.md index 685299f1..5729ac2c 100644 --- a/docs/design/nested-instruction-loading.md +++ b/docs/design/nested-instruction-loading.md @@ -4,6 +4,12 @@ Status: the head-plus-outline half is IMPLEMENTED (`engine/instructions_outline.go`). The nested attach-on-read half, ported from opencode's `resolve()`, is still design only — see "Proposed split". +This repository does not depend on the unimplemented half. Its root +`AGENTS.md` lists each scoped file and tells an agent to read the matching +scope before an edit. Each scoped file links back to the root because the +current loader selects only one closest file. This is an instruction +convention, not automatic nested attachment. + ## Problem `engine/instructions.go` injects one project instruction file into the system @@ -12,10 +18,12 @@ is now loud on both channels: the model reads a marker, the operator reads a WARN line. Loud is not enough. The content past the cap is still absent from the prompt, and the model must guess that it needs the rest. -Two files make this concrete. The boxes repository has a 408 KiB `AGENTS.md`: -84% of it never reaches the model. This repository's own `AGENTS.md` is -180,631 bytes over 2,866 lines with 51 headings: 116 KiB of binding -specification is dropped from every session today. +Two files made this concrete. The boxes repository had a 408 KiB +`AGENTS.md`: 84% of it never reached the model. This repository also had one +root instruction file above 180 KiB and 2,800 lines. Harness later split that +file into a concise root index, scoped files, and subject-based documentation. +The measurements below refer to the former monolithic fixture, not the +current root file. ## What the model sees @@ -31,7 +39,7 @@ does not fully contain, each line carrying the heading text, the exact `read_file` line range, and a short teaser from the section body: ``` -Project instructions from AGENTS.md (sections 17-51 are not in this prompt). +Project instructions from AGENTS.md (later sections are not in this prompt). Read a section with the read_file tool before you rely on it: read_file(path: /repo/AGENTS.md, offset: , limit: ) @@ -96,7 +104,7 @@ invariant the loud-truncation change established holds unchanged. The outline has its own budget so a pathological file cannot spend the prompt on an index: teasers are dropped first, then the list degrades to headings and -ranges only. Measured on this repository's `AGENTS.md`: 35 outlined sections +ranges only. Measured on the former monolithic `AGENTS.md`: 35 outlined sections cost 5,965 bytes with teasers, 1,424 bytes without. A 408 KiB file with the same heading density holds roughly 115 sections, so its outline lands near 19 KiB with teasers and near 4.7 KiB without — the budget picks the second. @@ -104,7 +112,7 @@ same heading density holds roughly 115 sections, so its outline lands near ## The 408 KiB boxes AGENTS.md, concretely Eager: the head, the first sections up to the last heading boundary under -64 KiB. Measured on this repository's file, that boundary is byte 41,648 — +64 KiB. In the former monolithic fixture, that boundary was byte 41,648 — sections 1-16 of 51. Cutting on a heading costs 23 KiB of eager text against a raw byte cut, and buys a head that never ends mid-sentence; the sections that pay that cost are listed in the outline, so they are reachable, not lost. @@ -139,10 +147,10 @@ the session, and never written to the session log. The test drives the real `read_file` tool with the outline's own ranges and compares the returned lines against the file, rather than comparing the outline to itself. -2. **Fenced code blocks.** This repository's `AGENTS.md` holds ```` ```bash ```` - blocks whose comment lines start with `#`. A naive heading scan reads them - as sections and emits ranges that point at shell comments. The scanner - tracks fences; a test file with a `#` line inside a fence pins it. +2. **Fenced code blocks.** A synthetic instruction file puts a shell comment + that starts with `#` inside a fenced block. A naive heading scan reads the + comment as a section and emits a wrong range. The scanner tracks fences; + the synthetic file pins the behavior. 3. **Head boundary.** The head must end on a heading boundary and must never exceed `MaxBytes`. A file whose first heading is past the cap has no boundary to cut on and falls back to the marker path. @@ -185,5 +193,5 @@ example in a four-backtick fence, and a boolean toggle closes the outer fence on the inner one, then reads the rest of the document as sections. `fenceState` in `engine/instructions_outline.go` carries the open fence's character and run length. A review of the implementation raised this shape; -this repository's own AGENTS.md does not hold such a fence today, so the rule -is hardening for other projects' files, not a fix for an observed break. +the original fixture did not hold such a fence, so the rule is hardening for +other projects' files, not a fix for an observed break. diff --git a/docs/development-interfaces.md b/docs/development-interfaces.md new file mode 100644 index 00000000..3bb7cc5e --- /dev/null +++ b/docs/development-interfaces.md @@ -0,0 +1,227 @@ +# Development interfaces + +This document describes the local hub and session monitor. + +## Development hub + +`harness hub` is a local, single-operator control surface over a FLEET of +`harness serve` boxes — a fleet dashboard for "what are my agents +doing right now" and for dispatching new goal-supervised sessions, not a +deployed product. It serves one embedded, single-file page +(`tools/hub/index.html`, `go:embed`) on +`localhost:7777` by default (`-addr` to change it). + +- **No server-side state.** The hub keeps no registry and reads no config + file: every box (name, base URL, run token) and the current selection + live only in that browser tab's URL fragment, base64-encoded JSON + (`#s=...`), kept in sync via `history.replaceState`. That makes a hub URL + bookmarkable and shareable between local tabs with zero persistence code + — and means **run tokens ride the URL by design**; treat a hub link like + a secret. +- **The page talks to boxes directly** from the browser, over each box's + normal HTTP+SSE API (`server/openapi.yaml`) — never proxied through the + hub's own server. Every box must therefore be started with `-cors-origin` + set to the hub's origin (or `*` for local hacking), e.g. `harness serve + -cors-origin http://localhost:7777`; a box without it will look + permanently unreachable from the hub. +- **The Go side is minimal on purpose**, exactly one API: `POST /spawn`. + It execs the command given by `-spawn-command` (or `$HARNESS_HUB_SPAWN`) + via `sh -c` and streams its combined stdout+stderr live to the page over + SSE. The **spawn-command contract** — the only coupling between this repo + and any deployment-specific provisioning tool — is plain lines anywhere + in that output: `TUNNEL_URL=` and `RUN_TOKEN=` (required to + add the box), and any number of `PORT_URL_=` lines (optional — + one per exposed port's own tunnel/preview URL, collected into a + `port_urls` map; see the process strip in `tools/hub/index.html`'s header + comment). Once the command exits, the stream ends with a summary carrying + those values (if found) and the exit code; the page adds the new box to + its own URL state itself. Nothing box-provisioning-specific lives in this + repo. + - **Box name passthrough.** `POST /spawn`'s JSON body optionally carries + `{"name": "..."}` — the page's generated (or, on a Respawn/ADOPT, reused) + box name. The Go handler sets it as `HARNESS_HUB_BOX_NAME` in the spawn + command's own environment (`tools/hub/spawn.go`'s `runSpawn`), exactly + the deployment-environment contract `docs/design/fleet-model.md` §8 + specifies: deployment tooling invoked by `-spawn-command` reads this + variable to derive per-name storage (typically setting + `HARNESS_SESSION_DIR` from it before `harness serve` starts) — harness's + own code never reads `HARNESS_HUB_BOX_NAME` at all. A request with no + body, or no `name` field, spawns exactly as before (no env var set). +- The hub binds loopback-only by default (`resolveAddr` in `tools/hub/hub.go`). +- **Browser-security hardening** (both in `tools/hub/hub.go`, tested in + `tools/hub/hub_test.go`). `POST /spawn` execs a real, costly provision + command, so `handleSpawn` rejects a browser cross-origin request before any + exec: if an `Origin` header is present it must match the request's `Host` + (OWASP verify-origin). Loopback binding alone does not stop this — any page + the operator visits can `fetch("http://localhost:7777/spawn",{method: + "POST"})` as a no-preflight CORS simple request — but the page's own + same-origin `fetch("/spawn")` (Origin == Host) and non-browser clients (no + Origin, so not a CSRF vector) pass unchanged. The served page also carries + a strict `Content-Security-Policy` (`default-src 'none'` + `'unsafe-inline'` + script/style — the page is a single no-build `go:embed`'d file with no + external resources and no per-response nonce hook — + `connect-src *`, + required because it fetches/streams from arbitrary operator-added box + origins the stateless hub cannot enumerate, + `frame-ancestors`/`base-uri`/ + `form-action` pinned to `'none'`): defense-in-depth for a page holding run + tokens in its URL fragment. +- **Pure hub logic is unit-tested** by `tools/hub/hub_test.mjs` (run: + `node --test tools/hub/*_test.mjs`). **End-to-end, against a real backend** + is `tools/hub/e2e` (see its README): a `go test -race ./...` subtree that + starts an actual `server.Server` + `hub.NewHandler` and drives the real, + served `index.html` with Node + jsdom and an unmocked `fetch` — no manual + setup step; it installs its own `npm` dependency on first run. + +### UI design language + +The hub is styled as **tactical telemetry** — a committed dark-only +brutalist archetype (derived from the public +[taste-skill](https://github.com/Leonxlnx/taste-skill) brutalist + +anti-slop skills). Any new hub UI — and future passes on the inspector, +which still wears the older soft theme — follows these rules: + +- **One substrate, no theme toggle**: `#0a0a0a` background, `#eaeaea` + phosphor foreground, `#2a2a2a` hairline borders. Never reintroduce a + light mode here; pick-one-and-commit is the point. +- **Two semantic colors only.** Hazard red (`--accent`, `#ff2a2a`) means + trouble or destructive action, nothing else. Terminal green (`--ok`, + `#4af626`) is reserved for exactly one semantic: live or succeeded goal + execution. Everything else is monochrome. +- **Monospace dominance**: body text is the `ui-monospace` stack; + headers are heavy uppercase system-ui. Micro-labels are uppercase with + `.06–.1em` tracking. No webfonts — the page is CSP-self-contained. +- **Geometry**: `border-radius: 0` absolutely everywhere; square status + markers; 1px compartment borders; inverted-video hover + (foreground/background swap). No gradients, soft shadows, or + translucency. The scanline overlay is static — motion requires a + stated purpose. +- **Copy discipline**: no emoji in UI strings, no em-dashes anywhere, and + every piece of "telemetry" displayed must be real data (vcs revisions, + seqs, PIDs, token counts) — never decorative or fabricated metadata. +- **Selectors are load-bearing**: the renderers create elements by class + name (`.sess`, `.box-card`, `.dot`, `.goalnarr`, …). Restyle classes; + never rename them in a styling pass. + +## Session monitor + +`tools/monitor` (`tools/monitor/index.html`) is the single-instance +counterpart to the hub above: where the hub answers "what are my boxes doing" +across a FLEET, the monitor answers "what is THIS `harness serve` instance +doing right now" — a live board of every session on that one box (phase, +current tool, staleness), a per-session detail view with a scrolling +transcript, and a composer to speak into a session (`prompt_async`). Like the +hub and the inspector, it is a build-free, dependency-free single HTML file +with no Go-side handler of its own. + +- **How to run it**: open the file directly (`file://`) or serve it from any + static host — nothing box-specific is baked in. The target box must be + started with `-cors-origin` covering the monitor's origin (or `*` for local + hacking), exactly like the hub's requirement above; a box without it looks + permanently unreachable. The base URL and run token are entered in the + page itself and persisted to `localStorage` in plaintext (same documented + tradeoff as the inspector — a dev tool, not for a shared origin with a + long-lived token) so a reload can reconnect automatically. Routing lives in + the URL fragment as small explicit params, not the hub's base64 blob: + `#b=` (box base URL) and `#s=` (open detail view) — both + bookmarkable, encoded/decoded by a tested pure helper. +- **Embedded serving, frictionless local**: every `harness serve` box also + offers its own copy same-origin, at `GET /monitor` — by default + `http://localhost:4096/monitor` (the port follows `-addr`, default + `localhost:4096`; the exact URL, with any `#t=` capability suffix, is + printed to the terminal on startup — `monitorTerminalHint`). The bare root + is a convenience redirect: `GET /` 302s to `/monitor` (via the `GET /{$}` + route — `{$}`-anchored to the root path only, never a catch-all — registered + under the same `MonitorPage` guard, so a pure-API box keeps `/` a clean 404). + `tools/monitor` + (package `monitor`, `embed.go`) `//go:embed`s the exact committed + `index.html`; `cmd/harness`'s `serveCmd` wires it into + `server.Options.MonitorPage`, which the server serves unauthenticated + (like `/health` — the page itself carries no secrets) with a + same-origin-scoped `Content-Security-Policy` (`connect-src 'self'`). + `server` itself never imports `tools/monitor` (layering: `server` must + not import `tools/*`); only `cmd/harness` does, the same pattern + `harness hub` already uses. The `file://`/static-host path is unaffected + — this is additive, and stays the only option for monitoring a box from a + different origin (the embedded route's CSP deliberately does not permit + that). + - **Unauthenticated-on-loopback** (`server.Options.Unauthenticated`, an + EXPLICIT opt-in never inferred from an empty `RunToken` on its own — + `New` still fails closed otherwise): `serveCmd` classifies `-addr` + (`isLoopbackAddr` — `localhost`, `127.0.0.1`/`::1`, any + `net.IP.IsLoopback()` address; a bare `:port`, `0.0.0.0`, `::`, or any + other routable address is NOT loopback). `HARNESS_RUN_TOKEN` unset + + loopback bind runs the box fully unauthenticated (every route, not just + `/health`/`/monitor`) and logs a clear warning; unset + non-loopback + still hard-errors `HARNESS_RUN_TOKEN is required` exactly as before. The + token guards network reachability, and loopback is a server-verifiable + proxy for that (unlike, say, `Origin`, which a client controls). + - **Unauthenticated on a non-loopback bind is also possible, but ONLY via + a SECOND, separate, EXPLICIT opt-in** — the `-unauthenticated` serve + flag, or `HARNESS_UNAUTHENTICATED=1` (`envUnauthenticated`, parsed with + `strconv.ParseBool`; an unset or unparsable value is false, fail-closed + on a malformed setting). `resolveUnauthenticated` (`cmd/harness/main.go`) + is the single decision point both serve flags feed: a non-empty token + always wins (token path unaffected, any bind); an empty token on a + loopback bind is unchanged from above; an empty token on a non-loopback + bind needs this opt-in or it still hard-errors exactly as before — the + opt-in is never inferred from the empty token alone, only from the flag + or env var. This is for a deployment where a trusted external gate + already restricts reachability (e.g. a Cloudflare Access-gated tunnel, + or a sandboxed network boundary) so the token is redundant with that + gate — `server.Options.Unauthenticated` itself is bind-address-agnostic + (see its own doc comment); the safety property lives entirely in + `cmd/harness` deciding WHEN to set it. Landing this on a non-loopback + bind logs a SEPARATE, distinctly worded loud warning ("serving + unauthenticated on a non-loopback bind") from the loopback one above, so + the two are distinguishable in a log search. + - **Same-origin auto-connect** (`embeddedConnectPlan`, index.html): opening + a box's own `/monitor` attempts the connection immediately against + `location.origin`, using whatever token is available (`#t=` fragment, + then a stored one, then none) — success lands straight on the board, no + panel, nothing typed (covers both the unauthenticated-loopback case and + a valid capability URL/returning operator with the SAME call). Only a + failed attempt (a real token is required and none was available) falls + back to a minimal token-only panel — host is already known, so the base + field never reappears. A `#t=` fragment (mirroring `tools/hub`'s + "run tokens ride the URL by design" precedent, `extractFragmentToken`) + is adopted into the SAME `localStorage` key manual entry uses and + immediately scrubbed from the visible URL via `history.replaceState`; a + fragment `#b=` naming a DIFFERENT origin than this box's own — which the + embedded route's CSP would block outright if tried — surfaces a + plain-text notice instead of attempting it, composing with (never + blocking) the own-origin auto-connect. None of this weakens the auth + model itself: a token is still required and still checked exactly as a + hand-typed one would be, on every box that hasn't explicitly opted into + `Unauthenticated`. + - On a TTY, `serveCmd` also prints a click-ready line to stderr after + "serve start" — `monitor: http:///monitor#t=` (or, when + running unauthenticated-loopback, the plain URL with no `#t=` at all, + since there's no credential to carry) — gated on `stderrIsTerminal` + (`os.ModeCharDevice`, stdlib-only) so a tokenized URL never lands in + piped/production stderr, only an interactive operator's own terminal. +- **Test layers.** Pure helpers (SSE parser, activity reducer, transcript + fold, route codec, formatters) live inside index.html's `/* TESTABLE-BEGIN + */ … /* TESTABLE-END */` region and are covered by + `tools/monitor/monitor_test.mjs` (run: `node --test tools/monitor/*_test.mjs`), + using the same extraction-and-`vm`-evaluate pattern as the inspector's + `inspector_test.mjs` (and now the hub's, above) — no build step, so the + tests read the region straight out of the committed HTML. End-to-end, + against a real backend, is `tools/monitor/e2e` (see its README): a `go + test` subtree that starts a real `server.Server` plus a plain static file + server for the actual committed `index.html`, and drives it with Node + + jsdom and an unmocked `fetch` — mirroring `tools/hub/e2e`'s structure and + conventions. A `window.__monitorTuning = {QUIET_MS, STALL_MS}` seam (set + via jsdom's `beforeParse` before the page's own script runs, a no-op in + production since nothing else ever sets it) lets both the unit and e2e + suites shrink the staleness thresholds so `quiet`/`stalled` transitions are + observable in test time instead of real minutes. +- **UI design language.** The monitor deliberately does NOT inherit the + hub's committed dark brutalist archetype above. It carries its own + "instrument sheet" language: light-first with a dark variant, both driven + by one OKLCH token set (`--surface`, `--text-1..3`, `--separator`, etc.); + semantic green/amber/red (`--ok`/`--warn`/`--critical`) are reserved + strictly for session/staleness state, never decoration; a single accent + color is owned by interaction (the composer's send affordance is the + page's only filled-accent control). `docs/design/monitor-mockup.html` is + the user-approved visual spec — its tokens, grid template, and markup + shapes are binding; restyle within that spec rather than importing the + hub's theme onto it. diff --git a/docs/engine-request-cycle.md b/docs/engine-request-cycle.md new file mode 100644 index 00000000..c6c4ee87 --- /dev/null +++ b/docs/engine-request-cycle.md @@ -0,0 +1,671 @@ +# Engine request cycle and tool behavior + +This document is the technical system of record for the engine request cycle +and tool behavior. Read only the sections relevant to the change. + +## Core invariants + +- **A session is an append-only log of typed events.** User messages, model deltas, tool calls, results, model switches — all events. UIs, JSON output, and plugins are subscribers to the same stream. +- **The session log stores the canonical message format, never a provider's.** Every request, the provider adapter transcodes canonical history → provider wire format from scratch (stateless transcoding). Mid-session model swap = next request uses a different transcoder. No migration step. +- **Provider-specific opaque data (reasoning/thinking blocks, encrypted reasoning items) is stored as provider-tagged attachments** on canonical messages: replayed verbatim to the same provider, dropped when crossing providers. Tool-call IDs are internal; each transcoder maps deterministically to provider-compliant IDs. Prompt-cache markers are injected at transcode time, never stored. +- **Model refs are `provider/model`** plus user-defined aliases (`fast`, `smart`) from config. Context-window metadata comes from the curated static `modelmeta` table. It never refreshes over the network. +- **A history repair that runs on live or persisted state is additive-only.** `LoadSession` writes the repaired slice back into live history, so a repair that deletes loses data permanently — not for one request, but for the life of the session. Add synthetic parts; never drop, reorder, or relocate a part another producer wrote. A transcode-time repair MAY be destructive, because it builds one throwaway request and never touches the record. Put every destructive rule on that side of the line. (Incident: a `ResolveOrphanToolCalls` rewrite deleted genuine tool output in three shapes and was reverted; see NEP-5293.) The concrete split is in "Wire normalization" below. +- **An empty tool result must never serialize as `null`.** The provider reads a null-content `tool_result` as ABSENT and rejects the whole request with "tool_use ids were found without tool_result blocks immediately after" — naming a block that IS in the payload. A tool that produces no output (a `grep` that matches nothing) is enough to wedge a session forever. `message.NoToolOutputText`, `ToolResult.SafeContent`, and `ToolResult.MarshalJSON` hold this line; every transcoder reads through `SafeContent`, never `Content`. (Incident: NEP-5272.) + +## Wire normalization + +Two functions repair `tool_use`/`tool_result` pairing. They sit on opposite +sides of the live-versus-transcode line in the invariant above. + +`message.ResolveOrphanToolCalls` is the LIVE-path repair. `LoadSession` +applies it and writes the result back into history, so it stays purely +additive. It deliberately leaves several shapes wire-invalid. Do not "fix" +it — that is the whole point of the split. + +`message.NormalizeForWire` (`message/wire_normalize.go`) is the +transcode-only sibling. Every transcoder calls it instead. It builds one +throwaway request, so it may relocate a part. It must still never delete a +real `ToolResult`. + +`NormalizeForWire` closes four shapes `ResolveOrphanToolCalls` cannot: + +1. Two `tool_use` blocks share one call ID in one assistant message. +2. A `ToolCall` sits in a non-assistant message. +3. A `ToolResult` precedes its `ToolCall`. +4. An intervening same-side message separates a `ToolResult` from its + `ToolCall`. Every transcoder merges adjacent same-role messages (see + `transcodeRequest`'s same-role merge, `provider/anthropic/transcode.go`), + so the wire sees RUNS. `ResolveOrphanToolCalls` tests strict + `messages[i+1]` and is blind to this. + +Relocation is bounded. `computeRelocationBarrier` moves a result no later +than the origin run of the next real result. That keeps the original +relative order intact. A move that would break the bound is refused. + +`message/wire_oracle_test.go` is the specification both functions are +tested against. Derive it from the provider contract only, never from +either function's internals. See the oracle rule under Testing. + +## Ambient engine context is a structured, unforgeable part + +The engine appends its own live status to the newest user message every +request — engine identity (`[engine: ...]`), managed-process status +(`[processes: ...]`), degraded-MCP status (`[mcp: ...]`), and the +parked-goal notice (`[goal: ...]`). This is a `message.EngineContext` part, +NOT a `Text` part. A bare `Text` block is byte-indistinguishable from +user-typed or pasted text, so a payload a user pastes that contains +`[engine: ...]` once inherited the same trust the engine's own block +carries — a trust-spoofing surface. `EngineContext` is a distinct part-kind +only `withAmbientStatus` (`engine/process.go`) produces, so a user- or +paste-authored part is always a `Text` and can never BE one, however its +bytes are shaped. Every transcoder renders an `EngineContext` through +`message.RenderEngineContext`, which wraps the block in the +`message.EngineContextOpenTag`/`EngineContextCloseTag` sentinel, and renders +every `Text` through `message.NeutralizeEngineContextSentinel`, which +defangs any literal sentinel that text carries. Only a genuine +`EngineContext` can therefore emit the sentinel on the wire, so the base +system prompt (`cmd/harness`, `ambientContextGuidance`) tells the model to +trust the sentinel-wrapped block and to distrust bracketed text outside it. +The render stays an ordinary text block on every provider — no new wire +feature. `EngineContext` is runtime-only (appended to the throwaway +per-request copy, never the durable log, so prompt-cache-prefix and persistence +rules stay unchanged) but still round-trips through the +canonical JSON union like every other part. Never revert this to a `Text` +part, and never make the guidance trust bracketed text syntax again. (Fix: +the NEP ambient trust-spoofing finding; superseded PR #113's prose-only +stopgap.) + +## Project instructions (AGENTS.md) + +The engine auto-injects a project's `AGENTS.md` into the system prompt. On the +first `Prompt` of a session (never at `NewSession` — the startup budget rule) +it walks up from `Config.WorkDir` for `AGENTS.md` (falling back to `AGENT.md`), +stopping at the git root or filesystem root; the closest file wins, per the +[agents.md](https://agents.md/) convention. The file is schema-less Markdown — +no headings are required or parsed. The segment is appended after +`Config.System` and before hook (`system.transform`) segments, cached for the +session, and never written to the session log (loaded fresh on resume). + +A present-but-unusable file (invalid UTF-8, or empty/whitespace-only) fails the +first `Prompt` — a project that meant to supply instructions must not run +silently without them. A missing file is fine. Disable with +`-no-instructions`, config `instructions: false`, or point at a specific file +with config `instructions_path`. + +An oversize file is truncated, and the truncation is LOUD on both channels. +`truncateInstructions` (`engine/instructions.go`) appends the in-band marker +`formatTruncationMarker` builds — it names the path, the original size, the +kept size, the dropped size, and the `read_file` tool that reads the rest — +and writes one WARN log line with the same counts. A silent cut was the +earlier behavior: a 408 KiB `AGENTS.md` reached the model as 64 KiB with no +sign of the missing 344 KiB, so the model followed a half specification and +believed it read the whole one. A truncated instruction file must always +announce itself; never make this cut quiet again. + +`InstructionsConfig.MaxBytes` sets the cap: 0 (the zero value) takes +`defaultMaxInstructionsBytes` (64 KiB), a positive value sets it, a NEGATIVE +value disables the cap so the whole file is injected. Config key +`instructions_max_bytes` (bytes) and the operator seam +`HARNESS_INSTRUCTIONS_MAX_KB` (kilobytes; negative disables) resolve it in +`cmd/harness`, the environment variable winning — the engine never reads an +environment variable itself. + +An oversize file is not merely marked, it is SPLIT. +`renderInstructions` (`engine/instructions_outline.go`) injects a HEAD plus an +OUTLINE: the head is every section that fits whole under the cap, and the +outline lists every section the head does not carry — one line each with the +heading, the exact `read_file` range that reads it +(`read_file(path=, offset=, limit=)`), and a short teaser +from the section body. The model pulls a section with `read_file`, whose +`offset`/`limit` are already 1-based line numbers, so this adds NO tool and no +schema to any request. The shape is the Agent Skills stage-1/stage-2 split +below, applied to one file: the outline is an index the model MUST read +through before it relies on a section. The former monolithic root exceeded +the cap, so its head carried the sections that fit and its outline advertised +every later section. Nothing was out of reach, where the marker alone had +left the whole tail unreachable. + +`scanSections` tracks fenced code blocks by FENCE CHARACTER AND RUN LENGTH, +not with a boolean. A `#` comment inside a ```` ```bash ```` block read as a +heading would advertise a range that points at a shell comment. The character +and run-length rules cover the next shape up: an instruction file that +documents Markdown wraps a three-backtick example in a four-backtick fence, +which a naive toggle closes early, turning the rest of the document into +false sections. + +The split composes with the cap in ONE direction: the cap decides what is +EAGER, the outline makes everything else REACHABLE, and nothing is dropped in +silence either way. Three shapes keep the marker. A file with fewer than two +sections (no heading, or one giant section) has nothing to outline and takes +the `truncateInstructions` path unchanged. `InstructionsConfig.Mode` +`InstructionsModeFull` selects that path for every file. And a file whose +FIRST section alone exceeds the cap has no boundary to cut on, so the head is +that truncated first section — marker and WARN line both firing, through +`truncateInstructionsOf`, which reports the WHOLE file's size and not the +first section's — with the outline still listing every later section. Never +let the head lose its marker in that shape: it is the one place where an +outline could hide a cut. Config key `instructions_mode` and the operator seam +`HARNESS_INSTRUCTIONS_MODE` resolve the mode in `cmd/harness`; only the value +`full` (case-insensitive) turns the outline off, because an unreadable knob +must not quietly drop an outline nobody asked to lose. Design: +docs/design/nested-instruction-loading.md. + +## Agent Skills + +The engine advertises [Agent Skills](https://agentskills.io) in the system +prompt following the spec's progressive-disclosure model. On the first `Prompt` +(alongside instructions loading, same load-once-cache-error pattern) it runs +`skill.Discover` over each configured directory, merges the results sorted by +name, and injects one system segment **after** the instructions segment and +before hook (`system.transform`) segments. That segment is stage 1 only: a +header telling the model it MUST read a skill's `SKILL.md` with the `read_file` +tool before relying on it, then one line per skill — `name — description (path: +)`. Stage 2 (the body) is deferred to that read. + +`Config.SkillsDirs` selects the directories: nil (the default) uses +`/.agents/skills` when it exists; an explicit empty slice disables +discovery. A malformed `SKILL.md` or a duplicate skill name across dirs fails +the first `Prompt` loudly (same semantics as a malformed AGENTS.md). Skills are +never written to the session log — a resumed session rediscovers them. Config +`skills_dirs` (array; a non-empty project value overrides the user value +entirely) and the repeatable `-skills-dir` run/serve flag drive it. + +## Tool-batching guidance + +The engine executes one assistant message's tool calls concurrently +(`engine/toolexec.go`, capped by `Config.ToolConcurrency`), but a model +that emits one call per turn never produces a batch wider than one. The +executor is only as useful as the model's willingness to batch, so the +engine asks for it: `toolBatchingSegment` (`engine/toolexec.go`) injects +one system segment telling the model to put independent calls in the same +message, and to wait when a call's arguments depend on an earlier call's +result. Both halves matter — the second is what stops a model +parallelizing genuinely dependent work, which no amount of executor +correctness can repair. + +The segment sits immediately after `Config.System` and before the +instructions segment (`engine/engine.go`): it describes how this engine +runs tools, not anything about the project. It is **gated on the +session's resolved concurrency and is empty at 1** — an operator who set +`HARNESS_SEQUENTIAL_TOOLS=1`, or an embedder who set `ToolConcurrency: 1`, +must not be told calls run concurrently when for that session they do not. +The cap in the text is rendered from `s.toolConcurrency`, so the number +the model reads is the number the executor enforces. Like every other +engine-injected segment it is never written to the session log. + +Adding a base segment shifts every later segment's index, so the +segment-layout assertions across `engine/*_test.go` pin it explicitly via +`isBatchingSegment` (`engine/toolbatching_test.go`); only that file pins +the wording itself. + +## read_file image support + +The built-in `read_file` tool (`engine/filetools.go`) can return an image +file as real visual content, not mangled text. `readPathContent` opens the +target path exactly once and classifies it by its magic bytes +(`http.DetectContentType` over at most the first 512 bytes) — never by its +extension: a `.txt` file that is actually a PNG is still recognized as an +image, and a `.png` file that is actually text stays a text read. On a +recognized image (`image/png`, `image/jpeg`, `image/gif`, `image/webp`), +`read_file` returns a `message.ToolResult` whose Content is `[Text, Blob]`: +a one-line Text summary (format, byte size, and pixel dimensions) followed +by a `message.Blob` carrying the real file bytes. This is the same +`Text`+`Blob` shape MCP's `mcpContentToParts` already produces +(`engine/mcp.go`) — `read_file` is a second producer of it — so every +transcoder's existing Blob handling and the imageclamp dimension/byte-size +pass (`imageclamp.Clamp`, called from every transcoder's +`transcodeRequest`) apply with no new wiring. `read_file` never bypasses +that clamp: it does not resize, re-encode, or otherwise touch pixels +itself. Because `imageclamp.Clamp` runs later, at transcode time, an image +it downscales or re-encodes can end up described by dimensions or a byte +size that no longer match the summary `read_file` reported when it read +the file; this is a known, accepted mismatch, not a defect to fix in +`read_file` itself. + +**Only the Anthropic route puts a tool-result image on the wire.** +`imageclamp.Limits.RecurseToolResults` is true for `provider/anthropic` +only; `provider/openai` and `provider/openaicompat` set it false and +instead replace a tool-result Blob with a text note, +`"[N image attachment(s) omitted]"` (`toolResultOutput`, +`provider/openai/transcode.go` and `provider/openaicompat/transcode.go`). +This is pre-existing wire-format behavior `read_file` inherits, not +something this feature introduces, but it means a `read_file` image reaches +the model as pixels only on the Anthropic route; on the other two the model +sees only the one-line Text summary. + +`readPathContent` applies three guards on the image path, in order: + +1. The sniff read uses `io.ReadFull`, not a single `Read`, so a short + `read(2)` — realistic on a pipe or FUSE mount — never misclassifies a + real image as plain text. +2. The read is bounded at `readFileMaxImageBytes` (20MB), checked + against an `io.LimitReader` over the same open handle, never against a + separately captured `os.Stat` size a concurrently growing file could + outrun. This cap is separate from and smaller than any provider's own + wire limit, which `imageclamp.Clamp` enforces at transcode time; it + exists only so `read_file` itself never loads an unbounded file into + memory. An over-cap image returns a plain text error and no Blob. +3. The body must decode with `image.DecodeConfig` before `read_file` + commits to the image outcome. A corrupt or truncated file that merely + opens with a matching magic-byte prefix fails this check; `read_file` + then reads the true remainder of the file (unbounded, same handle) and + returns it as ordinary text instead of shipping a Blob the model cannot + use. This guard is not airtight for GIF: the `GIF87a`/`GIF89a` header + carries no checksum, so text that happens to start with those exact six + bytes still "decodes" with fabricated dimensions. A real file colliding + with that prefix is vanishingly unlikely; this is a documented, accepted + residual. + +A non-image binary file (sniffed as `application/octet-stream` or similar) +keeps `read_file`'s existing (unbounded) text-read behavior; `readPathContent` +still reads it exactly once, through the same handle its sniff already +opened. + +**Known gap, filed as issue #129**: a transcode-time degrade of an image +Blob to a text placeholder for a model with no vision capability is not +implemented. No per-model vision-capability signal exists anywhere in the +codebase to gate it on — `modelmeta` carries context-window data but no vision +capability, and +`provider.Request` carries no capability flag comparable to `Effort` or +`SessionKey` that a caller could set from one. Building this now would mean +inventing an ad hoc, likely-wrong static model list, so it is deferred to +issue #129. Until it lands, a model with no vision support receives the +image Blob exactly as any vision-capable model does; how it handles that +block is between the model and its provider. + +## Bounding concurrent-read memory + +`read_file`'s text path is deliberately unbounded (`readPathContent`, +`engine/filetools.go`) — a coding agent legitimately reads whole files, and +no byte cap is right for every file. Only the IMAGE path is capped +(`readFileMaxImageBytes`), and `bash` caps its own output +(`defaultBashOutputCap`). + +That was safe while tool calls ran strictly one at a time: peak heap held +at most ONE file's raw bytes plus the line-numbered copy built from them. +The concurrent executor (`engine/toolexec.go`) removed that implicit bound +without replacing it, so a batch of N large reads holds N working sets at +once. Measured with eight 16MB files, retention swallowing the finals so +only the transient term shows: **~325MB peak parallel against ~73MB +sequential — ~4.3x, bounded only by `ToolConcurrency`**, with the model +choosing both the batch width and the file sizes. + +`toolReadBudget` (`engine/toolmem.go`) is the replacement bound. Each +`read_file` reserves its file's Stat size against a per-session byte +budget before touching the file, and holds it until the call returns — +spanning the line-numbering too, since for a large file `strings.Split` +is the single biggest allocation and releasing after the read would leave +the expansion outside the bound. With the budget set to one file's size +the same batch peaks at the SEQUENTIAL figure: ~73MB, **1.0x**. It bounds the **product** of read size and +concurrency, which is the actual hazard: a count limit still admits two +500MB reads, and a size limit breaks the legitimate large read. Ordinary +work never contends — a full-width batch of kilobyte reads reserves a +rounding error against the default and stays fully parallel +(`TestReadBudgetKeepsSmallReadsFullyParallel`). + +It bounds the TRANSIENT working set, not the ACCUMULATED results: +`runToolBatch` holds every call's output until the join in BOTH execution +modes, so N results occupy the same memory however they were produced. +That term is not a concurrency regression, and retention already collapses +oversized results where configured. + +Three properties keep it safe. A worker takes its pool slot first and then +reserves, which is head-of-line blocking but cannot deadlock: only a slot +holder ever holds budget, so someone is always making progress, and a +reservation is never held while acquiring another. A read larger than the +whole budget is CLAMPED to it, not refused, so it waits for the budget to +drain and runs alone — a batch always progresses. Waiters are served +strictly FIFO, because a retry-when-there-is-room loop lets a stream of +small reads starve one large read forever. + +`Config.ToolReadBudgetBytes`: 0 (the zero value) takes +`defaultToolReadBudgetBytes` (64 MiB), a negative value DISABLES the bound, +a positive value sets it. `HARNESS_TOOL_READ_BUDGET_MB` is the operator +seam (megabytes; negative disables), resolved in `cmd/harness` like +`HARNESS_TOOL_CONCURRENCY`. The budget is per SESSION: a process running +many sessions bounds each, not their sum — a process-wide budget is the +natural follow-up if that proves insufficient. + +## write_file read-before-overwrite guard + +`write_file` (`engine/filetools.go`) refuses to overwrite an EXISTING file +the session has not read. Before this guard, `write_file` overwrote any +existing file unconditionally — a model could destroy a file it never +opened, with no recovery path. `edit_file` never had this hole: its +`old_string` match is exact-content-required by construction, so it cannot +blindly clobber unseen content. Claude Code and opencode both close the +same gap on their own write tools (opencode's is a literal `"You must read +file X before overwriting it"` error); this guard gives `write_file` the +same property. + +`Session.recordRead`/`readHashFor` (`engine/filetools.go`) track, per live +session and in memory only, every path `read_file` has read or +`write_file`/`edit_file` has written, keyed on the RESOLVED absolute path +(`s.resolvePath`'s output, never the raw tool argument — two different +relative arguments that resolve to the same file must not be tracked as two +separate paths), mapped to the sha256 hash of that path's raw on-disk bytes +at the moment of that read or write. `read_file` hashes the complete raw +file bytes it already read off disk for its own content classification +(`readPathContent`'s `TextData`/`ImageData`) — never the offset/limit-sliced +text it returns to the model. This matches the reference guards (Claude +Code, opencode): the guard authorizes per successful OPEN, not per byte +displayed — a windowed read of a large file authorizes replacing the whole +file. The hash is recorded only at a `read_file` return that hands the +model content; a read that errors (offset past end-of-file) records +nothing. + +`write_file` on a path that `os.Stat` resolves to an existing regular file +requires, in order: (1) the path is present in this session's read set — +absent means `write_file: %s exists and has not been read this session; +read it first (or use edit_file)`; (2) hashing the path's CURRENT on-disk +bytes fresh (never trusting the recorded hash's age) matches the recorded +hash — a mismatch means `write_file: %s changed on disk since it was read; +read it again before overwriting`. A path that does not exist +(`fs.ErrNotExist`) is unguarded — creation is `write_file`'s main job, and +there is no prior content to protect. Any OTHER stat failure (permission, +transient metadata error) refuses the write with `write_file: cannot stat +%s to check the read-before-overwrite guard` — a failed stat cannot prove +no protected file exists there. A +successful `write_file` or `edit_file` records/updates the written path's +hash to the new content's hash, so a write immediately followed by another +write to the SAME path (the assistant overwriting its own just-written +content, or an `edit_file` followed by a `write_file` on the same path) +never spuriously re-triggers the guard — the session already knows exactly +what is on disk because it just put it there. + +The read set is runtime-only: never persisted, never folded by +`LoadSession`. A reloaded session starts with an EMPTY read set, so a +resumed session must `read_file` a path again before `write_file` can +overwrite it — even a path the session genuinely read in a prior process +life. This is deliberately conservative and matches the guard's purpose: +the guard exists to stop an overwrite of content the model never actually +saw THIS session, and a resumed model has no live memory of raw file bytes +from a prior process either, only whatever text happened to land in the +persisted transcript. The read set lives on `Session` state, not `Config`, +so `configSnapshot` (used to seed a spawned child's config) never copies +it — a spawned child starts with its own empty read set, correctly, since +it is a different session that has read nothing yet. + +Tool calls in one assistant batch execute concurrently. `filePathKey` gives +`read_file`, `write_file`, and `edit_file` calls on the same resolved path one +key, and `keyChain` runs those calls in FIFO order. Preserve that exclusion: +the read-set map's per-operation mutex does not make the +check-current-hash-then-write sequence atomic by itself. + +`bash` writes (a model redirecting output to an existing file via a shell +command) are explicitly OUT of scope: harness cannot classify an arbitrary +shell command as a file write versus anything else it might do, so this +guard covers only the two built-in tools that make a structured, typed +claim to write a file. + +## Base loop retry + +The base interactive `Prompt` loop retries a transient provider error itself, +so a plain box prompt never surfaces a one-off HTTP 500. `streamTurnWithRetry` +(`engine/prompt_retry.go`) wraps `streamTurn` at its single call site in +`runAgenticLoop` (`engine/engine.go`). It retries only when the error is +classified retryable through `provider.AsRetryable` — `server_error`, +`overloaded`, `rate_limited`, or `stream_truncated`, never by matching error +text — AND the budget has an attempt left. Every other error returns on the +first attempt with ZERO retries: a `context.Canceled` abort, an +`*interruptedTurnError` (whose partial `runAgenticLoop` must still append — +retrying would duplicate the model's already-emitted tool intent), a +`provider.AsPermanent` malformed-request shape, or any deterministic failure. +The final surfaced error still emits one `session.error` and drops the usage +exactly as before; an intermediate masked attempt emits neither. A masked +attempt is still a full `streamTurn`, so it DOES bump the per-session turn +counter (`s.turn`, reported by `session_info`) and re-fire the per-request +hooks (`chat.params`, `system.transform`) and `OnRequest` — one bump and one +hook pass per attempt, exactly like the goal loop's per-attempt behavior. Only +the `session.error` and usage are suppressed for a masked attempt. + +One class is retry-eligible WITHOUT `provider.AsRetryable`: a completed but +EMPTY turn (no non-empty text, no tool call — e.g. thinking consumed the +whole `max_tokens` ceiling; see `emptyTurnError`). Two deliberate deviations +from the masked-attempt rules above. First, a discarded empty attempt's +usage IS accumulated into cumulative `Usage()` (it was a fully billed +completion, unlike a transport failure — same principle as the empty +compaction summary), while `lastUsage` is left alone. Second, the nesting +math: an empty turn that survives all `PromptRetries+1` attempts surfaces a +deterministic error, which goal mode's worker tier retries +`goalWorkerRetries` more times — worst case `(PromptRetries+1) * +(goalWorkerRetries+1)` = 9 fully-billed calls — and then STOPS the goal +with the empty-turn reason. Before this class existed the same turn was a +silent success and a goal limped on with nothing appended; halting with a +legible reason is the intended trade. The fail-fast for the deterministic +`max_tokens`-exhaustion shape (cutting the 9 to 3) is a filed follow-up on +the PR that introduced this. + +Retrying `streamTurn` is idempotent for history and tool side effects: +`streamTurn` makes ONE model call and never executes a tool (`runAgenticLoop` +runs tools only AFTER `streamTurn` returns a `StopToolUse` message), so a +failed attempt ran no side effect to redo. The one shape that DID emit tool +intent before failing arrives as `*interruptedTurnError` and is excluded. + +The emit stream is NOT idempotent, so `streamTurnWithRetry` closes that gap. +A failed attempt can emit `EventTextDelta`/`EventReasoningDelta` for partial +text before its stream dies, and the retry re-streams that text from scratch. +`streamTurnWithRetry` emits one `EventTurnRestart` (`engine.go`) before each +retry, so a subscriber that renders deltas incrementally drops the stale +partial and rebuilds it from the retry — never the two runs concatenated +(`Hello wor` then `Hello world` shown as `Hello worHello world`). The server +forwards `EventTurnRestart` live over SSE (`server/journal.go`'s `Publish`); +the turn's final `EventMessage` still reconciles history regardless. + +`Config.PromptRetries` bounds it: additional attempts, zero (the engine zero +value) DISABLES retry, config/CLI default 2 via `config.Config`'s `*int` +`prompt_retries` key (`PromptRetriesValue`). The backoff +(`basePromptRetryDelay`: 1s, then 2s, `time.NewTimer`) is deliberately SMALLER +and SHORTER than the goal loop tiers in `docs/goal-loop.md` — an interactive user waits on +the turn, so this smooths a blip in a second or two, never the goal loop's +~30min weather schedule (`promptTurnWithRetry`, `goal.go`). The two are +distinct: the base loop wraps ONE model call and is inherently idempotent; the +goal loop's `promptTurnWithRetry` wraps a whole worker turn (with the +tool-executed non-idempotency gate) and parks on exhaustion. + +The two also NEST. A goal worker turn runs through `s.Prompt`/ +`s.runAgenticLoop` (`goal.go`), so every one of `promptTurnWithRetry`'s outer +attempts now issues up to `1+PromptRetries` inner `streamTurn` calls. For a +persistent retryable condition the worst case is `goalRetryableMaxAttempts` +(12) times `1+PromptRetries` (3) — about 36 full-input-price model calls, +where the goal-loop tiers alone assume ~12. This is deliberate: the fast inner +budget (1s, then 2s) smooths a one-off blip inside a single worker turn before +the outer weather tier ever counts it, so a goal loop rides a brief provider +blip without spending an outer attempt. `PromptRetries` 0 disables the inner +budget for a host that wants the outer tiers to be the only retry. + +## Max-tokens auto-continue + +A turn whose stop reason is `provider.StopMaxTokens` means the provider cut +the model off mid-emission — it did not choose to stop. Before this existed, +`runAgenticLoop`'s `if stop != provider.StopToolUse` branch +(`engine/engine.go`) treated every non-`tool_use` stop reason alike: append +`asst`, synthesize an is_error result for any orphaned `ToolCall` part via +`appendUnexecutedToolCallResults` (NEP-5272, see "Wire normalization" +above), and return. For `max_tokens` specifically that return settles the +session idle with nothing further ever prompting it. Incident: box +harness-parallel-tools — the model emitted a large tool call, the provider +stopped mid-emission with `max_tokens`, the engine synthesized the +unexecuted-call result exactly as designed, and the session then sat idle +until a human noticed and re-prompted it. On an autonomous fleet a silent +work stoppage is as bad as a crash. + +`runAgenticLoop` now branches on `stop == provider.StopMaxTokens` inside +that same `if`: `maybeAutoContinueMaxTokens` decides whether to `continue` +the loop (issuing a real follow-up model call in this SAME `Prompt` call) +instead of returning. This applies identically whether or not `asst` carried +a `ToolCall` part — a pure-text `max_tokens` truncation (Claude Code's own +behavior is to let the turn end and rely on the user re-prompting) gets the +same auto-continue an autonomous harness session needs, not a human-facing +half-answer. + +**A genuinely mid-emission tool call keeps its identity; only its Arguments +are cleared.** A cross-model adversarial review of this PR raised a +CRITICAL finding: a `StopMaxTokens` turn's trailing `ToolCall` can carry +non-empty but syntactically invalid `Arguments` — the raw, truncated +`partial_json` Anthropic's `assembledBlock.toolCall` +(`provider/anthropic/anthropic.go`) leaves behind when `max_tokens` lands +before the block's own `content_block_stop`, e.g. `{"comm` — and claimed +replaying that into the continuation request fails `json.Marshal` before it +reaches the provider. That premise does NOT hold: `message.Message.Normalize` +(`Session.append`'s `appendWithUsage`, run on every append) already coerces +the identical invalid-Arguments shape to nil in place, through the SAME +`*ToolCall` pointer `asst.Parts` already holds — so by the time the +continuation request is built, `Arguments` is already safe. This is not +incidental; it is the deliberate, incident-tested fix for a real production +defect (two goal sessions dead at "json: error calling MarshalJSON... "; see +`TestPersistTruncatedToolCallArguments`, `engine/tool_call_poison_test.go`). +The finding is REBUTTED WITH EVIDENCE, not implemented: no code change. See +`TestMaxTokensPartialJSONMarshalsThroughRealTranscoder` +(`engine/max_tokens_wire_test.go`), which drives a genuinely truncated +`partial_json` tool call through a REAL `anthropic.Client` (`provider/anthropic`, +via an `httptest` server, not a hand-rolled stand-in) and proves the +continuation request the client actually sends decodes cleanly server-side, +with the truncated call's identity preserved and its `Arguments` cleared — +this is the test that pins the rebuttal and must go red before any future +"drop the call entirely" change lands unchallenged. + +**The continuation nudge is a genuine new turn, never assistant prefill.** +The follow-up call carries the synthetic unexecuted-tool-call result (for +any COMPLETE call the engine chose not to execute) plus a one-shot nudge — +`s.pendingContinuationNudge`/`continuationNudgeSegment`. Unlike every other +ambient status segment (process, MCP, goal-parked, identity, task +notifications — see "Ambient engine context" above), this one is NOT glued +onto an existing message via `withAmbientStatus`: +`appendContinuationNudgeMessage` appends a genuine NEW `message.RoleUser` +message, carrying the nudge as its own `*message.EngineContext` part, to the +END of `streamTurn`'s own throwaway per-request message copy. +`withAmbientStatus` scans backward for the newest EXISTING `RoleUser` +message, which by the time a continuation request is built is an EARLIER +message than the just-truncated assistant turn (and its synthetic tool +result, if any) — leaving the canonical request ending in `RoleAssistant` or +`RoleTool`. Anthropic serializes that as assistant PREFILL: some models +reject it with a permanent 400, and even an accepting model sees a +"continue" instruction that chronologically precedes the output it refers +to. `appendContinuationNudgeMessage` never touches `s.history` — same as +every ambient segment — so a session reload, or any later unrelated +request, never sees the nudge. It still rides every attempt a +transient-error retry makes for that ONE follow-up call (mirroring +`checkoutTaskNotificationsSegment`'s idempotent-reread shape, +`taskdelivery.go`) and is cleared by `runAgenticLoop` the instant that whole +`streamTurnWithRetry` call returns, so it never bleeds into a later, +unrelated turn. + +**Queued operator input is drained before every continuation, not only at +tool-call boundaries.** `drainQueuedPromptsIntoHistory` (`engine/engine.go`) +is the shared implementation behind the tool-call-boundary drain (after a +`StopToolUse` round actually runs a tool) and the max_tokens continuation +branch (right before looping back for another follow-up call) — both are +points where `runAgenticLoop` is about to issue another provider request in +the SAME `Prompt` call, so both are valid mid-turn steering opportunities. +An operator prompt queued while a long truncated response streams is +delivered on the very next continuation request, not left undelivered for +the whole continuation chain. + +Loop safety is the critical part: `runAgenticLoop`'s local `maxTokensUsed` +is a PER-PROMPT BUDGET, spent by every continuation issued in the loop and +NEVER reset — including across an intervening `StopToolUse` round. (An +earlier version of this counter, `maxTokensStreak`, DID reset on any +non-`max_tokens` stop, which let a model alternate `max_tokens` and +`tool_use` — including denied, unknown, or failing tool calls, none of +which touch `toolExecCount` — indefinitely inside one `Prompt` call, +spending an unbounded number of continuations without ever tripping +`Config.MaxTokensContinuations`.) `maxTokensUsed` is bounded by +`Config.MaxTokensContinuations`: `Config.MaxTokensContinuations+1` +max_tokens stops used within the loop trips the bound. +`maybeAutoContinueMaxTokens` then returns a +`*maxTokensContinuationExhaustedError` naming the bound, wrapped +`provider.MarkPermanent`, instead of arming yet another doomed attempt; +`runAgenticLoop` emits `session.error` and returns it, the same "honest +terminal, never a silent success" shape `emptyTurnError`'s own budget +exhaustion uses (see "Base loop retry" above). + +**A goal-loop retry must never re-run an already-exhausted continuation +chain.** With the default bound of 3, one worker attempt that exhausts the +budget already makes 4 completed, fully billed `max_tokens` calls before +`*maxTokensContinuationExhaustedError` is even returned. +`maybeAutoContinueMaxTokens` wraps every value of that type +`provider.MarkPermanent` at its one construction site, so +`promptTurnWithRetry`'s existing `provider.AsPermanent` fail-fast branch +(`engine/goal.go`) stops after that ONE attempt instead of re-running the +whole exhausted chain up to `goalWorkerRetries` (2) additional times — which +would otherwise multiply 4 calls into 12 for one goal boundary. Like every +other permanent-classified worker error, this PARKS the goal (stays +resumable) rather than clearing it: the condition that produced the +exhaustion might not recur on a later resume. + +`Config.MaxTokensContinuations` follows `PromptRetries`'s own +unset-vs-zero config idiom: the engine field's zero value DISABLES +auto-continue entirely (a bare embedder-built `engine.Config`, and every +test that constructs one directly, keeps the exact pre-fix behavior — the +turn ends immediately on the first `max_tokens` stop). The config/CLI layer +(`config.Config.MaxTokensContinuations *int`, key +`max_tokens_continuations`, resolved via `MaxTokensContinuationsValue`) +supplies the product default of 3; an explicit `0` disables it the same way +`prompt_retries: 0` disables base-loop retry. + +A task child runs its turn through this exact same `runAgenticLoop` — a +child `Session` is a full `NewSession(childCfg)` +(`SessionManager.Spawn`, `engine/session_manager.go`), and `childCfg` comes +from `configSnapshot`, a whole-struct copy of the parent's `engine.Config` +— so `MaxTokensContinuations` (and every other engine.Config field) reaches +a child with no separate wiring. A child that hits `max_tokens` +auto-continues under the identical bound the root does. + +## Per-turn metrics + +`streamTurn` (`engine/engine.go`) emits one structured `turn_metrics` line per +COMPLETED model call — a stream that reached `EventDone`; a turn that errors +or is interrupted mid-stream (see `interruptedTurnError` above) reports +nothing, since there is no finished call to summarize. This is the box-fleet +answer to "why does this session feel slow": TTFT and stream duration, token +and prompt-cache accounting, and request shape, greppable straight off a +process's stderr. + +Fields: `session_id`, `model` (full `provider/model` ref), `ttft_ms` (elapsed +from just before `prov.Stream` to the first non-`EventActivity` stream event +— `EventActivity` carries no content, so a keep-alive ping or an in-progress +tool-argument chunk must never be mistaken for "first byte"; if `EventDone` +itself is the first event, `ttft_ms` covers the whole call and `stream_ms` is +0), `stream_ms` (first delta to `EventDone`), `input_tokens`/`output_tokens`/ +`cache_read_tokens`/`cache_write_tokens` (passed through from +`provider.Usage` verbatim), `system_len`/`tools_count`, and `retry` (the +1-indexed attempt number `streamTurnWithRetry` — `engine/prompt_retry.go` — +was on when this call completed; 1 for a turn that succeeded on its first +try). `system_len` is computed identically to the server's `request.meta` +record (`len(strings.Join(req.System, "\n"))`, see `server/journal.go`'s +`OnRequest`) — deliberately, not coincidentally: `session_id` + `model` + +`system_len` together are a natural join key between a `turn_metrics` stderr +line and the durable `request.meta` record for the same request, with no new +ID threaded through the provider boundary. + +`Config.OnTurnMetrics func(TurnMetrics)` is the seam. Unlike every other +`On*` callback in `Config` (`OnEvent`, `OnRequest`, `OnStorePhase`), nil is +NOT "disabled": `emitTurnMetrics` substitutes `defaultTurnMetricsLog` +(`engine/turn_metrics.go`), a `slog.NewJSONHandler` line written to +`os.Stderr` — the same stream every other structured log line in this repo +uses (see `cmd/harness/main.go`'s "Structured logging: JSON to stderr" +comment). Stderr keeps the line out of `harness run`'s stdout, which is +the model's answer channel, while a deployment's log pipeline (Kubernetes +captures both streams) scrapes it identically. A plain `harness run`/`harness serve` process +with no embedder wiring therefore still emits this line by default; an +embedder that wants a different sink (an OTel exporter, an in-memory test +recorder) sets `OnTurnMetrics` and never needs to suppress the default +first. + +`Config.Now func() time.Time` is the clock this measurement reads (nil +resolves to `time.Now` in `newSession`), scoped to this one seam rather than +a general engine clock — every other timestamp in the package still reads +`time.Now` directly. It exists so a test can script an exact instant sequence +instead of depending on real elapsed wall-clock time between two in-process +calls with nothing to wait on between them, per the Testing rule against real +sleeps. + +This was built for a deployment that ships a served process's stderr to a +log pipeline (a fleet of boxes running `harness serve`, each pod's stderr +collected by a Vector-style agent into BetterStack or an equivalent log +store). The intended query there filters `msg: "turn_metrics"` and groups by +`model`/`session_id` to compare TTFT and stream-duration distributions across +sessions — quantifying, with real numbers instead of a feeling, whether a +session "feels slow" because of provider latency, prompt-cache misses, or +something else entirely. diff --git a/docs/fleet-and-serve.md b/docs/fleet-and-serve.md new file mode 100644 index 00000000..d3eb210f --- /dev/null +++ b/docs/fleet-and-serve.md @@ -0,0 +1,281 @@ +# Fleet and serve diagnostics + +This document describes fleet state, task lineage, provider exhaustion, and +serve diagnostics. + +## Fleet model (the deploy story) + +The full build spec lives in `docs/design/fleet-model.md` — read it before +touching anything box-identity, session-lineage, or goal-pause related. The +short version this repo's code assumes: identity is an operator-chosen box +**NAME**; storage is one volume/directory per name (`HARNESS_SESSION_DIR` +points at it), never shared between concurrently-live servers; a box is +ephemeral compute serving one name (cattle), the name and its volume are +durable (pets). Respawning the same name over the same volume is **ADOPT** +— history restores, and any goal that was armed when the box died surfaces +as `paused`/`pause_reason: "restart"` (see the goal loop's paused +presentation, `engine/goal.go` and `server/journal.go`'s `goal.paused` +record) rather than a false "still running" reading. `parent_session` +(`POST /session`, see `engine/store.go`) is the lineage thread connecting a +re-dispatch to the task it continues from, so a fleet UI can group a box's +history by task across boxes. + +Subagent lineage is durable. `SessionManager.Spawn` records +`task_parent_id`, `task_agent_type`, and `task_depth` on the child's +session header (`engine/store.go`), and appends each child id to the +parent's own log. `LoadSession` restores all of them with no +SessionManager adoption needed. `GET /session/{id}.lineage` prefers the +durable `task_depth` over the live tree's derived depth, and merges live +children with the durable spawn list (`childIDsUnion`, +`server/handlers.go`) — so `lineage.depth` and `lineage.children` survive +`Reap` and a process restart. `childIDsUnion` merges both sides through +ONE de-duplicating loop and trusts neither side to be duplicate-free: an +id appears exactly once, whichever side carried it. Never re-add a +per-side fast path that skips the merge — an earlier one copied `live` +verbatim when `durable` was empty, so one repeated id survived or +collapsed depending only on whether the OTHER argument had anything in +it. A legacy header without `task_depth` +restores 0; `adoptReloadedLocked` then falls back to the `m.maxDepth` +refusal sentinel, exactly as before the field existed. + +A failed child's `fail_reason` carries the CAUSE, not only a class. +`classifySpawnFailure` (`engine/session_manager.go`) builds it as a fixed +classified prefix, then the underlying error message — masked with +`maskSecrets` and capped at `spawnErrorDetailCap` (500) runes. One prefix +covers a whole family of causes (a permanent 400 is a malformed request +AND a quota rejection AND a policy refusal), so a parent that reads only +the prefix must guess: a live incident measured that guess as "respawn a +sibling straight into the same fleet-wide provider wall". The #82 leak +rule still holds in its narrower form — never surface a provider error +RAW — through masking plus the cap, the same best-effort trade a retained +tool result already makes. `context.Canceled`/`context.DeadlineExceeded` +keep their short fixed `canceled`/`timed out` strings, with no cause +appended. The reason reaches the parent through the `[tasks: ...]` +notification, `SessionNode.FailReason` (so `task status` and +`GET /session/{id}.lineage.fail_reason`), and the journal's +`task_fail_reason`. + +Server-side session resolution has ONE entry point: `Server.resolveLive` +returns a `liveSession` snapshot (`server/live.go`) that holds the +residency half (`Server.sessions`, one `s.mu` hold) and the SessionManager +half (one `SessionAndInfo` hold) together. Read a session, its status, or +its lineage from that snapshot — never from `s.sessions` or `sessMgr` +directly, and never take a second manager read later in the same request. +The two halves are separate holds on purpose: `server.mu` is a leaf lock +with respect to `SessionManager.mu`, so one atomic hold over both would +build the cycle that rule forbids. Residency wins whenever it has an +answer, because a resident session's own `running` flag is authoritative +for itself (`freeRunSlotAndEmitIdle` clears it before `ReportTurnEnd` +flips the node). The manager half answers only what residency cannot: a +Spawn-driven child, which is never a residency key. + +**Provider exhaustion is not a child failure.** An ACCOUNT-level supply +wall — the API key's usage limit, quota, credit balance, or spend cap — is +FLEET-WIDE (every sibling on the same key hits the identical wall at the +identical moment) and TEMPORAL (the child's session and work are intact and +re-runnable once the provider's clock rolls over). A parent that reads it as +an ordinary failure respawns a replacement into the same wall, which a live +incident measured. Three layers carry it: + +- The ADAPTER classifies, never the engine. `provider/anthropic`'s + `parseUsageExhaustion` gates on the HTTP status (400/402/403/429, or none + at all for a mid-stream `error` event) and then matches + `usageExhaustionPatterns` — a deliberately flat, extensible list of + observed wall wordings, one regexp per shape, each of which must name a + spent SUPPLY, never a per-minute THROTTLE. It returns a + `provider.Error{Kind: ErrKindProviderExhausted, RecoverHint}` wrapped + permanent (no backoff outlives a spent quota). This is the second place + message matching is tolerated, under `parseContextOverflow`'s rules. Other + adapters opt in by producing the same kind; only anthropic does today. +- The ENGINE reads the typed classification, never text. + `classifySpawnFailure` (`engine/session_manager.go`) maps + `provider.AsProviderExhausted` — or a `RetryableRateLimited` class that + outlived the retry budget — to `FailKindProviderExhausted` + (`"provider_exhausted"`). Overload and 5xx weather deliberately do NOT + qualify: those clear in seconds and a sibling may well succeed. +- The STATUS VOCABULARY is unchanged. An exhausted child is `StatusFailed`, + with the kind in a SEPARATE `FailKind` field (`SessionNode`, + `taskNotification`, the durable `taskNotifyRecord`, `task status`'s + `fail_kind`, `GET /session/{id}.lineage.fail_kind`, the journal's + `task_fail_kind`). A sixth `SessionStatus` value would have forced every + cancellation/Reap/delivery/restore switch to grow an arm that behaves + exactly like `StatusFailed`; only the PARENT's next move differs. + +The rate-limit arm conflates a spent quota with a per-minute throttle that +outlived the child's small `PromptRetries` budget, one-directionally and on +purpose: a missed wall makes a parent respawn into it (the incident), while +a false wall costs one deferred resume of an intact child, and a hintless +guidance names no waiting period. An adapter that classifies its own quota +shape never reaches that arm. Both the cause and the recover-at hint go through +`boundedProviderText` (mask, then cap), so model-visible provider text on +this surface has one rule, not one per field. The hint is stated in ONE +engine-authored place — `taskFailureGuidance`'s "after " — never in +the reason prefix as well: the hint is extracted FROM the provider message +the reason already quotes, so naming it there made one rendered line +repeat the same time three times. It rides the durable record +(`taskNotifyRecord.FailHint`) because that guidance clause is now the only +carrier of the fact. + +`taskFailureGuidance` (`engine/taskdelivery.go`) appends the parent's +instructions to that child's own notification line — child preserved, do not +spawn a replacement, resume with `task send` on this session id, after the +recover-at hint when the provider gave one. Resuming is the existing +send-to-a-settled-descendant re-run path, unchanged. A turn that then +succeeds clears `failReason`/`failKind` on the node, so a resumed child +stops reporting a wall it already got past; `finalizeTurn`'s +`alreadyCanceled` branch clears them too, since a CANCELED re-run must not +keep snapshotting a classification no live cancellation sets and +`restoreKnownStatusLocked` restores as empty. + +A REAPED descendant is still resolvable, not "no such session." `Reap` +collects a done/failed/canceled LEAF the instant it settles (its own doc +comment), which a caller that spawned it has no way to observe before +asking about it again — a live incident hit exactly this: `task send` to +a settled child answered `no such session` depending on internal reap +timing the parent could not see. `resolveOrReviveDescendantLocked` +(`engine/session_manager.go`), the shared first step of all four verbs, +falls back to disk when a live-tree lookup misses: `LoadSession` the +target, then confirm ancestry from its own DURABLE `task_parent_id` +chain (`durableAncestorChainHas`) — never from live state alone, which is +exactly what `Reap` already erased. Only a target with no session log on +disk either, or whose durable lineage does not reach the caller, still +answers `ErrUnknownSession`/`ErrNotDescendant`. The four verbs then +diverge on what a successful disk resolution does: `send` RE-ADOPTS the +revived child into the tree (`adoptReloadedLocked`, the same +adopt-on-first-sight path `AdoptReloaded`/`handleSpawnChild`'s +parent-lookup fallback already use) and re-runs it exactly like a +settled-but-unreaped child — `budgetedByChild` surviving `Reap` by design +is what stops that re-adopt from double-crediting its already-spent +usage. `status`/`log` serve the disk-loaded state directly +(`durableSnapshot`/`deriveSettledStatus`) WITHOUT re-adopting: a +read-only, poll-shaped verb must not have the side effect of pinning a +reaped descendant back into memory. `cancel` on a reaped target is a +no-op success (nothing left to interrupt) reporting its real terminal +status, never `StatusCanceled`. The disk-bound half of this resolution +runs with `m.mu` released — one slow disk read must never stall every +other session's `Info`/`Reap`/`Spawn`/`Send` call — and re-validates +`m.nodes[targetID]` on reacquiring the lock, so a concurrent adopt of the +same id (another `Spawn`, `AdoptReloaded`, or a second racing revival) +always wins single-handedly: whichever adoption reaches `m.nodes` first +is authoritative, and the loser's own "already managed" is ignored, the +same rule `AdoptReloaded`'s existing callers already follow for that +race. + +A parent can read a dead child's tail. The `task` tool's `log` verb +(`runTaskLog`, `engine/task_tool.go`, over +`SessionManager.DescendantTranscript`) returns the last N transcript +entries of a descendant, LIVING OR DEAD, under the same ancestor gate +(`isDescendantLocked`, or the disk-backed lineage check above once +`Reap` has removed the live node) cancel/status/send use — a terminal +node keeps its `*Session`, history included, until `Reap`, so no reload +and no disk read is involved for a still-tracked descendant. It is +bounded on three axes, because its output lands in the +PARENT's context and replays on every later turn: `tail` (default 20, +clamped at 100, a negative value is an error), a per-entry rune cap, and a +total rune budget filled NEWEST-first so the messages nearest a death +always survive. The reply reports the descendant's whole message count +next to how many entries came back, so a model knows it is reading a +window, and it carries `fail_kind` alongside `fail_reason` — the same +structured half `task status` reports, so a reader with the tail in front +of it never needs a second call to learn a death was an account wall +rather than the child. Every non-text part is rendered rather than dropped — a tool call +with capped arguments, a tool result, a reasoning summary, and an +attachment COUNT that includes blobs nested inside a tool result, which +`Parts.Text()` itself drops. Content is deliberately NOT masked: parent +and child are the same operator's sessions in one process, and a child's +final text already reaches the parent verbatim in its completion +notification. + +`Config.OnRequest` receives the firing session's own id as its first +parameter (`engine/engine.go`). Never wire it as a closure over a +captured session variable: `configSnapshot` copies the func value into +every spawned child, which misattributes the child's `request.meta` +records to the closed-over session's id. + +**Hub spawn contract:** the hub that spawns boxes — `harness hub`, now +implemented in `tools/hub/` (see `docs/development-interfaces.md`) — passes the +generated box NAME to the spawn command's environment as +`HARNESS_HUB_BOX_NAME`, so deployment scripts can derive per-name storage +(e.g. mount/create a volume named after it) without the hub and the box +needing any other side channel to agree on identity. Harness itself never +reads this variable — it is a contract between the hub and deployment +tooling, documented in `docs/design/fleet-model.md` §8. + +## Serve-mode latency diagnostics + +A caller that waits seconds for a reply cannot tell, from outside the +process, whether `harness serve` was slow, the network in front of it was +slow, or the whole process was stopped by garbage collection. Three +threshold-gated WARN lines answer that, and nothing runs always-on. + +- **`slow request`** (`server/timing.go`). `serveTimed` wraps the mux + dispatch in `Server.ServeHTTP` and warns when this process took longer + than `slowRequestThreshold` (500ms) to answer, with `method`, `route`, + `status`, `duration_ms`, and the caller's `X-Request-Id` as + `request_id`. The route is `http.Request.Pattern`, which the mux sets + during the dispatch, so a session id never reaches a log line; a request + that matched no route logs the fixed `unmatched` label, because the path + is caller-controlled. `requestID` drops a header value over 64 bytes or + carrying anything outside one printable ASCII token — it is untrusted + input that lands in a log line. `longLivedRoutes` exempts `GET /event` + and `GET /session/{id}/wait`: both run for as long as their caller + wants, so timing them would warn for every healthy client. Keep that map + in step with any new streaming or long-poll route. +- **`long gc pause`** (`cmd/harness/gcwatch.go`). A stop-the-world pause + stops every goroutine, so the process logs nothing at all while it lasts + and looks exactly like a wedged handler. `gcWatcher` samples the + runtime's `/gc/pauses:seconds` histogram every 5 seconds and warns about + a new pause at or past 200ms. It reads `runtime/metrics`, never + `runtime.ReadMemStats` — `ReadMemStats` itself stops the world, so + sampling it would add the pause this watcher exists to find. + `longest_pause_ms` is the LOWER bound of the highest bucket that gained + a pause, since a histogram records a range. The first sample reports + nothing: the counts are cumulative for the whole process life. Same + lifecycle as `inFlightWatchdog` — one cancelable context, cancelled when + `serveCmd` returns. +- **`/debug/pprof/`** (`server/pprof.go`, `Options.PProf`, `harness serve + -pprof`). OFF by default, and authed like every other route when on. It + is the third step, not the first: `GET /debug/goroutines` needs no flag + and already answers "what is this process blocked on". Turn `-pprof` on + for a process under investigation when the CPU, heap, block, or mutex + profile is what is missing. + - **Never import `net/http/pprof` in this repository.** That package's + `init` registers `/debug/pprof/*` on `http.DefaultServeMux` for the + whole linked binary, so importing it — even to borrow its handler + functions behind this flag — exposes profiling in ANY program that + links `server` and serves the default mux + (`http.ListenAndServe(addr, nil)`), with no opt-in and no way for + `Options.PProf` to prevent it. Go runs a package's `init` on import; + there is no way to take the handlers without the side effect. So + `server/pprof.go` implements them on `runtime/pprof` and + `runtime/trace` directly. + `TestPProf_NotRegisteredOnDefaultServeMux` asserts the absence and is + the regression guard: a 404 test through this server's own mux passes + WITH the bad import and proves nothing. It exists TWICE, in `server/` + and in `cmd/harness/`, because each only covers its own package's + import graph — the binary links the engine, providers, plugins, MCP and + the tools, and any one of them pulling in `net/http/pprof` would + publish the endpoints for the whole process. + - `?seconds=N` on `profile`/`trace` is clamped to 1-60 (default 30); a + malformed or repeated value is a 400, through the same `intParam` every + other integer parameter uses. A second concurrent CPU profile or trace + is a 409, not a 500 — only one of each can run in a process — and the + refusal removes the download headers it had to set before starting, so + the error is JSON rather than a file a browser saves. A client + disconnect ends the profile early rather than holding a runtime-wide + lock for an abandoned request. + - `GET /debug/pprof/{name}` is in `longLivedRoutes` (`server/timing.go`): + a profile runs for exactly as long as the caller's `?seconds` asks, so + timing it would log a 30-second "slow request" every time an operator + ran `go tool pprof` against a box — their own tooling in the logs they + are reading. The index stays timed; it returns at once. + - The UNSLASHED `/debug/pprof` is registered explicitly, behind auth. + Left to the mux it takes an automatic 308 redirect issued before any + handler, which told an unauthenticated caller whether profiling is + enabled. Authed, the two states are 401-vs-404 — the shape every other + route in this API already has. + - `/debug/pprof/symbol` is deliberately not served: `go tool pprof` + symbolizes against the binary a profile came from. + +No metrics, no tracing, no always-on profiling. A new diagnostic in this +area is a threshold-gated log line or it does not land. diff --git a/docs/goal-loop.md b/docs/goal-loop.md index 98c83b41..0cfb691e 100644 --- a/docs/goal-loop.md +++ b/docs/goal-loop.md @@ -1,669 +1,403 @@ -# Goal-loop resilience: forensic root cause and the state-machine fix - -## Incident - -Two production sessions were found with an active goal that had stopped -making progress: `ses_41813d5a411c2ba5.jsonl` and, earlier, -`ses_55e4ae35d8344540.jsonl`. Both show the same shape. Immediately after a -`goal.eval` "NOT MET" verdict, the fixed-template guidance message is -appended to the log as the next directive — and then nothing. No assistant -turn, no error record, no further `goal.eval`. In `ses_41813d5a411c2ba5.jsonl` -the guidance message is timestamped `2026-07-09T05:20:12Z`; the very next -record in the file is a bare `goal.cleared`, followed by a message -timestamped `2026-07-09T12:12:52Z` — nearly seven hours later — that reads: - -> "Your goal loop was interrupted. Do exactly this and nothing else: cd -> /work/tssdk && git add -A && git commit ... && git push ..." - -That `goal.cleared` and the recovery message are not something the engine -produced. They are a human, seven hours later, noticing the session had gone -silent with an active goal, clearing it by hand, and manually steering the -agent to at least commit and push whatever it had. `ses_55e4ae35d8344540.jsonl` -shows the identical pattern twice in a row: a `goal.set` → guidance message → -silence → (manually cleared) → `goal.set` (a manual resume) → guidance -message → silence → (manually cleared) again. - -The goal's own directive in both sessions instructs the worker to `apt-get -install -y nodejs npm` and install a TypeScript SDK toolchain — commands that -can emit megabytes of installer/build output. The engine's `bash` tool had no -enforced output cap wired through `Config` (a 100KB post-hoc truncation -existed as a hardcoded constant, tail-only, not configurable), so a large -enough burst of tool output from exactly the kind of command these goals ran -is a plausible trigger for whatever made the next worker turn fail. But the -log is silent about *what* failed — because that is exactly the bug: nothing -recorded the failure at all. - -## Root cause - -`engine/goal.go`'s `PursueGoal` loop called the worker turn like this: - -```go -if _, err := s.Prompt(ctx, directive); err != nil { - return nil, err -} -``` - -Any error from `s.Prompt` — a provider timeout, a rate limit, a stream error -triggered while handling an oversized tool result, anything — propagated -straight out of `PursueGoal` with **no goal.eval, no goal.stalled, no -goal.cleared**. `goalActive` stayed `true` in memory and in the session log. -The server's `runGoal` (`server/handlers.go`) does journal a `session.error` -for such an error, but never clears the goal, so the session parks forever -with an active goal that no automated process will ever revisit — a zombie. -Only a human reading the log tail could tell the loop had died and clear it -by hand, which is exactly what happened, hours later, in both incidents. - -## The fix - -Two independent layers, both TDD red-first (see `engine/goal_test.go` and -`engine/bash_test.go`): - -1. **Goal-loop resilience** (`engine/goal.go`): a worker-turn error now goes - through `promptTurnWithRetry`, which retries the same directive up to - `goalWorkerRetries` (2) additional times, recording a durable - `goal.stalled` record/event (carrying the error and the attempt number) - for every failed attempt — the loop can never go silent again. If every - attempt fails, the goal is **cleared** (`goal.cleared`, carrying the error - as `GoalReason`) before the error is returned, so `goalActive` can never be - left `true` with nothing left to explain it. A `context.Canceled` error - (DELETE /goal, shutdown drain) is never retried or treated as a failure — - it is a deliberate, resumable stop, and the goal is left untouched. See the - state-machine diagram in the `goal.go` package doc. (**Superseded by the - Round 7 exit-park work**: exhausting this budget no longer clears — it PARKS, exactly - like the retryable-class budget below — see "Worker-turn exit-park and - activity-driven resume" further down.) - -2. **Bash output cap** (`engine/bash.go`): tool output is now bounded by a new - `Config.BashOutputCap` knob (default 96KB) enforced by a `cappedWriter` - during capture — not by buffering the full output and truncating - afterward — so a runaway command (an `apt-get`/`npm install` storm is the - real-world trigger) can allocate only `O(cap)` memory and can never dump - megabytes into a single message. The cap keeps both the head (so the - command and its early output stay visible) and the tail (so a trailing - error banner stays visible), joined by a `"N bytes truncated"` marker, - before the output ever reaches the message log or the next provider - request built from it. - -## Non-goals / things this does not change - -- `goal.stalled` is a pure trace record: it never flips `goalActive` by - itself (see `LoadSession`'s `scanLog` switch in `store.go`). -- `evaluateGoal`'s own error path — a provider error while asking the - evaluator, or two unparseable replies in a row — is **no longer** - unchanged from the worker-turn path. It used to clear the goal outright - (see "Round 3" below); as of Round 6 it is advisory, mirroring the - worker-turn retry machinery instead of bypassing it. See "Evaluator - resilience" below for the current behavior — this bullet exists only so a - reader following an old link lands somewhere accurate. - -## Round 3: closing the evaluator's own zombie path (superseded by Round 6) - -The paragraph below describes the fix as it shipped originally: any -`evaluateGoal` error — a provider error, or two unparseable replies in a -row — cleared the goal outright, on the theory that the goal's own state -was "still accurate and worth preserving for a human-triggered resume." -Production experience proved that theory wrong (see "Evaluator resilience" -below): a transient evaluator hiccup killed sessions where the worker model -was making fine progress. The clear-on-any-evaluator-error behavior is no -longer current; it is kept here as a record of what changed and why. - -Originally: an evaluator call that failed outright (a provider error, or two -unparseable replies in a row, see `errEvaluatorUnparseable`) was the one -edge out of ACTIVE that had no clear-and-explain treatment. One production -session (`ses_01kx3ts0pjfap950bmr9b2js0b.jsonl`) hit exactly this: the -worker turn succeeded, the evaluator returned unparseable output twice in a -row, `session.error` was emitted, and the goal stayed active in the log -forever — turns=0, no `goal.eval` ever, nothing to explain the silence -beyond that one error record. The original fix made a failing evaluator -call clear the goal (unless the error was a cancelled context) before -returning, the same "no third way out of ACTIVE" principle as the -worker-turn fix above. See `TestPursueGoalUnparseableTwiceClearsGoal` -(rewritten under Round 6 — see below). - -## Review follow-up: two findings on the initial fix - -The initial fix above shipped retries with no delay between attempts and no -discussion of what a retry actually re-runs. Both were flagged in review and -are fixed here, in `engine/goal.go`: - -### 1. Retries now wait — capped exponential backoff - -Back-to-back retries with zero delay do essentially nothing against the two -transient causes the doc above and the code comments name: a rate limit and a -momentary 5xx. Both usually need at least a little wall-clock time to clear. -`promptTurnWithRetry` now waits between attempts via `waitGoalRetryBackoff`, -on the schedule computed by `goalRetryDelay`: - -| after attempt | wait before next attempt | -|---|---| -| 1 | 1s | -| 2 | 4s | -| 3+ (hypothetical, `goalWorkerRetries` is 2 today) | ×4 each time, capped at 30s | - -The wait is context-cancellable (`select` on the timer and `ctx.Done()`), so -a deliberate abort (DELETE /goal, shutdown drain) ends it immediately instead -of sleeping out the rest of the schedule — same "leave the goal exactly as it -is" semantics as a `context.Canceled` from `s.Prompt` itself. - -Tested in `engine/goal_test.go` inside `testing/synctest` bubbles -(`TestPursueGoalRetriesTransientWorkerError` asserts the exact 1s+4s elapsed -schedule; `TestPursueGoalRetryBackoffCancellable` asserts a cancellation -arriving mid-wait cuts the schedule short) — per AGENTS.md, timer-dependent -logic is bubble-tested, never a real wall-clock sleep in the test binary. -`TestGoalRetryDelaySchedule` pins the schedule as a pure function, -independent of the loop. - -### 2. Retries are not idempotent, and that is now explicit and partially gated - -A retry does not resume the failed turn. `s.Prompt(ctx, directive)` is called -again with the identical directive text, which `Prompt` treats as a brand new -user turn from scratch. That is harmless if nothing happened yet — but -`Prompt`'s own loop is `model call -> tool calls -> model call -> ...` until -end-of-turn, so a single attempt can execute one or more tool calls, append -their results, and only then hit a provider error on a *later* model call -within that same attempt (exactly "a provider error after tool calls -executed" — the case this review finding names). Retrying that attempt -re-prompts a model that still believes it needs to satisfy the original -directive, and nothing stops it from re-issuing the same tool call(s) a -second time: a shell command re-run, a file re-written. Whether that is -actually safe is entirely tool-specific and this package cannot know it in -general. - -This is now stated prominently on `promptTurnWithRetry`'s doc comment (not -just here), and it is gated where it *is* detectable: `Session` tracks a -monotonic tool-execution counter (`toolExecCount`, incremented in -`runToolCall` in `engine.go`, once per tool call that actually executes). -`promptTurnWithRetry` snapshots it before each attempt and, if an attempt -fails after the count moved, treats the failure as non-retryable — it records -the `goal.stalled` for that attempt and returns immediately, without waiting -or trying again, rather than reissuing a directive that could re-run -whatever already ran. `TestPursueGoalNoRetryAfterToolExecution` is the -red-first test: a worker call executes a tool, the next worker call always -fails, and the test asserts the tool ran exactly once and no third provider -call — or fourth, fifth, ... — is ever made. - -**This is not a general fix and is documented as such, not implied as -safety it doesn't have.** A failure before an attempt's first tool call is -still retried (correctly — nothing to redo), but if *that* retry attempt -later executes a tool and then fails again on a still-later call, the -identical risk resurfaces one attempt later. There is no bound on how many -times this can recur short of `Prompt` gaining a resumable, sub-turn -checkpoint, which it does not have today. Tools that are not idempotent -(anything that mutates external state — `bash`, `write_file`, `edit_file`) -remain at risk of double execution whenever a worker-turn retry happens to -follow a tool call within the same attempt; this document and the doc -comment on `promptTurnWithRetry` are the explicit acknowledgment the review -asked for, not a claim that the risk is eliminated. - -## Retryable-class backoff and self-re-arm (GitHub issue #61) - -### Incident - -Two production days, one shared Anthropic overload wave: four separate goal -loops (`ses_01kx6423nef95t30vxgs36p80s`, `ses_01kx6423rne45s73xx1r816g1n`, and -two more on other sessions, all on volume `harness-dev-sessions-v2`) died -within minutes of each other with - -``` -engine: goal loop stalled: anthropic: Overloaded (overloaded_error) -``` - -The fix described above (`promptTurnWithRetry`) already treats every -worker-turn error identically: `goalWorkerRetries` (2) extra attempts, -~5 seconds of total backoff, then a permanent clear. That is exactly -backwards for `overloaded_error` — Anthropic-side capacity weather that -"routinely lasts several minutes," not the kind of failure five seconds of -patience was ever going to fix. Every one of the four incident goals resumed -cleanly the instant a human manually re-armed it once the wave passed — the -strongest possible evidence that these particular stalls were premature, not -genuine. - -### The fix: classify, then split the budget - -**1. Error classification lives at the provider layer, not the engine.** -`provider.RetryableError` (`provider/retryable.go`) is a typed wrapper an -adapter attaches to an error it recognizes as transient provider weather; the -engine recovers it with `errors.As` (`provider.AsRetryable`) — there is no -string-matching of error text anywhere in `engine/goal.go`. `RetryableError` -unwraps to the original error (so `errors.Is` and any existing `%w`-wrapping -still works, including through `engine`'s own `interruptedTurnError`) and its -`Error()` prefixes the message with the class (e.g. `[retryable:overloaded] -anthropic: Overloaded (...)`), so anything that only ever calls `.Error()` — a -journaled `goal.stalled` reason, a `turn.end` error — names the class for -free, with no extra plumbing. - -- `provider/anthropic` classifies HTTP 529 (`RetryableOverloaded`), HTTP 429 - (`RetryableRateLimited`), and any other 5xx (`RetryableServerError`) — both - from the ordinary HTTP-status error path and from the mid-stream `"error"` - SSE event (keyed on the wire's own `type` field: `overloaded_error`, - `rate_limit_error`, `api_error`), which is the exact shape the incident - hit. -- `provider/openaicompat` classifies HTTP 429 and any 5xx the same way (no - dedicated "overloaded" status on that generic wire). -- Everything else — 400s, authentication failures — is left unmarked and - fails exactly as fast as before. A bad request will never succeed no - matter how long the loop waits; only transient provider weather earns the - long budget below. - -**2. `promptTurnWithRetry` runs three independent budgets, chosen per -attempt** (the third — stream truncation — is a distinct tier described in -"A third tier: stream truncation" further down; this section describes the -original two-way deterministic/retryable-weather split as it shipped for -this issue). A failure that is *not* classified retryable takes the original, -completely unchanged fast path described above. A failure that *is* -classified retryable instead runs its own loop: - -- `goalRetryableMaxAttempts` (12) attempts, versus `goalWorkerRetries`'s 2. -- `goalRetryableBackoff`'s schedule: 5s, 10s, 20s, 40s, 80s, 160s, then - capped at 5 minutes — roughly 30 minutes of total waiting in the worst - case, versus ~5 seconds for the deterministic path. Each wait applies - "equal jitter" (half the scheduled delay fixed, half randomized) via the - `goalJitterFunc` seam, specifically so that many goal loops hitting the - *same* shared overload wave (as all four incident sessions did) don't all - retry in lockstep and re-hit the still-recovering provider at the same - instant. -- These retryable-class attempts **never increment `goalWorkerRetries`'s - counter.** A provider overload wave does not spend down the same - fast-fail allowance a bad request would; a goal that survives a long - outage via this path still has its full deterministic budget intact for - whatever comes next. -- The existing non-idempotency gate (stop retrying the instant a tool has - executed during the failing attempt — see the review-follow-up section - above) applies identically to both budgets. Retrying after a tool call ran - is unsafe regardless of why the next call failed. - -Every failed attempt — deterministic or retryable — still gets exactly one -`goal.stalled` record, so the loop can never go silent (the original, -`goal.go`-level invariant this whole document exists to protect). A -retryable-class record additionally carries `retryable: true`, -`retryable_class` (`overloaded` / `rate_limited` / `server_error` / -`stream_truncated` — see "A third tier: stream truncation" below), and -`waiting: true` — except the *final* one, if the retryable budget is -actually exhausted, which flips `waiting` to `false` to mark that the loop is -giving up on waiting and about to do something else (see below). This is -what lets a session log or a live SSE subscriber tell "waiting out provider -weather" apart from "genuinely stuck" without decoding `goal_reason` text — -`Session.goal` (`GET /session/{id}`, `GET /session/{id}/wait`) surfaces the -same three fields from the most recent `goal.stalled` record, reset by -`goal.set`/`goal.eval`/`goal.achieved` exactly like `attempt` already is. - -### Self-re-arm: park, don't die - -**Superseded by the Round 7 exit-park work — see "Worker-turn exit-park and -activity-driven resume" further down.** The section below describes the fix as it shipped -originally for issue #61: the retryable budget's exhaustion stayed *inside* -`PursueGoal`, retrying the same directive on the loop's own next iteration -without ever returning. That "park in-process" shape is kept here verbatim -as the historical record of the design this section chose over the rejected -server-timer alternative — the choice to retry via the ordinary turn loop -rather than invent new scheduling state is still the right call and is -unchanged. What changed is what happens once the budget is *actually* -exhausted: staying inside `PursueGoal` to retry turned out to have its own -cost in production — the run slot stayed pinned to the parked loop for the -whole outage, so a prompt queued during a long provider outage could only -ever be injected mid-turn into a doomed attempt, never dispatched as its own -ordinary turn the way it would against any other idle session. Round 7 -changes the exhaustion branch to exit `PursueGoal` instead (freeing the run -slot) while keeping this section's classification, schedule, and budget -(`goalRetryableMaxAttempts`, `goalRetryableBackoff`) completely intact. - -This is deliverable 4 of the issue, and the one with a real design choice -behind it. Two shapes were on the table: - -- **(chosen) Park the turn in-process, bounded by `MaxTurns`/wall-clock.** - When the retryable budget is exhausted, `promptTurnWithRetry` returns a - distinguished `*goalRetryableExhaustedError` (still wrapping the - underlying error and its class) instead of the bare error. - `PursueGoal` recognizes this type and, instead of clearing the goal the - way it clears a deterministic exhaustion, retries the *same directive* on - the next iteration of its own turn loop — which, because that consumes an - ordinary turn (the `for turn := 1; ...; turn++` loop's own increment), - reaches exactly the same already-durable, already-resumable "max turns - exhausted" terminal state (`goal` left **active**, `turn.end` outcome - `max_turns_exceeded`) that an ordinary long-running goal reaches today — - or, if `MaxTurns` is unlimited (0), keeps parking indefinitely, each cycle - bounded by real wall-clock backoff time rather than hot-spinning, which is - the same opt-in "no turn limit" contract `MaxTurns == 0` already carries. -- **(rejected) Have the server re-arm the goal automatically after a - cooldown.** A `Server`-side timer that re-POSTs `/session/{id}/goal` once - some cooldown elapses after an exhaustion. Rejected because it adds an - entirely new piece of state to the state machine (a scheduled, in-memory- - only re-arm timer, alongside `goalState`, that does not - survive a process restart and duplicates logic the loop already has) for a - case the chosen design already handles for free by reusing an existing, - already-durable terminal state. It would also require deciding a *second* - budget (how many cooldown-triggered re-arms before really giving up) on - top of the retryable-class budget this fix already introduces. - -The result: a retryable-class exhaustion is **never** a dead stall requiring -an operator to notice silence and re-POST by hand (the exact zombie shape the -original incident report for this document, above, describes) — it is -either an ordinary "keep working, same directive" continuation, or, once -`MaxTurns` is reached, the same resumable "max turns" pause every other -long-running goal can hit. Every state along the way is durably explained by -a `goal.stalled` record naming the retryable class, not inferred after the -fact. - -See `engine/goal.go`'s package doc ("Round 4") for the full state diagram -and `TestPursueGoalRetryableErrorLongBackoffThenRecovers` / -`TestPursueGoalRetryableBudgetExhaustedParksInsteadOfClearing` for the -tests (both run inside a `testing/synctest` bubble — no real sleeps, per -AGENTS.md). - -## Evaluator resilience: advisory failures, bounded terminal (Round 6) - -### Incident - -Round 3 above closed the "evaluator failure leaves a zombie goal" hole by -clearing the goal on ANY `evaluateGoal` error — a provider error, or two -unparseable replies in a row. That traded one incident for another: -production data showed two fleet boxes die mid-HEALTHY-work because the -tool-less evaluator call hit a transient provider hiccup (or, once, a -stretch of oddly-worded replies neither attempt could parse) while the -worker model itself was making fine progress. Unlike a worker-turn error — -expensive to retry blindly, see `promptTurnWithRetry`'s non-idempotency doc -above — a failing evaluator call risks nothing by being retried or, failing -that, simply skipped for one turn: the worker keeps working either way, and -the only thing a bad verdict can do wrong is delay noticing completion, not -corrupt anything. - -### The fix: in-boundary retry, then advisory failure, then a bounded terminal - -`evaluateGoal` now rides out a failure in-boundary before the boundary ever -counts as "failed," mirroring `promptTurnWithRetry`'s own error -classification: - -- A provider error classified `provider.AsRetryable` gets the SAME - retryable schedule and budget the worker turn uses - (`goalRetryableBackoff`, `goalRetryableMaxAttempts`) — the two paths - share provider weather, so they share a budget's shape, each keeping its - own counter. -- A non-retryable provider error is not retried in-boundary at all (the - call is cheap; a permanently broken provider needs the boundary-failure - path below, not a wasted second attempt). -- An unparseable reply still gets its original one extra attempt, but that - attempt now uses a STRICTER system prompt (`goalEvaluatorStrictSystem`) - instead of repeating the same instructions verbatim to a model that - already failed to follow them once. - -If `evaluateGoal` still errors after all that — the retryable budget -exhausted, a non-retryable error, or two unparseable replies even with the -stricter re-ask — the boundary "fails," but failing a boundary no longer -clears the goal. `PursueGoal` journals a durable `goal.eval_failed` record -(carrying the error and the CONSECUTIVE failure count), substitutes a fixed -evaluation-unavailable notice for the next turn's guidance — never the raw -error text, and never a stale NOT-MET reason from turns ago — waits a short -backoff (`goalRetryDelay`, keyed on the consecutive count), and `continue`s: -the worker gets another ordinary turn. A later boundary that DOES parse a -verdict (MET or NOT MET) resets the consecutive count to zero — the horizon -below is about a STREAK, not a lifetime total, so one good evaluation undoes -any number of prior bad ones. - -### The horizon: a bounded, loud terminal - -Infinite advisory failures would just be Round 3's zombie-goal risk wearing -a disguise (a goal that LOOKS active but whose evaluator has been dead for -hours, silently). After `goalEvalFailureLimit` (5) CONSECUTIVE failed -boundaries, `PursueGoal` clears the goal with a dedicated reason ("goal -evaluator failed at N consecutive turn boundaries") and returns a distinct -sentinel error type (`*goalEvaluatorExhaustedError`, recognized via -`IsGoalEvaluatorExhausted`) instead of a bare error — a caller can tell this -terminal apart from an ordinary worker-turn exhaustion via `errors.As`, -never by string-matching `GoalReason`. Unlike every advisory boundary below -the horizon, this terminal DOES emit `session.error`: it must be LOUD, since -past this point nothing else will ever explain the goal's silence. The -server (`server/journal.go`) maps it to a dedicated `turn.end` outcome, -`outcomeEvaluatorExhausted` ("evaluator_exhausted"), a sibling of -`outcomeContextExhausted`/`outcomeMaxTurnsExceeded` — consumers never have -to string-match `GoalReason` to distinguish it. - -`GoalSummary`/`GET /session` surface the current consecutive count as -`eval_failures` (omitted at zero), reset on any `goal.eval` / -`goal.achieved` / `goal.cleared` / `goal.updated` record, exactly mirroring -how `attempt`/`retryable`/`waiting` already reset on the worker-retry path. -`goal.eval_failed`, like `goal.stalled`, is a pure trace record on resume — -`LoadSession`'s fold does not change resume state from it, only the -in-memory consecutive counter used while the loop is live. - -Five is deliberately much smaller than `goalRetryableMaxAttempts` (12): by -the time a boundary counts as "failed" at all, the in-boundary retryable -budget has already ridden out one boundary's worth of ordinary provider -weather, so five separate TURNS of failure (each potentially minutes apart, -each after its own full worker turn) is a much stronger signal of a truly -broken evaluator than exhausting a single boundary's in-boundary retry -budget ever is. - -See `engine/goal.go`'s package doc ("Round 6") for the full narrative, -`TestPursueGoalEvaluatorUnparseableTwiceIsAdvisory`, -`TestPursueGoalEvaluatorRetryableErrorRecoversWithinBoundary`, and -`TestPursueGoalEvaluatorTerminalAfterConsecutiveFailureLimit` (all -`testing/synctest`-bubbled) for the tests, and -`server/goal_eval_resilience_test.go` for the server-side outcome/journal -coverage. - -## Worker-turn exit-park and activity-driven resume (Round 7) - -### Incident - -OpenRouter returned HTTP 404s for a worker turn — a genuinely non-retryable, -non-overload failure, so `provider.AsRetryable` correctly classified it as -the fast, 3-attempt/~5s deterministic budget (`goalWorkerRetries`), not the -long retryable one. That budget exhausted in seconds; the goal cleared -(`goal.cleared`), `session.error` fired, and the box then sat idle for -**hours** with nothing further ever explaining or resuming it — a human had -to notice the silence and manually re-`POST /goal`, the exact zombie-adjacent -failure mode "The fix" above was originally meant to close, just reached from -the "successfully explained, then abandoned" side instead of the "silently -zombied" side. - -The retryable tier's own #61 fix (see "Self-re-arm: park, don't die" above) -was too passive in the opposite direction: it never actually left -`PursueGoal`, so the run slot stayed pinned to the parked loop for the -**entire** outage — a prompt queued during that outage (see the Prompt queue -section of AGENTS.md) could only ever be injected mid-turn into a doomed -worker attempt, never dispatched as its own ordinary turn the way it would -against any other idle session. And every parked cycle re-spent a fresh -`goalWorkerRetries`-shaped schedule internally, with no cross-cycle memory of -how long the outage had already run. - -### The fix: exit-park both tiers, resume on activity - -Every way a worker turn can exhaust its retry budget — the deterministic -tier, the retryable tier, or the non-idempotency gate stopping retries early -once a tool has already executed this attempt (see "Retries are not -idempotent" above) — now returns OUT of `PursueGoal` entirely instead of -either clearing the goal or looping in place. The loop journals a durable, -generation-gated `goal.parked` record (gated exactly like `goal.stalled`/ -`goal.eval_failed` above — a park racing a concurrent `UpdateGoal` is -silently discarded, never attributed to a condition that is no longer -current) and returns a distinct sentinel, `*goalWorkerParkedError` -(`engine.IsGoalWorkerParked`), **without** ever calling `clearGoal` — -`goalActive` stays true, `ActiveGoal()` keeps reporting the same condition, -and `LoadSession` folds `goal.parked` as a pure trace record, exactly like -`goal.stalled`. Unlike `goal.stalled`/`goal.eval_failed`, whose `Reason` -carries the raw `err.Error()` text, `goal.parked`'s `Reason` is deliberately -CLASSIFIED (`classifyGoalWorkerError`) — never a provider's raw error text — -because a park is a durable, potentially long-lived terminal that an -operator-facing surface can read long after the triggering request and its -raw provider detail are gone, unlike the two per-attempt trace records that -are read close to the moment they were written. - -Freeing the run slot this way is what closes the #61 retryable-tier gap -above: a queued prompt dispatches as a normal turn the instant the slot is -free, and the server's **pre-existing** activity-driven auto-arm -(`maybeAutoArmGoal`, upstream of `engine` — see AGENTS.md's "Prompt queue" -section) re-enters the loop with a fresh `PursueGoal` call the next time any -ordinary prompt turn completes — no new timer, no new resume machinery, the -same mechanism an ordinary idle goal already relies on. `runGoal`'s own tail -deliberately never auto-arms itself (the pre-existing anti-churn property -documented at `server/handlers.go`'s `maybeAutoArmGoal` — a park does not -immediately respawn a loop against an empty queue). - -On the server side, a worker-parked sentinel maps to `session.error` plus a -distinct `turn.end outcome=worker_parked` (`server/journal.go`'s -`turnEndOutcome`), and `goalTracker` folds the `goal.parked` record into a -third `paused` presentation arm — `pause_reason: "worker_failure"` — sitting -between the existing `"restart"` (boot-time, no loop was ever attached) and -`"provider-backoff"` (the loop IS alive, merely waiting) arms in -`pauseView`'s precedence. `compositeState` forces `idle` for `worker_failure` -exactly like it does for `restart`: no loop is actually driving the goal -until the next auto-arm or an operator re-POST, unlike `provider-backoff`, -which keeps reading `goal-running`. The `worker_failure` presentation resets -everywhere `restart`'s does (`goal.set`/`achieved`/`cleared`/`updated`, -`handleGoal`'s re-arm branch) plus in `maybeAutoArmGoal`'s own successful -arm, so a resumed loop is never seen carrying a stale pause. - -There is deliberately **no streak horizon** on parking, unlike the -evaluator's `goalEvalFailureLimit` (5) bounded terminal above: parking is -immediate at exhaustion, every time, with no cross-park counter. A parked -goal stays parked-and-armed indefinitely until either activity resumes it or -an operator issues `DELETE /session/{id}/goal` — the only clear path a -parked goal has. - -### An ambient, model-facing signal - -The durable `goal.parked` record and the server's boot-only `goal.paused` -presentation both explain a park to an OPERATOR looking at the session from -outside. Neither says anything to the MODEL itself: an agent prompted -mid-outage — a queued prompt dispatching once the exit-park frees the run -slot, or any other ordinary turn — would otherwise see nothing indicating a -supervising goal exists, is still armed, and will resume on its own. -`engine/goal_parked_status.go` closes that gap the same way `mcp_status.go` -and `process.go` already do for their own degraded/live states: a small -ambient text block — a third occupant alongside the process and MCP -segments — computed fresh from live `Session` state -(`goalParked`/`goalParkedReason`/`goalParkedAttempts`) and appended only to -the newest user message of a request that is NOT itself one of this loop's -own worker turns (`PursueGoal`'s `clearGoalParkedAtEntry` call makes that -structural: the flag is always false again before this loop's very first -worker turn of a resumed run). The text is deliberately CLASSIFIED, matching -the record's own leak rule — never the raw provider error. - -This signal is **not persisted** and does not survive a process restart — -`LoadSession` never restores it. That is a real, accepted asymmetry: after a -restart mid-park, a fresh `Prompt` call sees no ambient block at all. -Visibility in that case comes from a different surface entirely — the -server's boot-only `goal.paused` presentation (`pause_reason: "restart"`), -which is operator-facing, not model-facing, and reads the durable -`goal.parked` trace record directly rather than this runtime-only field. - -### The asymmetry: context overflow still clears - -Context overflow (issue #62) is the one deliberate exception that keeps -clearing exactly as before this round. Every other worker-turn exhaustion -this round covers is a failure that MIGHT resolve if the loop simply waits -and tries again later (a dead provider that gets fixed, an outage that ends, -an operator intervening) — parking is a bet that time helps. Context -overflow can never resolve by waiting: the same, now-too-long request fails -identically on every future attempt no matter how long the goal sits parked, -so parking it would just be a slower-burning zombie, not a fix. Clearing -immediately, with a reason a human or automation can act on right away -(compact, shorten the goal, start over), is strictly more honest than a park -that can never self-resolve. - -See `engine/goal.go`'s package doc ("Round 7") for the full narrative and -state diagram, `TestPursueGoalWorkerFailsPermanentlyParksGoal`, -`TestPursueGoalRetryableBudgetExhaustedParksInsteadOfClearing`, and -`TestPursueGoalStaleWorkerFailureDiscarded` (`engine/goal_update_test.go`, -proving a park never lands for a stale generation) for the engine-side -tests; `server/goal_worker_park_test.go` -(`TestTurnEndOutcomeWorkerParked`, `TestForcesIdlePauseIncludesWorkerFailure`, -`TestGoalTrackerPauseViewPrecedence`, `TestGoalWorkerParkFreesRunSlotForQueuedPrompt`, -`TestGoalWorkerParkResumesOnNextPromptActivity`, -`TestGoalWorkerParkPauseSurvivesRestartAsRestartReason`) for the server-side -outcome/pause/resume coverage; and `engine/goal_parked_status_test.go` for the -ambient-segment tests. - -## A third tier: stream truncation (2026-08-06 incident) - -### Incident - -A gateway in front of one provider route was found to cut response streams -at a fixed ceiling well under two minutes: the connection closed with the -response still incomplete after a handful of chunks, HTTP status already a -clean 200, and no inline provider error event — nothing structured to -classify the failure from at all. The resulting error was a bare `io.EOF`, -which `provider.AsRetryable` correctly reported as NOT retryable (there was -nothing to mark it with), so it took the fast, `goalWorkerRetries`-shaped -deterministic path and parked in seconds. A prompt re-issued minutes later -on the same model succeeded cleanly — the cut was the gateway's own -per-response ceiling, not a dead or overloaded provider, so the fast park -was premature in exactly the same way the original `overloaded_error` -incident above was, just from an entirely different cause. - -Two problems, in the same shape as the #61 incident above: the truncation -wasn't classified as anything retryable at all (so it fell into the -deterministic bucket instead), and even if it had been, the 12-attempt/ -~30-minute weather schedule (`goalRetryableMaxAttempts`) is the wrong shape -for it regardless — waiting longer never raises a gateway's fixed stream -ceiling, and every retry re-prompts a full turn at full input cost, so a -long weather-style budget just burns tokens and wall-clock time waiting for -something that will never change. - -### The fix: a dedicated class, an idle-stream watchdog, and a short-schedule tier - -**Classification without a wire signal.** Every provider adapter now marks a -stream-read error that occurs *before* its terminal event was ever seen -(`provider.MarkStreamTruncated`, `provider/retryable.go`) as -`provider.RetryableStreamTruncated` — a fourth `RetryableClass` alongside -`overloaded`/`rate_limited`/`server_error`, and the one member of the family -with no structured provider response behind it: the bare transport error -(typically `io.EOF`, or a "connection reset" net error) is wrapped as-is, -with a message naming what actually happened. A context cancellation or -deadline is left unmarked — that is the caller's own abort (`POST /abort`, -shutdown, the watchdog's own parent deadline — see below), not provider -weather, so wrapping it would misclassify a deliberate stop as a transient -failure. - -**An idle-stream watchdog gives a silent-forever stream the same identity.** -Before this round, a stream that went completely silent — no bytes, no -`EventDone`, no error, ever — was unbounded: nothing in the engine or the -adapters would ever cut it, and the turn (and any goal loop driving it) -wedged forever, holding the run slot. `engine/stream_watchdog.go`'s -per-request watchdog (`Config.StreamIdleTimeout`, config key -`stream_idle_timeout_s`, default 5 minutes mirroring Codex's -`stream_idle_timeout_ms`, a negative value disabling it) resets on every -stream event and, on expiry, cancels the request's own child context and -converts the resulting cancellation into the same `RetryableStreamTruncated` -classification — deliberately never chained to `context.Canceled` itself, so -a retry loop's `errors.Is(err, context.Canceled)` abort check still means "a -real caller abort," never "the watchdog fired." It guards the worker turn, -the goal evaluator, and the compaction summarizer's streams identically -(`armIdleWatchdog` wraps all three). - -**A third, short-schedule tier — neither deterministic nor weather.** -`promptTurnWithRetry` (and `runEvaluatorWithRetry`, its evaluator-side -counterpart) gives a `RetryableStreamTruncated` failure its own -`goalStreamTruncatedMaxAttempts` (3) budget on the SAME short backoff -schedule (`goalRetryDelay`) the deterministic tier uses (~5s total): it -never spends the deterministic budget (the failure IS classified retryable), -and it never rides the long weather-tier schedule (waiting longer cannot fix -a fixed ceiling, and unlike a cheap evaluator poll every worker attempt is a -full-price re-prompt). Exhausting this tier parks exactly like the other -two — `goal.parked`'s `retryable_class` field reads `stream_truncated`, and -every `goal.stalled` record along the way carries the same class — so an -operator or a log reader can tell "waiting out a gateway ceiling" apart from -"waiting out an overload wave" apart from "a dead deterministic failure" -without ever decoding free text. - -See `engine/goal.go`'s `goalStreamTruncatedMaxAttempts` doc comment, -`provider/retryable.go`'s `RetryableStreamTruncated`/`MarkStreamTruncated`, -and `engine/stream_watchdog_test.go` for the watchdog's own coverage. - -## Operational reliability - -Goal-supervised turns are retried by the loop above and fail visibly with a -journaled reason (`goal.stalled`, then `goal.parked` — never `goal.cleared` -— if every retry budget, deterministic, retryable-weather, or -stream-truncated, is exhausted; see "Worker-turn exit-park and -activity-driven resume" above. Context overflow remains the one exception -that still clears). Plain `prompt_async` turns get -none of that: they are not retried, and a provider stream that dies mid-turn -silently ends them. The -signature of that silent death is a final assistant message containing -reasoning parts only — no text, no tool_call. Consequently, multi-step or -long-running work dispatched over an unreliable link should be wrapped in a -goal even when no evaluation condition is actually interesting, just for the -retry/visibility behavior; and anything polling a plain prompt must treat an -idle session whose last assistant message is reasoning-only as a failure to -investigate, not as completion. +# Goal loop + +This document is the technical system of record for the current goal-loop +contract. See `docs/history/goal-loop-resilience.md` for incident history and +the sequence of earlier fixes. + +## Control loop and evaluator + +`Session.PursueGoal(ctx, condition, GoalOptions)` drives the ordinary `Prompt` +loop toward a natural-language completion condition. Turn 1 prompts the raw +condition; after **every** turn an independent, TOOL-LESS evaluator model +(`GoalOptions.Evaluator`, resolved through the same `Config.Providers` registry, +`MaxTokens` 256) is asked to answer `MET: ` / `NOT MET: ` +(parsed leniently). The evaluator request always pins `message.EffortOff` +(`runEvaluator`, `engine/goal.go`) — it is a classifier, not a reasoning task, +and it never inherits the session's own effort level. On openaicompat, +`EffortOff` sends the literal `"off"`; on anthropic, it emits no thinking +block — both routes now spend none of the evaluator's 256-token budget on +reasoning. (Issue #124.) The openai Responses route is a known residual: +`reasoningEffort` (`provider/openai/transcode.go`) omits the `reasoning` +object for `EffortOff` exactly as it does for `EffortUnset`, and a +gpt-5-class model reasons by default with no adapter-level way to disable +it — so an evaluator on that route can still spend its budget on reasoning. +A NOT MET verdict re-prompts +with a fixed-template guidance message carrying the reason; MET returns +`Achieved`. `MaxTurns` (0 = unlimited) bounds it. Evaluation is advisory: a +retryable-class provider error from the +evaluator call rides the matching in-boundary backoff before the boundary +counts as failed — the long weather-tier schedule +(`goalRetryableMaxAttempts`, ~30min) for `overloaded`/`rate_limited`/ +`server_error`, or the short stream-truncation tier +(`goalStreamTruncatedMaxAttempts`, 3 attempts, ~5s) for a stream cut before +its terminal event — `runEvaluatorWithRetry` mirrors `promptTurnWithRetry`'s +own per-class split exactly (see below); two unparseable replies in a row +(the second re-asked with a stricter prompt) or a non-retryable provider +error also fail the boundary immediately. A failed boundary no longer +clears the goal — it journals a durable `goal.eval_failed` record (carrying the consecutive +failure count), substitutes a fixed evaluation-unavailable notice for the next +turn's guidance in place of the evaluator's text, and `continue`s: the worker +keeps working. Any later boundary that DOES parse a verdict (MET or NOT MET) +resets the consecutive count to zero — the horizon is a streak, not a +lifetime total. Only after `goalEvalFailureLimit` (5) consecutive failed +boundaries does the loop treat the evaluator as durably broken: it clears the +goal with a dedicated reason, and the server maps that terminal to a +`session.error` plus a distinct `turn.end outcome=evaluator_exhausted` — loud +and machine-distinguishable, since every failure below the horizon is +deliberately silent apart from the journaled record. +Durable `goal.set` / `goal.eval` / `goal.eval_failed` / `goal.parked` / +`goal.achieved` / `goal.cleared` records land in the session log, so +`LoadSession` restores an active goal (condition only; counters reset) via +`Session.ActiveGoal()` — resume never auto-runs it, the caller decides. The +loop also emits `goal.*` engine events so the server journals them. Config +`goal_evaluator_model` supplies the evaluator for `harness run -goal` and +`POST /session/{id}/goal`. + +## Evaluator transcript bounds + +The evaluator's own `CONVERSATION TRANSCRIPT` field is BOUNDED, independent +of whatever context window the MAIN session model has and independent of +whether automatic compaction (below) has fired at all. +`renderConversationBounded` (`engine/goal.go`, called from `runEvaluator`) +replaced the old unconditional `renderConversation(s.History())`: box +bx-01m0x8996, a real long session, died with "engine: goal evaluator failed +at 5 consecutive turn boundaries: context exhausted: prompt 245332 tokens > +limit ..." because the evaluator's prompt grows with the WHOLE session +transcript forever — unlike the main session, which automatic compaction +protects, the evaluator had no bound of its own at all. The budget comes +from `goalEvaluatorTranscriptBudgetBytes`, which resolves the EVALUATOR +model's own context window via `modelContextWindowLookup` +(`modelmeta.ContextWindow`) — the same table automatic compaction's +`resolveContextWindow` (`engine/context_window.go`) reads, called +DIRECTLY rather than through `resolveContextWindow` itself: that +function's `minAutoContextWindowTokens` floor answers "should automatic +compaction ARM for this window," so a real, small, KNOWN window (gpt-4's +documented 8,192 tokens) reports identically to a genuinely UNRECOGNIZED +model (0, disabled) — conflating them would give a real small-window +evaluator a budget roughly double its actual limit, the exact overflow +class this fix closes. `goalEvaluatorTranscriptBudgetBytes` trusts ANY +positive, known window from the table, however small, and falls back to +`goalEvaluatorFallbackContextWindowTokens` (mirroring +`minAutoContextWindowTokens`'s value) only when the model has NO entry at +all. It also reserves headroom for the system prompt and MaxTokens' output +budget, and applies a conservative fraction (`goalEvaluatorContextBudgetFraction`, +0.5) on top of the same crude ~4-bytes-per-token estimate +(`bytesPerTokenEstimate`) automatic compaction's own resilience fallback +uses — reused, not reinvented. +`renderConversationBounded` walks history from the NEWEST message backward, +accumulating rendered blocks until the budget would be exceeded, and +prefers "summary + tail" for free rather than summarizing a second time: +Compact (`engine/compact.go`) already splices its own summary message in +place of whatever range it folded, tagged with the `compactionSummaryIDTag` +prefix, so the backward walk simply STOPS the instant it includes such a +message — everything before it is already captured there. The newest +message is always kept regardless of budget (an empty transcript can never +be assessed); a truncated transcript is prefixed with +`goalEvaluatorTruncationNotice` so the evaluator, and an operator reading a +later `goal.eval` record, never mistakes a bounded view for the whole +session. + +## Retryable-class backoff + +A worker-turn error (`s.Prompt` failing) is retried by `promptTurnWithRetry` +on one of FOUR independent budgets, chosen by classification via +`provider.AsRetryable` — never by matching error text. + +One class skips every budget. Before it selects a budget, +`promptTurnWithRetry` tests `provider.AsPermanent` — its fail-fast check +(`engine/goal.go`) — and fails fast: a permanent error gets ONE attempt and +no retry. +`provider.MarkPermanent` marks a malformed request shape. The anthropic +adapter applies it to an HTTP 400 `invalid_request_error`, and to the same +error type mid-stream (`provider/anthropic/anthropic.go:114` and `:484`), +only after `parseContextOverflow` rules overflow out — the two are disjoint. +A retry never repairs a malformed request, and each attempt costs a full +turn at full input price. A permanent error still PARKS, exactly like every +budget exhaustion; it never clears. `permanent` is threaded through only to +select a more accurate classified reason and tier name +(`classifyGoalWorkerError`, `goalWorkerParkedError`), so an operator can +tell a single-attempt park from `goalWorkerRetries`+1 identical attempts. + +One shape is wrapped `provider.MarkPermanent` by the adapter but is +DELIBERATELY EXCLUDED from this fail-fast branch: `provider.AsProviderExhausted` +— an ACCOUNT-level usage/quota wall (PR #174's `provider.ErrKindProviderExhausted`, +originally added for task-child resumability; see `engine/session_manager.go`'s +`FailKindProviderExhausted`). An adapter marks it permanent for ordinary +HTTP-retry purposes (no short backoff schedule outlives a monthly quota), +but a wall lifts on its own, unchanged, so treating it as a doomed malformed +request silently kills goal supervision on the very first usage-limit +rejection. Live evidence: box bx-01m0x8996 parked after "1 permanent-tier +attempt(s)" on "You have reached your specified API usage limits" and never +resumed without an operator `DELETE` + re-register. `promptTurnWithRetry` +and `PursueGoal`'s worker-turn handling both check +`provider.AsProviderExhausted` explicitly and fold a positive result into +their local `retryable`/`class` bookkeeping (`class` set to the dedicated +marker `goalClassProviderExhausted`, never one of `provider.RetryableClass`'s +real values) — reusing the existing stall/park recording machinery rather +than adding a fourth field throughout. + +A deterministic +failure (not classified retryable, not permanent) gets `goalWorkerRetries` (2) additional +attempts on the short schedule (~5s total: 1s, then 4s). A provider error +classified `overloaded`/`rate_limited`/`server_error` gets a separately +budgeted `goalRetryableMaxAttempts` (12) backoff (~30min total, jittered, 5s +doubling to a 5min cap) that never spends the deterministic budget. A +provider error classified `provider.RetryableStreamTruncated` — a response +stream that died before its terminal event, with no HTTP status or inline +error to classify from (see the idle-stream watchdog below) — gets its own +`goalStreamTruncatedMaxAttempts` (3) budget on the SAME short schedule the +deterministic tier uses (~5s total): truncation is retryable, but it is not +weather — waiting longer never raises a stream ceiling, and every retry +re-prompts a full turn at full input cost — so it must ride neither the fast +deterministic budget nor the long weather-tier one. A `provider.AsProviderExhausted` +failure gets its OWN `goalProviderExhaustedMaxAttempts` budget (equal in +size to `goalRetryableMaxAttempts`, on the identical jittered schedule) — +never the ordinary weather counter, so a concurrent overload spell and an +account wall in the same turn can never silently share or steal from one +another's budget. It deliberately rides the SAME schedule ordinary weather +uses rather than computing a wait from the provider's own `RecoverHint` +("you regain access on "): `RecoverHint`'s format varies by provider +and by plan (see `provider.Error.RecoverHint`'s doc comment), so it is +never parsed into a duration, only ever quoted verbatim to a model-visible +caller. A wall that clears within the budget (a burst rate limit that +reached this classification, or a short-lived cap) resumes the worker turn +— and the whole goal — with NO operator action; a wall measured in hours or +days still exhausts the budget and parks, but honestly classified via +`classifyGoalWorkerError`'s dedicated branch ("provider account usage limit +exhausted the retry budget"), never as a permanent, unretriable request. +Every attempt records a +`goal.stalled` record regardless of tier, so the loop is never silent. +Exhausting ANY of the four budgets — or the non-idempotency gate stopping +retries early once a tool has already executed this attempt — PARKS the goal +instead of clearing it: `PursueGoal` exits, journals a durable, CLASSIFIED +`goal.parked` record (never raw provider error text — the same leak rule +`goal.eval_failed` follows), and returns a distinct `*goalWorkerParkedError` +sentinel (`engine.IsGoalWorkerParked`) WITHOUT calling `clearGoal` — +`goalActive` stays true, the condition is untouched, generation-gated exactly +like `goal.stalled`/`goal.eval_failed` so a park racing a concurrent +`UpdateGoal` is silently discarded rather than attributed to a condition the +model never saw. This supersedes both this package's earlier +deterministic-tier clear and GitHub issue #61's in-loop retryable-tier +self-re-arming `continue` — the latter pinned the run slot to the parked loop +for the whole outage; exiting instead frees the slot, so a queued prompt +dispatches as an ordinary turn during a long outage instead of only ever +being injected mid-turn into a doomed attempt. Context overflow (issue #62) +is the one deliberate exception and still clears immediately, never parks: +no amount of waiting fixes an oversized request, so parking it would just be +a slower-burning zombie instead of a fix. Parking has no streak horizon +(unlike the evaluator's 5-boundary terminal above) — every exhaustion parks +immediately, and `DELETE /session/{id}/goal` remains the only clear path for +a parked goal. + +## Directive reuse across retries + +Each retry re-issues the SAME directive, and `Prompt` appends whatever text +it gets as a brand-new user message — it has no notion of "this is a retry, +do not duplicate." Left alone, N failed attempts leave N unanswered copies of +one directive, and every LATER request pays for all of them. `Prompt` +persists each copy before the provider call that fails, so the duplicates +reach the durable log, not just live history. + +`promptTurnWithRetry` therefore never appends a second copy for the common +case. It tracks one `anchorID`, naming the point right before this turn's +CURRENT, still-unanswered directive — starting as `lastMessageID`, captured +once before attempt 1 — then dispatches each retry one of three ways +(`engine/goal.go`, `tailAfterAnchor` shares the anchor-to-tail lookup; see +docs/design/goal-retry-directive-reuse.md): + +- Attempt 1 calls `Prompt`, which appends the directive. +- A retry whose tail after `anchorID` is EXACTLY the previous attempt's + unanswered directive (`directiveReuseEligible`) calls `runAgenticLoop` + instead. That runs the turn loop against history as it stands and appends + nothing, so the existing message is answered rather than duplicated. +- Any other tail falls back to `dropUnansweredDirective` plus `Prompt`, then + re-anchors: `anchorID` moves to `lastMessageID`, the point right before + the fresh directive `Prompt` is about to append. A later attempt's reuse + check then measures from that new directive, never from the turn's + original start. + +`runAgenticLoop` is `Prompt`'s own loop body, split out unchanged +(`engine/engine.go`). `Prompt` still appends and then calls it, so `Prompt`'s +observable behavior is identical: same events, same `emitStatus`, same usage +accounting. Note that `maybeAutoCompact` stays in `Prompt` and does NOT run +on the reuse path. That is deliberate, and the reason is that history did +not grow: the reuse path is reachable only when the tail is exactly one +message, so no new completed turn appeared to fold since attempt 1 already +ran the check. (`maybeAutoCompact` folds only COMPLETED turns, so it would +never have folded the unanswered tail directive itself.) One narrow +residual: history sitting right at the threshold, where appending a +directive would tip it over, no longer triggers a mid-outage fold. That is +accepted — the outage that piles up retries is also when the summarizer's +own provider call fails, and compaction is best-effort anyway. + +`dropUnansweredDirective` remains the fallback for the interrupted-turn tail +(the directive plus a partial assistant message and its synthetic +tool-result message), and for any tail a denied tool call or delivered mail +makes undroppable. It anchors on a message ID, never on a history length, +and `isSafeToDropDirectiveTail` approves only that interrupted-turn shape +and the bare directive. Any other tail is left untouched — a denied tool's +result, or an already-delivered "OPERATOR MESSAGES" block, must never be +discarded. It mutates only live history and can never retract a journaled +record, which is why the reuse path above, not a retraction, is what keeps +the log clean. `promptTurnWithRetry`'s re-anchor above bounds an undroppable +residue's cost to ONE extra duplicate directive for the rest of the turn, +never one per remaining attempt: re-anchoring past it lets reuse resume on +the very next attempt instead of re-appending against a tail that can never +shrink back to a droppable shape again. + +## Idle-stream watchdog + +An idle provider stream — one that goes silent with no bytes, no +`EventDone`, no error, ever — is bounded by a per-request idle-stream +watchdog (`engine/stream_watchdog.go`, `Config.StreamIdleTimeout`, config key +`stream_idle_timeout_s`): every stream event resets its timer, and on expiry +it cancels the request's child context and converts the resulting +cancellation into a classified `provider.RetryableStreamTruncated` error +instead of an anonymous "context canceled" — this is what feeds the +stream-truncation tier above. It defaults to 5 minutes (mirroring Codex's +`stream_idle_timeout_ms`), a negative value disables it, and it guards the +worker turn, the goal evaluator, and the compaction summarizer's streams +alike (`armIdleWatchdog` wraps all three, so a silent stream at any of them +can no longer wedge the session forever while holding the run slot). + +## Automatic compaction fallback + +Automatic compaction's over-threshold check +(`maybeAutoCompact`/`estimatePromptTokensFromHistory`, `engine/compact.go`) +has its own resilience fallback: a provider route that reports all-zero +input usage on a turn that DID complete is treated as missing data, never as +"0 tokens, never over" — the check falls back to a crude ~4-bytes-per-token +estimate walked from the actual session history so the overflow-prevention +layer keeps functioning instead of going permanently dark on that route, +which otherwise runs to a hard context overflow that clears (never parks) an +active goal. + +## Context-window resolution + +The goal loop uses the session's normal context-window policy. A positive +`Config.ContextWindowTokens` value is pinned for the session. A negative value +is an explicit opt-out. Otherwise, `resolveContextWindow` derives the window +from the static `modelmeta` table and `SetModel` re-derives it after a model +switch. + +A known model below `minAutoContextWindowTokens` remains valid but does not arm +automatic compaction. A registry miss returns `ErrUnknownContextWindow`. When +`Config.RequireContextWindow` is true, session creation, model selection, and +prompting refuse that model. When it is false, the session retains the legacy +disabled-compaction behavior. + +Read `docs/models-and-providers.md` for the refusal policy and +`docs/design/context-compaction.md` for compaction behavior. + +## Server state and recovery + +On the server, a worker-parked sentinel maps to `session.error` plus a +distinct `turn.end outcome=worker_parked`, and `goalTracker` folds the +durable `goal.parked` record into a third `paused` arm (`pause_reason: +"worker_failure"`, alongside the existing boot-only `"restart"` and live +`"provider-backoff"`) — `compositeState` forces `idle` for it, and for a +restart pause, unless a turn is actually running, which reads `busy`: forced +idle must never mask a live turn (an ordinary prompt, or the resume prompt +that eventually re-arms the goal, can be streaming while the goal itself +sits parked), whereas provider-backoff's loop is merely waiting and keeps +reading `goal-running` regardless of whether a turn happens to be running. +Resume needs no new machinery: the existing activity-driven +`maybeAutoArmGoal` re-arms any active goal — parked or not — the next time an +ordinary prompt turn completes, resetting the `worker_failure` presentation; +`runGoal`'s own tail deliberately never auto-arms (the same anti-churn +property that already stops a freshly-parked goal from immediately +respawning a loop against an empty queue). + +## Structured server logging + +`harness serve` can also make this turn/goal lifecycle visible on stderr: +`server.Options.Logger`, when set (`cmd/harness/main.go` wires a +`slog.NewJSONHandler(os.Stderr, nil)` logger into it for `serveCmd`), emits a +structured line at every `recordTurnEnd` call (INFO for outcome "completed", +WARN otherwise) and at the `goal.*`/`session.error` durable-record choke +points — a heartbeat for the life of the box instead of logging only at +boot/config/MCP wiring, matching Codex's own structured stream-retry +logging. Nil (the default) disables all of it; every call site nil-guards +first, so an unset Logger is exactly the prior silent behavior. + +## Parked-goal ambient status + +A worker-parked goal is also surfaced in-session, model-facing: +`Session.goalParked` (set when a park lands, cleared at every `PursueGoal` +entry) drives a third ambient status segment — alongside the process and MCP +segments — appended to the newest user message of any turn that is NOT +itself one of this loop's own worker turns, naming the classified reason and +stating the goal resumes automatically. It is runtime-only and never +persisted; after a process restart, visibility reverts entirely to the +boot-only `goal.paused`/`pause_reason: "restart"` presentation instead — a +deliberate, documented asymmetry. + +## Updating an active goal + +The condition itself is adjustable mid-loop. `Session.UpdateGoal` rewrites an +active goal's condition, journals a durable `goal.updated` record, and emits +`EventGoalUpdated` — same lock-and-emit-under-`s.mu` shape as `RegisterGoal`; +a same-condition update is a silent no-op, updating an inactive goal errors. +`PursueGoal` takes a per-turn snapshot (condition, a runtime-only generation +counter, active) instead of closing over the original parameter, so a live +loop picks up new text at its very next turn boundary — both the worker +directive and the evaluator call. The generation counter guards stale +verdicts: if `UpdateGoal` lands while an evaluator call for generation N is +in flight, a MET (or stalled) verdict for N is discarded on return — no +`goal.achieved`, no `goal.eval`, the loop just continues against the new +condition, never a false-positive completion against text the model never +saw. `ClearGoal` is unaffected — it keys on `goalActive`, not condition +equality, so it still stops the loop at every point it does today. + +## Goal tool and host behavior + +A built-in `goal` session tool (gated on `Config.GoalTool`) lets the model +inspect or drive its own goal in-process: no HTTP round-trip, no run-slot +claim. `status` reports `{active, condition}`; `set` arms a new goal via +`RegisterGoal` (errors telling the model to use `adjust` if a goal is already +active); `adjust` rewrites an active goal's condition via `UpdateGoal`. There +is deliberately **no `clear` action** — see below. + +`Config.GoalTool` is on whenever `goal_evaluator_model` is configured, in +`harness run` and `harness serve` alike, entirely independent of the `-goal` +flag — a plain `harness run -p ...` with that config set still registers the +tool. But what happens after `set`/`adjust` differs by host: `harness serve` +auto-arms (see `maybeAutoArmGoal` below) — the loop actually starts running +once the current turn ends. Plain `harness run` (no `-goal`) has no such +auto-arm step: a tool-driven `set` call registers and journals the goal +(`goal.active` becomes true) but nothing ever calls `PursueGoal` for it, so +it never actually starts evaluating — the process runs its one `Prompt` call +and exits with the goal armed but inert. Only `harness run -goal ` +itself drives `PursueGoal` to completion. + +## HTTP goal updates and auto-arm + +`POST /session/{id}/goal` on a busy session no longer flatly 409s. A running +goal loop updates its condition in place (`status: "updated"`, 200 — no +second loop, no run-slot claim; the loop picks it up at its next turn +boundary). A plain prompt holding the slot with no goal yet active registers +the goal (`RegisterGoal` needs no run slot) and then retries the claim once, +closing the race against that same prompt's own `runPrompt` tail: if the +retry wins the now-freed slot, the loop spawns immediately and the response +reports `status: "started"` (202); otherwise the prompt's tail is still +ahead of us, its own auto-arm check (`maybeAutoArmGoal`) will claim the slot +and spawn the loop itself once that tail finishes, and the response reports +`status: "armed"` (202) — either way the loop starts exactly once, never +zero times, never twice, no further client action needed. This is also how +the `goal` tool's own `set` action takes effect: arming a goal mid-turn, the +same auto-arm path starts the loop the instant the current turn ends. A +workdir held by a genuinely different session still 409s, +unchanged. + +## Operator-only clear + +No self-clear is deliberate: a goal-supervised agent must never be able to +cancel its own supervision from inside a running turn, so the `goal` tool +has no `clear` action — `DELETE /session/{id}/goal` remains the only clear +path, and it is operator-only. + +## Deliberate exclusions + +The goal loop is a **plan-artifact-free, gate-free** control loop: it is +`Prompt` plus a read-only evaluator call, with no plan document, no edit/plan +mode, and no permission gate. diff --git a/docs/history/goal-loop-resilience.md b/docs/history/goal-loop-resilience.md new file mode 100644 index 00000000..3480eb76 --- /dev/null +++ b/docs/history/goal-loop-resilience.md @@ -0,0 +1,672 @@ +# Goal-loop resilience: forensic root cause and the state-machine fix + +This document records the implementation history. See `docs/goal-loop.md` for +the current goal-loop contract. + +## Incident + +Two production sessions were found with an active goal that had stopped +making progress: `ses_41813d5a411c2ba5.jsonl` and, earlier, +`ses_55e4ae35d8344540.jsonl`. Both show the same shape. Immediately after a +`goal.eval` "NOT MET" verdict, the fixed-template guidance message is +appended to the log as the next directive — and then nothing. No assistant +turn, no error record, no further `goal.eval`. In `ses_41813d5a411c2ba5.jsonl` +the guidance message is timestamped `2026-07-09T05:20:12Z`; the very next +record in the file is a bare `goal.cleared`, followed by a message +timestamped `2026-07-09T12:12:52Z` — nearly seven hours later — that reads: + +> "Your goal loop was interrupted. Do exactly this and nothing else: cd +> /work/tssdk && git add -A && git commit ... && git push ..." + +That `goal.cleared` and the recovery message are not something the engine +produced. They are a human, seven hours later, noticing the session had gone +silent with an active goal, clearing it by hand, and manually steering the +agent to at least commit and push whatever it had. `ses_55e4ae35d8344540.jsonl` +shows the identical pattern twice in a row: a `goal.set` → guidance message → +silence → (manually cleared) → `goal.set` (a manual resume) → guidance +message → silence → (manually cleared) again. + +The goal's own directive in both sessions instructs the worker to `apt-get +install -y nodejs npm` and install a TypeScript SDK toolchain — commands that +can emit megabytes of installer/build output. The engine's `bash` tool had no +enforced output cap wired through `Config` (a 100KB post-hoc truncation +existed as a hardcoded constant, tail-only, not configurable), so a large +enough burst of tool output from exactly the kind of command these goals ran +is a plausible trigger for whatever made the next worker turn fail. But the +log is silent about *what* failed — because that is exactly the bug: nothing +recorded the failure at all. + +## Root cause + +`engine/goal.go`'s `PursueGoal` loop called the worker turn like this: + +```go +if _, err := s.Prompt(ctx, directive); err != nil { + return nil, err +} +``` + +Any error from `s.Prompt` — a provider timeout, a rate limit, a stream error +triggered while handling an oversized tool result, anything — propagated +straight out of `PursueGoal` with **no goal.eval, no goal.stalled, no +goal.cleared**. `goalActive` stayed `true` in memory and in the session log. +The server's `runGoal` (`server/handlers.go`) does journal a `session.error` +for such an error, but never clears the goal, so the session parks forever +with an active goal that no automated process will ever revisit — a zombie. +Only a human reading the log tail could tell the loop had died and clear it +by hand, which is exactly what happened, hours later, in both incidents. + +## The fix + +Two independent layers, both TDD red-first (see `engine/goal_test.go` and +`engine/bash_test.go`): + +1. **Goal-loop resilience** (`engine/goal.go`): a worker-turn error now goes + through `promptTurnWithRetry`, which retries the same directive up to + `goalWorkerRetries` (2) additional times, recording a durable + `goal.stalled` record/event (carrying the error and the attempt number) + for every failed attempt — the loop can never go silent again. If every + attempt fails, the goal is **cleared** (`goal.cleared`, carrying the error + as `GoalReason`) before the error is returned, so `goalActive` can never be + left `true` with nothing left to explain it. A `context.Canceled` error + (DELETE /goal, shutdown drain) is never retried or treated as a failure — + it is a deliberate, resumable stop, and the goal is left untouched. See the + state-machine diagram in the `goal.go` package doc. (**Superseded by the + Round 7 exit-park work**: exhausting this budget no longer clears — it PARKS, exactly + like the retryable-class budget below — see "Worker-turn exit-park and + activity-driven resume" further down.) + +2. **Bash output cap** (`engine/bash.go`): tool output is now bounded by a new + `Config.BashOutputCap` knob (default 96KB) enforced by a `cappedWriter` + during capture — not by buffering the full output and truncating + afterward — so a runaway command (an `apt-get`/`npm install` storm is the + real-world trigger) can allocate only `O(cap)` memory and can never dump + megabytes into a single message. The cap keeps both the head (so the + command and its early output stay visible) and the tail (so a trailing + error banner stays visible), joined by a `"N bytes truncated"` marker, + before the output ever reaches the message log or the next provider + request built from it. + +## Non-goals / things this does not change + +- `goal.stalled` is a pure trace record: it never flips `goalActive` by + itself (see `LoadSession`'s `scanLog` switch in `store.go`). +- `evaluateGoal`'s own error path — a provider error while asking the + evaluator, or two unparseable replies in a row — is **no longer** + unchanged from the worker-turn path. It used to clear the goal outright + (see "Round 3" below); as of Round 6 it is advisory, mirroring the + worker-turn retry machinery instead of bypassing it. See "Evaluator + resilience" below for the current behavior — this bullet exists only so a + reader following an old link lands somewhere accurate. + +## Round 3: closing the evaluator's own zombie path (superseded by Round 6) + +The paragraph below describes the fix as it shipped originally: any +`evaluateGoal` error — a provider error, or two unparseable replies in a +row — cleared the goal outright, on the theory that the goal's own state +was "still accurate and worth preserving for a human-triggered resume." +Production experience proved that theory wrong (see "Evaluator resilience" +below): a transient evaluator hiccup killed sessions where the worker model +was making fine progress. The clear-on-any-evaluator-error behavior is no +longer current; it is kept here as a record of what changed and why. + +Originally: an evaluator call that failed outright (a provider error, or two +unparseable replies in a row, see `errEvaluatorUnparseable`) was the one +edge out of ACTIVE that had no clear-and-explain treatment. One production +session (`ses_01kx3ts0pjfap950bmr9b2js0b.jsonl`) hit exactly this: the +worker turn succeeded, the evaluator returned unparseable output twice in a +row, `session.error` was emitted, and the goal stayed active in the log +forever — turns=0, no `goal.eval` ever, nothing to explain the silence +beyond that one error record. The original fix made a failing evaluator +call clear the goal (unless the error was a cancelled context) before +returning, the same "no third way out of ACTIVE" principle as the +worker-turn fix above. See `TestPursueGoalUnparseableTwiceClearsGoal` +(rewritten under Round 6 — see below). + +## Review follow-up: two findings on the initial fix + +The initial fix above shipped retries with no delay between attempts and no +discussion of what a retry actually re-runs. Both were flagged in review and +are fixed here, in `engine/goal.go`: + +### 1. Retries now wait — capped exponential backoff + +Back-to-back retries with zero delay do essentially nothing against the two +transient causes the doc above and the code comments name: a rate limit and a +momentary 5xx. Both usually need at least a little wall-clock time to clear. +`promptTurnWithRetry` now waits between attempts via `waitGoalRetryBackoff`, +on the schedule computed by `goalRetryDelay`: + +| after attempt | wait before next attempt | +|---|---| +| 1 | 1s | +| 2 | 4s | +| 3+ (hypothetical, `goalWorkerRetries` is 2 today) | ×4 each time, capped at 30s | + +The wait is context-cancellable (`select` on the timer and `ctx.Done()`), so +a deliberate abort (DELETE /goal, shutdown drain) ends it immediately instead +of sleeping out the rest of the schedule — same "leave the goal exactly as it +is" semantics as a `context.Canceled` from `s.Prompt` itself. + +Tested in `engine/goal_test.go` inside `testing/synctest` bubbles +(`TestPursueGoalRetriesTransientWorkerError` asserts the exact 1s+4s elapsed +schedule; `TestPursueGoalRetryBackoffCancellable` asserts a cancellation +arriving mid-wait cuts the schedule short) — per the root `AGENTS.md`, timer-dependent +logic is bubble-tested, never a real wall-clock sleep in the test binary. +`TestGoalRetryDelaySchedule` pins the schedule as a pure function, +independent of the loop. + +### 2. Retries are not idempotent, and that is now explicit and partially gated + +A retry does not resume the failed turn. `s.Prompt(ctx, directive)` is called +again with the identical directive text, which `Prompt` treats as a brand new +user turn from scratch. That is harmless if nothing happened yet — but +`Prompt`'s own loop is `model call -> tool calls -> model call -> ...` until +end-of-turn, so a single attempt can execute one or more tool calls, append +their results, and only then hit a provider error on a *later* model call +within that same attempt (exactly "a provider error after tool calls +executed" — the case this review finding names). Retrying that attempt +re-prompts a model that still believes it needs to satisfy the original +directive, and nothing stops it from re-issuing the same tool call(s) a +second time: a shell command re-run, a file re-written. Whether that is +actually safe is entirely tool-specific and this package cannot know it in +general. + +This is now stated prominently on `promptTurnWithRetry`'s doc comment (not +just here), and it is gated where it *is* detectable: `Session` tracks a +monotonic tool-execution counter (`toolExecCount`, incremented in +`runToolCall` in `engine.go`, once per tool call that actually executes). +`promptTurnWithRetry` snapshots it before each attempt and, if an attempt +fails after the count moved, treats the failure as non-retryable — it records +the `goal.stalled` for that attempt and returns immediately, without waiting +or trying again, rather than reissuing a directive that could re-run +whatever already ran. `TestPursueGoalNoRetryAfterToolExecution` is the +red-first test: a worker call executes a tool, the next worker call always +fails, and the test asserts the tool ran exactly once and no third provider +call — or fourth, fifth, ... — is ever made. + +**This is not a general fix and is documented as such, not implied as +safety it doesn't have.** A failure before an attempt's first tool call is +still retried (correctly — nothing to redo), but if *that* retry attempt +later executes a tool and then fails again on a still-later call, the +identical risk resurfaces one attempt later. There is no bound on how many +times this can recur short of `Prompt` gaining a resumable, sub-turn +checkpoint, which it does not have today. Tools that are not idempotent +(anything that mutates external state — `bash`, `write_file`, `edit_file`) +remain at risk of double execution whenever a worker-turn retry happens to +follow a tool call within the same attempt; this document and the doc +comment on `promptTurnWithRetry` are the explicit acknowledgment the review +asked for, not a claim that the risk is eliminated. + +## Retryable-class backoff and self-re-arm (GitHub issue #61) + +### Incident + +Two production days, one shared Anthropic overload wave: four separate goal +loops (`ses_01kx6423nef95t30vxgs36p80s`, `ses_01kx6423rne45s73xx1r816g1n`, and +two more on other sessions, all on volume `harness-dev-sessions-v2`) died +within minutes of each other with + +``` +engine: goal loop stalled: anthropic: Overloaded (overloaded_error) +``` + +The fix described above (`promptTurnWithRetry`) already treats every +worker-turn error identically: `goalWorkerRetries` (2) extra attempts, +~5 seconds of total backoff, then a permanent clear. That is exactly +backwards for `overloaded_error` — Anthropic-side capacity weather that +"routinely lasts several minutes," not the kind of failure five seconds of +patience was ever going to fix. Every one of the four incident goals resumed +cleanly the instant a human manually re-armed it once the wave passed — the +strongest possible evidence that these particular stalls were premature, not +genuine. + +### The fix: classify, then split the budget + +**1. Error classification lives at the provider layer, not the engine.** +`provider.RetryableError` (`provider/retryable.go`) is a typed wrapper an +adapter attaches to an error it recognizes as transient provider weather; the +engine recovers it with `errors.As` (`provider.AsRetryable`) — there is no +string-matching of error text anywhere in `engine/goal.go`. `RetryableError` +unwraps to the original error (so `errors.Is` and any existing `%w`-wrapping +still works, including through `engine`'s own `interruptedTurnError`) and its +`Error()` prefixes the message with the class (e.g. `[retryable:overloaded] +anthropic: Overloaded (...)`), so anything that only ever calls `.Error()` — a +journaled `goal.stalled` reason, a `turn.end` error — names the class for +free, with no extra plumbing. + +- `provider/anthropic` classifies HTTP 529 (`RetryableOverloaded`), HTTP 429 + (`RetryableRateLimited`), and any other 5xx (`RetryableServerError`) — both + from the ordinary HTTP-status error path and from the mid-stream `"error"` + SSE event (keyed on the wire's own `type` field: `overloaded_error`, + `rate_limit_error`, `api_error`), which is the exact shape the incident + hit. +- `provider/openaicompat` classifies HTTP 429 and any 5xx the same way (no + dedicated "overloaded" status on that generic wire). +- Everything else — 400s, authentication failures — is left unmarked and + fails exactly as fast as before. A bad request will never succeed no + matter how long the loop waits; only transient provider weather earns the + long budget below. + +**2. `promptTurnWithRetry` runs three independent budgets, chosen per +attempt** (the third — stream truncation — is a distinct tier described in +"A third tier: stream truncation" further down; this section describes the +original two-way deterministic/retryable-weather split as it shipped for +this issue). A failure that is *not* classified retryable takes the original, +completely unchanged fast path described above. A failure that *is* +classified retryable instead runs its own loop: + +- `goalRetryableMaxAttempts` (12) attempts, versus `goalWorkerRetries`'s 2. +- `goalRetryableBackoff`'s schedule: 5s, 10s, 20s, 40s, 80s, 160s, then + capped at 5 minutes — roughly 30 minutes of total waiting in the worst + case, versus ~5 seconds for the deterministic path. Each wait applies + "equal jitter" (half the scheduled delay fixed, half randomized) via the + `goalJitterFunc` seam, specifically so that many goal loops hitting the + *same* shared overload wave (as all four incident sessions did) don't all + retry in lockstep and re-hit the still-recovering provider at the same + instant. +- These retryable-class attempts **never increment `goalWorkerRetries`'s + counter.** A provider overload wave does not spend down the same + fast-fail allowance a bad request would; a goal that survives a long + outage via this path still has its full deterministic budget intact for + whatever comes next. +- The existing non-idempotency gate (stop retrying the instant a tool has + executed during the failing attempt — see the review-follow-up section + above) applies identically to both budgets. Retrying after a tool call ran + is unsafe regardless of why the next call failed. + +Every failed attempt — deterministic or retryable — still gets exactly one +`goal.stalled` record, so the loop can never go silent (the original, +`goal.go`-level invariant this whole document exists to protect). A +retryable-class record additionally carries `retryable: true`, +`retryable_class` (`overloaded` / `rate_limited` / `server_error` / +`stream_truncated` — see "A third tier: stream truncation" below), and +`waiting: true` — except the *final* one, if the retryable budget is +actually exhausted, which flips `waiting` to `false` to mark that the loop is +giving up on waiting and about to do something else (see below). This is +what lets a session log or a live SSE subscriber tell "waiting out provider +weather" apart from "genuinely stuck" without decoding `goal_reason` text — +`Session.goal` (`GET /session/{id}`, `GET /session/{id}/wait`) surfaces the +same three fields from the most recent `goal.stalled` record, reset by +`goal.set`/`goal.eval`/`goal.achieved` exactly like `attempt` already is. + +### Self-re-arm: park, don't die + +**Superseded by the Round 7 exit-park work — see "Worker-turn exit-park and +activity-driven resume" further down.** The section below describes the fix as it shipped +originally for issue #61: the retryable budget's exhaustion stayed *inside* +`PursueGoal`, retrying the same directive on the loop's own next iteration +without ever returning. That "park in-process" shape is kept here verbatim +as the historical record of the design this section chose over the rejected +server-timer alternative — the choice to retry via the ordinary turn loop +rather than invent new scheduling state is still the right call and is +unchanged. What changed is what happens once the budget is *actually* +exhausted: staying inside `PursueGoal` to retry turned out to have its own +cost in production — the run slot stayed pinned to the parked loop for the +whole outage, so a prompt queued during a long provider outage could only +ever be injected mid-turn into a doomed attempt, never dispatched as its own +ordinary turn the way it would against any other idle session. Round 7 +changes the exhaustion branch to exit `PursueGoal` instead (freeing the run +slot) while keeping this section's classification, schedule, and budget +(`goalRetryableMaxAttempts`, `goalRetryableBackoff`) completely intact. + +This is deliverable 4 of the issue, and the one with a real design choice +behind it. Two shapes were on the table: + +- **(chosen) Park the turn in-process, bounded by `MaxTurns`/wall-clock.** + When the retryable budget is exhausted, `promptTurnWithRetry` returns a + distinguished `*goalRetryableExhaustedError` (still wrapping the + underlying error and its class) instead of the bare error. + `PursueGoal` recognizes this type and, instead of clearing the goal the + way it clears a deterministic exhaustion, retries the *same directive* on + the next iteration of its own turn loop — which, because that consumes an + ordinary turn (the `for turn := 1; ...; turn++` loop's own increment), + reaches exactly the same already-durable, already-resumable "max turns + exhausted" terminal state (`goal` left **active**, `turn.end` outcome + `max_turns_exceeded`) that an ordinary long-running goal reaches today — + or, if `MaxTurns` is unlimited (0), keeps parking indefinitely, each cycle + bounded by real wall-clock backoff time rather than hot-spinning, which is + the same opt-in "no turn limit" contract `MaxTurns == 0` already carries. +- **(rejected) Have the server re-arm the goal automatically after a + cooldown.** A `Server`-side timer that re-POSTs `/session/{id}/goal` once + some cooldown elapses after an exhaustion. Rejected because it adds an + entirely new piece of state to the state machine (a scheduled, in-memory- + only re-arm timer, alongside `goalState`, that does not + survive a process restart and duplicates logic the loop already has) for a + case the chosen design already handles for free by reusing an existing, + already-durable terminal state. It would also require deciding a *second* + budget (how many cooldown-triggered re-arms before really giving up) on + top of the retryable-class budget this fix already introduces. + +The result: a retryable-class exhaustion is **never** a dead stall requiring +an operator to notice silence and re-POST by hand (the exact zombie shape the +original incident report for this document, above, describes) — it is +either an ordinary "keep working, same directive" continuation, or, once +`MaxTurns` is reached, the same resumable "max turns" pause every other +long-running goal can hit. Every state along the way is durably explained by +a `goal.stalled` record naming the retryable class, not inferred after the +fact. + +See `engine/goal.go`'s package doc ("Round 4") for the full state diagram +and `TestPursueGoalRetryableErrorLongBackoffThenRecovers` / +`TestPursueGoalRetryableBudgetExhaustedParksInsteadOfClearing` for the +tests (both run inside a `testing/synctest` bubble — no real sleeps, per the +root `AGENTS.md`). + +## Evaluator resilience: advisory failures, bounded terminal (Round 6) + +### Incident + +Round 3 above closed the "evaluator failure leaves a zombie goal" hole by +clearing the goal on ANY `evaluateGoal` error — a provider error, or two +unparseable replies in a row. That traded one incident for another: +production data showed two fleet boxes die mid-HEALTHY-work because the +tool-less evaluator call hit a transient provider hiccup (or, once, a +stretch of oddly-worded replies neither attempt could parse) while the +worker model itself was making fine progress. Unlike a worker-turn error — +expensive to retry blindly, see `promptTurnWithRetry`'s non-idempotency doc +above — a failing evaluator call risks nothing by being retried or, failing +that, simply skipped for one turn: the worker keeps working either way, and +the only thing a bad verdict can do wrong is delay noticing completion, not +corrupt anything. + +### The fix: in-boundary retry, then advisory failure, then a bounded terminal + +`evaluateGoal` now rides out a failure in-boundary before the boundary ever +counts as "failed," mirroring `promptTurnWithRetry`'s own error +classification: + +- A provider error classified `provider.AsRetryable` gets the SAME + retryable schedule and budget the worker turn uses + (`goalRetryableBackoff`, `goalRetryableMaxAttempts`) — the two paths + share provider weather, so they share a budget's shape, each keeping its + own counter. +- A non-retryable provider error is not retried in-boundary at all (the + call is cheap; a permanently broken provider needs the boundary-failure + path below, not a wasted second attempt). +- An unparseable reply still gets its original one extra attempt, but that + attempt now uses a STRICTER system prompt (`goalEvaluatorStrictSystem`) + instead of repeating the same instructions verbatim to a model that + already failed to follow them once. + +If `evaluateGoal` still errors after all that — the retryable budget +exhausted, a non-retryable error, or two unparseable replies even with the +stricter re-ask — the boundary "fails," but failing a boundary no longer +clears the goal. `PursueGoal` journals a durable `goal.eval_failed` record +(carrying the error and the CONSECUTIVE failure count), substitutes a fixed +evaluation-unavailable notice for the next turn's guidance — never the raw +error text, and never a stale NOT-MET reason from turns ago — waits a short +backoff (`goalRetryDelay`, keyed on the consecutive count), and `continue`s: +the worker gets another ordinary turn. A later boundary that DOES parse a +verdict (MET or NOT MET) resets the consecutive count to zero — the horizon +below is about a STREAK, not a lifetime total, so one good evaluation undoes +any number of prior bad ones. + +### The horizon: a bounded, loud terminal + +Infinite advisory failures would just be Round 3's zombie-goal risk wearing +a disguise (a goal that LOOKS active but whose evaluator has been dead for +hours, silently). After `goalEvalFailureLimit` (5) CONSECUTIVE failed +boundaries, `PursueGoal` clears the goal with a dedicated reason ("goal +evaluator failed at N consecutive turn boundaries") and returns a distinct +sentinel error type (`*goalEvaluatorExhaustedError`, recognized via +`IsGoalEvaluatorExhausted`) instead of a bare error — a caller can tell this +terminal apart from an ordinary worker-turn exhaustion via `errors.As`, +never by string-matching `GoalReason`. Unlike every advisory boundary below +the horizon, this terminal DOES emit `session.error`: it must be LOUD, since +past this point nothing else will ever explain the goal's silence. The +server (`server/journal.go`) maps it to a dedicated `turn.end` outcome, +`outcomeEvaluatorExhausted` ("evaluator_exhausted"), a sibling of +`outcomeContextExhausted`/`outcomeMaxTurnsExceeded` — consumers never have +to string-match `GoalReason` to distinguish it. + +`GoalSummary`/`GET /session` surface the current consecutive count as +`eval_failures` (omitted at zero), reset on any `goal.eval` / +`goal.achieved` / `goal.cleared` / `goal.updated` record, exactly mirroring +how `attempt`/`retryable`/`waiting` already reset on the worker-retry path. +`goal.eval_failed`, like `goal.stalled`, is a pure trace record on resume — +`LoadSession`'s fold does not change resume state from it, only the +in-memory consecutive counter used while the loop is live. + +Five is deliberately much smaller than `goalRetryableMaxAttempts` (12): by +the time a boundary counts as "failed" at all, the in-boundary retryable +budget has already ridden out one boundary's worth of ordinary provider +weather, so five separate TURNS of failure (each potentially minutes apart, +each after its own full worker turn) is a much stronger signal of a truly +broken evaluator than exhausting a single boundary's in-boundary retry +budget ever is. + +See `engine/goal.go`'s package doc ("Round 6") for the full narrative, +`TestPursueGoalEvaluatorUnparseableTwiceIsAdvisory`, +`TestPursueGoalEvaluatorRetryableErrorRecoversWithinBoundary`, and +`TestPursueGoalEvaluatorTerminalAfterConsecutiveFailureLimit` (all +`testing/synctest`-bubbled) for the tests, and +`server/goal_eval_resilience_test.go` for the server-side outcome/journal +coverage. + +## Worker-turn exit-park and activity-driven resume (Round 7) + +### Incident + +OpenRouter returned HTTP 404s for a worker turn — a genuinely non-retryable, +non-overload failure, so `provider.AsRetryable` correctly classified it as +the fast, 3-attempt/~5s deterministic budget (`goalWorkerRetries`), not the +long retryable one. That budget exhausted in seconds; the goal cleared +(`goal.cleared`), `session.error` fired, and the box then sat idle for +**hours** with nothing further ever explaining or resuming it — a human had +to notice the silence and manually re-`POST /goal`, the exact zombie-adjacent +failure mode "The fix" above was originally meant to close, just reached from +the "successfully explained, then abandoned" side instead of the "silently +zombied" side. + +The retryable tier's own #61 fix (see "Self-re-arm: park, don't die" above) +was too passive in the opposite direction: it never actually left +`PursueGoal`, so the run slot stayed pinned to the parked loop for the +**entire** outage — a prompt queued during that outage (see +`docs/session-storage-and-queue.md`) could only ever be injected mid-turn into a doomed +worker attempt, never dispatched as its own ordinary turn the way it would +against any other idle session. And every parked cycle re-spent a fresh +`goalWorkerRetries`-shaped schedule internally, with no cross-cycle memory of +how long the outage had already run. + +### The fix: exit-park both tiers, resume on activity + +Every way a worker turn can exhaust its retry budget — the deterministic +tier, the retryable tier, or the non-idempotency gate stopping retries early +once a tool has already executed this attempt (see "Retries are not +idempotent" above) — now returns OUT of `PursueGoal` entirely instead of +either clearing the goal or looping in place. The loop journals a durable, +generation-gated `goal.parked` record (gated exactly like `goal.stalled`/ +`goal.eval_failed` above — a park racing a concurrent `UpdateGoal` is +silently discarded, never attributed to a condition that is no longer +current) and returns a distinct sentinel, `*goalWorkerParkedError` +(`engine.IsGoalWorkerParked`), **without** ever calling `clearGoal` — +`goalActive` stays true, `ActiveGoal()` keeps reporting the same condition, +and `LoadSession` folds `goal.parked` as a pure trace record, exactly like +`goal.stalled`. Unlike `goal.stalled`/`goal.eval_failed`, whose `Reason` +carries the raw `err.Error()` text, `goal.parked`'s `Reason` is deliberately +CLASSIFIED (`classifyGoalWorkerError`) — never a provider's raw error text — +because a park is a durable, potentially long-lived terminal that an +operator-facing surface can read long after the triggering request and its +raw provider detail are gone, unlike the two per-attempt trace records that +are read close to the moment they were written. + +Freeing the run slot this way is what closes the #61 retryable-tier gap +above: a queued prompt dispatches as a normal turn the instant the slot is +free, and the server's **pre-existing** activity-driven auto-arm +(`maybeAutoArmGoal`, upstream of `engine` — see +`docs/session-storage-and-queue.md`) re-enters the loop with a fresh `PursueGoal` call the next time any +ordinary prompt turn completes — no new timer, no new resume machinery, the +same mechanism an ordinary idle goal already relies on. `runGoal`'s own tail +deliberately never auto-arms itself (the pre-existing anti-churn property +documented at `server/handlers.go`'s `maybeAutoArmGoal` — a park does not +immediately respawn a loop against an empty queue). + +On the server side, a worker-parked sentinel maps to `session.error` plus a +distinct `turn.end outcome=worker_parked` (`server/journal.go`'s +`turnEndOutcome`), and `goalTracker` folds the `goal.parked` record into a +third `paused` presentation arm — `pause_reason: "worker_failure"` — sitting +between the existing `"restart"` (boot-time, no loop was ever attached) and +`"provider-backoff"` (the loop IS alive, merely waiting) arms in +`pauseView`'s precedence. `compositeState` forces `idle` for `worker_failure` +exactly like it does for `restart`: no loop is actually driving the goal +until the next auto-arm or an operator re-POST, unlike `provider-backoff`, +which keeps reading `goal-running`. The `worker_failure` presentation resets +everywhere `restart`'s does (`goal.set`/`achieved`/`cleared`/`updated`, +`handleGoal`'s re-arm branch) plus in `maybeAutoArmGoal`'s own successful +arm, so a resumed loop is never seen carrying a stale pause. + +There is deliberately **no streak horizon** on parking, unlike the +evaluator's `goalEvalFailureLimit` (5) bounded terminal above: parking is +immediate at exhaustion, every time, with no cross-park counter. A parked +goal stays parked-and-armed indefinitely until either activity resumes it or +an operator issues `DELETE /session/{id}/goal` — the only clear path a +parked goal has. + +### An ambient, model-facing signal + +The durable `goal.parked` record and the server's boot-only `goal.paused` +presentation both explain a park to an OPERATOR looking at the session from +outside. Neither says anything to the MODEL itself: an agent prompted +mid-outage — a queued prompt dispatching once the exit-park frees the run +slot, or any other ordinary turn — would otherwise see nothing indicating a +supervising goal exists, is still armed, and will resume on its own. +`engine/goal_parked_status.go` closes that gap the same way `mcp_status.go` +and `process.go` already do for their own degraded/live states: a small +ambient text block — a third occupant alongside the process and MCP +segments — computed fresh from live `Session` state +(`goalParked`/`goalParkedReason`/`goalParkedAttempts`) and appended only to +the newest user message of a request that is NOT itself one of this loop's +own worker turns (`PursueGoal`'s `clearGoalParkedAtEntry` call makes that +structural: the flag is always false again before this loop's very first +worker turn of a resumed run). The text is deliberately CLASSIFIED, matching +the record's own leak rule — never the raw provider error. + +This signal is **not persisted** and does not survive a process restart — +`LoadSession` never restores it. That is a real, accepted asymmetry: after a +restart mid-park, a fresh `Prompt` call sees no ambient block at all. +Visibility in that case comes from a different surface entirely — the +server's boot-only `goal.paused` presentation (`pause_reason: "restart"`), +which is operator-facing, not model-facing, and reads the durable +`goal.parked` trace record directly rather than this runtime-only field. + +### The asymmetry: context overflow still clears + +Context overflow (issue #62) is the one deliberate exception that keeps +clearing exactly as before this round. Every other worker-turn exhaustion +this round covers is a failure that MIGHT resolve if the loop simply waits +and tries again later (a dead provider that gets fixed, an outage that ends, +an operator intervening) — parking is a bet that time helps. Context +overflow can never resolve by waiting: the same, now-too-long request fails +identically on every future attempt no matter how long the goal sits parked, +so parking it would just be a slower-burning zombie, not a fix. Clearing +immediately, with a reason a human or automation can act on right away +(compact, shorten the goal, start over), is strictly more honest than a park +that can never self-resolve. + +See `engine/goal.go`'s package doc ("Round 7") for the full narrative and +state diagram, `TestPursueGoalWorkerFailsPermanentlyParksGoal`, +`TestPursueGoalRetryableBudgetExhaustedParksInsteadOfClearing`, and +`TestPursueGoalStaleWorkerFailureDiscarded` (`engine/goal_update_test.go`, +proving a park never lands for a stale generation) for the engine-side +tests; `server/goal_worker_park_test.go` +(`TestTurnEndOutcomeWorkerParked`, `TestForcesIdlePauseIncludesWorkerFailure`, +`TestGoalTrackerPauseViewPrecedence`, `TestGoalWorkerParkFreesRunSlotForQueuedPrompt`, +`TestGoalWorkerParkResumesOnNextPromptActivity`, +`TestGoalWorkerParkPauseSurvivesRestartAsRestartReason`) for the server-side +outcome/pause/resume coverage; and `engine/goal_parked_status_test.go` for the +ambient-segment tests. + +## A third tier: stream truncation (2026-08-06 incident) + +### Incident + +A gateway in front of one provider route was found to cut response streams +at a fixed ceiling well under two minutes: the connection closed with the +response still incomplete after a handful of chunks, HTTP status already a +clean 200, and no inline provider error event — nothing structured to +classify the failure from at all. The resulting error was a bare `io.EOF`, +which `provider.AsRetryable` correctly reported as NOT retryable (there was +nothing to mark it with), so it took the fast, `goalWorkerRetries`-shaped +deterministic path and parked in seconds. A prompt re-issued minutes later +on the same model succeeded cleanly — the cut was the gateway's own +per-response ceiling, not a dead or overloaded provider, so the fast park +was premature in exactly the same way the original `overloaded_error` +incident above was, just from an entirely different cause. + +Two problems, in the same shape as the #61 incident above: the truncation +wasn't classified as anything retryable at all (so it fell into the +deterministic bucket instead), and even if it had been, the 12-attempt/ +~30-minute weather schedule (`goalRetryableMaxAttempts`) is the wrong shape +for it regardless — waiting longer never raises a gateway's fixed stream +ceiling, and every retry re-prompts a full turn at full input cost, so a +long weather-style budget just burns tokens and wall-clock time waiting for +something that will never change. + +### The fix: a dedicated class, an idle-stream watchdog, and a short-schedule tier + +**Classification without a wire signal.** Every provider adapter now marks a +stream-read error that occurs *before* its terminal event was ever seen +(`provider.MarkStreamTruncated`, `provider/retryable.go`) as +`provider.RetryableStreamTruncated` — a fourth `RetryableClass` alongside +`overloaded`/`rate_limited`/`server_error`, and the one member of the family +with no structured provider response behind it: the bare transport error +(typically `io.EOF`, or a "connection reset" net error) is wrapped as-is, +with a message naming what actually happened. A context cancellation or +deadline is left unmarked — that is the caller's own abort (`POST /abort`, +shutdown, the watchdog's own parent deadline — see below), not provider +weather, so wrapping it would misclassify a deliberate stop as a transient +failure. + +**An idle-stream watchdog gives a silent-forever stream the same identity.** +Before this round, a stream that went completely silent — no bytes, no +`EventDone`, no error, ever — was unbounded: nothing in the engine or the +adapters would ever cut it, and the turn (and any goal loop driving it) +wedged forever, holding the run slot. `engine/stream_watchdog.go`'s +per-request watchdog (`Config.StreamIdleTimeout`, config key +`stream_idle_timeout_s`, default 5 minutes mirroring Codex's +`stream_idle_timeout_ms`, a negative value disabling it) resets on every +stream event and, on expiry, cancels the request's own child context and +converts the resulting cancellation into the same `RetryableStreamTruncated` +classification — deliberately never chained to `context.Canceled` itself, so +a retry loop's `errors.Is(err, context.Canceled)` abort check still means "a +real caller abort," never "the watchdog fired." It guards the worker turn, +the goal evaluator, and the compaction summarizer's streams identically +(`armIdleWatchdog` wraps all three). + +**A third, short-schedule tier — neither deterministic nor weather.** +`promptTurnWithRetry` (and `runEvaluatorWithRetry`, its evaluator-side +counterpart) gives a `RetryableStreamTruncated` failure its own +`goalStreamTruncatedMaxAttempts` (3) budget on the SAME short backoff +schedule (`goalRetryDelay`) the deterministic tier uses (~5s total): it +never spends the deterministic budget (the failure IS classified retryable), +and it never rides the long weather-tier schedule (waiting longer cannot fix +a fixed ceiling, and unlike a cheap evaluator poll every worker attempt is a +full-price re-prompt). Exhausting this tier parks exactly like the other +two — `goal.parked`'s `retryable_class` field reads `stream_truncated`, and +every `goal.stalled` record along the way carries the same class — so an +operator or a log reader can tell "waiting out a gateway ceiling" apart from +"waiting out an overload wave" apart from "a dead deterministic failure" +without ever decoding free text. + +See `engine/goal.go`'s `goalStreamTruncatedMaxAttempts` doc comment, +`provider/retryable.go`'s `RetryableStreamTruncated`/`MarkStreamTruncated`, +and `engine/stream_watchdog_test.go` for the watchdog's own coverage. + +## Operational reliability + +Goal-supervised turns are retried by the loop above and fail visibly with a +journaled reason (`goal.stalled`, then `goal.parked` — never `goal.cleared` +— if every retry budget, deterministic, retryable-weather, or +stream-truncated, is exhausted; see "Worker-turn exit-park and +activity-driven resume" above. Context overflow remains the one exception +that still clears). Plain `prompt_async` turns get +none of that: they are not retried, and a provider stream that dies mid-turn +silently ends them. The +signature of that silent death is a final assistant message containing +reasoning parts only — no text, no tool_call. Consequently, multi-step or +long-running work dispatched over an unreliable link should be wrapped in a +goal even when no evaluation condition is actually interesting, just for the +retry/visibility behavior; and anything polling a plain prompt must treat an +idle session whose last assistant message is reasoning-only as a failure to +investigate, not as completion. diff --git a/docs/mcp-tool-loading.md b/docs/mcp-tool-loading.md new file mode 100644 index 00000000..812fe7bb --- /dev/null +++ b/docs/mcp-tool-loading.md @@ -0,0 +1,142 @@ +# MCP tool loading + +This document describes deferred MCP schemas and deterministic tool ordering. + +## Lazy MCP tools (deferred schemas) + +An MCP server's tools reach the model as full JSON Schemas in the tools +array. A box that wires several large servers therefore pays for hundreds +of schemas on every turn, at the FRONT of the cached prefix. The MCP +CONNECTION was already lazy; the schema cost was not. + +`engine/mcp_lazy.go` defers that cost, opt-in. A DEFERRED server's tools +leave the tools array and appear instead as a name-only catalog — one +`name — one-line description` line each — in a system segment placed after +the Agent Skills catalog and before hook (`system.transform`) segments. It +is the same progressive-disclosure staging Agent Skills already use. The +model loads a schema with the `mcp` tool's `select` action, and the loaded +def is back in the tools array on the next request, so a selected tool is +called exactly like a statically registered one. `runAgenticLoop` rebuilds +the request per tool round, so a `select` takes effect inside the same +turn. + +Config: `mcp_tool_loading` is `eager` (the default, and today's behaviour +byte for byte), `auto` (defer once the live catalog exceeds +`mcp_tool_loading_threshold`, default 20 tools), or `lazy` (always). +`mcp_servers..tool_loading` pins one server `eager` or `lazy`; +`auto` is global-only, because the threshold measures whole-catalog +pressure. A zero threshold resolves to the default. User/project config +rejects a negative threshold; a direct engine embedder that supplies a +non-positive `Engine.Config` value also receives the default, never a floor +of 1 — that value would defer every catalog. + +Four rules are load-bearing. Do not relax them: + +- **A session that does not hold the `mcp` tool defers nothing.** Never + defer what the session cannot select. A subagent restricted by an agent + definition that omits `"mcp"` (`restrictTools`) would otherwise lose + every MCP schema AND the only path to load one back. +- **The `auto` threshold counts the WHOLE catalog**, including a server + pinned `eager`. A pin says "always keep these loaded", never "ignore + their cost". +- **The catalog listing sorts by full tool name, in the engine**, not by + the registry's server-then-tool slice order (the two differ for servers + `a` and `a0`). The tools array stays byte-stable because the partition + preserves the registry's order and changes only when a selection does. +- **`streamTurn` resolves the provider BEFORE it computes the tool plan.** + The plan's `Tools(ctx)` call is what dials a server for the first time + and spawns a child process for every stdio server. A turn naming an + unconfigured provider must return before any of that. The plan still + runs before `mcpStatusSegment`, which is the pre-existing rule that a + first-attempt failure is reported in its own turn. + +The same reorder moved one hook. `chat.params` still runs first and still +fires on every turn. `system.transform` now runs AFTER provider +resolution, so a turn naming an unconfigured provider returns without +firing it — it used to fire, then fail. Building a system prompt for a +request that is never sent buys nothing, and a plugin that counts +`system.transform` calls now counts sent requests. `chat.params` is +unaffected because provider resolution needs the model it returns. + +A stale selection is reaped at plan time: a selected name whose server is +CONNECTED and whose catalog lacks it is dropped. That is what keeps an +invented name — accepted while a server was unconnected, where a real name +and an invented one are indistinguishable — out of the effective set. A +selection whose server is still unconnected is KEPT, so it arms itself on +reconnect. The reap is memory-only; replay re-unions the log and prunes +again. + +The `mcp` session tool carries two extra actions when the session can +defer, and only then — a session that defers nothing must not advertise an +action with nothing to act on. `search(query)` ranks the live catalog by +keyword: substring matching over lowercased text, scored once per DISTINCT +query token per field (remote name 50, description 10, server name 5, plus +100 once when the whole query equals a name), sorted by score then name. +Tokens split on Unicode letter/digit classes, never the ASCII ranges — an +ASCII split truncates `café` to `caf` and reduces a CJK query to nothing. A +blank query errors rather than dumping the catalog. Both actions are +refused at DISPATCH, not only omitted from the advertised enum, on a +session that can defer nothing. `select(tools)` loads +schemas, and every name lands in exactly one bucket, tested TOP TO BOTTOM: +`already`, `selected`, `pending` (its server is configured but not +connected — it arms on reconnect), `missing` (no connected server holds it, +or the name is malformed). Its `note` is conditional on that outcome: a +`pending`-only batch must not claim its tools are callable next request. +`select` returns NO schemas: the tools array is the one authoritative copy, +and echoing them would write every schema a second time into durable +history. + +**Use implies selection.** An MCP tool call that ROUTES records its own +name. Without it, a tool of an eager server — which needs no `select`, and +which the model is told not to select — would lose its schema the moment an +`auto` flip deferred its server mid-task. The gate is per SERVER, not per +session: a server pinned `eager` can never flip, so a record for its tools +could never pay for itself, even in a session that defers a different +server. A plain `eager` config therefore records nothing at all. + +**Both writers of the record apply that same gate.** `select` records a +name only when its server could ever defer, exactly as a routed call does. +A record exists only to survive a flip, so "can this server ever flip" has +one answer whichever writer asks. A pinned-`eager` server's tool is still +reported `selected` — it is loaded and callable — and simply records +nothing. + +A selection is durable. `mcp.tools_selected` (`recMCPToolsSelected`, +`engine/store.go`) records the names that ENTER the set, and `LoadSession` +unions every record back. It follows `recToolResultRetained`: +engine-internal state, journaled and folded, with no engine event and no +server journal mapping. **Two writers produce it** — `select`, and a routed +MCP call through use-implies-selection. Wiring only the first silently +loses a tool the model used but never selected. + +Recovery degrades in one direction. A restored name whose server is absent +or parked is KEPT, so it arms on reconnect. One whose server connects +WITHOUT it is reaped. A malformed name is skipped on replay, exactly as +`select` refuses to record one — one rule at both ends of the record's +life. + +Full design, including the durable record: `docs/design/mcp-lazy-tools.md`. + +## The tool array is byte-stable across requests + +`Session.toolDefs` (`engine/engine.go`) sorts the BUILT-IN tool group by +name. That sort is a prompt-cache requirement, not cosmetics. `Session.tools` +is a map, Go randomizes map iteration on every range, and tools sit at the +FRONT of the cached prefix on every provider — Anthropic caches tools, then +system, then messages. An unsorted build therefore emitted a different tools +array on every request and invalidated the WHOLE prefix each turn, which no +TTL can help. + +The defect is invisible to a unit test that checks the tool SET, and it +appears only in live traffic: consecutive turns of one session each report a +full cache write and no cache read, for a byte-identical system prompt. A +new test must therefore assert the byte-stability of the array, not its +membership. The commit that introduced the sort carries the measured +before/after evidence. + +Group order stays built-ins, then MCP, then plugins. The other two groups were +already deterministic — `MCPManager.rebuildToolsLocked` sorts by server then +tool, and `plugin.Host.Tools` walks the configured instance slice — so the +sort applies WITHIN the built-in group only. Adding an MCP server must never +reshuffle the built-in block ahead of it. Any new tool source must be +deterministic before it joins this list. diff --git a/docs/models-and-providers.md b/docs/models-and-providers.md new file mode 100644 index 00000000..b660150f --- /dev/null +++ b/docs/models-and-providers.md @@ -0,0 +1,444 @@ +# Models and providers + +This document describes model switching, context windows, reasoning effort, +cache affinity, and provider configuration. + +## Model switching + +`Session.SetModel` swaps the MAIN session model for later requests. History +transcodes from scratch every request, so there is no migration step. Three +routes reach `SetModel`: the built-in `model` session tool, a per-request +`prompt_async` model override, and `POST /session/{id}/model`. + +`SetModel` is the single event choke point. On a real change (never a no-op +set to the current model) it persists the durable `recModel` resume record +AND emits `EventModelChanged` (carrying the new model), both while holding +`s.mu` — the same persist-and-emit-under-`s.mu` shape `RegisterGoal` uses. +The server's `Publish` maps `EventModelChanged` to the durable `model` +journal record. Every swap route funnels through this ONE emit, so a swap +journals exactly once — the handlers never emit `model` themselves. `recModel` +is the resume record `LoadSession` restores; `EventModelChanged` is the +observability event. They are separate and both fire on one swap. + +The `model` session tool (gated on `Config.ModelTool`) has two actions: +`status` reports the current model, the configured aliases, and the configured +provider names; `set {model}` resolves a one-level alias (from +`Config.ModelAliases`, which mirrors `config.Aliases` — the engine never +imports config), parses the ref, VALIDATES the provider is configured +(`s.cfg.Providers.For`), then calls `SetModel`. A `set` to an unconfigured +provider returns a tool error listing the valid aliases and provider names and +changes nothing. There is deliberately NO `clear` action — a session always +has a model. Scope is the MAIN model only; the goal-evaluator and subagent +models are untouched. + +`Config.ModelTool` is on by default. Config key `model_tool` (a `*bool`, +default true — like `instructions`) lets a host opt OUT; `harness run`, +`harness serve`, and the server `mkCfg` all set it from +`config.ModelToolEnabled()`. This differs from `GoalTool`, which opts IN only +when an evaluator is configured. + +`POST /session/{id}/model` is the network counterpart: a client/dashboard swap +decoupled from prompting, so it never claims the run slot (it applies even +while a turn is running, taking effect on the next request). It validates a +non-empty `{model}` and rejects an unconfigured provider (400), an unknown +session (404), or an empty model (400) — the same validation as the tool — then +calls `SetModel`. Aliases are not resolved at this endpoint; resolve them +client-side, as the CLI does. + +## An unknown model's context window is a refusal, not a shrug + +`modelmeta.ContextWindow` answers "how big is this model's context window". +When it does not recognize a ref, `resolveContextWindow` +(`engine/context_window.go`) used to fold that into the SAME answer a +deliberate opt-out produces: source `disabled`, `compaction_armed=false`, and +a session that started anyway and ran with **no context management at all** — +until it died with "context exhausted" instead of compacting. An unrecognized +model is not a state to degrade into; it is a configuration an operator has to +fix. + +`resolveContextWindow` now REPORTS the miss (an error wrapping +`ErrUnknownContextWindow`, whose text always names the offending ref) instead +of swallowing it, and does not decide what to do about it. +`Config.RequireContextWindow` does, through the single policy point +`requiredContextWindowErr` — so the definition of a miss lives in one place +and the policy lives with the session that must honor it, and the ERROR log +line fires once per miss however it was reached. + +Only a REGISTRY MISS refuses. Four ways to have no window stay legitimate and +silent: an explicit positive `ContextWindowTokens` (naming the window IS the +missing information, so it satisfies the requirement for any model), an +explicit NEGATIVE one (`contextWindowSourceOptOut` — a stated choice, told +apart from `disabled` precisely because `disabled` can mean "unrecognized"), a +ZERO model ref (nothing to look up; the refusal belongs to whatever later +names a model), and a model the registry KNOWS whose window is below +`minAutoContextWindowTokens` (a known model, not a gap). + +The refusal is recorded at the earliest point of use and surfaced everywhere a +model starts being used: `newSession`, `SetModel`, and `LoadSession`'s +post-replay re-derive set `Session.contextWindowErr`; `ContextWindowErr()` lets +a create route refuse before the session is durable or resident; `CheckModel` +is `ModelSupported`'s sibling gate, called at the same three `SetModel` routes +BEFORE the swap so a rejected ref never reaches the durable `recModel` record; +and every `Prompt` returns it before touching history, the provider, or the +instructions read. A RESUME is deliberately not fatal: a session that cannot +load cannot be listed, read, or exported either, and an operator would lose the +transcript along with the ability to fix the config. + +`Config.RequireContextWindow` false — the engine zero value — keeps the +pre-fix behavior, so a bare embedder-built `engine.Config` and every test in +the package are unaffected; the config/CLI layer supplies the product default +of TRUE (`context_window_required`), the same unset-versus-explicit split +`prompt_retries` uses. + +## Reasoning effort + +`message.Effort` is the unified, provider-agnostic reasoning-effort level: +`off`, `minimal`, `low`, `medium`, `high`, plus the zero value `EffortUnset` +(empty string) that sends NO control at all. It rides one `provider.Request` +field (`Request.Effort`), the same way `MaxTokens` does, and each adapter maps +it to that provider's own wire shape at transcode time — so an effort swap, +like a model swap, needs no migration step: + +- `provider/anthropic` enables extended thinking with a `thinking.budget_tokens` + budget (minimal 1024, low 4096, medium 8192, high 16384). The API requires + `max_tokens > budget_tokens` and rejects an explicit `temperature`/`top_p` + while thinking is on, so `transcodeRequest` bumps `max_tokens` above the + budget and drops both. `off`/unset emit no `thinking` block. +- `provider/openai` (Responses) sets `reasoning.effort` to the level string + (minimal/low/medium/high). `off`/unset omit the `reasoning` object. +- `provider/openaicompat` sets the top-level `reasoning_effort` string; a + gateway (Bifrost) maps it to the upstream provider's own knob. A non-off + level sends the level string; `EffortOff` sends the literal string `"off"`, + not an omitted field — several gateway upstreams reason BY DEFAULT when + the field is absent, so omitting it cannot express "disabled." Measured + (2026-08-12): Fireworks kimi-k3 through Bifrost streamed a full reasoning + block (266 chars) with the field absent, and zero reasoning content (0 + chars, 8 vs 133 completion tokens) with the literal `"off"` sent. Only + `EffortUnset` omits the field, leaving the gateway/model default in force. + It surfaces returned reasoning from EITHER wire field — Bifrost/DeepSeek + `reasoning_content` or OpenRouter `reasoning` — as a `Reasoning` part; a + gateway sends one field, never both. + +`Effort` does NOT police which model accepts which level — that is a +provider-and-model fact the engine cannot know from the ref alone. The adapter +sends the requested level and the provider is the final judge. A caller that +must gate levels per model (a dashboard picker) holds its OWN mapping. + +**Downgrade strip — DELIBERATELY asymmetric between the two reasoning +adapters.** A stored thinking block (anthropic) or reasoning item (openai +Responses) can be a transcode-time destructive drop (throwaway request, intact +record); a later reasoning-ON turn replays the part from the unchanged history. +A strip is ever needed because a stored block shipped while the request omits +the reasoning control can be rejected, and durable in history it 400s every +later turn — a permanent wedge. But WHEN each adapter strips differs, because +the two providers default differently: + +- `provider/anthropic` strips whenever the request enables no reasoning + (`off`/unset, or a swap to a non-reasoning model). This is safe: Claude emits + NO thinking block unless the control is sent, so an unset turn carries none + to preserve. +- `provider/openai` (Responses) strips ONLY on an EXPLICIT `off` (a genuine + "reasoning disabled" intent), NEVER on `EffortUnset`. OpenAI reasoning models + (gpt-5) reason BY DEFAULT, so an unset turn — the default of every `harness + run`/`serve` session, since nothing sets `Config.Effort` — still produces + encrypted reasoning items, and those items are REQUIRED for stateless + (`Store:false`) multi-turn tool use. Stripping them on unset wedged every + turn-2+ gpt-5 tool continuation; an unset session now replays them exactly as + every pre-effort-control build did (`stripReasoning` in + `provider/openai/transcode.go`, gated on `req.Effort == EffortOff`). So + `unset != off` here — do NOT re-fold the openai strip back onto + `!Reasoning()`. (Regression: NEP-5272 review of PR #117.) One residual the + off-only strip cannot enforce: a `SetModel` swap to a NON-reasoning openai + model (gpt-5 -> gpt-4o) at unset effort still replays the stored items — the + same per-model gating punt the enable direction has, so the caller (a + dashboard picker) clears/re-validates effort on a model swap, NOT this + transcoder. + +The reverse (ENABLE) direction — turning reasoning ON over a prior tool_use +that lacks a thinking block — stays a documented limitation, since a signed +thinking block cannot be synthesized (see `provider/anthropic/ +transcode.go`). + +`Session.SetEffort` is the single event choke point, mirroring `SetModel` +exactly. On a real change (never a no-op set to the current level) it persists +the durable `recEffort` resume record AND emits `EventEffortChanged`, both under +`s.mu`. The server's `Publish` maps `EventEffortChanged` to the durable `effort` +journal record. That record ALWAYS carries the `effort` field, even on a clear: +`server/journal.go`'s `Event.Effort` is a `*message.Effort` (the same +explicit-zero-vs-absent pattern `QueueLen` uses), so a clear to `EffortUnset` +renders as an explicit `"effort":""`, never a dropped key — "cleared to the +provider default" stays byte-distinguishable from a malformed record. +`LoadSession` restores the level: the create-time level rides +the session header record, and every later `SetEffort` writes a `recEffort` +record. `Session.Effort()` reads it back. + +`POST /session/{id}/thinking` is the network counterpart: a client/dashboard +swap decoupled from prompting, so it never claims the run slot. It validates the +`{effort}` value with `message.ParseEffort` (400 on an unknown level), accepts +an empty string as "clear to provider default", and rejects an unknown session +(404). Unlike the model endpoint it has NO provider gate (see above). The +current level is read back on `GET /session/{id}` (`effort`), the same way the +current model is. + +**Effort at the three request-build sites is NOT uniform, by design.** The +main turn (`streamTurn`, `engine/engine.go`) sends `s.Effort()` — the +session's current level, read fresh every request. The two internal +tool-less calls diverge from that and from each other (issue #124): the +goal-loop evaluator (`runEvaluator`, `engine/goal.go`) always pins +`EffortOff` — see `docs/goal-loop.md` — because it is a classifier the model +must answer in one line, and reasoning-by-default gateway models can burn +its 256-token budget before ever emitting a verdict. The compaction +summarizer (`runCompactionSummary`, `engine/compact.go`) instead inherits +`s.Effort()`, the same as the main turn, because summarization is a real +writing task that benefits from the session's own quality setting; +`EffortUnset` stays `EffortUnset` there. Do not fold these two internal +sites onto one shared rule — one is a classifier, the other is prose. +Known residual (not addressed by issue #124, filed as issue #126): a +non-off session effort can raise the summarizer's effective output cap +above `compactionMaxTokens` (the anthropic and openai adapters both bump +the cap for reasoning — up to ~20480 tokens at `EffortHigh`, versus the +documented 1024 cap), and openaicompat applies no cap floor at all, so a +reasoning-heavy summary can truncate silently — `runCompactionSummary` has +no `StopReason` guard to catch it. A raised cap also delivers less context +reduction from this call, at the layer whose own failure runs to a hard +overflow that clears an active goal. A second, related residual (issue +#127): the summarizer sends folded history containing `ToolCall` parts +from turns that ran with no thinking block, and a non-off level here +enables thinking over that same history — the documented ENABLE-direction +"thinking blocks expected before tool_use" reject case, just reached from +compaction instead of a live turn. + +**The summarization request always ends in a trailing `RoleUser` message, +never the folded range's own last message verbatim** (2026-08-19 incident, +session `ses_jumpy-pizza`). `foldEnd` (`Session.Compact`) is the last +message before the next KEPT turn's leading `RoleUser` message — ordinarily +that folded turn's own final assistant reply, `RoleAssistant` — so sending +`folded` as `req.Messages` verbatim ordinarily ends the wire request in an +assistant-role message, which the Anthropic Messages API treats as +assistant message prefill; some models reject prefill outright (400 +`invalid_request_error`, "This model does not support assistant message +prefill. The conversation must end with a user message."). `runCompactionSummary` +builds its request via `compactionRequestMessages`, which appends one +trailing `RoleUser` instruction message (`compactionInstructionText`) after +`folded`, unconditionally — never a conditional check on the folded range's +last role, since a `RoleTool` message (a `message.ResolveOrphanToolCalls` +synthetic repair, or an ordinary tool result) also wire-transcodes to +Anthropic's `"user"` role and would otherwise mask the same bug depending on +where a fold boundary happens to land, exactly as it did live (`keep_turns=8` +happened to succeed on the same session where `keep_turns=20` failed). + +**An empty summary is a graceful no-op, never an error surfaced to the +caller.** A summarization call that completes without a transport/stream +error but returns no usable text (`errEmptyCompactionSummary`) is reported +by `Session.Compact` as the same `TurnsFolded == 0` "nothing worth folding" +shape the too-few-turns case above already uses — no history mutation, no +journal write, no error returned — though `EventCompactionFailed` still +fires so the attempt stays visible to anything tailing events, and the +call's real usage is still accumulated into cumulative `Usage()` (it was a +billed call even though it produced nothing — this accumulation is +live-only, not journaled, since no compact record exists for a skipped +fold). Before ever calling the provider, `Compact` also skips a fold range +whose entire content is a single earlier compaction's own summary message +(`isLoneExistingSummary`): re-summarizing an already-compressed summary with +nothing new alongside it has nothing to gain, and was the live incident's +concrete trigger (a small `keep_turns` landed a fold range dominated by a +prior summary). Do not conflate this with a REAL summarization failure +(rate limit, transient 5xx, a truncated stream, a range too large to +summarize) — those still abort with an error, per §2 "Failure handling" in +`docs/design/context-compaction.md`. + +`CompactResult.SkipReason` names WHICH of the three `TurnsFolded == 0` +shapes occurred (`SkipReasonNotEnoughTurns`, `SkipReasonLoneExistingSummary`, +`SkipReasonSummarizerEmpty`) — they used to be wire-identical, which hid two +real defects (review follow-up on PR #136, Findings A/B/C, fixed before +merge): + +- **Hysteresis must latch on `SkipReasonSummarizerEmpty`, never on the two + free skip reasons.** `maybeAutoCompact` only armed its churn-guard + hysteresis when `TurnsFolded > 0`. A summarizer that always returns empty + therefore never latched it: every subsequent over-threshold turn + re-triggered a full, billed summarization call, indefinitely, at full + input price — the "free" no-op was actually a recurring-spend bug + (Finding A). It now also latches when `SkipReason == + SkipReasonSummarizerEmpty`, since that reason DID cost a call; it must + still NOT latch on `SkipReasonNotEnoughTurns`/`SkipReasonLoneExisting + Summary` — both are free, and latching there would permanently disarm + compaction for an over-threshold session that simply lacks enough turns + yet, since the guard only clears once `LastUsage()` dips back under + threshold. +- **`isLoneExistingSummary` gates on the summary message's `ID`, never on + `CompactionSummaryBanner`'s text.** The banner is a display convention; a + user-typed or pasted message that happens to start with the exact banner + string is a genuine turn with real content, not a lone existing summary — + matching on text alone false-positived on it, skipped it forever without + ever calling the provider, and under the automatic trigger the session + never compacted again (Finding B). Every compaction summary's `ID` is now + minted with the `cmpsum_` prefix (`compactionSummaryIDTag`) instead of the + ordinary `msg_` prefix every other message gets, and `isCompactionSummaryID` + tests exactly that prefix — a structural, unforgeable marker of + compaction origin, the same pattern `message.IsSyntheticOrphanID` already + establishes for a different synthetic-message kind. No text-based + fallback exists for a summary minted by an earlier pre-fix build of this + same PR (still `msg_`-prefixed): the miss is bounded and self-healing — + `Compact` just re-summarizes that one old-style range like any other real + content, and the fresh summary it produces carries the new ID tag from + then on. +- **The `skip_reason` field on `POST /session/{id}/compact`'s response** + (`compactResponseJSON`, `server/handlers.go`) surfaces + `CompactResult.SkipReason` directly, `omitempty` (absent on a real fold) — + see `docs/design/context-compaction.md` §1 for the wire shape (Finding + C). + +## Session affinity (prompt-cache routing hint) + +`provider.Request.SessionKey` carries a stable, opaque session identifier on +every request the engine builds — one field on the same per-request struct +`Effort` rides, though unlike `Effort` (set at one call site), the engine +sets `SessionKey` to `Session.ID` at all three request-build sites: +`streamTurn` (`engine/engine.go`, the main turn), `runEvaluator` +(`engine/goal.go`, the goal-loop evaluator), and `runCompactionSummary` +(`engine/compact.go`, the compaction summarizer). The field itself is never +persisted; the value it carries (`Session.ID`) already is, as the session's +own identity. + +Two adapters forward it, each to its own field, because each provider +documents its own affinity hint: + +- `provider/openaicompat` sets the wire top-level `user` field. This is a + generic chat-completions gateway adapter (fronting Bifrost, OpenRouter, + and similar); `user` is the field a Fireworks-style backend behind such a + gateway reads for routing. OpenAI itself has deprecated `user` on its own + API in favor of `prompt_cache_key`/`safety_identifier` (see the next + bullet), but that deprecation is OpenAI's, not the gateway's: the + openaicompat route keeps sending `user` because `user` is the field the + measured Bifrost/Fireworks path above actually reads. Do not "fix" this + adapter by swapping in `prompt_cache_key` — that field is specific to + OpenAI's own API, and the openaicompat adapter targets non-OpenAI + backends behind a gateway, whose measured path reads `user`. Swapping it + would silently drop the measured cache-affinity win. The adapter now sends + `prompt_cache_key` ALONGSIDE `user`, set from the same `SessionKey`: a + gateway fronts several upstream shapes, and an OpenAI-shaped upstream + behind it reads `prompt_cache_key` while the measured Fireworks path reads + `user`. Both fields carry the identical value, one extra field costs + nothing, and an upstream that knows neither ignores both. Add, never swap + — the rule above still binds. Config key `no_prompt_cache_key` on an + `openai-compat` providers entry suppresses that ONE field for a strict + self-hosted upstream that rejects an unknown top-level parameter; `user` + keeps carrying the session key, so the opt-out never costs the measured + affinity win. It is rejected on any entry that is not `openai-compat` — + the native openai adapter always sends `prompt_cache_key`, its own + documented field. +- `provider/openai` (Responses API) sets the wire top-level + `prompt_cache_key` field — the Responses API's own documented routing/ + cache-affinity hint, distinct from `user`. OpenAI combines it with the + request's prefix hash to raise the chance repeat requests land on the + same cache-holding backend. + +Both follow the same omit-on-empty rule: a non-empty `SessionKey` sets the +field; an empty key omits it entirely, never an empty string. +`provider/anthropic` ignores `SessionKey` — it already uses explicit +`cache_control` markers, so a routing hint would add nothing; a live probe +through Bifrost (2026-08-12) confirmed a 41k-token cache write followed by a +41k-token cache read on the very next turn with no `SessionKey` involved. + +The reason `SessionKey` exists at all is measured, not theoretical: Fireworks +serverless prompt caching is prefix-based, automatic, and PER-REPLICA. +Without a routing hint, a re-sent request can land on a different replica +and miss its own prefix cache. A live probe through Bifrost (2026-08-12) +sent a byte-identical 150k-token prompt twice: with no `user` field, the +second call still read `cached_tokens=0` at 10.8s time-to-first-token; with +a stable `user` field, the second call read `cached_tokens=150,300` at 2.8s +time-to-first-token, through the same gateway. Harness sessions re-send the +whole history every request (stateless transcoding), so a long session on +the openaicompat route (a gateway to Fireworks kimi-k3 and similar models) +pays full prefill on nearly every turn without this hint. + +## Anthropic cache TTL (default 1 hour) + +`provider/anthropic` marks two prompt-cache breakpoints on every request — +the last system block and the last content block of the final message — and +never stores a marker in the session log (`transcodeRequest`, +`provider/anthropic/transcode.go`). The marker's TTL defaults to the +EXTENDED 1-hour cache, not the API's own 5-minute default. + +This is an opt-OUT default, and it changes the wire for an operator who +configures nothing: every anthropic request carries the beta header and +writes 1h entries. Two deployments must know it. A proxy that rejects an +unknown `anthropic-beta` value fails every request, and a workload of short +one-shot sessions pays the 2x incremental write premium with no later turn +to read the entry back. Both set `cache_ttl: "5m"`, which restores the +previous bytes exactly. + +`Client.CacheTTL` selects it: `"5m"`, `"1h"`, or empty for +`DefaultCacheTTL` (`"1h"`). Config key `cache_ttl` on the NATIVE `anthropic` +providers entry sets it, and `cmd/harness`'s `registry` passes it to the +client. The value is validated twice, and both checks fail loudly rather +than fall back: `config.validateCacheTTL` rejects an unknown value, and +rejects `cache_ttl` on any entry that is not the native anthropic adapter — +matching on IDENTITY, the map key `anthropic` with no `type`, never on the +key alone, since an entry keyed `anthropic` but typed `openai-compat` builds +an openaicompat client that would never read the value. `anthropic. +resolveCacheTTL` then rejects an unknown value again at the first `Stream` +call, like a missing API key. A typo must never silently ship different +cache economics. + +Wire shapes, by TTL: + +- `"1h"` sends `cache_control: {"type":"ephemeral","ttl":"1h"}` on both + breakpoints, plus the request header `anthropic-beta: + extended-cache-ttl-2025-04-11`. That header is the documented gate for the + extended TTL. Some endpoints no longer enforce the gate and accept the TTL + without it. Harness sends it regardless, because an endpoint that DOES + enforce it must not fail. +- `"5m"` sends `cache_control: {"type":"ephemeral"}` and NO beta header — + byte-identical to a build with no TTL support at all. This is the escape + hatch for a gateway that rejects an unknown beta. + +The default is 1h because of cost. Cache READS price the same at both TTLs. +A 1h WRITE costs 2x base input where a 5m write costs 1.25x, and that +premium applies only to the INCREMENTAL tokens each turn adds to the prefix. +A 5m expiry on a mature session, by contrast, rewrites the WHOLE prefix — +the entire history, at full input price. One such miss costs more than the +1h write premium over hundreds of turns. Agentic sessions exceed 5 minutes +by construction: one build, one live probe, or one subagent runs longer than +the window, and a user reads an answer before sending the next turn. The +commit that introduced this default carries the measured evidence. + +## A second native Responses provider + +`provider/openai` speaks the OpenAI Responses API. Other vendors speak the +same wire at their own host, under their own request path. Two config +fields let a deployment reach one without new adapter code. + +`Provider.Type` accepts `"openai"` (`config.TypeOpenAI`) under ANY providers +map key. The key becomes the provider family, routed by the first segment of +a `provider/model` ref, exactly like an `"openai-compat"` entry. +`cmd/harness`'s `registerOpenAIProviders` builds one `openai.Client` per such +entry. `base_url` is required: an arbitrary endpoint under a caller-chosen +key has no sensible built-in default, the same rule `"openai-compat"` +follows. The bare `openai` key with an empty type keeps its own built-in +default and is unchanged. + +`Provider.ResponsesPath` (`responses_path`) sets `Client.ResponsesPath`, the +path appended to the base URL. Empty means `/v1/responses`, the path the +Responses API documents and the only path this adapter could reach before. +The field is valid ONLY on an entry that builds this adapter — the native +`openai` key with an empty type, or any key with type `"openai"`. +`config.validateResponsesPath` rejects it elsewhere, matching on IDENTITY +(`buildsResponsesAdapter`) rather than on the map key alone, for the reason +`validateCacheTTL` already documents: the key `openai` with type +`"openai-compat"` builds an openaicompat client that would never read the +value. + +`openai.Client.Family` overrides the family key `Name()` reports AND the +`ProviderData` tag the transcoder reads and the stream writes. Empty means +the package `Family` constant, so every existing caller is unchanged; +`registerOpenAIProviders` sets it to the providers map key. The tag matters +beyond routing. A Responses reasoning item is opaque, usually ENCRYPTED, and +scoped to the endpoint that minted it, and history replays it verbatim on +every later request. One shared `"openai"` tag across two endpoints would +make the canonical family match succeed between them, so a session that +swapped models would replay one endpoint's ciphertext to the other. A +per-client family makes that a cross-family DROP instead — the canonical +crossing rule — which costs one turn of reasoning continuity and nothing +else. diff --git a/docs/plans/2026-07-20-goal-eval-resilience.md b/docs/plans/2026-07-20-goal-eval-resilience.md index f27302e8..31ee0eac 100644 --- a/docs/plans/2026-07-20-goal-eval-resilience.md +++ b/docs/plans/2026-07-20-goal-eval-resilience.md @@ -46,7 +46,7 @@ - Reason pairing: `reason`/`reasonGen` :529-532, 577-584, 755-756; `goalAdjustedNotice` :1265. - Server: `runGoal` error classification server/handlers.go:1426-1458; `turnEndOutcome` server/journal.go:218-228 (add the new outcome alongside `outcomeContextExhausted`); Publish switch :234-258; `publishGoal` :286-362 / `foldGoalRecordLocked` :751-804 (lockstep); `goalTracker` server/server.go:289-308; `goalJSON` server/handlers.go:194-237; openapi Event enum :623-638, GoalSummary :290-368. - Tests changing meaning: `TestPursueGoalUnparseableTwice` (engine/goal_test.go:329), `TestPursueGoalUnparseableTwiceClearsGoal` (:361), `TestClearGoalDuringPendingEvaluatorFailureIsCleanStop` (:914 — its doc comment locks in the old asymmetry; rewrite deliberately). `TestPursueGoalUnparseableThenRecovers` (:414) survives but re-check against the stricter re-ask. -- Doc drift to fix while there: `docs/goal-loop.md:84-92` says evaluator path "unchanged" — stale relative to current code; update alongside AGENTS.md. +- Doc drift fixed during implementation: the former `docs/goal-loop.md:84-92` said the evaluator path was "unchanged." The superseded text and incident sequence now live in `docs/history/goal-loop-resilience.md`. --- @@ -66,7 +66,7 @@ Commit: `feat(server): surface goal.eval_failed and the evaluator_exhausted term ### Task 3: Docs + review + validation -`AGENTS.md` goal-loop section (evaluator resilience paragraph: advisory evaluation, N-boundary horizon, loud terminal); fix `docs/goal-loop.md`'s stale "unchanged" claim; hub check (unknown `goal.eval_failed` event must not break rendering — same three dispatch sites as prior PRs; add `eval_failures` display ONLY if trivially cheap, else skip). Full gates + `node --test tools/hub/*_test.mjs`. Then Opus full-branch review; then live e2e (drive a real serve with an evaluator pointed at a garbage-returning stub via openaicompat to force unparseable output against a REAL working model doing the work — prove the box keeps working through evaluator failure and terminates loudly at the horizon). +Record the evaluator resilience contract and incident sequence in `docs/history/goal-loop-resilience.md`; hub check (unknown `goal.eval_failed` event must not break rendering — same three dispatch sites as prior PRs; add `eval_failures` display ONLY if trivially cheap, else skip). Full gates + `node --test tools/hub/*_test.mjs`. Then Opus full-branch review; then live e2e (drive a real serve with an evaluator pointed at a garbage-returning stub via openaicompat to force unparseable output against a REAL working model doing the work — prove the box keeps working through evaluator failure and terminates loudly at the horizon). --- diff --git a/docs/plans/2026-07-21-goal-worker-park.md b/docs/plans/2026-07-21-goal-worker-park.md index e34bd35c..48ad3db3 100644 --- a/docs/plans/2026-07-21-goal-worker-park.md +++ b/docs/plans/2026-07-21-goal-worker-park.md @@ -14,7 +14,7 @@ ## Locked design decisions -- **Both tiers exit-park.** Deterministic (3 attempts, ~5s) and retryable (12 attempts, ~30min in-turn backoff) exhaustion both journal `goal.parked` + return the sentinel. This CHANGES #61's retryable-exhausted semantics from continue-in-loop to exit — deliberately: exiting frees the run slot during long outages (queued prompts run as normal turns instead of only injecting), and resume-on-activity re-enters the loop cleanly. Document the supersession in goal.go's round-history style and docs/goal-loop.md. +- **Both tiers exit-park.** Deterministic (3 attempts, ~5s) and retryable (12 attempts, ~30min in-turn backoff) exhaustion both journal `goal.parked` + return the sentinel. This CHANGES #61's retryable-exhausted semantics from continue-in-loop to exit — deliberately: exiting frees the run slot during long outages (queued prompts run as normal turns instead of only injecting), and resume-on-activity re-enters the loop cleanly. Document the supersession in goal.go's round-history style and `docs/history/goal-loop-resilience.md`. - **`goal.parked` is an ENGINE record/event** (`recGoalParked`/`EventGoalParked`), persisted+emitted under `s.mu` before `PursueGoal` returns, carrying `{Reason: , Attempts, RetryableClass?}` — distinct from the boot-only server-synthesized `goal.paused` (which stays as-is). Generation-gated like `goal.stalled` (a park racing `UpdateGoal` → stale-discard, loop continues against the new condition instead of parking). - **The goal stays armed**: no `clearGoal`, `ActiveGoal()` true, LoadSession replay unchanged (goal.parked folds as trace; active goal restores; boot pause presentation then applies as today). - **Server terminal is loud + distinct**: sentinel + exported `engine.IsGoalWorkerParked` predicate (the #81 evaluator_exhausted wiring pattern end-to-end): `runGoal` default-branch emits `session.error` and `turn.end outcome=worker_parked`; openapi outcome enum + Event docs updated. Event order: goal.parked < session.error < turn.end < idle. @@ -62,7 +62,7 @@ Session.goalParked lifecycle + third ambient occupant + tests (invariant 6). Com ### Task 4: Docs, review, e2e, PR -AGENTS.md (goal-loop section: park-both-tiers, supersede the #61 in-loop-park description; the deliberate context-overflow asymmetry), docs/goal-loop.md new section, full gates, Opus review, live incident replay (stub provider returning 404s → park not clear → GET shows worker_failure → prompt the box → auto-arm resumes → flip healthy → achieves; also retryable-tier park under synctest-less live with short schedule if feasible, else covered by tests), PR, converge, merge on approval. +Record park-both-tiers, the superseded #61 in-loop behavior, and the deliberate context-overflow asymmetry in `docs/history/goal-loop-resilience.md`; full gates, Opus review, live incident replay (stub provider returning 404s → park not clear → GET shows worker_failure → prompt the box → auto-arm resumes → flip healthy → achieves; also retryable-tier park under synctest-less live with short schedule if feasible, else covered by tests), PR, converge, merge on approval. ## Execution notes diff --git a/docs/plugins-and-protocols.md b/docs/plugins-and-protocols.md new file mode 100644 index 00000000..dbe3f22b --- /dev/null +++ b/docs/plugins-and-protocols.md @@ -0,0 +1,109 @@ +# Plugins and external protocols + +This document describes plugin lifecycle, hooks, client APIs, and external +protocol boundaries. + +## Plugin System + +Plugins are separate processes (any language; Go SDK provided) speaking a versioned JSON-RPC protocol over stdio. + +- **Manifest cache**: `harness plugin probe` runs a bounded manifest probe and caches the manifest (name, protocol version, hooks subscribed, tool definitions) with executable identity and plugin-spec identity. Run and serve startup trust a matching entry; a missing or stale entry performs one bounded probe before host construction. The long-lived plugin process does not start at boot. +- **Lazy spawn**: a plugin process starts on first hook dispatch or tool call, then stays warm for later calls during the host lifetime (module-level caches in plugins are expected and fine). +- Sync hooks chain across plugins in config order — each sees the previous plugin's mutations — and every sync dispatch carries a deadline so a hung plugin can't wedge a session. +- **Plugin visibility**: `Host.Plugins()` reports every CONFIGURED plugin — name, spawn state (`not-spawned`/`running`/`errored`/`stopped`), registered tools, subscribed hooks — from the cached manifest plus live spawn state. The `session_info` tool (field `plugins`) and `GET /session/{id}` (field `plugins`) both surface it, so a not-yet-spawned plugin still appears. The engine reads it through the `Hooks.Plugins()` interface method, nil-guarded exactly like the other `s.cfg.Hooks` dispatch sites. The state read is lock-free (`instance.liveState`, `plugin/host.go`): `instance.start` holds `inst.mu` for the whole dial-plus-handshake, and `Host` is a box-scoped singleton shared by every session on the box, so a read gated on `inst.mu` would let one session's plugin spawn stall `GET /session`/`session_info` for every other session too — the same "a hung plugin can't wedge a session" rule above, applied to a status read instead of a hook dispatch. `errored` also covers a plugin that died AFTER a successful spawn (its connection closed, detected via the existing `conn.closed` signal), not only a failed start. + +### Hook protocol v1 + +| Hook | Mode | Purpose | +|---|---|---| +| `event` | async, fire-and-forget | full event stream (batched) | +| `chat.params` | sync, mutating | model, temperature, etc. per request | +| `chat.message` | sync, mutating | messages before they enter the log | +| `system.transform` | sync, additive | append segments to the system prompt (runs after `chat.params`) | +| `shell.env` | sync, mutating | inject env vars into shell/tool commands | +| `tool.execute.before` | sync, mutating/blocking | rewrite args or block with `{deny: "message"}` | +| `tool.execute.after` | sync, mutating | rewrite/annotate tool results | + +Plugins may also register **custom tools** (defs in manifest, execution via RPC). + +### Plugin client API + +Plugins are API clients over the same channel: `Session.Messages`, `MCP.Call`, `Generate` (LLM calls through the harness provider layer — plugins never carry their own API keys), and `plugin.HTTPClient()` (outbound HTTP with harness-configured headers, e.g. workspace attribution). + +Events v1: `session.status`, `question.asked`, `file.edited`, +`tool.execute.start`, `tool.execute.end`, `session.error`. Message-delta +events are deliberately deferred (see plugin/PROTOCOL.md) pending a +throttling design. + +Capability parity bar: the protocol must be able to express the plugin +patterns common in opencode setups — event-driven activity tracking, token +refresh via `shell.env`, tool-call rewriting/vetoing and result guards via +`tool.execute.*`, path-scoped system prompt injection, and custom tools that +call back into the platform. + +## External Protocol Surfaces + +Standards we conform to at the edges. The internal model (event log, canonical +messages, hook protocol) is ours; these are adapters, never the internal +representation. + +- **ACP (Agent Client Protocol, agentclientprotocol.com)** — the editor ↔ agent + standard (Zed, JetBrains, Neovim, Emacs). Harness does not implement an ACP + adapter today. If one lands, keep it thin: map the event log to + `session/update` notifications and prefer ACP names where our vocabulary is + arbitrary. Harness has no permission system, so an adapter must not invent + `session/request_permission`. This is Zed's Agent *Client* Protocol, not + IBM's former Agent Communication Protocol. +- **MCP** — client (consume tool servers) and server (expose sessions/tools) + modes. + + A server's first connect (Initialize+ListAllTools) stays lazy — + triggered by a session's first `Tools()`/`CallTool()`, bounded by a + per-server `connect_timeout_s` config field (`MCPServerSpec`, integer + seconds, <= 0/absent defaults to the engine's own 15s). A server whose + first attempt fails is never dropped for the process's life: it gets a + detached background retry on a capped exponential backoff (~1s doubling + to a 5min cap, jittered) — but bounded to `mcpRetryMaxAttempts` (3) + further attempts (under ~10s of background effort total). Once those + are exhausted the entry is marked Parked and the retry goroutine exits + for good — no further attempt ever fires spontaneously; only an + explicit re-trigger (the `mcp` tool's `connect` action, below) can move + it again. A HEALTHY server, by contrast, connects exactly once and is + never re-probed. `Tools()` always reads live state, so a server that + recovers mid-session — background retry or explicit reconnect — + contributes tools on the very next turn automatically, no new session + required. `CallTool`/`CallServerTool` split the old combined error into + two: a server name absent from config errors "not configured" (never + recoverable); a configured-but-unconnected server (still retrying, or + parked) errors naming that state explicitly (recoverable — retrying may + still self-heal, parked needs the `mcp` tool). While at least one + server is degraded, request assembly appends an ambient `[mcp: + unavailable — (; retrying), ...]` block to the newest + user message only — computed fresh every turn, never persisted, + self-correcting as retries succeed; a Parked server's clause instead + reads ` (; use the mcp tool action "connect" to retry)` — + sharing its append-only-to-the-newest-message mechanism + (`withAmbientStatus`) with the managed-processes status block described in + `docs/session-storage-and-queue.md`. + + A built-in `mcp` session tool is registered in `newSession` whenever + the session's MCP registry reports at least one configured server (no + config flag, unlike `GoalTool`). `status` reports every configured + server's live state — `{name, connected, attempts, parked, reason}`; + `connect {server}` makes ONE bounded, synchronous attempt for a named + server — the only path back for a Parked server, though it works + against a still-retrying or never-yet-attempted one too. An + already-connected server is a friendly no-op; an unknown name errors + listing the configured names. A per-server in-flight guard (under the + manager's own lock) serializes a tool-triggered connect against both a + concurrent `connect` call and `retryServer`'s own background attempt + for the same server — whichever gets there first dials, the other + reports "attempt already in progress." Every model-visible string on + this surface — the ambient block, `status`'s `reason`, `connect`'s + failure result — is `classifyMCPConnectError`'s output, never a raw + error (which can embed the server's endpoint URL and any secret it + carries). +- **OpenTelemetry GenAI semantic conventions** — for span/metric naming when + observability lands. Configuration via standard `OTEL_*` env vars only. +- **A2A** — deliberately not implemented. Cross-org agent meshes are a + different layer; revisit only if a concrete need appears. diff --git a/docs/session-storage-and-queue.md b/docs/session-storage-and-queue.md new file mode 100644 index 00000000..094a498c --- /dev/null +++ b/docs/session-storage-and-queue.md @@ -0,0 +1,411 @@ +# Session storage, reads, queue, and processes + +This document describes session persistence, paging, prompt queues, and +managed-process invariants. Read the matching section before changing those +paths. + +## Session metadata index + +`GET /session` and `GET /session/{id}` do not replay a session journal. Each +session log has a sidecar `.index.json` holding one +`engine.SessionIndex`. The index carries every wire `Session` field with a +durable source: timestamps, model, effort, workdir, parent session, task +lineage, message count, usage, durable goal state, queue depth, and +compaction counters. + +Before the index, a read of a non-live session called `LoadSession`. That +call decodes every message body and rebuilds the whole history. The handler +then reported a dozen scalars and dropped the rest. The list endpoint paid +that cost once per non-live session (workstream 1 in the +`console-read-path.md` design from the meetneptune/boxes repository). + +The index is a fold of the journal (`engine/index.go`). Three rules keep it +honest. + +**It folds records, not memory.** `Session.writeRecord` folds every record +it appends. `ensureLog` folds the header records it writes directly, which +is the one write that does not pass `writeRecord`. Memory is never the +source: `EnqueuePromptDurable` writes its record before it mutates the +queue, so a fold of memory at that instant would disagree with the log it +claims to summarize. + +**It is a cache, never an authority.** `ReadSessionIndex` serves a stored +index only when three checks pass: a checksum over the stored bytes, the +journal's byte length, and the journal's modification time. Anything else +refolds from byte 0. There is no repair path, so no repair path can be +wrong. The checksum catches a torn read — both writers replace the sidecar +in place, and a reader in another process can otherwise mix old bytes with +new ones that parse. Length and modification time are a staleness key, not a +proof: they rest on the journal's own contract of one writer and append-only +writes. + +**It reports what a full load reports.** `Messages` and `LastActivityAt` run +through `message.ResolveOrphanToolCalls`, over a skeleton of ids, roles, and +tool-call ids. A crash between a tool call and its result therefore counts +the same on both paths. `DurableMessages` counts the records alone, for a +reader that must map a message to a byte offset; paginated message reads are +numbered against it. + +Some journals cannot be answered by a fold at all. A legacy header records +no workdir, and a crash can tear away the initial model record. `LoadSession` +answers those from the loading `Config`; a fold has no `Config`. +`SessionIndex.Complete` reports the difference, and +`Server.coldSessionJSON` uses the authoritative load path for such a +session. + +Three folds have real state machines. Each has one implementation, shared by +`LoadSession` and the index: `applyCompactRecord` (`compact.go`), +`applyGoalRecord` (`store.go`), and `promptQueueFold` (`queue.go`). +`engine/index_test.go`'s oracle test pins every index field against a full +`LoadSession`. + +A refold is far cheaper than a full `LoadSession`, and a current index is +cheaper again. Write-through costs one marshal and one rewrite per record. +The sidecar never gets an `fsync`: losing it in a crash costs one refold. + +`server.Options.Plugins` supplies a session's plugins to a cold read. +Plugins are process configuration, not durable session state, and a cold +read has no `Session` to ask. + +## Journal snapshotting + +`LoadSession` does not always replay a whole journal. Beside the log sits +`.snap`, a **seq-anchored checkpoint**: the fold-produced state as of +journal line N, plus a CRC-32 and a format version. Recovery loads the +snapshot, applies the session HEADER record (line 1, always — it is what +carries `created_at`, workdir, and task lineage, whose restore rules turn on +absent-versus-present and are not worth reproducing twice), and then replays +only lines `> N`. The tail scan reads through `scanLogRaw`, so a covered +record is never DECODED: skipping the decode of a message record's whole +part tree is where the saving is. Design: +docs/design/journal-snapshotting.md. + +The snapshot schema is EXPLICIT (`sessionSnapshot`, `engine/snapshot.go`), +never `json.Marshal` of a `*Session`: every field but `ID` is unexported, +and the config carries live callbacks, a `SessionManager` pointer, and open +file handles that must never be serialized. The rule for what belongs in it +is **exactly what the folds reconstruct** — a fold added without a matching +snapshot field is silently dropped, which is what +`TestSnapshotCarriesEveryFoldedField` and the replay-equivalence tests pin. +Two exclusions are deliberate: header-derived state (replayed instead), and +`turn`/`lastSystem`, which have NO durable source at all — capturing them +would make a snapshot-loaded session disagree with a full replay of the same +journal and make an observable field depend on whether a snapshot happened +to exist. + +The invariant is `state(snapshot@N) + replay(N+1..head) ≡ +full-replay(0..head)`. Every doubt falls back to a full replay: no snapshot, +a torn one, a checksum mismatch, a wrong version, another session's id, a +`seq` ahead of the journal head, or a journal whose first record is not a +session header. **Slower, never wrong.** The journal is never truncated; +snapshots are derived and can be deleted at any time. + +**The trigger is at the APPEND BOUNDARY, never inside `writeRecord`.** A +snapshot pairs a memory image with a journal position, and inside +`writeRecord` the two do not yet agree: some callers persist their record +BEFORE applying their own memory mutation (`EnqueuePromptDurable`, +deliberately), so a capture there would anchor past a record whose effect +memory has not applied — and the reload would skip that record and lose the +effect forever. `appendWithUsage` (after `persistMessage`, still under +`s.mu`) and the on-idle trigger (`runAgenticLoop`'s defer, and +`ReleaseFiles` on eviction) are the two boundaries where the caller has +completed both halves. + +The OPPOSITE direction is guarded by `snapshotSafeLocked`: +`SessionManager` splits some mutations into an in-memory half and a +DEFERRED durable half (`appendMemoryOnly`/`persistAppendedMessage`, +`enqueueTaskNotificationMemoryOnly*`/`persistQueuedTaskNotification`, +`queueRecordDeferredLocked`), so memory can be AHEAD of the journal. A +snapshot taken in that window carries the mutation AND leaves its record in +the tail, and the reload applies it twice — a duplicated message, or a +child-completion notification the parent renders to the model twice. +`Session.durableDebt` counts the outstanding halves; a capture is refused +while any is open, which merely postpones the snapshot to the next +boundary. The debt clamps at zero: a leaked increment stops this session +snapshotting (degrading to today's full replay), where a negative count +would ARM a capture in exactly the unsafe window. + +`Config.SnapshotEveryRecords` is the cadence. Zero — the engine zero value — +disables snapshot WRITING, so a bare embedder-built `engine.Config` keeps +the pre-snapshot behavior; the config/CLI layer supplies the product default +of 64 (`snapshot_every_records`), the same unset-versus-explicit-zero split +`prompt_retries` uses. READING a snapshot is never gated on it: recovery is +a property of the files on disk, not of the loading Config. A snapshot write +is background, coalesced (one in flight per session), and atomic (temp → +fsync → rename), and its failure lands in `lastSnapshotErr`, never +`lastPersistErr` — a snapshot is derived acceleration, not a durability +promise. + +A load that takes the snapshot path marks the metadata-index fold BROKEN +rather than building a partial one: the index summarizes EVERY record, and +this load deliberately did not see most of them. The index is a cache with +no repair path, so a reader that finds none refolds and `ensureLog` re-seeds +the fold from the journal on this session's next write. Snapshotting the +index fold itself is the obvious follow-up; nothing may guess at it. + +## Paginated message reads + +`GET /session/{id}/message?before_seq=N&limit=K` answers one bounded page of +a session's messages, read from the journal's tail. The unparameterized call +is unchanged, byte for byte: no `before_seq` and no `limit` still returns the +bare array of the whole history every existing caller expects. A request that +names either parameter gets a `MessagePage` envelope instead — `messages`, +`first_seq`, `last_seq`, `total`, `has_more` — because a client that pages +needs the page's position and a client that does not must not have to learn a +new shape. A console loads the tail and pages older messages in on scroll +(workstream 2 and directive 1 in the `console-read-path.md` design from the +meetneptune/boxes repository). Before this, every console open transferred +the whole transcript. + +**Seq is an ordinal over the DURABLE message sequence**: message records in +log order, with each compact record's fold applied (the folded range replaced +by that record's summary). `SessionIndex.DurableMessages` counts that same +sequence, so the newest message's seq equals it. `SessionIndex.Messages` can +be larger — it also counts the synthetic tool results +`message.ResolveOrphanToolCalls` derives — and a derived message has no +record, so it has no seq and no page carries it. This definition is what +makes a bounded read possible: an ordinal over durable records can be counted +backwards from the end of a file, while an ordinal over a materialized +history cannot be known without materializing it. + +`engine.ReadMessagePage` (`engine/messagepage.go`) serves a page two ways, +and both are numbered by the same index: + +- The **tail walk** reads `revChunkBytes` blocks backwards from + `SessionIndex.LogSize` and numbers message records down from the total. It + touches only the tail however long the journal is. It gives up the moment + it meets a compact record, because the messages a fold KEPT sit in the log + between the folded range and the compact record itself — undoing that + backwards would be a second, subtly different implementation of a fold. +- The **fold path** then reuses `indexFold` — the same forward fold and the + same compact-range occurrence selection — to learn which messages occupy + the requested seqs. `indexFold` carries each surviving message's journal + record ordinal beside its skeleton; `foldedPage` reads back by that ordinal, + never by message ID alone. This matters for hand-written or externally + assembled journals that repeat an ID: one occurrence can be compacted away + while another survives, and the surviving sequence entry must decode the + surviving record. The path costs one slim pass (ids, roles, and ordinals, + never message bodies), which is still three orders of magnitude below + materializing the history. + +The scan is bounded by `SessionIndex.LogSize`, never the file's current size, +so a turn appending records while a page is read cannot renumber that page +under it. + +Two properties are deliberate. A page carries durable messages **verbatim**: +it never runs `message.ResolveOrphanToolCalls`. That repair exists to keep a +provider REQUEST valid, this endpoint builds no request, and fabricating a +tool failure in a read view has real production history (see `Server.lookup`'s +doc comment: a healthy child's in-flight tool call rendered as failed in the +console for as long as it kept running). And compaction **renumbers** — a fold +replaces N messages with one summary, so every later seq shifts down by N-1. +A client paging across a compaction can see one page overlap another; message +ids are stable and are the way to de-duplicate. + +A page is read from the durable records even when the session is resident, so +one seq definition covers both cases: a resident history can carry messages +the log does not (load-time repairs, recovery's memory-only closers), and +numbering those would give one message two different seqs depending on +residency. A session with no journal at all — created through the API and +never prompted — falls back to its resident history, which for such a session +IS the durable sequence. + +## Prompt queue + +`POST /session/{id}/prompt_async` against a session already busy (another +prompt, or a running goal loop) no longer 409s — it queues. The prompt is +enqueued durably (`engine.Session.EnqueuePrompt`, persisting a `prompt.queued` +record and assigning a session-monotonic ID) synchronously, before any +response is written — the same enqueue-durable-before-202 shape `RegisterGoal` +already uses for goals, closing the accept-vs-lose race structurally. The +response is 202 either way: `status: "started"` when a turn is now running for +this request's own prompt (an idle claim against an EMPTY queue, or a +freed-slot retry that happens to win and dispatch this same prompt), or +`status: "queued"` (carrying the current depth) when it is durably waiting — +including the idle-claim case where the queue is already non-empty (a +restart refold, or any other drain gap that ever left a prompt stranded): +`handlePrompt` enqueues the incoming text behind whatever is already waiting, +then dispatches the queue's HEAD — not necessarily this request's own text — +into the run slot it just claimed, so a fresh arrival can never cut the +line ahead of prompts already queued. The workdir-held-by-another-session 409 +is unchanged — only same-session busy gets queue semantics. + +The queue drains FIFO, by queue ID, at every run-slot release, with no +exceptions: `runPrompt`'s, `runGoal`'s, and `handleCompact`'s tails all call +`maybeDispatchQueued`, which claims the freed slot, dequeues the head +(`reason: "delivered"`), and spawns it as a normal prompt turn — whose own +tail repeats the check, so the whole queue drains one turn at a time before +anything else gets a look. `handlePrompt`'s own claim-success path (previous +paragraph) is the one non-tail drain site: an admission-time head-dispatch +for the idle-with-non-empty-queue case, closing the gap a tail-only drain +would otherwise leave open between "session goes idle with a queue still +non-empty" and "the next prompt/goal/compact activity happens to touch it." +This is also where +**queue beats goal auto-arm**: `runPrompt`'s and `handleCompact`'s tails call +`maybeDispatchQueued` *before* `maybeAutoArmGoal` (see above), so a prompt +sitting in the queue when a turn or a compact call ends is dispatched first — +direct user input outranks the background objective — and the goal only +auto-arms once the queue is empty. + +**Delivery granularity is per tool-call boundary, not per turn.** Inside +`Session.Prompt`'s agentic loop (`engine/engine.go`), the instant a +tool-result message is appended — after the model made one or more tool +calls and before the next provider request in that SAME turn — the loop +drains the ENTIRE queue, FIFO, in one locked op (`DequeueAllPrompts +("injected")`) and appends the drained batch as a single, durable user +message: the same labeled "OPERATOR MESSAGES" block template +(`operatorMessagesBlock`, `engine/queue.go`, shared by every drain site so a +batch renders identically apart from one parameterized word — this +call site passes `operatorContextTask`, so its header says "continue the +task", never "continue the goal", even when this drain happens to fire +inside a goal loop's worker turn; only goal.go's own turn-boundary drain +below passes `operatorContextGoal`). This only ever +APPENDS — never rewrites an earlier message — so a provider's prompt-cache +prefix stays intact, the same principle the managed-processes ephemeral +status block below relies on, except this message is a REAL, durable +delivery, not a disposable status line. A turn that ends WITHOUT any tool +call never reaches this drain point at all (the model's own end-of-turn +return precedes it), so that path — and anything still queued when it +happens — is left entirely to the mechanisms below. Because `PursueGoal`'s +worker turns run through this exact same `Prompt` loop +(`promptTurnWithRetry`), goal loops inherit tool-call-boundary injection +automatically, with no separate wiring: a prompt queued while a goal's +worker turn is mid-tool-call is delivered inside that SAME worker turn — +matching Claude Code's mid-turn steering granularity — rather than waiting +for the goal's own turn boundary described next. + +`PursueGoal` keeps a second, complementary drain at its own turn boundary: +at the top of every turn (the same `snapshotGoal` boundary #77's +condition-update snapshot uses, and before that turn's own tool-call-boundary +drain above has any chance to run) it drains the *entire* queue, FIFO — +catching anything still queued from a turn that made no tool calls at all, or +that arrived in the gap between one turn ending and the next one's snapshot — +and prepends it to that turn's directive as the same labeled "OPERATOR +MESSAGES" block (`operatorMessagesBlock`, `operatorContextGoal` — so its +header says "continue the goal"), ahead of — never replacing — the +ordinary condition/guidance text. The evaluator's condition string is +unchanged by this — it is built from the condition alone, never from the +block or the turn's rendered directive — so goal injection judges only the +goal there; the evaluator's separate transcript field does render the full +history, so it does see the block once the worker turn that received it has +run. Every drained prompt journals its own `prompt.dequeued(injected)` record +before the turn's directive is even built, so it counts as delivered at that +point even if the turn's outcome later turns out stale and gets discarded — +an injected prompt is never re-queued, at either drain site. This means an +abort (`POST /abort`) or a goal clear (`DELETE /session/{id}/goal`) racing a +goal turn boundary consumes an entire just-injected batch at once: every +prompt the boundary drained is already journaled `dequeued(injected)` before +the worker turn even starts, so a turn that gets cancelled or whose outcome is +later discarded as stale still loses all of them together — several operator +messages, not just one — the same exposure class an ordinary in-flight prompt +already has, just multiplied across the whole drained batch. The two drain +sites can never double-deliver the same prompt: `DequeueAllPrompts` is one +atomic, locked pop of the whole queue, so whichever site runs first against a +given prompt is the only one that ever sees it. + +Every enqueue/dequeue is a durable record — `prompt.queued` and +`prompt.dequeued`, the latter carrying a `reason` of `"delivered"` (idle +drain), `"injected"` (tool-call-boundary or goal-turn-boundary injection — +both drain sites share the reason, see above), or `"cleared"` (see below) — +journaled and emitted (`EventPromptQueued`/`EventPromptDequeued`) under +`s.mu` in the same critical section, mirroring `RegisterGoal`/`ClearGoal` +exactly. Dequeue always journals *before* the text enters any turn, so a crash +between that journal write and the dispatched turn's completion cannot +double-deliver — the prompt is simply gone from the queue on replay, the same +exposure any in-flight prompt already has today. **Boot never auto-dispatches +a resumed queue**: `LoadSession` folds `prompt.queued`/`prompt.dequeued` +records back into the exact undelivered set, `GET /session`'s `queued` count +reflects it immediately, and it sits there until the next natural drain +trigger (an idle prompt, the next tool-call boundary inside a running turn, or +a goal loop's next turn boundary) — the same settled boot rule goals follow. +`DELETE /session/{id}/queue` is the one explicit clear surface: it journals +`prompt.dequeued(cleared)` for every pending item then 204, idempotent on an +empty queue, and never touches a currently running turn — `POST /abort` is +unrelated and does not touch the queue either way (it only cancels the +in-flight turn's context). + +Two v1 limits are deliberate, not gaps: **text-only** (queued prompts carry a +plain string — `QueuedPrompt{ID, Text}` — no attachment machinery, matching +the plain-prompt contract's `parts` being text-only already), and **a +per-request `model` override is silently dropped when the prompt is queued** +— there is no slot in `QueuedPrompt` to carry it through to a future drain, so +a caller that needs a model swap to take effect must re-issue the request once +it is confirmed `started`. + +`POST /session/{id}/enqueue` (docs/plans/2026-07-21-durable-enqueue.md) is +`prompt_async`'s durable, idempotent sibling for a caller whose own upstream +ack rides on this call succeeding — an inbox poller or coordinator relay, +not an interactive client. `Session.EnqueuePromptDurable` extends +`EnqueuePrompt` with three properties the plain path deliberately lacks: +write-ahead durability (the `prompt.queued` record is written and, in the +default `session_sync: "fsync"` mode, *fsynced* before any in-memory +mutation or response, so a 2xx is an honest attestation rather than a +best-effort ack — a write/fsync failure returns 500 "enqueue not durable" +instead of the swallowed `lastPersistErr` every other persist path uses), a +caller-issued session-monotonic `seq` deduplicated against a durable +high-water mark (`Session.EnqueueSeq()`, journaled on the record and +rebuilt by `LoadSession` — a seq at or below the mark is a clean 200 +`duplicate` no-op, so retries are always safe, including across a process +restart), and torn-write healing (a burned-but-failed queue ID is never +reused, and replay folds same-seq records last-writer-wins). Delivery is the +exact same FIFO/tool-boundary/goal-boundary machinery described above — this +is a new *acceptance* contract, not a new delivery path: durable means +accepted into the queue, and delivery-out is still the queue's normal +at-most-once-per-dequeue machinery, so a crash between dequeue and turn +completion loses that delivery once rather than redelivering it, exactly +like any in-flight prompt (`maybeDispatchQueued`'s "No-double-delivery +equivalence", invariant 7, in server/handlers.go). `GET +/session/{id}/queue` is the paired reconciliation read: the watermark plus +the pending queue (FIFO, `seq` present only on durable-enqueue entries), for +an upstream recovering from its own crash to check what's already inside the +durability domain instead of re-sending blind. `prompt_async` remains the +right choice for an interactive client that has no upstream ack to protect — +it is not going away, and `POST /session/{id}/enqueue` adds no new limits +beyond what queued prompts already have (text-only, no model override). + +The `fsync` in "write-ahead durability" above is itself mode-selectable: +config's `session_sync` ("fsync", the default, or "volume") gates both this +durable-enqueue fsync and the one-time session-create directory fsync +(`ensureLog`'s fresh-file `syncDir` call, store.go) — nothing else changes. +"volume" is for a session store on a continuously-synced network volume +whose own commit layer is the documented durability boundary: fsync adds no +durability there, and some FUSE/9p transports deadlock permanently on it +(`fsync(dirfd)` especially — a wedge that hangs every later file op on the +mount, not just the one call). In that mode the write(2) landing out of +`EnqueuePromptDurable`/`ensureLog` is itself the attestation; the write +ordering, torn-write healing, and replay/fold logic above are byte-for-byte +identical in both modes — a volume can still lose an unsynced tail on abrupt +death exactly like a torn fsync can, and the same last-writer-wins fold +repairs both. See docs/deploy-modal.md for the recommended setting on Modal +Volume v2 deployments. + +## Managed processes + +`config.Config.Processes` (`processes` in JSON) declares named long-lived +dev/support processes (`pnpm dev`, a local DB) that a `process` session +tool can start/stop/restart/inspect without an agent reinventing PID +tracking. `*process.Manager` (package `process`, not `engine`) is a +box-scoped singleton — built once per harness process and shared across +every session, exactly like `engine.MCPManager` — with a +starting/ready/running/exited/stopped state machine, unix process-GROUP +kill on stop (mirroring `engine/bash_unix.go`'s Setpgid/kill-pgroup/ +WaitDelay pattern), and asynchronous death detection (a waiter goroutine +flips state to `exited` with no client asking). Logs land at +`/.harness/proc/.log`. + +The tool can also `declare`/`undeclare` NEW process definitions at +runtime (server-lifetime only, never written to `.harness.json`) — see +`docs/design/managed-processes.md` for the full validation and origin +(`config` vs `runtime`) rules. `harness serve` always builds a +`*process.Manager`, even with zero configured processes, so the tool is +present on every served box; `harness run` keeps the zero-cost-when- +unconfigured rule. + +Once at least one declared process has EVER been started (this server +process's lifetime), request assembly appends an ephemeral `[processes: +...]` status block to the newest user message ONLY — as a +`message.EngineContext` part (see the "Ambient engine context" section in +`docs/engine-request-cycle.md`), never persisted into the durable session log, +never touching any earlier message so a provider's prompt cache prefix +stays intact. See `docs/design/managed-processes.md` §4 for the exact +mechanism and why it is safe. diff --git a/e2e/AGENTS.md b/e2e/AGENTS.md new file mode 100644 index 00000000..d7817320 --- /dev/null +++ b/e2e/AGENTS.md @@ -0,0 +1,25 @@ +# End-to-end test instructions + +These rules apply to `e2e/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. + +## Cross-process timing exception + +End-to-end tests may observe out-of-process state with a deadline-bounded poll. +No in-process signal crosses that boundary. + +Every poll must use `internal/testpoll`. Do not write an inline sleep loop. +The timeout is a failure bound. Return on the first successful check. + +Do not use this exception for engine, manager, queue, or server state that has +an in-process notification seam. + +## Test scope + +Start a real subprocess only when the process boundary is the behavior under +test. Keep fixtures local and deterministic. Do not require live provider +credentials in the ordinary suite. + +Run end-to-end tests with `-race`. Preserve useful subprocess output on +failure, and mask secret values. diff --git a/e2e/CLAUDE.md b/e2e/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/e2e/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/engine/AGENTS.md b/engine/AGENTS.md new file mode 100644 index 00000000..7467a5dd --- /dev/null +++ b/engine/AGENTS.md @@ -0,0 +1,175 @@ +# Engine instructions + +These rules apply to `engine/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. + +Read `message/AGENTS.md`, `provider/AGENTS.md`, or `server/AGENTS.md` when a +change crosses those boundaries. + +Detailed behavior and rationale live in `docs/`. This file contains edit-time +constraints for the engine. + +## Read first + +- Request assembly, tools, retries, or metrics: + `docs/engine-request-cycle.md`. +- Goal supervision: `docs/goal-loop.md`. +- Journals, indexes, pages, queues, or processes: + `docs/session-storage-and-queue.md`. +- Models, effort, affinity, or provider caches: + `docs/models-and-providers.md`. +- Task lineage or provider exhaustion: `docs/design/fleet-model.md`. +- Compaction: `docs/design/context-compaction.md`. +- Project instruction loading: + `docs/design/nested-instruction-loading.md`. +- MCP schema deferral: `docs/design/mcp-lazy-tools.md`. +- MCP connection recovery: `docs/plugins-and-protocols.md`. +- Process lifecycle: `docs/design/managed-processes.md`. + +## Sessions, history, and ambient context + +- Treat the session log as append-only. +- Store canonical `message.Message` values, not provider wire values. +- Keep live and persisted history repairs additive-only. +- Hold `Session.mu` while persistence and emission must form one observation. +- Keep runtime-only state out of the journal unless replay needs it. +- Pass the firing session ID into callbacks copied into child sessions. + +Only engine code creates `message.EngineContext`. Use it for trusted ambient +status and continuation nudges. Never replace it with `message.Text` or persist +it. Add ambient status only to a throwaway request copy. + +## Project instructions and Agent Skills + +`loadInstructions` searches upward from `Config.WorkDir`. It returns the first +`AGENTS.md` or `AGENT.md` and stops at the Git root or filesystem root. It runs +on the first prompt, not at `NewSession`. + +- Treat a missing instruction file as valid. +- Reject an empty or invalid UTF-8 instruction file. +- Make truncation visible in the prompt and logs. +- Keep omitted sections reachable through the generated outline. +- Do not claim that nested attach-on-read is implemented. + +Discover Agent Skills on the first prompt. Inject only the catalog. Require the +model to load a selected `SKILL.md`. Reject malformed or duplicate skills. +Rediscover skills after resume and never persist the catalog. + +## Tool execution and file tools + +- Run independent calls concurrently up to `ToolConcurrency`. +- Preserve serial barriers and original result order. +- Run calls with one non-empty key in call order. +- Use resolved file keys for `read_file`, `write_file`, and `edit_file`. +- Do not let a keyed waiter hold a worker slot. +- Return exactly one result for every tool call, including cancellation. +- Keep the model-facing batching segment equal to resolved concurrency. +- Do not claim that file keys cover Bash side effects. + +Keep the tool array byte-stable. Sort built-ins by name. Preserve built-in, MCP, +then plugin group order. Make every new tool source deterministic. + +Classify images by magic bytes from one open handle. Enforce the 20 MiB limit +and inspect dimensions through that handle. Keep non-images on the text path. + +Reserve `read_file` memory from the stat size before reading. Serve reservations +FIFO. Let one oversized read use the full budget alone. + +Before overwriting an existing regular file, require a successful live-session +read or write record. Compare its saved SHA-256 with current bytes. Track +resolved absolute paths and serialize parallel mutations by file key. + +## Requests, retries, and metrics + +- Retry only typed retryable errors and completed empty turns. +- Never retry cancellation, permanent request errors, or interrupted tool intent. +- Emit `EventTurnRestart` before retrying after partial streamed output. +- Do not journal a failed attempt or run its tools. +- Keep interactive backoff shorter than goal-worker retry tiers. + +Treat `StopMaxTokens` as incomplete. Continue within one `Prompt` and one +`MaxTokensContinuations` budget. Preserve a partial tool call's identity, +drain operator input first, and end with a user-role engine-context nudge. +Mark budget exhaustion permanent. + +Emit one `TurnMetrics` record per completed provider call. Do not emit one for +a failed or interrupted stream. Preserve server join fields. Inject `Config.Now` +in tests. + +## Goal supervision + +- Keep the evaluator tool-less and force `message.EffortOff`. +- Bound evaluator context independently from main-model context. +- Select retry tiers with typed error classes. +- Clear on context overflow. Park on worker retry exhaustion. +- Persist goal transitions and generation-check stale results. +- Reuse one unanswered directive across retries. +- Keep the `goal` tool free of a `clear` action. +- Require operator evidence for completion. + +## Persistence, queues, and processes + +Treat the sidecar index as a cache of the journal fold. Validate its checksum, +journal size, and modification time. Refold on doubt. Share fold logic with +full replay. + +Treat a snapshot as a versioned checkpoint, never as authority. Capture only +at a durable append boundary. Reject capture while `durableDebt` is non-zero. +Fall back to full replay for every invalid snapshot. + +Number `MessagePage` values by durable folded message sequence. Keep the legacy +unparameterized response as a full array. Never add synthetic orphan repairs +to a page. Bound scans to the indexed log size. + +- Keep the prompt queue durable and FIFO. +- Persist enqueue before acceptance and dequeue before delivery. +- Let queued input beat goal auto-arm. +- Deliver each item once across tool and goal-turn drains. +- Do not auto-dispatch a restored queue at boot. +- Keep queued prompts text-only and model-override-free. +- Preserve durable sequence deduplication and its high-water mark. + +Keep the process manager box-scoped and shared. Runtime declarations are not +configuration writes. Preserve process-group termination and asynchronous exit +detection. Render status only as runtime `EngineContext`. + +## Children, models, compaction, and effort + +- Persist child lineage, depth, agent type, and spawn links. +- Keep parent notifications durable and bounded. +- Preserve provider-exhausted children for later resume. +- Mask and cap provider failure details. +- Resolve reaped descendants through durable ancestry. +- Run slow disk recovery outside the manager-wide lock. + +Use `Session.SetModel` and `Session.SetEffort` as the only change choke points. +Persist and emit real changes; write nothing for no-ops. Re-evaluate a derived +context window after a model switch. Refuse a required registry miss. + +- End summarization requests with a new `RoleUser` instruction. +- Never send a folded assistant message as provider prefill. +- Detect summaries by structural message-ID prefix. +- Keep empty completed summaries as non-mutating no-ops. +- Keep failed summarizer calls as errors. + +The main turn uses session effort. The evaluator uses `EffortOff`. The +summarizer inherits session effort. Do not combine these policies. + +## MCP + +- Keep first connection lazy and bounded by the configured timeout. +- Park after the bounded background retry schedule. +- Let only explicit `mcp connect` revive a parked server. +- Read live state when publishing tools. +- Serialize background and explicit connections per server. +- Show only classified, secret-safe errors in runtime `EngineContext`. +- Never defer schemas without the `mcp` selector tool. +- Sort the complete catalog by full tool name. +- Resolve the provider before a plan can dial MCP. +- Keep selections durable and reap them only after proved absence. + +## Deliberately absent + +The engine has no tool permission gate and no plan mode. Do not add either as +a local convenience. diff --git a/engine/CLAUDE.md b/engine/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/engine/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/engine/bash.go b/engine/bash.go index 077afb7b..8dd6abbe 100644 --- a/engine/bash.go +++ b/engine/bash.go @@ -19,7 +19,7 @@ import ( // runs, git logs, file dumps) while bounding the worst case — an apt-get or // npm install storm that would otherwise dump megabytes into a single // message, bloating the session log and the next provider request built from -// it (see AGENTS.md and docs/goal-loop.md for the incident this fixed). +// it (see docs/history/goal-loop-resilience.md for the incident this fixed). const defaultBashOutputCap = 96 * 1024 // bashWaitDelay bounds how long cmd.Wait may block on the command's output diff --git a/engine/compact.go b/engine/compact.go index 6b9a42a2..3d873abd 100644 --- a/engine/compact.go +++ b/engine/compact.go @@ -96,9 +96,9 @@ const compactionSummaryIDTag = "cmpsum" // the worst case (the summarizer returns empty for that one old-style // range) is now bounded to a single extra billed call, not an unbounded // loop: SkipReasonSummarizerEmpty latches maybeAutoCompact's hysteresis -// immediately. This repo is pre-production (see AGENTS.md's "Do not -// over-engineer a pre-production system"), so no persisted session -// predates this PR's own compaction feature — there is nothing to migrate. +// immediately. No persisted session predates this PR's own compaction +// feature, so there is no old session shape to migrate (see +// docs/design/context-compaction.md). func isCompactionSummaryID(id string) bool { return strings.HasPrefix(id, compactionSummaryIDTag+"_") } @@ -561,10 +561,10 @@ func (s *Session) runCompactionSummary(ctx context.Context, model message.ModelR // EffortUnset here — this only forwards, never overrides. (Issue // #124.) // - // Known residual, deliberately not addressed here (see AGENTS.md's - // scope-discipline rule): a non-off level lets the anthropic and - // openai adapters raise this request's effective output cap above - // compactionMaxTokens (anthropic's thinking-budget bump, openai's + // Known residual, deliberately not addressed here (see the root + // AGENTS.md "Change discipline" section): a non-off level lets the + // anthropic and openai adapters raise this request's effective output + // cap above compactionMaxTokens (anthropic's thinking-budget bump, openai's // reasoningOutputFloor — up to ~20480 tokens at EffortHigh, versus // the documented 1024 cap), and openaicompat sends reasoning_effort // with no such floor at all, so a reasoning-heavy summary can be diff --git a/engine/context_window_test.go b/engine/context_window_test.go index 9d761b80..b9b8e305 100644 --- a/engine/context_window_test.go +++ b/engine/context_window_test.go @@ -294,8 +294,8 @@ func TestSetModelClearsStaleHysteresisOnWindowChange(t *testing.T) { } // TestSetModelSameWindowDoesNotLog is the red-first regression test for -// Finding 5: context_window.go's logContextWindowArmed doc comment (and the -// AGENTS.md addendum) promise the "model_switch" INFO line fires only when +// Finding 5: context_window.go's logContextWindowArmed doc comment and +// docs/models-and-providers.md promise the "model_switch" INFO line fires only when // the effective window actually changes, but SetModel logged on every // non-no-op model change regardless — two models that happen to share the // same modelmeta-derived window still produce a spurious "model_switch" diff --git a/engine/engine.go b/engine/engine.go index 59eecc94..766de232 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -105,7 +105,7 @@ type Tool struct { } // Event is one entry in the session's event stream. Event types follow ACP -// naming where a choice is arbitrary (see AGENTS.md). +// naming where a choice is arbitrary (see docs/plugins-and-protocols.md). type Event struct { Type string `json:"type"` SessionID string `json:"session_id"` @@ -546,9 +546,10 @@ type Config struct { // Nil (the default) resolves to time.Now in newSession. This exists so a // test can inject a scripted sequence of instants instead of depending // on real elapsed wall-clock time between two calls to a fake stream's - // Next() — the AGENTS.md testing rule against real sleeps in tests - // applies here exactly as it does to any other timer-dependent code, and - // a real duration between two in-process function calls with no actual + // Next(). The per-turn metrics contract in docs/engine-request-cycle.md + // requires tests to avoid real sleeps, and it applies here exactly as it + // does to any other timer-dependent code. A real duration between two + // in-process function calls with no actual // I/O between them would otherwise round to ~0 and prove nothing. Scoped // to this one measurement, not a general engine clock seam: every other // timestamp in this package (CreatedAt, etc.) still reads time.Now @@ -637,7 +638,7 @@ type Config struct { // (the default) installs no `goal` tool at all, exactly like a nil // Config.Processes installs no `process` tool. The server/CLI wiring // that sets this true when a goal evaluator is configured is a later - // task (see docs/design/2026-07-19-goal-self-adjust.md) — this field + // task (see docs/plans/2026-07-19-goal-self-adjust.md) — this field // only gates registration. GoalTool bool @@ -1348,8 +1349,8 @@ type Session struct { // rather than a per-process one. Guarded by mu. toolResultBytes int - // readHashes backs the write_file read-before-overwrite guard (see the - // "write_file read-before-overwrite guard" section of AGENTS.md and + // readHashes backs the write_file read-before-overwrite guard (see + // docs/engine-request-cycle.md and // writeFileTool's doc comment in filetools.go). It maps a resolved // absolute path (s.resolvePath's output, never a raw relative tool // argument) to the sha256 hash of that path's raw on-disk bytes as of @@ -2163,7 +2164,7 @@ func (s *Session) persistAppendedMessage(m message.Message) { // This mirrors the accounting compact.go's errEmptyCompactionSummary skip // path already established for the sibling "the call ran and cost real // tokens even though it produced nothing usable" shape (see runCompaction's -// doc comment and AGENTS.md's "An empty summary is a graceful no-op..."): +// doc comment and docs/models-and-providers.md): // the call was real, the provider billed it in full (for an empty turn, // typically a full input prefill plus the entire max_tokens output // ceiling), and no tokens were refunded just because streamTurnWithRetry diff --git a/engine/engine_test.go b/engine/engine_test.go index a72cd7e8..09e473cd 100644 --- a/engine/engine_test.go +++ b/engine/engine_test.go @@ -282,10 +282,11 @@ func TestHooksIntegration(t *testing.T) { afterSuffix: "[annotated]", } s := NewSession(Config{ - Providers: provider.Registry{"test": prov}, - Model: message.ModelRef{Provider: "test", Model: "m1"}, - System: []string{"base"}, - Hooks: hooks, + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + System: []string{"base"}, + Hooks: hooks, + Instructions: &InstructionsConfig{Disabled: true}, }) if _, err := s.Prompt(context.Background(), "go"); err != nil { diff --git a/engine/filetools.go b/engine/filetools.go index 8b37bdfd..c44e8133 100644 --- a/engine/filetools.go +++ b/engine/filetools.go @@ -300,7 +300,7 @@ func filePathKey(s *Session, args json.RawMessage) string { // s.resolvePath output — every caller in this file passes one, so the map // never keys on a raw, unresolved tool argument. See the readHashes field // doc comment (engine.go) and the "write_file read-before-overwrite guard" -// section of AGENTS.md for the full design. +// section of docs/engine-request-cycle.md for the full design. func (s *Session) recordRead(resolvedPath string, hash [sha256.Size]byte) { s.mu.Lock() defer s.mu.Unlock() @@ -477,7 +477,7 @@ func writeFileTool() Tool { // gated — creation is write_file's main job, so a path that // does not exist falls straight through unguarded. Any OTHER // stat failure refuses the write: it cannot prove no protected - // file exists there. See AGENTS.md's "write_file + // file exists there. See docs/engine-request-cycle.md's "write_file // read-before-overwrite guard" section for the full design. info, statErr := os.Stat(path) if statErr != nil && !errors.Is(statErr, fs.ErrNotExist) { diff --git a/engine/filetools_test.go b/engine/filetools_test.go index 1be19c68..4c5d6e68 100644 --- a/engine/filetools_test.go +++ b/engine/filetools_test.go @@ -223,8 +223,7 @@ func runToolParts(t *testing.T, tool Tool, workDir, args string) (message.Parts, } // tinyPNG builds a real, compliant, tiny PNG — a 2x2 solid image — so image -// tests exercise genuine image bytes without a committed binary fixture -// (AGENTS.md's fixture-size lesson from #101: keep test images tiny). +// tests exercise genuine image bytes without a committed binary fixture. func tinyPNG(t *testing.T) []byte { t.Helper() img := image.NewRGBA(image.Rect(0, 0, 2, 2)) @@ -318,7 +317,8 @@ func TestReadFileImageExtensionLieTextNamedPNGStaysText(t *testing.T) { // TestReadFileImageExtensionLiePNGNamedTxtIsSniffedAsImage is the missing- // direction half: a file named .txt that actually holds PNG magic bytes must // still be recognized as an image. Extension is a hint only; magic bytes are -// authoritative (AGENTS.md: read_file classifies "never by its extension"). +// authoritative (see docs/engine-request-cycle.md's "read_file image +// support" section). func TestReadFileImageExtensionLiePNGNamedTxtIsSniffedAsImage(t *testing.T) { dir := t.TempDir() data := tinyPNG(t) @@ -398,8 +398,8 @@ func TestReadFileImageTruncatedPNGFallsBackToText(t *testing.T) { // closing the gap between the engine-level Tool.Run tests above and the // transcode-level golden test in provider/anthropic/transcode_test.go, // which hand-builds a ToolResult shaped like read_file's output rather than -// obtaining one from read_file itself (AGENTS.md's "verification drives -// the production entry point" rule). +// obtaining one from read_file itself (see the root AGENTS.md testing rule +// to drive the production entry point). func TestReadFileImageToolCallProducesBlobToolResult(t *testing.T) { dir := t.TempDir() data := tinyPNG(t) diff --git a/engine/goal.go b/engine/goal.go index 50c0e5af..150af962 100644 --- a/engine/goal.go +++ b/engine/goal.go @@ -7,7 +7,7 @@ // next turn until the condition is met or the turn budget runs out. // // This is a plan-artifact-free, gate-free loop: it introduces no plan mode and -// no permission gate (see AGENTS.md, "Deliberately absent"). It is a control +// no permission gate (see docs/goal-loop.md). It is a control // loop over Prompt plus a read-only evaluator call, nothing more. // // Durable goal.* records land in the session log so a resumed session can tell @@ -63,7 +63,8 @@ // (ses_41813d5a411c2ba5.jsonl, ses_55e4ae35d8344540.jsonl): each died right // after a "goal not met" guidance message was appended, mid-turn, with no // goal.eval and no error record — the log simply stopped, for hours, until a -// human forced a goal.cleared. See docs/goal-loop.md for the write-up. +// human forced a goal.cleared. See docs/history/goal-loop-resilience.md for +// the write-up. // // # Round 3: the same escape path, the other half of the loop // @@ -298,7 +299,7 @@ // closes the second bullet above: a queued prompt dispatches as a normal // turn the instant the slot is free, and the server's PRE-EXISTING // activity-driven auto-arm (maybeAutoArmGoal, upstream of this package — -// see AGENTS.md's "Prompt queue" section) re-enters the loop with a fresh +// see docs/session-storage-and-queue.md) re-enters the loop with a fresh // PursueGoal call the next time any ordinary prompt turn completes — no new // timer, no new resume machinery, the same mechanism an ordinary idle goal // already relies on. @@ -1916,7 +1917,7 @@ func (s *Session) lastMessageID() string { // touch the durable session log. Prompt's own append already persisted a // recMessage record for the interrupted-turn trio via appendWithUsage // before promptTurnWithRetry ever saw the failure — that record is on disk, -// and the append-only session log (see AGENTS.md's core invariants) has no +// and the append-only session log (see docs/session-storage-and-queue.md) has no // sanctioned mechanism this package can reach for retracting or amending an // already-journaled record without engine.go/store.go changes, which // docs/design/goal-retry-directive-reuse.md §4 rejects outright (a new @@ -2447,8 +2448,8 @@ func (s *Session) runEvaluator(ctx context.Context, condition string, evaluator // a routing/cache-affinity hint (see provider.Request.SessionKey). SessionKey: s.ID, // The evaluator is a classifier, not a reasoning task: it always - // pins EffortOff, never the session's own level (see AGENTS.md's - // "Goal loop" section). Since a7c5cce, EffortOff sends the literal + // pins EffortOff, never the session's own level (see docs/goal-loop.md). + // Since a7c5cce, EffortOff sends the literal // "off" on openaicompat and no thinking block on anthropic — both // routes now spend none of the evaluator's MaxTokens 256 budget on // reasoning. openai Responses is a known residual: reasoningEffort diff --git a/engine/instructions_outline.go b/engine/instructions_outline.go index a8a18ca7..8abe2004 100644 --- a/engine/instructions_outline.go +++ b/engine/instructions_outline.go @@ -69,9 +69,9 @@ type section struct { // line and byte accounting stays complete. // // The scan tracks fenced code blocks: a "# ..." line inside a ``` fence is -// body text, never a heading. This repository's own AGENTS.md holds bash -// blocks whose comments start with '#', and reading one as a section would -// advertise a range that points at a shell comment. +// body text, never a heading. The regression fixture puts a shell comment in +// such a block; reading it as a section would advertise a range that points at +// the comment. func scanSections(data []byte) []section { lines := strings.Split(string(data), "\n") // A trailing newline produces one empty final element; it is not a line. @@ -113,10 +113,9 @@ func scanSections(data []byte) []section { // Both rules are load-bearing for an instruction file that documents // Markdown: such a file wraps a three-backtick example in a four-backtick // fence, and a naive toggle closes the OUTER fence on the inner one, then -// reads the rest of the document as headings. This repository's own AGENTS.md -// does not currently hold such a fence — it holds bash blocks, which the -// simpler rule already handles — so this is hardening against a shape any -// documentation-heavy project produces, not a fix for an observed break. +// reads the rest of the document as headings. This is hardening against a +// shape any documentation-heavy project can produce, not a fix for an +// observed repository break. type fenceState struct { open bool char byte diff --git a/engine/mcp_lazy.go b/engine/mcp_lazy.go index cfe2d4d5..5278825a 100644 --- a/engine/mcp_lazy.go +++ b/engine/mcp_lazy.go @@ -28,8 +28,8 @@ // // Two properties this file must not break: // -// - The tools array is byte-stable across requests (see AGENTS.md and -// Session.toolDefs). The partition preserves the registry's order, and +// - The tools array is byte-stable across requests (see +// docs/mcp-tool-loading.md and Session.toolDefs). The partition preserves the registry's order, and // the catalog listing is sorted by full tool name -- by THIS file, not // inherited from the registry -- so identical state always renders // identical bytes. diff --git a/engine/mcp_lazy_test.go b/engine/mcp_lazy_test.go index 8182a2ed..049de616 100644 --- a/engine/mcp_lazy_test.go +++ b/engine/mcp_lazy_test.go @@ -328,8 +328,8 @@ func TestMarkMCPToolsSelectedRejectsMalformedNames(t *testing.T) { } // TestToolDefsByteStableUnderDeferral is the prompt-cache property, asserted -// on BYTES rather than membership (see AGENTS.md, "The tool array is -// byte-stable across requests"): repeated builds that change no selection +// on BYTES rather than membership (see docs/mcp-tool-loading.md, "The tool +// array is byte-stable across requests"): repeated builds that change no selection // must serialize identically, and one selection must change the array // exactly once and then hold still again. func TestToolDefsByteStableUnderDeferral(t *testing.T) { diff --git a/engine/prompt_retry_test.go b/engine/prompt_retry_test.go index 7d8a3557..6c354062 100644 --- a/engine/prompt_retry_test.go +++ b/engine/prompt_retry_test.go @@ -514,9 +514,8 @@ func emptyMaxTokensTurnUsage(usage provider.Usage) []provider.Event { // not a provider failure — the call ran to completion and billed real // tokens (a full input prefill plus the max_tokens output ceiling) — so // those tokens must still land in cumulative Session.Usage(), exactly like -// the #136 empty-compaction-summary precedent AGENTS.md documents ("the -// call's real usage is still accumulated into cumulative Usage() ... it was -// a billed call even though it produced nothing"). Dropping it silently +// the discarded-empty-attempt contract in docs/engine-request-cycle.md (the +// call's real usage still accumulates because it was billed). Dropping it silently // would undercount GET /session by the full cost of every discarded // attempt. // diff --git a/engine/searchtools.go b/engine/searchtools.go index e96d2b2b..6d774df8 100644 --- a/engine/searchtools.go +++ b/engine/searchtools.go @@ -53,7 +53,7 @@ const ( // against a separately captured os.Stat size a concurrently growing // file (a live log, a build artifact still being written) could // outrun between the stat and a later read — the exact TOCTOU shape - // AGENTS.md's read_file guidance forbids, and an earlier revision of + // docs/engine-request-cycle.md's read_file guidance forbids, and an earlier revision of // this file used. A file over the cap is skipped entirely — grep's // job is finding SMALL, textual matches, not partially searching one // giant file, so skipping is the right trade, not a truncated read. diff --git a/engine/searchtools_test.go b/engine/searchtools_test.go index 00e9b1aa..0e696304 100644 --- a/engine/searchtools_test.go +++ b/engine/searchtools_test.go @@ -232,7 +232,8 @@ func TestGrepToolSkipsBinaryFiles(t *testing.T) { // file). The oversized file is skipped entirely, bounded via // io.LimitReader(f, maxGrepFileBytes+1) over one open handle (not a // separately captured os.Stat size — an earlier revision used exactly -// that TOCTOU-prone shape, which AGENTS.md's read_file guidance forbids, +// that TOCTOU-prone shape, which docs/engine-request-cycle.md's read_file +// guidance forbids, // and a second review round caught it); a small file alongside it still // matches normally. func TestGrepToolSkipsOversizedFiles(t *testing.T) { diff --git a/engine/session_info_test.go b/engine/session_info_test.go index 42507ee8..c9ff874b 100644 --- a/engine/session_info_test.go +++ b/engine/session_info_test.go @@ -212,8 +212,7 @@ func TestSessionInfoReportsEffortAfterSetEffort(t *testing.T) { } // rawSessionInfoPlugins re-marshals just the plugins field so the test can -// assert the empty case serializes as [] (not null) — the NoToolOutputText / -// empty-slice trap AGENTS.md warns about. +// assert the empty case serializes as [] (not null). func rawSessionInfoPlugins(t *testing.T, info decodedSessionInfo) string { t.Helper() b, err := json.Marshal(info.Plugins) diff --git a/engine/session_key_test.go b/engine/session_key_test.go index 8e57537e..af0f9fee 100644 --- a/engine/session_key_test.go +++ b/engine/session_key_test.go @@ -14,8 +14,8 @@ import ( // builds, so an adapter that forwards it (openaicompat's "user" field) can // pin a session's requests to one provider replica for prompt-cache // affinity. This drives the same Session.Prompt entry point production -// calls, not a hand-built Request (see AGENTS.md, "Verification drives the -// production entry point"). +// calls, not a hand-built Request (see the root AGENTS.md "Testing" +// section). func TestPromptSetsSessionKeyOnRequest(t *testing.T) { prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ asstTurn(provider.StopEndTurn, &message.Text{Text: "done"}), diff --git a/engine/session_manager.go b/engine/session_manager.go index 6ebc09ff..29347408 100644 --- a/engine/session_manager.go +++ b/engine/session_manager.go @@ -96,7 +96,8 @@ var ( // A live review finding: an earlier version of this gate compared // only input+output against the budget, while usageByRoot itself // already accumulated all four fields — a cache-heavy tree (the - // openaicompat/Fireworks and anthropic routes AGENTS.md calls out, + // openaicompat/Fireworks and anthropic routes described in + // docs/models-and-providers.md, // where a large prompt resent every turn reads mostly from cache) // could keep spawning children well past the operator's real // intended ceiling, because the largest component of its actual diff --git a/engine/session_manager_test.go b/engine/session_manager_test.go index 41afcfa3..195e8d76 100644 --- a/engine/session_manager_test.go +++ b/engine/session_manager_test.go @@ -1896,8 +1896,8 @@ func TestSpawnBudgetExceeded(t *testing.T) { // InputTokens+OutputTokens against SetMaxTreeTokens, while usageByRoot // itself already accumulated all four provider.Usage fields — a // cache-heavy child (a large prompt resent every turn, reading mostly -// from cache, the shape AGENTS.md calls out for the openaicompat/ -// Fireworks and anthropic routes) could spend well past the operator's +// from cache, the shape docs/models-and-providers.md describes for the +// openaicompat/Fireworks and anthropic routes) could spend well past the operator's // real intended ceiling with the gate never noticing, because cache // read/write tokens were silently exempt from the very check meant to // bound them. Gives a child a small input+output total (20) but a large diff --git a/engine/stream_watchdog.go b/engine/stream_watchdog.go index bb67e600..d294d261 100644 --- a/engine/stream_watchdog.go +++ b/engine/stream_watchdog.go @@ -136,8 +136,8 @@ func (w *streamWatchdog) stop() { // context.Canceled arriving in the same instant the timer fires is a real // provider failure that must keep its own identity — most pointedly a // context-overflow classification, whose deliberate clear-don't-park -// semantics (see AGENTS.md) would otherwise be laundered into a retryable -// truncation. +// semantics (see docs/goal-loop.md) would otherwise be laundered into a +// retryable truncation. func (w *streamWatchdog) explain(err error) error { if w == nil || err == nil || !w.fired.Load() || !errors.Is(err, context.Canceled) { return err diff --git a/engine/toolexec.go b/engine/toolexec.go index f61ca23b..b3a0e293 100644 --- a/engine/toolexec.go +++ b/engine/toolexec.go @@ -219,8 +219,9 @@ // // ctx cancellation (an aborted turn) cancels every in-flight call's own // context, but every call still yields exactly one ToolResult — the -// NEP-5272 invariant (AGENTS.md's "empty tool result" rule, and the -// orphan tool_use rule generally) holds regardless of how the batch ends. +// NEP-5272 invariant (docs/engine-request-cycle.md's empty-tool-result rule, +// and the orphan tool_use rule generally) holds regardless of how the batch +// ends. // A call whose ctx is already cancelled before it starts still runs // (executeTool/runToolCall are unchanged; a cancelled context is not a // license to skip a call, only a signal the call's own logic may check — @@ -358,8 +359,8 @@ func (s *Session) runToolBatch(ctx context.Context, asst *message.Message) messa // tool did not return a normal result: the call was refused after the // turn was canceled, no execution path filled its slot, or the tool // panicked. All three exist to hold the pairing invariant: a tool_use -// block with no tool_result wedges a session permanently (AGENTS.md, -// NEP-5272). +// block with no tool_result wedges a session permanently (see +// docs/engine-request-cycle.md, NEP-5272). const ( toolCallCanceledText = "tool call not started: the turn was canceled" toolCallNoResultText = "tool call produced no result" diff --git a/engine/toolexec_test.go b/engine/toolexec_test.go index 3094f794..a3e7e0bf 100644 --- a/engine/toolexec_test.go +++ b/engine/toolexec_test.go @@ -588,8 +588,8 @@ func TestBatchPartialFailureLetsSiblingsFinish(t *testing.T) { } // TestBatchCancellationStillYieldsOneResultPerCall is the orphan-pairing -// guard (AGENTS.md, NEP-5272): a tool_use block with no tool_result wedges -// a session forever, so an aborted turn must still produce exactly one +// guard (docs/engine-request-cycle.md, NEP-5272): a tool_use block with no +// tool_result wedges a session forever, so an aborted turn must still produce exactly one // result per call — never fewer, and never a duplicate. // // Two shapes. In "already canceled" no call runs at all. In "canceled in diff --git a/engine/toolresult.go b/engine/toolresult.go index 00853d2f..afb82f19 100644 --- a/engine/toolresult.go +++ b/engine/toolresult.go @@ -15,8 +15,8 @@ // about how the engine BUILT the result, never a new part kind, never a new // wire shape. // -// It is also why retention does not violate AGENTS.md's additive-only -// live-repair rule. That rule governs a repair that runs over live or +// It is also why retention does not violate docs/engine-request-cycle.md's +// additive-only live-repair rule. That rule governs a repair that runs over live or // persisted history (ResolveOrphanToolCalls). Retention is not a repair: it // runs once, on a message that does not yet exist in history, at the one // point the engine already decides what the ToolResult's content is. diff --git a/imageclamp/AGENTS.md b/imageclamp/AGENTS.md new file mode 100644 index 00000000..44b3fd61 --- /dev/null +++ b/imageclamp/AGENTS.md @@ -0,0 +1,30 @@ +# Image clamp instructions + +These rules apply to `imageclamp/`. Harness does not merge ancestor files. If +root guidance is not active, locate the Git root and read +`/AGENTS.md`. Resolve repository paths from that root. Read +`provider/AGENTS.md` before changing adapter limits. + +## Transcode-time normalization + +`Clamp` repairs a throwaway request. It must not mutate canonical history. +Return the original slice without allocation when no image changes. + +Keep output deterministic so repeated transcodes remain prompt-cache stable. + +## Bounds + +Enforce both dimension and encoded-byte limits. Reject absurd dimensions or +pixel counts before a full decode. Use the text placeholder when safe decode or +useful downscale is impossible. + +Do not remove the decode-memory guards. Do not rewrite the durable source blob. + +The caller decides whether to recurse into tool results. Preserve adapter +differences documented in `provider/AGENTS.md`. + +## Tests + +Use small generated fixtures when practical. Cover copy-on-write behavior, +deterministic bytes, dimension limits, byte limits, many-image thresholds, and +placeholder fallbacks. diff --git a/imageclamp/CLAUDE.md b/imageclamp/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/imageclamp/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/mcp/AGENTS.md b/mcp/AGENTS.md new file mode 100644 index 00000000..b99dde9e --- /dev/null +++ b/mcp/AGENTS.md @@ -0,0 +1,37 @@ +# MCP transport instructions + +These rules apply to `mcp/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. +Read `engine/AGENTS.md` for connection policy and lazy schema loading. + +## Package boundary + +Keep this package independent from engine, server, and command integration. It +implements the MCP client protocol and transports only. + +Keep JSON-RPC framing dependency-free. Preserve request ID correlation and +notification handling. + +## Transports + +- Stdio uses one JSON-RPC message per line over the child process streams. +- Streamable HTTP accepts a JSON response or SSE response. +- Preserve `MCP-Session-Id` continuity. +- Preserve paginated `tools/list` cursors. +- Keep static request headers on every HTTP call. + +Do not add OAuth, client-served capabilities, legacy HTTP+SSE fallback, or +other MCP feature families without an explicit scope change. + +## Content + +Preserve text, image, audio, resource-link, embedded-resource, and `isError` +tool-result fields. Do not collapse structured content into text inside this +package. + +## Tests + +Use `net.Pipe` for protocol framing and `httptest` for HTTP. Test split frames, +multiple SSE events, pagination, cancellation, and malformed responses. Do not +call a remote MCP server in the unit suite. diff --git a/mcp/CLAUDE.md b/mcp/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/mcp/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/message/AGENTS.md b/message/AGENTS.md new file mode 100644 index 00000000..40a1f5fc --- /dev/null +++ b/message/AGENTS.md @@ -0,0 +1,77 @@ +# Message instructions + +These rules apply to `message/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. Read `provider/AGENTS.md` for wire +adapters and `engine/AGENTS.md` for live history. + +## Canonical representation + +Canonical messages are the durable representation. Do not add provider wire +objects to the message union. + +Provider-specific opaque data uses a provider-family tag. The same family can +replay it. A different family drops it at transcode time. + +Keep tool-call IDs provider-neutral in history. Adapters own deterministic +wire-ID mapping. + +## Empty tool results + +An empty tool result must not serialize as `null`. + +- Keep `NoToolOutputText` as the canonical fallback. +- Keep `ToolResult.SafeContent` as the read path for consumers. +- Keep `ToolResult.MarshalJSON` safe for direct serialization. +- Add a regression test for each new serializer or transcoder path. + +## Wire normalization + +`ResolveOrphanToolCalls` runs on live or persisted state. It is additive-only. +It may add a synthetic result. It must not delete, move, or reorder a real part. + +`NormalizeForWire` runs on a throwaway request. It may relocate data to meet +a provider contract. It must never delete a real `ToolResult`. + +Keep support for these wire-only shapes: + +1. Duplicate tool-call IDs in one assistant message. +2. A tool call in a non-assistant message. +3. A tool result before its call. +4. A same-role run between a call and its result. + +Keep relocation within `computeRelocationBarrier`. Derive +`wire_oracle_test.go` from the provider contract, not either implementation. + +Read the "Wire normalization" section in +`docs/engine-request-cycle.md` before changing this logic. + +## EngineContext trust boundary + +`EngineContext` is a structured part that only the engine creates. Keep it +distinct from `Text`. + +`RenderEngineContext` wraps trusted context with the sentinel. +`NeutralizeEngineContextSentinel` defangs the same bytes in user text. Only a +real `EngineContext` may emit the trusted sentinel on the wire. + +Keep the part in canonical JSON for runtime round trips. Do not turn it into a +persisted ambient-status mechanism. + +## Normalization and mutation + +When `Message.Normalize` sanitizes invalid tool arguments, preserve the +existing part pointer when callers rely on in-place cleanup before request +assembly. + +Do not use a cleansing marshal as proof that resident state was clean. A +marshal can hide invalid in-memory input. + +## Tests + +- Use round-trip tests for every part variant. +- Use property tests for repair invariants. +- Assert that real tool output is never lost. +- Assert both missing and surplus tool results. +- Drive the real `LoadSession` or provider entry point when the defect occurs + there. diff --git a/message/CLAUDE.md b/message/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/message/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/modelmeta/AGENTS.md b/modelmeta/AGENTS.md new file mode 100644 index 00000000..d3477bfc --- /dev/null +++ b/modelmeta/AGENTS.md @@ -0,0 +1,38 @@ +# Model metadata instructions + +These rules apply to `modelmeta/`. Harness does not merge ancestor files. If +root guidance is not active, locate the Git root and read +`/AGENTS.md`. Resolve repository paths from that root. Read +`engine/AGENTS.md` before changing context-window policy. + +## Static metadata + +Keep model metadata static and deterministic. This package must not perform a +network request or refresh in the background. + +Curate context-window values from the documented source. Keep zero unavailable +for non-chat models because zero also means unknown to callers. + +## Lookup + +Keep provider-family matching explicit. Preserve documented handling for dated +model variants and aliases. An unknown model returns no window; the engine owns +the refusal or opt-out policy. + +Do not add capability guesses from model-name substrings unless a design and +tests define the contract. + +## Server-side tool search + +`SupportsToolSearch` uses an explicit first-party Anthropic allowlist. Return +false for other provider families and for Bedrock-style Anthropic refs. An +unknown ref must keep the portable client-side search path instead of emitting +a provider tool that the route can reject. + +Keep its Bifrost namespace stripping aligned with context-window lookup. + +## Tests + +Test exact known refs, supported variant patterns, near misses, unknown +families, and tool-search refusals. A metadata update must include the source +and lookup regression tests. diff --git a/modelmeta/CLAUDE.md b/modelmeta/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/modelmeta/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/plugin/AGENTS.md b/plugin/AGENTS.md new file mode 100644 index 00000000..08bcf116 --- /dev/null +++ b/plugin/AGENTS.md @@ -0,0 +1,66 @@ +# Plugin instructions + +These rules apply to `plugin/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. + +Read `plugin/PROTOCOL.md` before a wire or hook change. Read +`docs/plugins-and-protocols.md` for extended rationale. + +## Process model + +A plugin is a separate process that speaks versioned JSON-RPC over stdio. + +- `harness plugin probe` runs a bounded manifest probe and caches it with + executable identity and plugin-spec identity. +- Run and serve startup trust a matching cached manifest. A missing or stale + entry performs one bounded probe before host construction. +- Spawn a plugin on its first hook dispatch or tool call. +- Keep one warm process for later calls. +- Bound every synchronous dispatch with a deadline. +- A hung plugin must not block unrelated sessions or status reads. + +## Manifest and visibility + +A configured plugin appears in `Host.Plugins()` before it starts. Report +manifest tools and hooks with live spawn state. + +Keep status reads lock-free with respect to a dial or handshake. A plugin that +dies after startup becomes `errored`. + +## Hook protocol v1 + +| Hook | Contract | +|---|---| +| `event` | Asynchronous, batched, fire-and-forget | +| `chat.params` | Synchronous request-parameter mutation | +| `chat.message` | Synchronous message mutation before logging | +| `system.transform` | Synchronous additive system segments | +| `shell.env` | Synchronous command environment mutation | +| `tool.execute.before` | Synchronous argument rewrite or deny | +| `tool.execute.after` | Synchronous result rewrite | + +Run synchronous hooks in configured plugin order. Each plugin sees prior +mutations. Keep `system.transform` after provider resolution. + +## Plugin tools and client API + +Tool definitions come from the cached manifest. Tool execution uses RPC. + +Plugins can call `Session.Messages`, `MCP.Call`, `Generate`, and the +configured HTTP client. Plugins must not carry provider API keys. + +Do not add message-delta events without a throttling and backpressure design. + +## Settled protocol exclusions + +Do not add permission hooks or auth hooks. Network-layer deployment controls +own credentials. + +Do not add a JavaScript runtime or opencode compatibility shim. + +## Tests + +Use `net.Pipe` for framing and protocol tests. Use `testing/synctest` for +queue and deadline behavior. Do not spawn a real fixture unless process +lifecycle is the subject of the test. diff --git a/plugin/CLAUDE.md b/plugin/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/plugin/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/plugin/PROTOCOL.md b/plugin/PROTOCOL.md index ba2ed3f1..1e51151f 100644 --- a/plugin/PROTOCOL.md +++ b/plugin/PROTOCOL.md @@ -11,9 +11,9 @@ own requests independently. ## Lifecycle -1. Harness spawns the plugin process (lazily, on first hook dispatch or tool - call — never at startup; manifests are cached at install time, keyed by - binary hash). +1. Harness spawns the long-lived plugin process lazily on the first hook + dispatch or tool call. Before host construction, a matching cached manifest + is reused; a missing or stale entry gets one bounded probe. 2. Harness → `initialize` (request) with `InitializeParams`. Plugin responds with its `Manifest`. A protocol-version mismatch is an initialize error. The harness verifies the live manifest matches the cached one. @@ -30,15 +30,15 @@ API to reach. See "Trust model" below. ### Trust model Plugins are **trusted local processes**, not third parties: the harness -spawns them itself, over stdio, from a manifest cached at install time -(binary-hash keyed). `run_token` is therefore the exact same bearer token +spawns them itself over stdio from a probed and validated manifest. +`run_token` is therefore the exact same bearer token the orchestrator holds for this run — not a separate, narrower-scoped credential minted per plugin. A plugin that can reach `serve_url` can do anything the orchestrator can do over the HTTP API for this process (create sessions, prompt, abort, read any session it owns). This mirrors the `shell.env` hook, which already hands plugins a seat at env-var injection -for tool commands, and the "no auth hooks" decision in AGENTS.md (credential -scoping happens at the network layer, not inside the harness). A plugin +for tool commands. Credential scoping happens at the network layer, not inside +the harness. A plugin that should not have this reach simply should not be installed. A plugin that fails to start or errors on a dispatch is skipped — **hook @@ -90,10 +90,10 @@ dispatch path and type-checks end to end, but is not implemented yet: provider-layer routing for plugin-initiated LLM calls is a separate PR, and it returns a clear RPC error until then. -Any language implementing this protocol (the Go SDK in this package, a -future TypeScript SDK, etc.) gets `serve_url`/`run_token` for free once it -decodes `InitializeParams` — no protocol-version bump was needed since they -are additive, optional fields (see Versioning below). +Any language implementing this protocol (the Go helpers in this package, the +TypeScript SDK under `sdk/typescript`, etc.) gets `serve_url`/`run_token` for +free once it decodes `InitializeParams` — no protocol-version bump was needed +since they are additive, optional fields (see Versioning below). ## Chaining semantics @@ -179,12 +179,10 @@ The rules, for a plugin in any language: Concurrency is not new to this version. `plugin.Host` is a box-scoped singleton shared by every session, so two sessions have always been able to dispatch a hook to one plugin at the same time. Parallel tool execution in -the engine will add the same concurrency WITHIN one session: the tool calls -of one assistant message will run as a batch, so their -`tool.execute.before` and `tool.execute.after` dispatches will overlap. -Today the engine still runs those calls one at a time -(`Session.runToolCalls`), so a plugin sees within-session overlap only -after that change lands. +the engine also creates concurrency WITHIN one session: the tool calls of one +assistant message run as a bounded batch, so their `tool.execute.before` and +`tool.execute.after` dispatches can overlap. A plugin must therefore tolerate +overlap both across sessions and across independent calls in one session. This section documents an existing wire property. It changes no payload shape, so `ProtocolVersion` stays 1. @@ -204,5 +202,5 @@ new optional fields) bump the minor behavior but not the version — unknown hooks are simply never subscribed, and unknown fields are ignored. Breaking changes to existing payload shapes bump the version. -Deliberately absent from this protocol, by design (see AGENTS.md): permission -hooks, plan mode, and auth hooks. +Deliberately absent from this protocol by design: permission hooks, plan mode, +and auth hooks. diff --git a/plugin/hooks.go b/plugin/hooks.go index dcf43bac..438fe253 100644 --- a/plugin/hooks.go +++ b/plugin/hooks.go @@ -33,9 +33,9 @@ const ( func (h Hook) method() string { return hookMethodPrefix + string(h) } // Manifest describes a plugin: what it's called, which hooks it subscribes -// to, and which tools it provides. The harness caches manifests at install -// time (keyed by binary hash) so that routing is known at startup without -// spawning anything. +// to, and which tools it provides. The harness caches probed manifests with +// executable and plugin-spec identity so routing is known before the +// long-lived plugin process starts. type Manifest struct { Name string `json:"name"` Version string `json:"version,omitempty"` diff --git a/plugin/host.go b/plugin/host.go index 6bedb161..0fb95e11 100644 --- a/plugin/host.go +++ b/plugin/host.go @@ -36,8 +36,8 @@ var ErrGenerateNotImplemented = errors.New("plugin: generate not implemented yet // Spec configures one plugin for a Host. // -// Manifest comes from the install-time cache (see Probe), which is what lets -// the Host route hooks and advertise tools without spawning anything: a +// Manifest comes from the probe cache (see ProbeSpec), which lets the Host +// route hooks and advertise tools without starting the long-lived process. A // plugin process starts on its first hook dispatch or tool call, then stays // warm. type Spec struct { @@ -783,9 +783,9 @@ func NewTestSpec(name string, hooks *Hooks) Spec { // ProbeSpec spawns a plugin binary using the full spec — command, Env, Dir, // and Config — performs the initialize handshake, and returns its manifest. -// This is the install-time step ("harness plugin install") that populates -// the manifest cache; the cache should be keyed by binary hash so a changed -// binary is re-probed. +// This is the bounded probe used by `harness plugin probe` and by a run/serve +// cache miss. The command layer records executable identity and plugin-spec +// identity so a changed binary or spec is re-probed. // // Probing with the full spec (rather than the bare command — see Probe) // matters: a plugin can behave differently (even report a different diff --git a/process/AGENTS.md b/process/AGENTS.md new file mode 100644 index 00000000..a46b39c3 --- /dev/null +++ b/process/AGENTS.md @@ -0,0 +1,30 @@ +# Managed process instructions + +These rules apply to `process/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. Read +`docs/design/managed-processes.md` before a lifecycle change. + +## Manager lifecycle + +The process manager is box-scoped and shared across sessions. + +- Preserve the starting, ready, running, exited, and stopped states. +- Detect child exit asynchronously. +- Stop the Unix process group, not only its leader. +- Keep runtime declarations in memory. Do not write them into project config. +- Keep logs under the configured work directory. +- A restarted name is a new instance. `WaitExit` must return the terminal state + of the instance that the caller observed. + +## Status and engine integration + +The engine exposes process state through a runtime-only `EngineContext` part. +Do not make the process package depend on `engine` or `message` to produce it. + +## Tests + +This package tests real subprocess machinery, so it can use the root +cross-process timing exception. Route polling through `internal/testpoll`. +Never add an inline sleep loop. Use in-process signals when a state is already +observable without crossing the OS process boundary. diff --git a/process/CLAUDE.md b/process/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/process/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/provider/AGENTS.md b/provider/AGENTS.md new file mode 100644 index 00000000..4e63bfc3 --- /dev/null +++ b/provider/AGENTS.md @@ -0,0 +1,116 @@ +# Provider instructions + +These rules apply to `provider/` and its adapters. Harness does not merge +ancestor files. If root guidance is not active, locate the Git root and read +`/AGENTS.md`. Resolve repository paths from that root. Read +`message/AGENTS.md` for canonical data rules. + +## Adapter boundary + +Each adapter receives a canonical `provider.Request` and builds a new wire +request. Never store provider wire state in session history. + +Every transcoder must: + +- Call `message.NormalizeForWire`. +- Read tool output through `ToolResult.SafeContent`. +- Apply `imageclamp.Clamp` with adapter-specific limits. +- Map internal tool-call IDs deterministically. +- Replay opaque `ProviderData` only for the matching family. +- Preserve request order after same-role merging. +- Keep prompt-cache markers out of canonical history. + +Use golden JSON tests for wire shape and ordering. + +## Error classification + +Classify errors with typed `provider.Error` values. Engine retry code must not +match provider error text. + +Mark malformed requests permanent. Keep context overflow separate. Use text +matching only inside a provider parser for a documented provider shape, such as +context overflow or account exhaustion. + +A stream that ends without its terminal event is +`RetryableStreamTruncated`. Do not report it as an ordinary cancellation. + +## Images in tool results + +Anthropic can recurse into tool-result blobs. Native OpenAI and +OpenAI-compatible adapters replace those blobs with an omission note. Preserve +this adapter difference until the wire contracts change. + +Do not add a static model-name vision list. The repository has no complete +capability signal. + +## Reasoning effort + +`message.EffortUnset` means "send no control." It is not equal to +`message.EffortOff`. + +- Anthropic maps enabled levels to thinking budgets. It raises `max_tokens` + above the budget and drops temperature and top-p. +- Native OpenAI Responses maps enabled levels into `reasoning.effort`. +- OpenAI-compatible chat sends the literal `"off"` for `EffortOff`. + Gateways can reason by default when the field is absent. + +Reasoning-history stripping is intentionally asymmetric: + +- Anthropic strips stored thinking when reasoning is not enabled. +- Native OpenAI strips stored reasoning only for explicit `EffortOff`. +- Native OpenAI must replay encrypted reasoning on `EffortUnset` for + stateless multi-turn tool use. + +Do not replace this with one shared `!Reasoning()` condition. + +Read `docs/models-and-providers.md` before changing effort or +compaction request behavior. + +## Session affinity + +`Request.SessionKey` is the stable session routing hint. + +- OpenAI-compatible sends `user` and, unless disabled, `prompt_cache_key`. +- Native OpenAI Responses sends `prompt_cache_key`. +- Anthropic ignores `SessionKey` and uses explicit cache markers. + +Omit empty keys. Do not replace the gateway `user` field with the native +OpenAI field. + +## Anthropic cache TTL + +Anthropic uses two cache breakpoints. The default TTL is one hour. + +- `"1h"` adds the TTL and the required beta header. +- `"5m"` restores the short cache shape without the beta header. +- Reject unknown values. +- Reject `cache_ttl` on an entry that does not build the native Anthropic + adapter. + +## Native OpenAI Responses endpoints + +A configured provider with type `"openai"` builds the Responses adapter under +that provider-map key. Require `base_url` for a non-built-in family. + +Keep `Client.Family` equal to the configured family. The family is both the +router name and the opaque-data isolation boundary. Do not replay encrypted +reasoning between two Responses endpoints. + +`responses_path` is valid only for an entry that builds this adapter. + +## Stable request bytes + +Prompt caches depend on stable bytes, not set equality. Keep tool order, system +segment order, message merge behavior, and JSON field behavior deterministic. + +A test for cache-sensitive data must compare ordered wire bytes or ordered +decoded objects. A membership assertion is insufficient. + +## Tests + +- Test shared behavior through each affected real adapter. +- Use provider-contract oracles that do not call production normalization. +- Cover malformed HTTP responses and mid-stream errors. +- Assert unknown optional fields are omitted, not emitted as empty strings, + when the contract requires omission. +- Never make a live provider call in the ordinary unit suite. diff --git a/provider/CLAUDE.md b/provider/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/provider/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/provider/openai/session_affinity_test.go b/provider/openai/session_affinity_test.go index ff827f7e..06935040 100644 --- a/provider/openai/session_affinity_test.go +++ b/provider/openai/session_affinity_test.go @@ -10,8 +10,8 @@ import ( // TestSessionKeySetsPromptCacheKey: a non-empty Request.SessionKey sets the // Responses API top-level "prompt_cache_key" field to the same string, for -// per-replica prompt-cache routing affinity (see AGENTS.md, "Session -// affinity" section). This adapter uses prompt_cache_key, the Responses +// per-replica prompt-cache routing affinity (see docs/models-and-providers.md, +// "Session affinity" section). This adapter uses prompt_cache_key, the Responses // API's own routing hint — NOT the "user" field the openaicompat adapter // uses for a generic chat-completions gateway. func TestSessionKeySetsPromptCacheKey(t *testing.T) { diff --git a/provider/openai/transcode.go b/provider/openai/transcode.go index a50c084b..538510f9 100644 --- a/provider/openai/transcode.go +++ b/provider/openai/transcode.go @@ -50,7 +50,8 @@ type apiRequest struct { // so the model runs its own default. Reasoning *apiReasoning `json:"reasoning,omitempty"` // PromptCacheKey is the Responses API's documented routing/cache-affinity - // hint, set from Request.SessionKey (see AGENTS.md, "Session affinity" + // hint, set from Request.SessionKey (see docs/models-and-providers.md, + // "Session affinity" // section, for the Fireworks/Bifrost measured evidence this mechanism is // modeled on). OpenAI combines it with the prefix hash to raise the // chance repeat requests land on the same cache-holding backend. Empty diff --git a/provider/openaicompat/session_affinity_test.go b/provider/openaicompat/session_affinity_test.go index 111da429..844a50e0 100644 --- a/provider/openaicompat/session_affinity_test.go +++ b/provider/openaicompat/session_affinity_test.go @@ -10,8 +10,8 @@ import ( // TestSessionKeySetsUserField: a non-empty Request.SessionKey sets the // top-level "user" field to the same string, for Fireworks-style -// per-replica prompt-cache affinity through a gateway (see AGENTS.md, -// "Session affinity" section). +// per-replica prompt-cache affinity through a gateway (see +// docs/models-and-providers.md, "Session affinity" section). func TestSessionKeySetsUserField(t *testing.T) { req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) req.SessionKey = "sess_abc" diff --git a/provider/openaicompat/transcode.go b/provider/openaicompat/transcode.go index 5497f7f3..ba368585 100644 --- a/provider/openaicompat/transcode.go +++ b/provider/openaicompat/transcode.go @@ -47,7 +47,8 @@ type apiRequest struct { // gateway (Bifrost) maps it to the upstream provider's own thinking knob. ReasoningEffort string `json:"reasoning_effort,omitempty"` // User is the OpenAI-compatible top-level routing/cache-affinity hint, - // set from Request.SessionKey (see AGENTS.md, "Session affinity" + // set from Request.SessionKey (see docs/models-and-providers.md, + // "Session affinity" // section, for the Fireworks per-replica prompt-cache evidence). Empty // sends no field. User string `json:"user,omitempty"` diff --git a/sdk/AGENTS.md b/sdk/AGENTS.md new file mode 100644 index 00000000..377d31f8 --- /dev/null +++ b/sdk/AGENTS.md @@ -0,0 +1,32 @@ +# SDK instructions + +These rules apply to `sdk/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths and commands from that root. + +Read `plugin/AGENTS.md` and `plugin/PROTOCOL.md` before an SDK protocol change. + +## Protocol parity + +The TypeScript SDK and Go plugin host must speak the same versioned NDJSON +protocol. Keep method names, field names, hook behavior, tool results, and +shutdown behavior in parity. + +Do not add an SDK-only wire extension. Change the protocol document and Go +implementation in the same change. + +## TypeScript SDK + +- Keep `sdk/typescript/harness-plugin.mjs` zero-dependency ESM. +- Use Node built-ins only. +- Keep stdout exclusive to protocol frames. Send logs to stderr. +- Preserve snake_case wire fields. +- Derive manifest hooks and tools from the supplied definition. +- Keep Node 18 compatibility unless the README changes the support floor. + +Run: + +```bash +node --test sdk/typescript/test/*.test.mjs +go test -race ./plugin/... +``` diff --git a/sdk/CLAUDE.md b/sdk/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/sdk/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/server/AGENTS.md b/server/AGENTS.md new file mode 100644 index 00000000..bb2bdc7e --- /dev/null +++ b/server/AGENTS.md @@ -0,0 +1,137 @@ +# Server instructions + +These rules apply to `server/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. +Read `engine/AGENTS.md` for session state machines. + +## Layering + +The server exposes the headless engine through HTTP and SSE. It must not import +`tools/*`. The CLI injects embedded tool pages through +`server.Options`. + +Update `server/openapi.yaml` with an API contract change. + +## Live session resolution + +Use `Server.resolveLive` as the single resolution entry point for a live +session, status, or lineage. Read all values from one `liveSession` snapshot. + +Do not hold `server.mu` while you acquire `SessionManager.mu`. Residency is +authoritative for the session's own running state. The manager supplies child +state that residency cannot. + +## Journal, index, and cold reads + +Durable journal records are the source of truth. An index or snapshot is a +cache. + +- Use the session index for cold metadata only when its validation passes. +- Fall back to `LoadSession` for incomplete legacy metadata. +- Keep record folds shared with engine replay. +- Serve parameterized message pages from the durable sequence. +- Keep the unparameterized message endpoint response unchanged. +- Do not synthesize orphan results into a read-only transcript view. + +Read `docs/design/journal-snapshotting.md` and +`docs/session-storage-and-queue.md` before changing these +paths. + +## Prompt queue + +`prompt_async` accepts same-session busy work into a durable FIFO queue. +Another session that owns the workdir still conflicts. + +- Return `started` only for the prompt that received the run slot. +- Do not let a fresh prompt jump ahead of a restored queue head. +- Never auto-dispatch a restored queue during boot. Leave it for the next + natural drain trigger. +- Dispatch queued input before goal auto-arm. +- Keep abort independent from queue clear. +- Keep `POST /enqueue` write-ahead and idempotent. +- Keep `GET /queue` as the reconciliation surface. + +## Goal and turn state + +Map a worker park to a distinct terminal outcome and keep the goal active. +Map context overflow to a clear, not a park. + +A paused goal must not force the composite session idle while a real turn runs. +Activity can re-arm a parked goal only through the existing auto-arm path. +Resume needs no new mechanism: completion of an ordinary prompt uses that +same auto-arm path. + +Log classified reasons. Do not journal raw provider errors or secrets. + +`POST /session/{id}/thinking` parses the effort, accepts an empty string as +`EffortUnset`, and calls `Session.SetEffort` without claiming the run slot. + +## Session lineage and fleet state + +Read `docs/design/fleet-model.md` before changing box identity, session +lineage, revival, or goal pause behavior. + +Merge live and durable child IDs through one de-duplicating path. Resolve a +reaped descendant from durable lineage before you report it missing. + +Use `fail_kind` for machine behavior and bounded, masked `fail_reason` for +operator context. + +Treat `provider_exhausted` as a recoverable account wall. Preserve the child +and guide the parent to resume it. Do not present the condition as a reason to +spawn a replacement. + +## Authentication and browser surfaces + +Fail closed unless the CLI explicitly selects an allowed unauthenticated mode. +Do not infer unauthenticated service from an empty token inside `server.New`. + +Keep monitor HTML unauthenticated because it contains no secret. Keep API +routes under the normal auth policy. Apply CORS only from configured origins. + +## Session monitor + +`GET /monitor` serves bytes supplied through `Options.MonitorPage`. +`GET /{$}` redirects only when that page exists. Do not add a catch-all +route. + +Keep the embedded page's CSP same-origin. Cross-origin monitoring uses a +separately hosted page. + +Read `tools/AGENTS.md` before changing monitor behavior. + +## Serve-mode latency diagnostics + +A diagnostic must be opt-in or threshold-gated. + +- Exclude streaming and long-poll routes from slow-request warnings. +- Log the mux route pattern, never the caller-controlled path. +- Bound and validate `X-Request-Id` before logging it. +- Keep pprof disabled by default and authenticated when enabled. +- Never import `net/http/pprof`. Its `init` mutates the default mux. +- Register the unslashed pprof path explicitly behind auth. +- Keep profile and trace durations bounded. +- Do not warn that a requested profile is a slow request. + +The tests in both `server/` and `cmd/harness/` must prove that the default +mux has no pprof routes. + +## External protocols + +The current implementation exposes Harness HTTP/SSE and MCP-related surfaces. +ACP naming can guide a future adapter, but no ACP adapter currently exists. +Do not describe ACP as implemented until routes and tests exist. + +A2A remains a non-goal. + +If telemetry lands, use OpenTelemetry GenAI semantic conventions and standard +`OTEL_*` environment variables. + +## Concurrency tests + +Use `testing/synctest` for in-process timeout logic. Use notification seams +for status and queue transitions. Do not poll server state or add guessed +deadline wrappers. + +Keep lock-order tests when a change introduces a new lock edge. diff --git a/server/CLAUDE.md b/server/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/server/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/server/goal_worker_park_test.go b/server/goal_worker_park_test.go index 36f02e39..e0df06cf 100644 --- a/server/goal_worker_park_test.go +++ b/server/goal_worker_park_test.go @@ -547,7 +547,7 @@ func TestGoalWorkerParkFreesRunSlotForQueuedPrompt(t *testing.T) { // TestGoalWorkerParkResumesOnNextPromptActivity is invariant 4: after a // worker-park, a plain prompt completing auto-arms the still-active goal // (maybeAutoArmGoal, the pre-existing activity-driven resume mechanism — -// see AGENTS.md's "Resume needs zero new machinery" design note), the +// see server/AGENTS.md's "Goal and turn state" section), the // paused/worker_failure presentation resets, and a now-healthy provider // lets the goal achieve. It also proves the anti-churn property: with an // EMPTY queue, nothing re-arms the goal immediately at park time — runGoal's diff --git a/server/handlers.go b/server/handlers.go index 85a5457a..962f1847 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -602,7 +602,8 @@ func (s *Server) handleHealth(w http.ResponseWriter, _ *http.Request) { // origin-agnostic (it drives arbitrary, operator-added box origins it keeps // no state about); the embedded monitor is the opposite — GET /monitor // exists specifically so a box can offer a same-origin, zero-CORS way to -// watch ITSELF (see AGENTS.md's "Session monitor" section), so this CSP +// watch ITSELF (see docs/development-interfaces.md's "Session monitor" +// section), so this CSP // scopes fetch/EventSource targets to that one origin, blocking it from // being used to reach anywhere else even if the served copy were somehow // pointed at a different base URL. An operator who genuinely wants @@ -1690,7 +1691,8 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // retargeted the session's model even when a DIFFERENT, already-queued // head was what actually got dispatched — contradicting the documented // "a per-request model override is silently dropped when the prompt is - // queued" rule (see AGENTS.md's Prompt queue section and + // queued" rule (see docs/session-storage-and-queue.md's "Prompt queue" + // section and // enqueueOrDispatch's identical rule for the same-session-busy branch). // See TestQueuedArrivalDoesNotRetargetSessionModel. if !body.Model.IsZero() { diff --git a/server/journal.go b/server/journal.go index b4518925..debcf0fb 100644 --- a/server/journal.go +++ b/server/journal.go @@ -558,8 +558,8 @@ func (s *Server) publishGoal(ev engine.Event) { // waiting on external activity to resume it, exactly the "operator // tailing the log concluded the box was dead" scenario, so both log at // WARN with the same turn/attempt/reason/retry-budget shape the - // AGENTS.md brief asks for (mirroring Codex's structured stream-retry - // warn!). set/achieved/cleared are ordinary lifecycle transitions, not + // docs/goal-loop.md's structured-logging contract requires. The + // set/achieved/cleared transitions are ordinary lifecycle transitions, not // failures, so INFO. // // goal.eval also logs at INFO, one line per completed worker turn (the diff --git a/server/openapi.yaml b/server/openapi.yaml index c08214f4..29669a2c 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -2818,7 +2818,7 @@ paths: description: > Present only when this instance was started with the monitor page embedded (cmd/harness's `serve` always does this; see - tools/monitor's AGENTS.md section); a box that omits it 404s here + docs/development-interfaces.md); a box that omits it 404s here exactly as it always has. Serves tools/monitor/index.html verbatim, byte for byte — the same static, build-free, credential-free page that can also be opened via file:// or hosted separately. diff --git a/server/queue_test.go b/server/queue_test.go index b2ad1028..67dd8c74 100644 --- a/server/queue_test.go +++ b/server/queue_test.go @@ -1209,8 +1209,8 @@ func TestIdlePromptWithQueueDispatchDoesNotRaceQueuedCountInResponse(t *testing. // even though a DIFFERENT, already-queued head is what actually gets // dispatched into the run slot -- contradicting the documented "a per-request // model override is silently dropped when the prompt is queued" rule (see -// AGENTS.md's Prompt queue section and enqueueOrDispatch's identical rule for -// the same-session-busy branch). +// docs/session-storage-and-queue.md's "Prompt queue" section and +// enqueueOrDispatch's identical rule for the same-session-busy branch). // // The override here names a provider that is NOT registered // ("bogus/doesnotexist"): if the leak were still present, the dispatched diff --git a/server/server.go b/server/server.go index f6b699dd..7d40debd 100644 --- a/server/server.go +++ b/server/server.go @@ -276,7 +276,7 @@ type Options struct { Processes engine.ProcessRegistry // MonitorPage, when non-nil, is served verbatim at GET /monitor (and // /monitor/) — the single-file board+detail+composer UI documented in - // AGENTS.md's "Session monitor" section, letting a box serve its own + // docs/development-interfaces.md's "Session monitor" section, letting a box serve its own // copy same-origin instead of requiring a separately hosted one. Nil // (the default) registers no route at all: GET /monitor 404s exactly // as it always has, so an existing deployment that never sets this is @@ -491,7 +491,8 @@ type Server struct { // // queueDrainPending is what lets waitSnapshot (wait.go) tell this // transient, self-resolving window apart from a session resumed after - // a restart with a non-empty queue and nothing running — AGENTS.md: + // a restart with a non-empty queue and nothing running — see + // docs/session-storage-and-queue.md's "Prompt queue" section: // "Boot never auto-dispatches a resumed queue... it sits there until // the next natural drain trigger." That case has no pending drain // trigger at all (freeRunSlotAndEmitIdle, the only setter, never runs diff --git a/server/session_journal.go b/server/session_journal.go index 61155e76..71cb425d 100644 --- a/server/session_journal.go +++ b/server/session_journal.go @@ -39,8 +39,8 @@ type JournalResponse struct { // durable engine log (engine.LoadJournal), reshaped and sanitized, oldest // first — read-only, paginated via `from`/`limit` query parameters // (mirroring the SSE stream's `from` cursor convention). This is the -// endpoint the restart-recovery debugging this repo's own AGENTS.md history -// records (PR #145/#147) kept needing pod-exec into a box to answer by hand +// endpoint that restart-recovery debugging for PR #145 and PR #147 kept +// needing pod-exec into a box to answer by hand // — "was a task-notification checkout/commit/requeue ever recorded for this // child" or "did a recovery marker fire on this turn" — now answerable over // the wire. diff --git a/server/wait.go b/server/wait.go index e33e2410..c67173ec 100644 --- a/server/wait.go +++ b/server/wait.go @@ -185,7 +185,8 @@ func (waitTimeoutError) Error() string { return "timeout_s must be a positive in // OPPOSITE order) — made the running/queued reads non-atomic, a narrower // version of the same false-idle. And gating naively on queue depth alone // (empty or not) is simply wrong: a session resumed after a restart with a -// non-empty queue and nothing running is genuinely idle right now (AGENTS.md: +// non-empty queue and nothing running is genuinely idle right now (see +// docs/session-storage-and-queue.md's "Prompt queue" section: // "Boot never auto-dispatches a resumed queue... it sits there until the // next natural drain trigger") — loadJournal never sets queueDrainPending, // so that case is unaffected here and still returns idle immediately. diff --git a/server/wait_test.go b/server/wait_test.go index 9197dd79..b56493dc 100644 --- a/server/wait_test.go +++ b/server/wait_test.go @@ -795,7 +795,8 @@ func TestWaitDisconnectDoesNotLeakWaiter(t *testing.T) { // TestWaitUntilIdleDoesNotWakeEarlyOnQueuedFollowUp (queue_test.go): gating // until=idle naively on "is the queue non-empty" is wrong for a session // resumed after a restart with a prompt still durably queued and nothing -// running — AGENTS.md: "Boot never auto-dispatches a resumed queue... it +// running — server/AGENTS.md: "Never auto-dispatch a restored queue during +// boot. Leave it for the next natural drain trigger"; it // sits there until the next natural drain trigger (an idle prompt, the next // tool-call boundary inside a running turn, or a goal loop's next turn // boundary)." That session has no pending drain trigger at all and is diff --git a/skill/AGENTS.md b/skill/AGENTS.md new file mode 100644 index 00000000..0b600e66 --- /dev/null +++ b/skill/AGENTS.md @@ -0,0 +1,42 @@ +# Agent Skill instructions + +These rules apply to `skill/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths from that root. +Read `engine/AGENTS.md` for prompt integration. + +## Progressive disclosure + +Keep the two-stage contract. + +- `Load` validates frontmatter without retaining or interpreting the Markdown + body. +- `Skill.Instructions` reads the file again and returns the body on demand. +- Engine discovery advertises only validated stage-one metadata. + +Do not retain, inject, or interpret skill bodies during discovery or session +startup. + +## Frontmatter parser + +Keep the parser dependency-free and limited to the supported Agent Skills +subset. Reject unknown top-level keys and unsupported nested structures. Do not +silently interpret general YAML features. + +Require the skill name to match its parent directory. Preserve rune-based field +limits. + +## Discovery + +Sort discovered skills by name. Reject duplicate names across configured +directories. A malformed `SKILL.md` fails discovery loudly. + +In the resolved `engine.Config`, an explicit empty directory list disables +discovery and a nil list keeps the project default. Preserve `config` package +layering semantics before that resolved value reaches the engine. + +## Tests + +Keep parser tables and fuzz coverage for frontmatter boundaries. Test stage-one +reads separately from body reads. Assert deterministic discovery order and +duplicate failure. diff --git a/skill/CLAUDE.md b/skill/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/skill/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/tools/AGENTS.md b/tools/AGENTS.md new file mode 100644 index 00000000..ecfdf3ab --- /dev/null +++ b/tools/AGENTS.md @@ -0,0 +1,97 @@ +# Local tool instructions + +These rules apply to `tools/`. Harness does not merge ancestor files. If root +guidance is not active, locate the Git root and read `/AGENTS.md`. +Resolve repository paths and commands from that root. + +The hub, monitor, and inspector are operator tools. They are not deployed +multi-user products. Keep each page build-free and dependency-free unless a +separate design changes that constraint. + +## Shared browser rules + +- Keep API data real. Do not invent telemetry. +- Treat run tokens as secrets. +- Keep state and route codecs in pure helpers. +- Test the exact committed HTML. +- Wait for conditions in Node tests. Do not use fixed sleeps. +- Keep CORS and CSP requirements explicit. +- Do not rename renderer-owned CSS classes during a styling-only change. + +## Development hub + +The hub is a stateless fleet control surface. + +- Browser URL-fragment state owns the box registry and current selection. +- The browser calls each box directly. +- The Go server exposes only the embedded page and `POST /spawn`. +- Bind loopback by default. +- Verify browser `Origin` against `Host` before a spawn. +- Keep the page CSP strict while allowing connections to operator-added boxes. +- Treat a shared hub URL as a secret because its fragment contains run tokens. + +### Spawn-command contract + +The spawn command emits `TUNNEL_URL`, `RUN_TOKEN`, and optional +`PORT_URL_` lines. Pass the selected box name through +`HARNESS_HUB_BOX_NAME`. Harness itself does not consume that variable. + +Run: + +```bash +node --test tools/hub/*_test.mjs +go test -race ./tools/hub/... +``` + +Read `tools/hub/e2e/README.md`, `docs/design/fleet-model.md`, and the +"Development hub" section in +`docs/development-interfaces.md` before a behavior change. + +### Hub UI design language + +The hub uses a dark tactical-telemetry style. + +- Use the existing black, phosphor, hairline, square geometry. +- Reserve red for hazards and destructive actions. +- Reserve green for live or successful goal execution. +- Keep body text monospace and headings heavy uppercase. +- Do not add gradients, soft shadows, rounded corners, or decorative metadata. +- Do not add emoji or em dashes to hub UI strings. + +## Session monitor + +The monitor is a single-box live board. + +- Keep the standalone `file://` and static-host path working. +- Keep `GET /monitor` as the same-origin embedded path. +- Store manual connection settings in the documented local-storage keys. +- Keep route state in explicit fragment parameters. +- Adopt a `#t=` token into storage, then scrub it from the visible URL. +- Do not connect an embedded page to a different origin under its same-origin + CSP. +- Keep the testable helper region stable. + +Run: + +```bash +node --test tools/monitor/*_test.mjs +go test -race ./tools/monitor/... +``` + +Read `tools/monitor/e2e/README.md`, +`docs/development-interfaces.md`, and the approved monitor +mockup before a layout change. + +### Monitor UI design language + +The monitor uses the instrument-sheet design, not the hub design. + +- Preserve the light-first OKLCH token system and its dark variant. +- Reserve green, amber, and red for state. +- Reserve the filled accent for the send action. +- Keep `docs/design/monitor-mockup.html` as the visual specification. + +## Inspector + +Follow the same build-free, pure-helper, and no-fixed-sleep rules. Do not apply +the hub theme unless a dedicated inspector design requests it. diff --git a/tools/CLAUDE.md b/tools/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/tools/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/tools/hub/hub.go b/tools/hub/hub.go index e667ad07..ba47089d 100644 --- a/tools/hub/hub.go +++ b/tools/hub/hub.go @@ -20,7 +20,7 @@ import ( var indexHTML []byte // defaultAddr is deliberately loopback-only: the hub is a local, single- -// operator dev tool (see AGENTS.md, "Development hub") and never listens on +// operator dev tool (see tools/AGENTS.md, "Development hub") and never listens on // every interface by default. const defaultAddr = "localhost:7777" @@ -54,7 +54,7 @@ type Options struct { } // NewHandler builds the hub's HTTP handler: the embedded page at "/" and -// the single POST /spawn API described in AGENTS.md. Everything else the +// the single POST /spawn API described in tools/AGENTS.md. Everything else the // page needs (session state, box CRUD) is client-side — see index.html. func NewHandler(opts Options) http.Handler { mux := http.NewServeMux() @@ -83,7 +83,7 @@ func handleIndex(w http.ResponseWriter, r *http.Request) { // spawnRequest is POST /spawn's optional JSON body: {"name": "..."} passes // the box name the page generated (or is re-using for a Respawn/ADOPT) — -// see AGENTS.md's spawn contract section and runSpawn's `name` parameter. +// see tools/AGENTS.md's "Spawn-command contract" and runSpawn's `name` parameter. // A missing or empty body is fine (no name passthrough), matching every // spawn command that predates this field. type spawnRequest struct { @@ -191,7 +191,7 @@ func Run(args []string) error { var addr string fs.StringVar(&addr, "addr", defaultAddr, "listen address (loopback by default — this is a local, single-operator tool)") var spawnCommand string - fs.StringVar(&spawnCommand, "spawn-command", "", "shell command (run via `sh -c`) that POST /spawn execs to bring up a new box; falls back to $"+spawnCommandEnv+"; see AGENTS.md's spawn-command contract") + fs.StringVar(&spawnCommand, "spawn-command", "", "shell command (run via `sh -c`) that POST /spawn execs to bring up a new box; falls back to $"+spawnCommandEnv+"; see docs/design/fleet-model.md for the spawn contract") if err := fs.Parse(args); err != nil { return err } diff --git a/tools/hub/hub_test.go b/tools/hub/hub_test.go index 93385d5c..5d751b32 100644 --- a/tools/hub/hub_test.go +++ b/tools/hub/hub_test.go @@ -184,7 +184,7 @@ func TestHandleSpawnStreamsSSEFrames(t *testing.T) { // TestHandleSpawnPassesNameAsBoxNameEnv is the HTTP-level half of // TestRunSpawnSetsBoxNameEnv (spawn_test.go): POST /spawn's JSON body // {"name": "..."} must reach the spawn command's own environment as -// HARNESS_HUB_BOX_NAME — see AGENTS.md's spawn contract section. +// HARNESS_HUB_BOX_NAME — see tools/AGENTS.md's "Spawn-command contract". func TestHandleSpawnPassesNameAsBoxNameEnv(t *testing.T) { srv := httptest.NewServer(NewHandler(Options{SpawnCommand: `echo "NAME=$HARNESS_HUB_BOX_NAME"`})) defer srv.Close() diff --git a/tools/hub/index.html b/tools/hub/index.html index 764d015e..c3100ad7 100644 --- a/tools/hub/index.html +++ b/tools/hub/index.html @@ -316,7 +316,7 @@ .overflow-menu button { border: none; border-bottom: 1px solid var(--border); text-align: left; width: 100%; } .overflow-menu button:last-child { border-bottom: none; } button.respawn { border-color: var(--warn); color: var(--warn); } - /* PROCESS STRIP: a compact row per managed process (see AGENTS.md/this + /* PROCESS STRIP: a compact row per managed process (see tools/AGENTS.md and this page's header comment). Hidden entirely (display:none, set in JS) when a box reports no processes or doesn't serve GET /process at all. Green stays reserved for goal execution: a ready process is lit @@ -711,7 +711,7 @@

harness hub

// §7: goal.paused — boot-time, always "restart"; goal.stalled while // goal_retryable && goal_waiting — "provider-backoff"; goal.parked — always // "worker_failure" (a worker turn exit-parked the goal instead of -// clearing it, see AGENTS.md's Goal loop section) — all three carry +// clearing it, see docs/goal-loop.md) — all three carry // goal_paused/goal_pause_reason). Non-goal events return prev unchanged // (same reference); goal events return a fresh object so the input is // never mutated. An unrecognized future goal.* event type falls through @@ -1112,7 +1112,7 @@

harness hub

} // canRedispatch reports whether a session is a candidate for the -// Re-dispatch action (distinct from in-place Re-arm, see AGENTS.md/this +// Re-dispatch action (distinct from in-place Re-arm, see tools/AGENTS.md and this // page's header comment): it must be idle in the composite sense — not // currently busy or driving a goal — which covers both a plainly-ended // session and one whose last turn errored out. @@ -1251,7 +1251,7 @@

harness hub

} // normalizePorts tolerates every shape a managed process's not-yet-final -// "ports" field (arriving on a parallel branch — see AGENTS.md/this page's +// "ports" field (arriving on a parallel branch — see tools/AGENTS.md and this page's // header comment) might take: absent entirely, an array of port numbers or // numeric strings, or an object keyed by port. Always returns an array of // port strings; never throws. @@ -1761,7 +1761,7 @@

harness hub

const busyChip = el("span", { class: "count-chip busy" }); const goalChip = el("span", { class: "count-chip goal-running" }); const counts = el("div", { class: "box-counts" }, idleChip, busyChip, goalChip); - // procStrip: the PROCESS STRIP (see AGENTS.md/this page's header comment) + // procStrip: the PROCESS STRIP (see tools/AGENTS.md and this page's header comment) // — one compact row per managed process this box reports via GET // /process. Absent/removed entirely for a box that doesn't serve that // endpoint at all (older boxes) — see updateProcessStrip. @@ -1957,7 +1957,7 @@

harness hub

const expandedLineages = new Set(); // updateSessList reconciles the session rows under one (expanded) box card. -// LINEAGE GROUPING (see AGENTS.md/this page's header comment, +// LINEAGE GROUPING (see tools/AGENTS.md and this page's header comment, // docs/design/fleet-model.md §6): sessions sharing a parent_session chain // collapse into one "task" row keyed by the lineage's TIP (most recent) // session — buildLineages does the pure grouping; a lone session (no @@ -2207,7 +2207,7 @@

harness hub

`/event?from=0&session=` — distinct from the box-wide stream used for fleet cards above. That replays the session's ENTIRE durable history (every message + every goal.* record, in order) before continuing live, - which is exactly the narrative the goal workflow wants (see AGENTS.md's + which is exactly the narrative the goal workflow wants (see tools/AGENTS.md's "Development hub" section) and lets the drill-down bootstrap itself from one stream instead of juggling a separate REST fetch plus a filtered live tail. */ @@ -2501,7 +2501,7 @@

harness hub

case "goal.parked": { // Round 7, live event: a worker turn exhausted either exhaustion // tier and PursueGoal exit-parked instead of clearing (see - // AGENTS.md's Goal loop section) — always pause_reason + // docs/goal-loop.md) — always pause_reason // "worker_failure", never calm (unlike provider-backoff, this loop // has genuinely exited; it resumes on the next ordinary activity, // not on its own next retry). diff --git a/tools/hub/spawn.go b/tools/hub/spawn.go index 4876e835..09256393 100644 --- a/tools/hub/spawn.go +++ b/tools/hub/spawn.go @@ -16,15 +16,15 @@ import ( ) // boxNameEnv is the environment variable the spawn command's own process -// sees the hub-chosen (or operator-chosen) box NAME in — see AGENTS.md's -// spawn contract section and docs/design/fleet-model.md §8. Deployment +// sees the hub-chosen (or operator-chosen) box NAME in — see tools/AGENTS.md's +// "Spawn-command contract" and docs/design/fleet-model.md §8. Deployment // tooling invoked by -spawn-command reads this to derive per-name storage // (e.g. HARNESS_SESSION_DIR); harness's own code never reads it. const boxNameEnv = "HARNESS_HUB_BOX_NAME" // spawnEvent is one frame of the /spawn SSE stream, JSON-encoded as the // `data:` payload. This is the entire spawn-output contract described in -// AGENTS.md: a "stdout" event per line of the spawn command's combined +// tools/AGENTS.md: a "stdout" event per line of the spawn command's combined // stdout+stderr, and exactly one terminal "done" event carrying the exit // status plus whatever TUNNEL_URL / RUN_TOKEN lines were found along the // way. The page needs nothing else to add the new box to its own state. diff --git a/tools/hub/spawn_test.go b/tools/hub/spawn_test.go index 89c7f30c..6914ed17 100644 --- a/tools/hub/spawn_test.go +++ b/tools/hub/spawn_test.go @@ -15,7 +15,7 @@ func collectSpawn(t *testing.T, ctx context.Context, command string) []spawnEven } // collectSpawnNamed is collectSpawn plus the box-name passthrough (see -// TestRunSpawnSetsBoxNameEnv below and AGENTS.md's spawn contract section). +// TestRunSpawnSetsBoxNameEnv below and tools/AGENTS.md's "Spawn-command contract"). func collectSpawnNamed(t *testing.T, ctx context.Context, command, name string) []spawnEvent { t.Helper() var mu sync.Mutex @@ -166,7 +166,7 @@ func TestRunSpawnCancelKillsProcess(t *testing.T) { } // TestRunSpawnSetsBoxNameEnv verifies the box-name passthrough documented in -// AGENTS.md's spawn contract section and docs/design/fleet-model.md §8: a +// tools/AGENTS.md's "Spawn-command contract" and docs/design/fleet-model.md §8: a // non-empty name is set as HARNESS_HUB_BOX_NAME in the spawn command's own // environment (not just some sidecar field), visible to the child process // like any other env var — this is what lets deployment tooling invoked by From aeead5c5363a9151e07bc4390a760f8f2992ebd1 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Sat, 29 Aug 2026 10:27:29 -0400 Subject: [PATCH 24/95] provider/openai,config: allow a provider to omit Responses request params (#213) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ChatGPT Codex subscription backend (chatgpt.com/backend-api/codex/responses) is not a drop-in OpenAI Responses endpoint: it 400s outright on max_output_tokens, temperature, top_p, and metadata, one at a time, each reported as "Unsupported parameter". provider/openai always sends max_output_tokens (the engine defaults MaxTokens to 8192 when a caller sets none), so any provider entry routed to that backend 400ed on every single turn with no way to stop it. Design: a per-provider allowlisted omit list, not arbitrary field surgery. config.Provider gains OmitResponseParams []string, validated against a small, closed set of the optional params this adapter can conditionally drop (max_output_tokens, temperature, top_p, metadata) — an unknown name fails config validation loudly, the same "typo must not vanish" rule every other allowlisted Provider field already follows. The field is valid only on an entry that builds the native Responses adapter, gated by the same buildsResponsesAdapter identity check #197 added for responses_path, and it merges the same way: a non-empty project-layer list replaces the user-layer list wholesale. provider/openai.Client gains the matching OmitResponseParams field; transcodeRequestFamily clears each named field last, after every other adjustment (including the reasoning-floor bump that can raise MaxOutputTokens), so an entry that omits a param never sends it regardless of what an earlier step computed. harness's internal MaxTokens/Temperature/TopP bookkeeping is untouched — only the wire copy is affected. Rejected alternatives: a global default change (would strip these params for every provider, including ones that require them); mutating the request in a gatekeeper/proxy layer (the params are correct for the platform API — this is one specific subscription backend's narrower contract, not a general policy); matching on hostname (fragile, and provider/openai has no concept of "which backend" beyond its configured base URL/path already). Investigated, not assumed: - temperature/top_p: harness does emit these when a caller sets them (session params reach provider.Request.Temperature/TopP via engine params, server handlers, and plugin hooks, and transcode.go passes them straight through). metadata has no corresponding field anywhere in the request pipeline today — harness never emits it regardless of this list, so listing it is pure future-proofing against the day a Metadata field is added. - maybeAutoContinueMaxTokens (the max-tokens continuation loop) keys off the RESPONSE's incomplete_details.reason, never off the value this adapter put on the wire, so omitting max_output_tokens does not touch that path. - the Codex backend also rejects some tool-schema keywords a stricter validator enforces (confirmed against a live probe, UNSUPPORTED_FIELDS + normalizeToolSchemas in ~/dev/web/.opencode/plugins/codex-request-normalize.ts). harness's own built-in tool schemas (bash, file, search, etc.) use only keywords that validator accepts, but engine/mcp.go forwards a THIRD-PARTY MCP server's tool schema onto the wire completely unsanitized — a real, separate gap if a box's tool roster ever includes an MCP server with a richer schema routed through this backend. Flagging it; not building a schema sanitizer here. Verification: go build/vet/test all green (config, provider/openai, cmd/harness). New tests cover: config validation accepts all four names on an entry that builds the Responses adapter and rejects an unknown name or the field set anywhere else; config merge replaces the list wholesale on a non-empty override and inherits it otherwise; the adapter emits none of the four listed fields end-to-end over real HTTP, a partial list omits only what it names, and omission wins over the reasoning-floor bump; cmd/harness threads the config value into the built client for both the native "openai" key and a keyed type:"openai" entry. --- cmd/harness/main.go | 31 +++- cmd/harness/omit_response_params_test.go | 120 ++++++++++++++++ config/config.go | 94 +++++++++++- config/omit_response_params_test.go | 142 +++++++++++++++++++ provider/openai/omit_response_params_test.go | 112 +++++++++++++++ provider/openai/openai.go | 11 +- provider/openai/transcode.go | 42 +++++- 7 files changed, 538 insertions(+), 14 deletions(-) create mode 100644 cmd/harness/omit_response_params_test.go create mode 100644 config/omit_response_params_test.go create mode 100644 provider/openai/omit_response_params_test.go diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 68d95c11..76af648f 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -937,8 +937,9 @@ func registry(cfg *config.Config) provider.Registry { anthropic.Family: &anthropic.Client{APIKey: akey, BaseURL: abase, CacheTTL: anthropicCacheTTL(cfg)}, // The built-in openai entry deliberately leaves Family empty: it IS // the package default, and naming it here would only invite the two - // to drift. Its ResponsesPath comes from the native entry, if any. - openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg)}, + // to drift. Its ResponsesPath/OmitResponseParams come from the + // native entry, if any. + openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg), OmitResponseParams: nativeOmitResponseParams(cfg)}, } registerOpenAICompatProviders(reg, cfg) registerOpenAIProviders(reg, cfg) @@ -993,10 +994,11 @@ func registerOpenAIProviders(reg provider.Registry, cfg *config.Config) { } apiKey := os.Getenv(keyEnv) reg[name] = &openai.Client{ - Family: name, - APIKey: apiKey, - BaseURL: p.BaseURL, - ResponsesPath: p.ResponsesPath, + Family: name, + APIKey: apiKey, + BaseURL: p.BaseURL, + ResponsesPath: p.ResponsesPath, + OmitResponseParams: p.OmitResponseParams, } } } @@ -1017,6 +1019,23 @@ func nativeResponsesPath(cfg *config.Config) string { return p.ResponsesPath } +// nativeOmitResponseParams reads omit_response_params configured on the +// NATIVE "openai" entry (map key "openai", no type — the same shape +// config.validateProviders permits omit_response_params on). Empty leaves +// the adapter sending every param it always has. A keyed type:"openai" +// entry carries its own list and is wired by registerOpenAIProviders +// instead. Mirrors nativeResponsesPath exactly. +func nativeOmitResponseParams(cfg *config.Config) []string { + if cfg == nil { + return nil + } + p := cfg.Providers[openai.Family] + if p.Type != "" { + return nil + } + return p.OmitResponseParams +} + // registerOpenAICompatProviders builds a provider/openaicompat client for // every config.Providers entry of config.TypeOpenAICompat, keyed by its // providers map name — that name is what routes "name/model" refs to it, diff --git a/cmd/harness/omit_response_params_test.go b/cmd/harness/omit_response_params_test.go new file mode 100644 index 00000000..21595503 --- /dev/null +++ b/cmd/harness/omit_response_params_test.go @@ -0,0 +1,120 @@ +package main + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/majorcontext/harness/config" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" + "github.com/majorcontext/harness/provider/openai" +) + +// TestRegistryTypeOpenAIThreadsOmitResponseParams: a `type: "openai"` entry's +// omit_response_params must reach the built *openai.Client, exactly as +// ResponsesPath does — the seam a keyed second Responses provider routed to +// a strict upstream (e.g. the ChatGPT Codex backend) actually needs. +func TestRegistryTypeOpenAIThreadsOmitResponseParams(t *testing.T) { + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: "https://gateway.example", + APIKeyEnv: "SECONDARY_API_KEY", + OmitResponseParams: []string{"max_output_tokens", "temperature", "top_p", "metadata"}, + }, + }}) + c, ok := reg["secondary"].(*openai.Client) + if !ok { + t.Fatalf("secondary provider is %T, want *openai.Client", reg["secondary"]) + } + want := []string{"max_output_tokens", "temperature", "top_p", "metadata"} + if len(c.OmitResponseParams) != len(want) { + t.Fatalf("OmitResponseParams = %v, want %v", c.OmitResponseParams, want) + } + for i, v := range want { + if c.OmitResponseParams[i] != v { + t.Errorf("OmitResponseParams[%d] = %q, want %q", i, c.OmitResponseParams[i], v) + } + } +} + +// TestRegistryNativeOpenAIHonorsOmitResponseParams: the bare "openai" key +// builds the same adapter, so its omit_response_params must reach the +// client too — mirrors TestRegistryNativeOpenAIHonorsResponsesPath. +func TestRegistryNativeOpenAIHonorsOmitResponseParams(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "openai": {OmitResponseParams: []string{"max_output_tokens"}}, + }}) + c := reg[openai.Family].(*openai.Client) + if len(c.OmitResponseParams) != 1 || c.OmitResponseParams[0] != "max_output_tokens" { + t.Errorf("OmitResponseParams = %v, want [max_output_tokens]", c.OmitResponseParams) + } +} + +// TestRegistryOmitResponseParamsEmitsNoneOnWire drives the production path +// end to end: a provider configured to omit all four params must actually +// send a request body without them, over real HTTP through Client.Stream — +// the direct proof this feature fixes the Codex-backend 400, not just that +// config threads a value through. +func TestRegistryOmitResponseParamsEmitsNoneOnWire(t *testing.T) { + var gotBody map[string]json.RawMessage + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil { + t.Errorf("decoding request body: %v", err) + } + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_1\"}}\n\n") //nolint:errcheck + })) + defer srv.Close() + + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: srv.URL, + APIKeyEnv: "SECONDARY_API_KEY", + OmitResponseParams: []string{"max_output_tokens", "temperature", "top_p", "metadata"}, + }, + }}) + ref, err := message.ParseModelRef("secondary/gpt-5") + if err != nil { + t.Fatalf("ParseModelRef: %v", err) + } + p, err := reg.For(ref) + if err != nil { + t.Fatalf("reg.For: %v", err) + } + temp := 0.5 + stream, err := p.Stream(context.Background(), &provider.Request{ + Model: ref, + Messages: []message.Message{{ID: "msg_1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hello"}}}}, + MaxTokens: 4096, + Temperature: &temp, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer stream.Close() + for { + if _, err := stream.Next(); err == io.EOF { + break + } else if err != nil { + t.Fatalf("Next: %v", err) + } + } + + for _, field := range []string{"max_output_tokens", "temperature", "top_p", "metadata"} { + if _, ok := gotBody[field]; ok { + t.Errorf("request body has field %q, want it omitted", field) + } + } + if _, ok := gotBody["model"]; !ok { + t.Error("request body is missing model — omission must not touch required fields") + } +} diff --git a/config/config.go b/config/config.go index e10e5bf3..721c11a8 100644 --- a/config/config.go +++ b/config/config.go @@ -565,6 +565,32 @@ type Provider struct { // CacheTTL follow. Merge semantics are additive like every other // Provider field (see NoPromptCacheKey's doc comment). ResponsesPath string `json:"responses_path,omitempty"` + // OmitResponseParams names optional Responses request params the native + // openai adapter must NOT send on the wire for this provider. Each entry + // must be one of OmitResponseParamValues (max_output_tokens, temperature, + // top_p, metadata) — an unknown name fails validation loudly rather than + // silently doing nothing, the same "typo must not vanish" rule every + // other allowlisted field in this struct follows. + // + // It exists because some Responses-API-compatible endpoints reject + // params the OpenAI API itself accepts: the ChatGPT Codex backend + // 400s on all four (verified against a live probe), and this adapter + // always sent max_output_tokens (engine defaults MaxTokens=8192), so a + // provider routed there 400s on every turn with no way to stop it. + // Omitting a param here is wire-only — harness keeps its internal + // MaxTokens/Temperature/TopP for accounting and continuation logic + // regardless of what an entry omits (see provider/openai/transcode.go). + // + // Valid ONLY on an entry that builds the Responses adapter, exactly + // like ResponsesPath — the same buildsResponsesAdapter identity check + // gates both fields, and validateOmitResponseParams follows + // validateResponsesPath's shape. Merge semantics are additive like + // every other Provider field (see NoPromptCacheKey's doc comment): a + // non-empty project-layer list replaces the user-layer list wholesale, + // the same rule SkillsDirs and Plugins follow, rather than merging + // entry by entry (a project layer un-listing a param the user layer + // added would otherwise be unrepresentable). + OmitResponseParams []string `json:"omit_response_params,omitempty"` // CacheTTL selects the Anthropic prompt-cache breakpoint lifetime: // "5m" (the Anthropic API default) or "1h" (the extended TTL, beta // extended-cache-ttl-2025-04-11). Empty (the default) leaves the @@ -584,6 +610,31 @@ const ( CacheTTL1h = "1h" ) +// OmitResponseParam* name the optional OpenAI Responses request params the +// native openai adapter can conditionally omit on Provider.OmitResponseParams. +// This is a BOUNDED allowlist, not an arbitrary-field-name mechanism: it +// names the specific optional params a real upstream is known to reject +// (the ChatGPT Codex backend 400s on all four), so a config typo cannot +// silently ask the adapter to drop a REQUIRED field like model or input. +const ( + OmitResponseParamMaxOutputTokens = "max_output_tokens" + OmitResponseParamTemperature = "temperature" + OmitResponseParamTopP = "top_p" + OmitResponseParamMetadata = "metadata" +) + +// OmitResponseParamValues returns every value validateOmitResponseParams +// accepts, in a fresh slice. validateOmitResponseParams iterates this same +// list, so the set and the validator cannot drift from each other. +func OmitResponseParamValues() []string { + return []string{ + OmitResponseParamMaxOutputTokens, + OmitResponseParamTemperature, + OmitResponseParamTopP, + OmitResponseParamMetadata, + } +} + // Load reads a single config file. A missing file yields a zero-value Config // and a nil error (config is optional). Malformed JSON or an unknown field // (json.Decoder with DisallowUnknownFields, so typos surface) yields an error @@ -668,6 +719,9 @@ func validateProviders(providers map[string]Provider) error { if err := validateResponsesPath(name, p); err != nil { return err } + if err := validateOmitResponseParams(name, p); err != nil { + return err + } } return nil } @@ -705,6 +759,28 @@ func validateResponsesPath(name string, p Provider) error { return nil } +// validateOmitResponseParams fails loudly on an omit_response_params entry +// that cannot possibly be honored: a name outside OmitResponseParamValues (a +// typo that would otherwise silently keep sending the very param the +// deployment configured it to stop sending), or any value at all on an +// entry that does not build the Responses adapter — the same +// buildsResponsesAdapter identity check validateResponsesPath uses, since +// only that adapter reads the field. Empty (the default) is always valid. +func validateOmitResponseParams(name string, p Provider) error { + if len(p.OmitResponseParams) == 0 { + return nil + } + if !buildsResponsesAdapter(name, p) { + return fmt.Errorf("providers.%s: omit_response_params is only valid on an entry that builds the OpenAI Responses adapter (map key %q with no type, or any key with type %q); no other adapter reads it", name, "openai", TypeOpenAI) + } + for _, param := range p.OmitResponseParams { + if !slices.Contains(OmitResponseParamValues(), param) { + return fmt.Errorf("providers.%s: unknown omit_response_params entry %q (valid values: %q)", name, param, OmitResponseParamValues()) + } + } + return nil +} + // ValidateProviderCacheTTL reports whether ttl is a value this package // accepts for Provider.CacheTTL. It is the exported seam cmd/harness's // parity test uses to prove this list and provider/anthropic's own list @@ -1092,10 +1168,11 @@ func merge(base, over *Config) *Config { if n := len(base.Providers) + len(over.Providers); n > 0 { m := make(map[string]Provider, n) for k, v := range base.Providers { - // Deep-copy ExtraHeaders here too: a base-only key (never - // touched by the loop below, e.g. no matching over.Providers - // entry) would otherwise leave m[k] aliasing base's map, so - // mutating the merged config's headers would corrupt base's. + // Deep-copy ExtraHeaders and OmitResponseParams here too: a + // base-only key (never touched by the loop below, e.g. no + // matching over.Providers entry) would otherwise leave m[k] + // aliasing base's map/slice, so mutating the merged config's + // copy would corrupt base's. if len(v.ExtraHeaders) > 0 { hm := make(map[string]string, len(v.ExtraHeaders)) for hk, hv := range v.ExtraHeaders { @@ -1103,6 +1180,9 @@ func merge(base, over *Config) *Config { } v.ExtraHeaders = hm } + if len(v.OmitResponseParams) > 0 { + v.OmitResponseParams = append([]string(nil), v.OmitResponseParams...) + } m[k] = v } for k, v := range over.Providers { @@ -1125,6 +1205,9 @@ func merge(base, over *Config) *Config { if v.ResponsesPath != "" { ex.ResponsesPath = v.ResponsesPath } + if len(v.OmitResponseParams) > 0 { + ex.OmitResponseParams = append([]string(nil), v.OmitResponseParams...) + } if v.NoPromptCacheKey { ex.NoPromptCacheKey = true } @@ -1147,6 +1230,9 @@ func merge(base, over *Config) *Config { } v.ExtraHeaders = hm } + if len(v.OmitResponseParams) > 0 { + v.OmitResponseParams = append([]string(nil), v.OmitResponseParams...) + } m[k] = v } } diff --git a/config/omit_response_params_test.go b/config/omit_response_params_test.go new file mode 100644 index 00000000..b38f8107 --- /dev/null +++ b/config/omit_response_params_test.go @@ -0,0 +1,142 @@ +package config + +import ( + "strings" + "testing" +) + +// TestOmitResponseParamsOnNativeOpenAIKeyOK: the bare "openai" key (empty +// type) builds the Responses adapter, so it may carry omit_response_params +// too — the field is gated on the ADAPTER an entry builds, not on its type +// string, exactly like ResponsesPath. +func TestOmitResponseParamsOnNativeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "openai": {OmitResponseParams: []string{ + OmitResponseParamMaxOutputTokens, + OmitResponseParamTemperature, + OmitResponseParamTopP, + OmitResponseParamMetadata, + }}, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + got := merged.Providers["openai"].OmitResponseParams + want := []string{"max_output_tokens", "temperature", "top_p", "metadata"} + if len(got) != len(want) { + t.Fatalf("OmitResponseParams = %v, want %v", got, want) + } + for i, v := range want { + if got[i] != v { + t.Errorf("OmitResponseParams[%d] = %q, want %q", i, got[i], v) + } + } +} + +// TestOmitResponseParamsOnTypeOpenAIKeyOK: a TypeOpenAI entry under an +// arbitrary key builds the same adapter, so it may carry the field too. +func TestOmitResponseParamsOnTypeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "secondary": { + Type: TypeOpenAI, + BaseURL: "https://gateway.example", + OmitResponseParams: []string{OmitResponseParamMaxOutputTokens}, + }, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if got := merged.Providers["secondary"].OmitResponseParams; len(got) != 1 || got[0] != "max_output_tokens" { + t.Errorf("OmitResponseParams = %v, want [max_output_tokens]", got) + } +} + +// TestOmitResponseParamsOnWrongAdapterFails: only the Responses adapter +// reads omit_response_params. Set anywhere else it would vanish silently +// into a client that never looks at it, so it is rejected loudly instead — +// the same rule responses_path, no_prompt_cache_key, and cache_ttl follow. +func TestOmitResponseParamsOnWrongAdapterFails(t *testing.T) { + for _, tc := range []struct { + name string + key string + p Provider + }{ + {"openai-compat", "mycompat", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", OmitResponseParams: []string{"max_output_tokens"}}}, + {"native anthropic", "anthropic", Provider{OmitResponseParams: []string{"max_output_tokens"}}}, + } { + t.Run(tc.name, func(t *testing.T) { + c := &Config{Providers: map[string]Provider{tc.key: tc.p}} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatalf("mergeAndValidate accepted omit_response_params on a %s entry", tc.name) + } + if !strings.Contains(err.Error(), "omit_response_params") { + t.Errorf("error %q does not name the offending field", err) + } + if !strings.Contains(err.Error(), tc.key) { + t.Errorf("error %q does not name the offending key", err) + } + }) + } +} + +// TestOmitResponseParamsUnknownNameFails: a typo'd param name must not +// silently do nothing — the line between this bounded allowlist and an +// arbitrary-field-deletion mechanism. +func TestOmitResponseParamsUnknownNameFails(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "openai": {OmitResponseParams: []string{"max_output_tokens", "store"}}, + }} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on an unknown omit_response_params entry") + } + if !strings.Contains(err.Error(), `"store"`) { + t.Errorf("error %q does not name the offending value", err) + } +} + +// TestMergeProviderOmitResponseParams: omit_response_params layers wholesale +// like SkillsDirs/Plugins, not additively like ExtraHeaders — a project +// layer's non-empty list replaces the user layer's entirely, so a project +// can also SHRINK the set (un-list a param the user layer added), which an +// additive merge could never represent. +func TestMergeProviderOmitResponseParams(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "openai": {OmitResponseParams: []string{"temperature", "top_p"}}, + }} + over := &Config{Providers: map[string]Provider{ + "openai": {OmitResponseParams: []string{"max_output_tokens"}}, + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + got := merged.Providers["openai"].OmitResponseParams + if len(got) != 1 || got[0] != "max_output_tokens" { + t.Errorf("OmitResponseParams = %v, want [max_output_tokens] (project layer replaces wholesale)", got) + } +} + +// TestMergeProviderOmitResponseParamsInheritsWhenOverEmpty: an override +// layer that names no omit_response_params must inherit the base layer's +// list unchanged, exactly as ResponsesPath and every other Provider field +// does when the override leaves it zero. +func TestMergeProviderOmitResponseParamsInheritsWhenOverEmpty(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "openai": {OmitResponseParams: []string{"temperature", "top_p"}}, + }} + over := &Config{Providers: map[string]Provider{ + "openai": {}, + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + got := merged.Providers["openai"].OmitResponseParams + if len(got) != 2 || got[0] != "temperature" || got[1] != "top_p" { + t.Errorf("OmitResponseParams = %v, want [temperature top_p]", got) + } +} diff --git a/provider/openai/omit_response_params_test.go b/provider/openai/omit_response_params_test.go new file mode 100644 index 00000000..97c71b60 --- /dev/null +++ b/provider/openai/omit_response_params_test.go @@ -0,0 +1,112 @@ +package openai + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/majorcontext/harness/message" +) + +// float64Ptr is a tiny helper so tests can take the address of a literal. +func float64Ptr(f float64) *float64 { return &f } + +// TestOmitResponseParamsNoneListedUnchanged: a request transcoded with no +// omit list behaves exactly as before the field existed — the "changed +// nothing for anyone" half of this feature, mirrored on ResponsesPath's own +// default test. +func TestOmitResponseParamsNoneListedUnchanged(t *testing.T) { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.Temperature = float64Ptr(0.5) + req.TopP = float64Ptr(0.9) + + out, err := transcodeRequestFamily(req, Family, nil) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if out.MaxOutputTokens != 4096 { + t.Errorf("MaxOutputTokens = %d, want 4096 (unchanged)", out.MaxOutputTokens) + } + if out.Temperature == nil || *out.Temperature != 0.5 { + t.Errorf("Temperature = %v, want 0.5", out.Temperature) + } + if out.TopP == nil || *out.TopP != 0.9 { + t.Errorf("TopP = %v, want 0.9", out.TopP) + } +} + +// TestOmitResponseParamsAllFourOmitsFromWire is the reason the field +// exists: the ChatGPT Codex backend 400s on max_output_tokens, temperature, +// top_p, and metadata. Listing all four (the config allowlist) must clear +// every field this adapter actually sends for them — MaxOutputTokens (via +// its omitempty int tag), Temperature, and TopP; "metadata" has no +// corresponding apiRequest field yet, so it is accepted but a no-op here +// (see applyOmitResponseParams's doc comment). +func TestOmitResponseParamsAllFourOmitsFromWire(t *testing.T) { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.Temperature = float64Ptr(0.5) + req.TopP = float64Ptr(0.9) + + omit := []string{"max_output_tokens", "temperature", "top_p", "metadata"} + out, err := transcodeRequestFamily(req, Family, omit) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if out.MaxOutputTokens != 0 { + t.Errorf("MaxOutputTokens = %d, want 0 (omitted)", out.MaxOutputTokens) + } + if out.Temperature != nil { + t.Errorf("Temperature = %v, want nil (omitted)", out.Temperature) + } + if out.TopP != nil { + t.Errorf("TopP = %v, want nil (omitted)", out.TopP) + } + + body, err := json.Marshal(out) + if err != nil { + t.Fatalf("marshal: %v", err) + } + for _, field := range []string{`"max_output_tokens"`, `"temperature"`, `"top_p"`} { + if strings.Contains(string(body), field) { + t.Errorf("wire body contains %s, want it omitted entirely: %s", field, body) + } + } +} + +// TestOmitResponseParamsPartialList: an entry that lists only SOME of the +// four params must omit exactly those and leave the rest unchanged — the +// per-field independence a config.Provider.OmitResponseParams entry needs +// (a deployment omitting only max_output_tokens still wants an explicit +// temperature honored). +func TestOmitResponseParamsPartialList(t *testing.T) { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.Temperature = float64Ptr(0.5) + + out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if out.MaxOutputTokens != 0 { + t.Errorf("MaxOutputTokens = %d, want 0 (omitted)", out.MaxOutputTokens) + } + if out.Temperature == nil || *out.Temperature != 0.5 { + t.Errorf("Temperature = %v, want 0.5 (not listed, must be unchanged)", out.Temperature) + } +} + +// TestOmitResponseParamsWinsOverReasoningFloor: reasoningOutputFloor raises +// MaxOutputTokens for a reasoning turn BEFORE applyOmitResponseParams runs. +// An entry that omits max_output_tokens must still send none — the omit +// list is the caller's assertion that the upstream rejects the field +// outright, which no amount of internal floor-raising changes. +func TestOmitResponseParamsWinsOverReasoningFloor(t *testing.T) { + req := reasoningRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + + out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if out.MaxOutputTokens != 0 { + t.Errorf("MaxOutputTokens = %d, want 0 (omitted even though reasoning would otherwise raise it)", out.MaxOutputTokens) + } +} diff --git a/provider/openai/openai.go b/provider/openai/openai.go index 9d8ef599..6ff6e0d4 100644 --- a/provider/openai/openai.go +++ b/provider/openai/openai.go @@ -53,6 +53,15 @@ type Client struct { // a turn of reasoning continuity and nothing else. Family string HTTPClient *http.Client // defaults to http.DefaultClient + // OmitResponseParams names optional Responses request params this + // client must NOT send on the wire, e.g. "max_output_tokens", + // "temperature", "top_p", "metadata" — see config.Provider's field of + // the same name for the full rationale (some Responses-API-compatible + // endpoints, e.g. the ChatGPT Codex backend, reject params the OpenAI + // API itself accepts). This is wire-only: it never affects the + // canonical provider.Request this client is handed, only the JSON body + // transcodeRequestFamily builds from it. + OmitResponseParams []string } // familyOrDefault resolves a configured family override to the family key @@ -77,7 +86,7 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St if c.APIKey == "" { return nil, fmt.Errorf("openai: no API key configured (set OPENAI_API_KEY)") } - wire, err := transcodeRequestFamily(req, c.family()) + wire, err := transcodeRequestFamily(req, c.family(), c.OmitResponseParams) if err != nil { return nil, err } diff --git a/provider/openai/transcode.go b/provider/openai/transcode.go index 538510f9..5d8222a2 100644 --- a/provider/openai/transcode.go +++ b/provider/openai/transcode.go @@ -154,7 +154,7 @@ func wireCallID(id string) string { // under the package Family constant — the default for a client that // configures no family of its own. func transcodeRequest(req *provider.Request) (*apiRequest, error) { - return transcodeRequestFamily(req, Family) + return transcodeRequestFamily(req, Family, nil) } // transcodeRequestFamily is transcodeRequest with the ProviderData tag made @@ -163,8 +163,12 @@ func transcodeRequest(req *provider.Request) (*apiRequest, error) { // endpoint configured under its own providers-map key reads back only the // items it produced itself, and drops the other endpoint's (the canonical // cross-family rule — those items are opaque, usually encrypted, and -// endpoint-scoped). -func transcodeRequestFamily(req *provider.Request, family string) (*apiRequest, error) { +// endpoint-scoped). omitParams names optional request params to leave off +// the wire entirely (Client.OmitResponseParams / config.Provider's field of +// the same name) — applied last, after every other field of out is +// computed, so it always wins over the reasoning-floor bump and every other +// adjustment above. +func transcodeRequestFamily(req *provider.Request, family string, omitParams []string) (*apiRequest, error) { out := &apiRequest{ Model: req.Model.Model, Instructions: strings.Join(req.System, "\n\n"), @@ -250,9 +254,41 @@ func transcodeRequestFamily(req *provider.Request, family string) (*apiRequest, if len(out.Input) == 0 { return nil, fmt.Errorf("openai: request has no transcodable messages") } + applyOmitResponseParams(out, omitParams) return out, nil } +// applyOmitResponseParams clears each field named in omitParams so its +// omitempty tag drops it from the marshaled request — the same allowlist +// config.OmitResponseParamValues validates. Applied last, it always wins +// over every earlier adjustment (e.g. the reasoning-floor bump raising +// MaxOutputTokens): an entry that lists max_output_tokens sends none, even +// for a reasoning turn that would otherwise raise it. +// +// harness's own accounting is untouched — this only ever mutates the wire +// copy (out), never req or the engine's internal MaxTokens/Temperature/TopP +// bookkeeping, and the fields it can clear are all optional on the +// Responses API; nothing here can drop a required field like model or +// input. +// +// "metadata" is accepted by config validation (it is a known-bad param on +// at least one real upstream) but apiRequest has no Metadata field yet — +// harness never emits it regardless of this list, so there is nothing to +// clear for it here. Listing it is still valid and cheap future-proofing; +// add a case here if a Metadata field is ever added. +func applyOmitResponseParams(out *apiRequest, omitParams []string) { + for _, p := range omitParams { + switch p { + case "max_output_tokens": + out.MaxOutputTokens = 0 + case "temperature": + out.Temperature = nil + case "top_p": + out.TopP = nil + } + } +} + // transcodeMessage expands one canonical message into a sequence of Responses // input items. Contiguous text/image parts are grouped into a single message // item; tool calls, tool results, and reasoning are each their own item. From 048cbe2be06250e0b61a66e0574d498c0a504830 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Sat, 29 Aug 2026 20:52:30 -0400 Subject: [PATCH 25/95] engine,provider: delegate a turn to the Claude Code CLI over stream-json (#214) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A box running on a user's Claude subscription must run through Anthropic's own client — Anthropic blocks third-party harnesses from authenticating against a subscription directly. harness cannot be that client for a subscription-backed turn: it has no sanctioned way to hold or use the credential. It can, however, stay the session/journal/HTTP-API front door and delegate the turn to the real `claude` CLI itself, driven headlessly, bridging its event stream back into harness's own journal so the console and every other read-path consumer are unaffected. Design: a new execution path, not a provider.Provider adapter. Claude Code runs its own inner tool loop, its own retries, and its own context management — collapsing a whole delegated turn into one opaque provider.Provider.Stream call (the rejected alternative) would lose per-tool-call journal fidelity and force a fake StopReason onto a turn shape that never used one. Instead, a session's model ref (message.ModelRef{Provider: "claude-code", Model: "sonnet"}) is checked at the two points that actually drive a turn — engine.go's PromptWithOrigin (before ContextWindowErr/ensureInstructions/ ensureSkills/maybeAutoCompact, all native-loop-only) and runAgenticLoop itself (goal.go's promptTurnWithRetry directive-reuse retry calls this directly, bypassing PromptWithOrigin) — and dispatches to the new engine/claude_code_backend.go driver instead of the native streamTurn/runToolCalls machinery. Every message the driver decodes is appended through the ordinary Session.append/ appendWithUsage path and emitted through the ordinary Session.emit, so the journal, SSE stream, and transcript read-path treat a delegated turn's records exactly like a native one's, distinguished only by message.OriginClaudeCode (presentation metadata, never read by a transcoder). Selection: config.Provider gains Type "claude-code-cli" plus its non-HTTP fields (BinaryPath, ExtraArgs, PermissionMode) — validateProviders accepts the type with no base_url requirement (this backend spawns a process, never dials an endpoint) and rejects the three new fields on any other type. provider/claudecode.Client registers the family in cmd/harness's provider registry so Session.ModelSupported accepts a swap to a claude-code ref; its Stream method is a defensive backstop that errors loudly; it is never actually called; the dispatch above always intercepts first. modelmeta.ContextWindow gained a claude-code case (200k, a plausible stand-in — the real number is unused, since harness's own auto-compaction is unconditionally skipped for a delegated session) so RequireContextWindow's default hard refusal doesn't reject session create for the very ref this backend exists to serve. Event mapping (engine/claude_code_backend.go's package doc has the full mapping and every documented assumption about the CLI's wire shape): system/init captures the CLI's own session id, persisted (recClaudeCodeSessionID) and passed as --resume on every later turn, surviving a process restart; assistant events become an appended RoleAssistant message (Text/ToolCall parts) plus EventMessage/ EventTextDelta/EventToolStart; user events (Claude Code's own tool_result delivery, in the Anthropic API's "user"-role convention) become an appended RoleTool message plus EventToolEnd; a result event applies the turn's aggregate usage via a new durable, message- independent record (recClaudeCodeUsage — mirrors recCompact's own "usage with no single owning message" precedent) rather than re-appending a duplicate terminal message purely to give appendWithUsage somewhere to attach it. Usage maps input/output tokens directly; Claude Code's cache_read_input_tokens/ cache_creation_input_tokens map onto provider.Usage's CacheRead/ CacheWrite fields (same accounting, different naming convention); total_cost_usd has no home in provider.Usage (no adapter carries a cost field) and is deliberately dropped. Abort/shutdown cancels via an escalating SIGINT-then-SIGTERM-then-Kill cascade against the child, bounded by a grace window per signal, so a wedged child is always eventually reaped. Deferred, flagged rather than half-built: MCP passthrough (--mcp-config) — translating engine/mcp.go's server specs into the CLI's own config shape and reconciling two permission models is separable follow-on work; --append-system-prompt is not auto-populated (Claude Code discovers its own CLAUDE.md/AGENTS.md, and harness's native-tool-shaped instructions would mislead a CLI with different tools) but reachable via the new ExtraArgs escape hatch; goal-loop error classification (provider.RetryableError-driven backoff/park decisions) does not extend to delegated-turn failures, which surface as plain errors — the directive-reuse retry path is verified to dispatch correctly and not break, but basic interactive Prompt-driven delegation is the supported v1 shape. Verification: go build/vet/test all green across the repo (one pre-existing, unrelated flaky test — TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhenLogCannotReconstruct — reproduces identically on origin/main with -count=5, confirmed before and after this change). New tests: config validation and merge semantics for the new type/fields; cmd/harness registry wiring and the config-to-engine translation; a compiled fake `claude` stand-in (engine/testdata/fakeclaude) driving real pipes and a real child process to prove the full event mapping, --resume across a session reload, error-result handling with usage still accounted, and that a canceled context interrupts a genuinely hanging child within bounds; a selection test proving a claude-code ref never touches the native provider and an ordinary ref never touches the delegated backend. --- cmd/harness/claude_code_test.go | 84 +++ cmd/harness/main.go | 57 ++ config/claude_code_cli_test.go | 180 ++++++ config/config.go | 110 +++- engine/claude_code_backend.go | 753 +++++++++++++++++++++++++ engine/claude_code_backend_test.go | 408 ++++++++++++++ engine/engine.go | 79 +++ engine/store.go | 69 +++ engine/testdata/fakeclaude/main.go | 137 +++++ message/message.go | 14 + modelmeta/modelmeta.go | 29 + provider/claudecode/claudecode.go | 51 ++ provider/claudecode/claudecode_test.go | 34 ++ 13 files changed, 2003 insertions(+), 2 deletions(-) create mode 100644 cmd/harness/claude_code_test.go create mode 100644 config/claude_code_cli_test.go create mode 100644 engine/claude_code_backend.go create mode 100644 engine/claude_code_backend_test.go create mode 100644 engine/testdata/fakeclaude/main.go create mode 100644 provider/claudecode/claudecode.go create mode 100644 provider/claudecode/claudecode_test.go diff --git a/cmd/harness/claude_code_test.go b/cmd/harness/claude_code_test.go new file mode 100644 index 00000000..6e9ae512 --- /dev/null +++ b/cmd/harness/claude_code_test.go @@ -0,0 +1,84 @@ +package main + +import ( + "testing" + + "github.com/majorcontext/harness/config" + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider/claudecode" +) + +// TestRegistryClaudeCodeCLIRegistersStubUnderKey proves a claude-code-cli +// entry registers a claudecode.Client under its providers-map key, exactly +// like registerOpenAICompatProviders does for its own type — this is what +// makes Session.ModelSupported accept a swap to a claude-code model ref +// (see registerClaudeCodeProviders' own doc comment). +func TestRegistryClaudeCodeCLIRegistersStubUnderKey(t *testing.T) { + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "claude-code": {Type: config.TypeClaudeCodeCLI, BinaryPath: "/usr/local/bin/claude"}, + }}) + if _, ok := reg["claude-code"].(claudecode.Client); !ok { + t.Fatalf("claude-code provider is %T, want claudecode.Client", reg["claude-code"]) + } + ref, err := message.ParseModelRef("claude-code/sonnet") + if err != nil { + t.Fatalf("ParseModelRef: %v", err) + } + if _, err := reg.For(ref); err != nil { + t.Errorf("reg.For(%s): %v, want a registered adapter", ref, err) + } +} + +// TestClaudeCodeConfigForTranslatesProviderFields proves +// claudeCodeConfigFor carries BinaryPath/ExtraArgs/PermissionMode from a +// config.Provider entry into engine.ClaudeCodeConfig — the one translation +// point between the file-config Provider type and the engine's own +// backend-agnostic Config (engine deliberately does not import config; see +// claudeCodeConfigFor's own doc comment). +func TestClaudeCodeConfigForTranslatesProviderFields(t *testing.T) { + cfg := &config.Config{Providers: map[string]config.Provider{ + "claude-code": { + Type: config.TypeClaudeCodeCLI, + BinaryPath: "/opt/claude/bin/claude", + ExtraArgs: []string{"--mcp-config", "/tmp/mcp.json"}, + PermissionMode: "acceptEdits", + }, + }} + got := claudeCodeConfigFor(cfg, claudecode.Family) + want := engine.ClaudeCodeConfig{ + BinaryPath: "/opt/claude/bin/claude", + ExtraArgs: []string{"--mcp-config", "/tmp/mcp.json"}, + PermissionMode: "acceptEdits", + } + if got.BinaryPath != want.BinaryPath || got.PermissionMode != want.PermissionMode { + t.Errorf("claudeCodeConfigFor = %+v, want %+v", got, want) + } + if len(got.ExtraArgs) != len(want.ExtraArgs) { + t.Fatalf("ExtraArgs = %+v, want %+v", got.ExtraArgs, want.ExtraArgs) + } + for i := range want.ExtraArgs { + if got.ExtraArgs[i] != want.ExtraArgs[i] { + t.Errorf("ExtraArgs[%d] = %q, want %q", i, got.ExtraArgs[i], want.ExtraArgs[i]) + } + } +} + +// TestClaudeCodeConfigForAbsentEntryYieldsZeroValue proves a config with no +// matching entry (or a nil *config.Config, the same defensive shape every +// other cmd/harness helper here handles) yields the zero ClaudeCodeConfig +// rather than panicking — engine.newSession's own BinaryPath default +// ("claude") then applies. +func TestClaudeCodeConfigForAbsentEntryYieldsZeroValue(t *testing.T) { + assertZero := func(t *testing.T, got engine.ClaudeCodeConfig) { + t.Helper() + if got.BinaryPath != "" || got.PermissionMode != "" || len(got.ExtraArgs) != 0 { + t.Errorf("claudeCodeConfigFor = %+v, want the zero value", got) + } + } + assertZero(t, claudeCodeConfigFor(nil, claudecode.Family)) + cfg := &config.Config{Providers: map[string]config.Provider{ + "anthropic": {APIKeyEnv: "X"}, + }} + assertZero(t, claudeCodeConfigFor(cfg, claudecode.Family)) +} diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 76af648f..394f9a03 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -32,6 +32,7 @@ import ( "github.com/majorcontext/harness/message" "github.com/majorcontext/harness/provider" "github.com/majorcontext/harness/provider/anthropic" + "github.com/majorcontext/harness/provider/claudecode" "github.com/majorcontext/harness/provider/openai" "github.com/majorcontext/harness/provider/openaicompat" "github.com/majorcontext/harness/server" @@ -778,6 +779,7 @@ func runCmd(args []string) error { ModelTool: cfg.ModelToolEnabled(), ModelAliases: cfg.Aliases, SessionManager: sessMgr, + ClaudeCode: claudeCodeConfigFor(cfg, claudecode.Family), }, opts.resume, opts.cont, modelSet) if err != nil { return err @@ -943,10 +945,60 @@ func registry(cfg *config.Config) provider.Registry { } registerOpenAICompatProviders(reg, cfg) registerOpenAIProviders(reg, cfg) + registerClaudeCodeProviders(reg, cfg) ensureDefaultOpenRouter(reg, cfg) return reg } +// registerClaudeCodeProviders registers a claudecode.Client — a +// provider.Provider stand-in never expected to actually stream, see that +// package's own doc comment — for every config.Providers entry of +// config.TypeClaudeCodeCLI, keyed by its providers map name, exactly like +// registerOpenAICompatProviders. This is what makes Session.ModelSupported +// (engine/engine.go, consulted by the `model` tool, POST +// /session/{id}/model, and Spawn's model-override validation) accept a +// swap to a claude-code model ref: without an entry in the registry under +// that key, ModelSupported would reject the very refs +// engine.ClaudeCodeProviderFamily's delegated-turn dispatch exists to +// serve, even though that dispatch never actually calls into this +// registered client's Stream method. +func registerClaudeCodeProviders(reg provider.Registry, cfg *config.Config) { + if cfg == nil { + return + } + for name, p := range cfg.Providers { + if p.Type != config.TypeClaudeCodeCLI { + continue + } + reg[name] = claudecode.Client{} + } +} + +// claudeCodeConfigFor resolves the engine.ClaudeCodeConfig for the given +// providers-map key (by convention claudecode.Family, "claude-code") from a +// config.TypeClaudeCodeCLI entry, translating config.Provider's +// BinaryPath/ExtraArgs/PermissionMode fields into their engine.Config +// counterpart. Absent (no such entry, or cfg nil) yields the zero value, +// which engine.newSession defaults BinaryPath from ("claude" — see that +// function). Package engine deliberately does not import package config +// (see engine.Config's own field-by-field translation precedent, e.g. +// SessionSync/ContextWindowTokens above), so this narrow translation lives +// here, at the one boundary that already does it for every other field. +func claudeCodeConfigFor(cfg *config.Config, name string) engine.ClaudeCodeConfig { + if cfg == nil { + return engine.ClaudeCodeConfig{} + } + p, ok := cfg.Providers[name] + if !ok || p.Type != config.TypeClaudeCodeCLI { + return engine.ClaudeCodeConfig{} + } + return engine.ClaudeCodeConfig{ + BinaryPath: p.BinaryPath, + ExtraArgs: p.ExtraArgs, + PermissionMode: p.PermissionMode, + } +} + // registerOpenAIProviders builds a native provider/openai (Responses API) // client for every config.Providers entry of config.TypeOpenAI, keyed by // its providers map name — that name is what routes "name/model" refs to @@ -1606,6 +1658,11 @@ func serveCmd(args []string) error { // tool-driven `set` resolves an alias like the CLI does. ModelTool: cfg.ModelToolEnabled(), ModelAliases: cfg.Aliases, + // ClaudeCode carries the BinaryPath/ExtraArgs/PermissionMode a + // config.TypeClaudeCodeCLI entry configures (see + // claudeCodeConfigFor); zero value when none is configured, + // which engine.newSession defaults BinaryPath from ("claude"). + ClaudeCode: claudeCodeConfigFor(cfg, claudecode.Family), } } // monitorPage is a named local (rather than inlining monitor.Page below) diff --git a/config/claude_code_cli_test.go b/config/claude_code_cli_test.go new file mode 100644 index 00000000..26a6abef --- /dev/null +++ b/config/claude_code_cli_test.go @@ -0,0 +1,180 @@ +package config + +import ( + "strings" + "testing" +) + +// TestLoadProviderClaudeCodeCLI proves a minimal, valid entry parses and +// keeps its BinaryPath/ExtraArgs/PermissionMode fields intact — the same +// shape TestLoadProviderOpenAICompat establishes for that type. +func TestLoadProviderClaudeCodeCLI(t *testing.T) { + dir := t.TempDir() + p := dir + "/config.json" + writeFile(t, p, `{ + "providers": { + "claude-code": { + "type": "claude-code-cli", + "binary_path": "/usr/local/bin/claude", + "extra_args": ["--mcp-config", "/tmp/mcp.json"], + "permission_mode": "acceptEdits" + } + } + }`) + c, err := Load(p) + if err != nil { + t.Fatalf("Load: %v", err) + } + pr, ok := c.Providers["claude-code"] + if !ok { + t.Fatal("providers.claude-code missing") + } + if pr.Type != TypeClaudeCodeCLI { + t.Errorf("Type = %q, want %q", pr.Type, TypeClaudeCodeCLI) + } + if pr.BinaryPath != "/usr/local/bin/claude" { + t.Errorf("BinaryPath = %q", pr.BinaryPath) + } + if len(pr.ExtraArgs) != 2 || pr.ExtraArgs[0] != "--mcp-config" { + t.Errorf("ExtraArgs = %+v", pr.ExtraArgs) + } + if pr.PermissionMode != "acceptEdits" { + t.Errorf("PermissionMode = %q", pr.PermissionMode) + } +} + +// TestClaudeCodeCLINoBaseURLRequired proves the one deliberate divergence +// from TypeOpenAICompat/TypeOpenAI: this type spawns a process rather than +// dialing an HTTP endpoint, so validateProviders must accept an entry with +// no base_url at all. +func TestClaudeCodeCLINoBaseURLRequired(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI}, + }} + if _, err := mergeAndValidate(c, &Config{}); err != nil { + t.Fatalf("mergeAndValidate: %v, want no error for a base_url-less claude-code-cli entry", err) + } +} + +// TestClaudeCodeCLIUnknownPermissionModeFails proves a typo'd +// permission_mode fails loudly at config-load time rather than reaching +// the `claude` child's own --permission-mode flag and failing there, +// mirroring TestLoadProviderOpenAICompatMissingBaseURLFails's own +// "fail close to the mistake" shape. +func TestClaudeCodeCLIUnknownPermissionModeFails(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI, PermissionMode: "yolo"}, + }} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on unknown permission_mode") + } + if !strings.Contains(err.Error(), "yolo") { + t.Errorf("error %q does not name the offending value", err) + } +} + +// TestClaudeCodeCLIEmptyPermissionModeOK proves the field is optional: an +// entry naming no permission_mode at all is valid (the CLI's own default +// applies, no flag sent). +func TestClaudeCodeCLIEmptyPermissionModeOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI}, + }} + if _, err := mergeAndValidate(c, &Config{}); err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } +} + +// TestClaudeCodeFieldsRejectedOnOtherTypes proves BinaryPath/ExtraArgs/ +// PermissionMode are type-scoped, exactly like ResponsesPath/CacheTTL are +// for their own adapters — a config author setting one of these on, say, +// an openai-compat entry gets a loud, specific error instead of a value +// that silently vanishes into a client that never reads it. +func TestClaudeCodeFieldsRejectedOnOtherTypes(t *testing.T) { + tests := []struct { + name string + p Provider + }{ + {"binary_path", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", BinaryPath: "/bin/claude"}}, + {"extra_args", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", ExtraArgs: []string{"--foo"}}}, + {"permission_mode", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", PermissionMode: "acceptEdits"}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + c := &Config{Providers: map[string]Provider{"mycompat": tc.p}} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatalf("mergeAndValidate did not fail on %s set for a non-claude-code-cli entry", tc.name) + } + if !strings.Contains(err.Error(), TypeClaudeCodeCLI) { + t.Errorf("error %q does not name %q as the only valid type", err, TypeClaudeCodeCLI) + } + }) + } +} + +// TestClaudeCodeCLIUnknownTypeErrorListsIt proves the unknown-type error +// message advertises claude-code-cli alongside the other valid types, so a +// config author typo'ing the type string is told the real option exists. +func TestClaudeCodeCLIUnknownTypeErrorListsIt(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "mystery": {Type: "carrier-pigeon"}, + }} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatal("mergeAndValidate did not fail on unknown provider type") + } + if !strings.Contains(err.Error(), TypeClaudeCodeCLI) { + t.Errorf("error %q does not list %q as a valid type", err, TypeClaudeCodeCLI) + } +} + +// TestMergeClaudeCodeExtraArgsNotAliased mirrors +// TestMergeProviderExtraHeadersBaseOnlyKeyNotAliased for the new slice +// field: a base-only key's ExtraArgs must not alias the base config's own +// backing array once merged. +func TestMergeClaudeCodeExtraArgsNotAliased(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI, ExtraArgs: []string{"--a"}}, + }} + merged := merge(base, &Config{}) + pr := merged.Providers["claude-code"] + pr.ExtraArgs[0] = "mutated" + if base.Providers["claude-code"].ExtraArgs[0] != "--a" { + t.Error("merge aliased the base provider's ExtraArgs slice") + } +} + +// TestMergeClaudeCodeFieldsOverride proves a project-layer override +// replaces BinaryPath/PermissionMode and replaces ExtraArgs wholesale +// (never element-wise merges it), exactly like OmitResponseParams' +// documented merge semantics. +func TestMergeClaudeCodeFieldsOverride(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "claude-code": { + Type: TypeClaudeCodeCLI, + BinaryPath: "/user/claude", + ExtraArgs: []string{"--user-flag"}, + PermissionMode: "default", + }, + }} + over := &Config{Providers: map[string]Provider{ + "claude-code": { + BinaryPath: "/project/claude", + ExtraArgs: []string{"--project-flag"}, + PermissionMode: "acceptEdits", + }, + }} + merged := merge(base, over) + pr := merged.Providers["claude-code"] + if pr.BinaryPath != "/project/claude" { + t.Errorf("BinaryPath = %q, want project override", pr.BinaryPath) + } + if len(pr.ExtraArgs) != 1 || pr.ExtraArgs[0] != "--project-flag" { + t.Errorf("ExtraArgs = %+v, want wholesale project override", pr.ExtraArgs) + } + if pr.PermissionMode != "acceptEdits" { + t.Errorf("PermissionMode = %q, want project override", pr.PermissionMode) + } +} diff --git a/config/config.go b/config/config.go index 721c11a8..ff5c600c 100644 --- a/config/config.go +++ b/config/config.go @@ -413,6 +413,21 @@ const TypeOpenAICompat = "openai-compat" // built-in default. const TypeOpenAI = "openai" +// TypeClaudeCodeCLI selects the Claude Code CLI delegated-turn backend +// (engine/claude_code_backend.go) for a Provider config entry, under +// whatever providers map key names it (by convention "claude-code" — that +// key becomes the ModelRef.Provider a session's model ref must name to be +// routed to it, e.g. "claude-code/sonnet"). +// +// Unlike TypeOpenAICompat/TypeOpenAI, this is NOT an HTTP adapter: the +// engine spawns the `claude` binary as a child process and bridges its +// `--output-format stream-json` event stream into the session's own +// journal/event pipeline directly (see that file's package doc) — it never +// goes through provider.Provider.Stream in normal operation. BaseURL is +// therefore never required for this type (see validateProviders); the +// fields it DOES read are BinaryPath, ExtraArgs, and PermissionMode, below. +const TypeClaudeCodeCLI = "claude-code-cli" + // nativeProviderKeys are the only providers map keys allowed an empty Type // with no further defaulting: the built-in adapters cmd/harness's registry // wires directly by name (provider/anthropic.Family and @@ -599,6 +614,47 @@ type Provider struct { // ONLY on the native "anthropic" entry: no other adapter reads it, so // validateProviders rejects it elsewhere rather than ignoring it. CacheTTL string `json:"cache_ttl,omitempty"` + + // BinaryPath is the executable this entry's Claude Code CLI child + // process is spawned from — resolved via PATH like any exec, exactly + // as PluginSpec.Command's Command[0]. Empty (the default) spawns + // "claude", the CLI's own published binary name. Valid ONLY on a + // TypeClaudeCodeCLI entry — no other adapter spawns a process at all — + // so validateProviders rejects it elsewhere, the same rule every other + // type-scoped field in this struct follows. + BinaryPath string `json:"binary_path,omitempty"` + // ExtraArgs are appended verbatim to the `claude` invocation, after + // every flag the engine itself constructs (--input-format, + // --output-format, --verbose, --model, --resume, --permission-mode — + // see engine/claude_code_backend.go). This is the escape hatch for a + // flag this Provider struct has no dedicated field for (e.g. + // --append-system-prompt, --mcp-config, --allowedTools) — the engine + // deliberately does not auto-populate either of those two itself for + // v1 (see that file's package doc for why). Valid ONLY on a + // TypeClaudeCodeCLI entry. Merge semantics are additive like every + // other Provider slice field (see NoPromptCacheKey's doc comment): a + // non-empty project-layer list replaces the user-layer list wholesale. + ExtraArgs []string `json:"extra_args,omitempty"` + // PermissionMode selects the `claude` child's --permission-mode flag + // (one of ClaudeCodePermissionModeValues — "default", "acceptEdits", + // "bypassPermissions", "plan"; see the Claude Code CLI's own + // documentation for what each does). Empty (the default) omits the + // flag entirely, leaving the CLI's own default in place. Valid ONLY on + // a TypeClaudeCodeCLI entry. + PermissionMode string `json:"permission_mode,omitempty"` +} + +// ClaudeCodePermissionModeValues returns every value validatePermissionMode +// accepts, in a fresh slice — the Claude Code CLI's own --permission-mode +// choices. validatePermissionMode iterates this same list, so the set and +// the validator cannot drift from each other. +func ClaudeCodePermissionModeValues() []string { + return []string{ + "default", + "acceptEdits", + "bypassPermissions", + "plan", + } } // Cache TTL values accepted by Provider.CacheTTL. They mirror @@ -699,7 +755,7 @@ func validateProviders(providers map[string]Provider) error { switch p.Type { case "": if !nativeProviderKeys[name] { - return fmt.Errorf("providers.%s: type is required (empty type is only valid for the built-in %q/%q entries); valid types: \"\" (native anthropic/openai override), %q, %q", name, "anthropic", "openai", TypeOpenAICompat, TypeOpenAI) + return fmt.Errorf("providers.%s: type is required (empty type is only valid for the built-in %q/%q entries); valid types: \"\" (native anthropic/openai override), %q, %q, %q", name, "anthropic", "openai", TypeOpenAICompat, TypeOpenAI, TypeClaudeCodeCLI) } // Legacy/native provider entry (anthropic or openai); no // further validation here. @@ -707,8 +763,12 @@ func validateProviders(providers map[string]Provider) error { if p.BaseURL == "" { return fmt.Errorf("providers.%s: base_url is required for type %q", name, p.Type) } + case TypeClaudeCodeCLI: + // No base_url requirement: this type spawns a child process + // (engine/claude_code_backend.go), it never dials an HTTP + // endpoint — see TypeClaudeCodeCLI's own doc comment. default: - return fmt.Errorf("providers.%s: unknown type %q (valid types: \"\" (native anthropic/openai override), %q, %q)", name, p.Type, TypeOpenAICompat, TypeOpenAI) + return fmt.Errorf("providers.%s: unknown type %q (valid types: \"\" (native anthropic/openai override), %q, %q, %q)", name, p.Type, TypeOpenAICompat, TypeOpenAI, TypeClaudeCodeCLI) } if err := validateCacheTTL(name, p); err != nil { return err @@ -722,6 +782,37 @@ func validateProviders(providers map[string]Provider) error { if err := validateOmitResponseParams(name, p); err != nil { return err } + if err := validateClaudeCodeFields(name, p); err != nil { + return err + } + } + return nil +} + +// validateClaudeCodeFields fails loudly on BinaryPath/ExtraArgs/ +// PermissionMode set on any entry that is not TypeClaudeCodeCLI — no other +// adapter reads them, the same silent-misconfiguration class +// validateResponsesPath and validateCacheTTL refuse to allow — and on an +// unrecognized PermissionMode value on an entry that IS that type (a typo +// would otherwise reach the `claude` child's --permission-mode flag +// verbatim and fail there instead, far from the config that caused it). +// Empty PermissionMode is always valid: it omits the flag, leaving the +// CLI's own default in place. +func validateClaudeCodeFields(name string, p Provider) error { + if p.Type == TypeClaudeCodeCLI { + if p.PermissionMode != "" && !slices.Contains(ClaudeCodePermissionModeValues(), p.PermissionMode) { + return fmt.Errorf("providers.%s: unknown permission_mode %q (valid values: %q)", name, p.PermissionMode, ClaudeCodePermissionModeValues()) + } + return nil + } + if p.BinaryPath != "" { + return fmt.Errorf("providers.%s: binary_path is only valid on a %q entry (only the Claude Code CLI backend spawns a process)", name, TypeClaudeCodeCLI) + } + if len(p.ExtraArgs) > 0 { + return fmt.Errorf("providers.%s: extra_args is only valid on a %q entry", name, TypeClaudeCodeCLI) + } + if p.PermissionMode != "" { + return fmt.Errorf("providers.%s: permission_mode is only valid on a %q entry", name, TypeClaudeCodeCLI) } return nil } @@ -1183,6 +1274,9 @@ func merge(base, over *Config) *Config { if len(v.OmitResponseParams) > 0 { v.OmitResponseParams = append([]string(nil), v.OmitResponseParams...) } + if len(v.ExtraArgs) > 0 { + v.ExtraArgs = append([]string(nil), v.ExtraArgs...) + } m[k] = v } for k, v := range over.Providers { @@ -1211,6 +1305,15 @@ func merge(base, over *Config) *Config { if v.NoPromptCacheKey { ex.NoPromptCacheKey = true } + if v.BinaryPath != "" { + ex.BinaryPath = v.BinaryPath + } + if len(v.ExtraArgs) > 0 { + ex.ExtraArgs = append([]string(nil), v.ExtraArgs...) + } + if v.PermissionMode != "" { + ex.PermissionMode = v.PermissionMode + } if n := len(ex.ExtraHeaders) + len(v.ExtraHeaders); n > 0 { hm := make(map[string]string, n) for hk, hv := range ex.ExtraHeaders { @@ -1233,6 +1336,9 @@ func merge(base, over *Config) *Config { if len(v.OmitResponseParams) > 0 { v.OmitResponseParams = append([]string(nil), v.OmitResponseParams...) } + if len(v.ExtraArgs) > 0 { + v.ExtraArgs = append([]string(nil), v.ExtraArgs...) + } m[k] = v } } diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go new file mode 100644 index 00000000..34446f9c --- /dev/null +++ b/engine/claude_code_backend.go @@ -0,0 +1,753 @@ +// The Claude Code CLI delegated-turn backend. +// +// # Why this exists +// +// A box running on a user's Claude SUBSCRIPTION (rather than metered API +// access) must run through Anthropic's own client — Anthropic blocks +// third-party harnesses from authenticating against a subscription +// directly. harness cannot BE that client for a subscription-backed turn. +// It can, however, stay the session/journal/HTTP-API front door and +// DELEGATE the turn to the real `claude` CLI, driven headlessly over its +// documented `--input-format stream-json --output-format stream-json` +// protocol, bridging that event stream into this package's own +// Session.append/emit/journal pipeline — so the console and every other +// read-path consumer see an ordinary transcript, unaware that a particular +// turn's tool loop ran inside `claude` rather than inside this package's +// own runToolCalls. +// +// # The seam +// +// A session is "delegated" purely by its model ref: message.ModelRef{ +// Provider: ClaudeCodeProviderFamily, Model: "sonnet"} (or "opus", "haiku", +// any name the `claude` binary's own --model flag accepts). Two entry +// points check claudeCodeDelegated() and dispatch here, BEFORE any +// native-loop-only step runs (see each call site's own doc comment for why +// both exist rather than one): +// +// - PromptWithOrigin (engine.go), for every ordinary Prompt/ +// PromptEngineResume call — dispatches before ContextWindowErr, +// ensureInstructions, ensureSkills, and maybeAutoCompact, none of +// which apply to a turn Claude Code drives with its own context +// management. +// - runAgenticLoop (engine.go), for goal.go's promptTurnWithRetry +// directive-reuse retry, which calls runAgenticLoop DIRECTLY, +// bypassing PromptWithOrigin entirely. +// +// Both roads converge on runClaudeCodeTurn below, which reads the tail of +// s.History() — always exactly one appended, not-yet-answered RoleUser +// message, whichever caller appended it — and drives ONE `claude` turn +// against it. +// +// # Event mapping +// +// Each line of the CLI's stream-json stdout is one JSON object with a +// discriminating "type" field (claudeCodeEnvelope below). This mapping is +// the best-faithful reading of the CLI's own documented stream-json +// protocol; any shape this file has not verified live against a real +// `claude` binary is called out in the relevant type/function's own +// comment, and decoding is deliberately permissive (unknown fields +// ignored, an unrecognized top-level "type" or "subtype" treated as +// inert activity rather than a hard failure) so a future CLI version +// that adds a field or event this file doesn't yet know about degrades to +// "nothing observable happened" instead of crashing a turn. +// +// - "system"/"init": captures Claude Code's OWN session id +// (claudeCodeCLISessionID) for --resume on this harness session's next +// delegated turn. Any other subtype (e.g. "api_retry") is activity +// only — observed, never fatal. +// - "assistant": one COMPLETE API-level assistant message (text and/or +// tool_use content blocks together) — NOT a token-by-token delta; this +// driver does not pass --include-partial-messages, so there is nothing +// more granular to forward. Decoded into one message.Message (Text and +// ToolCall parts, in order), appended via plain Session.append (no +// usage — see the usage-mapping note below) and emitted as +// EventMessage, with one EventTextDelta per non-empty text part +// (folding the whole block's text into the message in a single +// "delta", the closest honest match to the native EventTextDelta +// contract given the CLI hands over complete blocks) and one +// EventToolStart per tool_use part. +// - "user": Claude Code's own tool execution results, arriving in the +// Anthropic API's own convention of a "user"-role message carrying +// tool_result content blocks (Claude Code executes its OWN tools here +// — this package never calls runToolCalls for a delegated turn). +// Decoded into one RoleTool message.Message (one ToolResult part per +// block), appended and emitted as EventMessage, with one EventToolEnd +// per part. +// - "result": the turn's terminal event. Never itself appended as a +// message (the assistant text it summarizes was already appended by +// the last "assistant" event above) — it instead carries the turn's +// AGGREGATE usage, applied once via applyClaudeCodeUsage (a durable, +// message-independent record — see recClaudeCodeUsage in store.go). +// An IsError result becomes this call's returned error; TotalCostUSD +// has no home in provider.Usage (no adapter carries a cost field — +// every consumer derives cost from token counts) and is deliberately +// dropped, not persisted. +// +// # Deferred for v1 (flagged, not silently skipped) +// +// - MCP passthrough (`--mcp-config`): a delegated turn does not forward +// harness's configured MCP servers to the `claude` child. Wiring this +// correctly means translating engine/mcp.go's server specs into the +// CLI's own --mcp-config JSON shape and reconciling two independent +// permission/tool-approval models — real, separable follow-on work. +// - `--append-system-prompt`: not auto-populated from s.cfg.System. +// Harness's system-prompt assembly (project instructions, Agent +// Skills, tool-batching guidance) is deliberately native-loop-only +// (see PromptWithOrigin's dispatch comment) — Claude Code already +// discovers its own CLAUDE.md/AGENTS.md in the box workspace, and +// re-injecting harness's OWN native-tool-shaped instructions into a +// CLI that has different tools would be actively misleading. An +// operator who wants extra injected wording can still reach the flag +// via config.Provider.ExtraArgs. +// - Full goal-loop support: the directive-reuse retry path (see above) +// IS dispatched correctly, so a goal loop driving a delegated session +// does not error out — but goal.go's retryable-error CLASSIFICATION +// (provider.RetryableError, promptTurnWithRetry's backoff/park +// decisions) is shaped entirely around native provider.Stream errors. +// An error this file returns is a plain error, never classified +// retryable, so a goal loop treats every delegated-turn failure as a +// deterministic stall rather than transient provider weather. Basic +// interactive Prompt-driven delegation is the verified, supported +// shape for v1. +package engine + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os/exec" + "strings" + "syscall" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// ClaudeCodeProviderFamily is the message.ModelRef.Provider value that +// selects this delegated-turn backend — see claudeCodeDelegated. It +// matches config.TypeClaudeCodeCLI's conventional providers-map key and +// provider/claudecode.Family (that package cannot import this one — see +// its own doc comment for why the string is duplicated there rather than +// imported, and TestClaudeCodeProviderFamilyMatchesClaudecodePackage for +// the parity check). +const ClaudeCodeProviderFamily = "claude-code" + +// defaultClaudeCodeBinaryPath is ClaudeCodeConfig.BinaryPath's zero-value +// default (newSession) — the CLI's own published binary name, resolved via +// PATH like any exec. +const defaultClaudeCodeBinaryPath = "claude" + +// claudeCodeInterruptGrace bounds how long runClaudeCodeTurn waits for the +// `claude` child to exit on its own after each escalating signal, before +// sending the next one — see runClaudeCodeTurn's signal-cascade goroutine. +// A var, not a const, so a test can shrink it rather than paying the real +// wall-clock cost. +var claudeCodeInterruptGrace = 5 * time.Second + +// claudeCodeStderrCap bounds how much of the `claude` child's stderr this +// file retains for an error message — enough to be a useful diagnostic +// (a missing binary, a permission error, an early crash) without letting +// a runaway or malicious child exhaust memory buffering it. +const claudeCodeStderrCap = 4096 + +// ClaudeCodeConfig configures the delegated-turn backend — see +// Config.ClaudeCode's own doc comment. It is engine's own minimal +// translation target for config.Provider's BinaryPath/ExtraArgs/ +// PermissionMode fields (cmd/harness's claudeCodeConfigFor does the +// translation): package engine does not import package config, the same +// separation every other Config field already keeps. +type ClaudeCodeConfig struct { + // BinaryPath is the `claude` executable to spawn, resolved via PATH + // like any exec. Empty defaults to "claude" (newSession). + BinaryPath string + // ExtraArgs are appended verbatim after every flag this file + // constructs itself. See config.Provider.ExtraArgs's doc comment for + // the escape-hatch flags this is for (--append-system-prompt, + // --mcp-config, --allowedTools, ...). + ExtraArgs []string + // PermissionMode, if non-empty, becomes --permission-mode . + PermissionMode string +} + +// claudeCodeDelegated reports whether s's CURRENT model routes to this +// delegated backend rather than a native provider.Provider — see +// ClaudeCodeProviderFamily. +func (s *Session) claudeCodeDelegated() bool { + return s.Model().Provider == ClaudeCodeProviderFamily +} + +// claudeCodeSessionID returns the Claude Code CLI's own session id +// captured on this session's most recent delegated turn (see +// Session.claudeCodeCLISessionID's own doc comment), or "" before the +// first one has completed an init event. +func (s *Session) claudeCodeSessionID() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.claudeCodeCLISessionID +} + +// recordClaudeCodeSessionID durably records id as this session's Claude +// Code CLI session id, for --resume on every later delegated turn. A no-op +// when id is empty or already recorded, so a repeat init event (there is +// exactly one per turn, but nothing prevents a future CLI version from +// emitting more) never writes a redundant journal record. +func (s *Session) recordClaudeCodeSessionID(id string) { + if id == "" { + return + } + s.mu.Lock() + defer s.mu.Unlock() + if s.claudeCodeCLISessionID == id { + return + } + s.claudeCodeCLISessionID = id + s.persistClaudeCodeSessionID(id) +} + +// applyClaudeCodeUsage folds a delegated turn's AGGREGATE usage (the +// "result" event's own usage field, covering every internal API call +// Claude Code made across the whole turn — not just the closing one) into +// Session.Usage()/LastUsage(), durably (recClaudeCodeUsage, store.go). +// +// This is deliberately NOT routed through appendWithUsage: by the time a +// "result" event arrives, every message this turn produced has already +// been appended (plain Session.append, no usage attached — see this +// file's package doc, "final result" bullet) in receipt order, matching +// each one to the EventMessage/EventToolStart/EventToolEnd emit it needs +// at the moment it actually happened; retroactively re-appending a +// duplicate "terminal" message purely to give appendWithUsage somewhere to +// attach Usage would double the journal's message count for every +// delegated turn. Unlike compact.go's recCompact precedent (which folds +// its own Usage into cumulative ONLY, deliberately leaving lastUsage +// alone, because compact's caller must keep sizing auto-compaction off the +// last REAL prompt), this DOES set lastUsage: harness's own auto- +// compaction never runs for a delegated session (see PromptWithOrigin's +// dispatch), so there is no native trigger signal here to protect, and +// GET /session should still report an accurate last-turn size. +func (s *Session) applyClaudeCodeUsage(usage provider.Usage) { + s.mu.Lock() + defer s.mu.Unlock() + s.usage.InputTokens += usage.InputTokens + s.usage.OutputTokens += usage.OutputTokens + s.usage.CacheReadTokens += usage.CacheReadTokens + s.usage.CacheWriteTokens += usage.CacheWriteTokens + s.lastUsage = usage + s.haveLastUsage = true + s.persistClaudeCodeUsage(usage) +} + +// runClaudeCodeTurn drives ONE turn through the `claude` CLI against s's +// CURRENT history tail — the single, most-recently-appended, not-yet- +// answered RoleUser message, appended by whichever of PromptWithOrigin or +// goal.go's directive-reuse retry called in (see this file's package doc). +// It returns the final assistant message.Message on success. +// +// Every message this turn produces is appended to s.History() as it is +// decoded from the child's stdout, so a turn that fails PARTWAY THROUGH — +// after some tool calls already ran — still leaves an accurate, ungapped +// transcript behind; only the (*message.Message, error) return itself +// reports the failure, exactly like the native path's +// interruptedTurnError partial-append behavior (engine.go). +func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, error) { + text := lastUserMessageText(s.History()) + if text == "" { + return nil, errors.New("engine: claude-code delegated turn found no pending user message to answer") + } + + cfg := s.cfg.ClaudeCode + binary := cfg.BinaryPath + if binary == "" { + binary = defaultClaudeCodeBinaryPath + } + model := s.Model() + + args := []string{ + "--input-format", "stream-json", + "--output-format", "stream-json", + "--verbose", + } + if model.Model != "" { + args = append(args, "--model", model.Model) + } + if resumeID := s.claudeCodeSessionID(); resumeID != "" { + args = append(args, "--resume", resumeID) + } + if cfg.PermissionMode != "" { + args = append(args, "--permission-mode", cfg.PermissionMode) + } + args = append(args, cfg.ExtraArgs...) + + cmd := exec.Command(binary, args...) //nolint:gosec // binary/args are operator config, not request input + cmd.Dir = s.cfg.WorkDir + // Auth is deliberately NOT this package's job: no ANTHROPIC_API_KEY is + // set, no ~/.claude/.credentials.json is read or written here. The + // child inherits harness's own ambient environment verbatim — on the + // boxes platform, that is whatever placeholder credential material and + // gatekeeper routing the BOX already set up before harness ever ran + // (see this file's package doc). cmd.Env left nil means exactly that: + // os/exec's own documented behavior is to inherit os.Environ(). + + stdin, err := cmd.StdinPipe() + if err != nil { + return nil, fmt.Errorf("engine: claude-code: creating stdin pipe: %w", err) + } + stdout, err := cmd.StdoutPipe() + if err != nil { + return nil, fmt.Errorf("engine: claude-code: creating stdout pipe: %w", err) + } + var stderr capBuffer + stderr.cap = claudeCodeStderrCap + cmd.Stderr = &stderr + + if err := cmd.Start(); err != nil { + return nil, fmt.Errorf("engine: claude-code: starting %q: %w", binary, err) + } + + inputLine, err := json.Marshal(claudeCodeInputMessage{ + Type: "user", + Message: claudeCodeInputInnerMessage{ + Role: "user", + Content: text, + }, + }) + if err != nil { + _ = cmd.Process.Kill() + _ = cmd.Wait() + return nil, fmt.Errorf("engine: claude-code: encoding turn input: %w", err) + } + if _, err := stdin.Write(append(inputLine, '\n')); err != nil { + _ = cmd.Process.Kill() + _ = cmd.Wait() + return nil, fmt.Errorf("engine: claude-code: writing turn input: %w", err) + } + // Close stdin: this driver sends exactly one turn per `claude` child + // (continuity across harness turns is --resume, not a long-lived + // child — see this file's package doc), and an unclosed stdin would + // leave the CLI waiting indefinitely for a second message that is + // never coming, wedging cmd.Wait() below forever. + if err := stdin.Close(); err != nil { + _ = cmd.Process.Kill() + _ = cmd.Wait() + return nil, fmt.Errorf("engine: claude-code: closing stdin: %w", err) + } + + // The signal-abort cascade: SIGINT first (Claude Code's own docs + // describe this as ending the current turn gracefully, leaving it + // --resume-able), escalating to SIGTERM and finally an unconditional + // Kill if the child does not exit within claudeCodeInterruptGrace of + // each — so a harness-side abort/shutdown always eventually reaps a + // wedged child rather than leaking it. done is closed once this + // call's own Wait returns, so the goroutine never fires a signal at a + // process this call has already reaped. + done := make(chan struct{}) + defer close(done) + go func() { + select { + case <-ctx.Done(): + case <-done: + return + } + proc := cmd.Process + if proc == nil { + return + } + _ = proc.Signal(syscall.SIGINT) + select { + case <-done: + return + case <-time.After(claudeCodeInterruptGrace): + } + _ = proc.Signal(syscall.SIGTERM) + select { + case <-done: + return + case <-time.After(claudeCodeInterruptGrace): + } + _ = proc.Kill() + }() + + finalMsg, turnErr := s.consumeClaudeCodeStream(stdout, model) + + waitErr := cmd.Wait() + + if ctx.Err() != nil { + // An abort/shutdown-driven cancellation always wins over whatever + // the stream decoded — the same precedence the native path's + // context.Canceled handling gives ctx (engine.go's + // streamTurnWithRetry/runAgenticLoop treat a canceled context as a + // deliberate stop, not an ordinary failure). + return nil, ctx.Err() + } + if turnErr != nil { + return nil, turnErr + } + if waitErr != nil { + msg := fmt.Sprintf("engine: claude-code: %q exited with error: %v", binary, waitErr) + if tail := stderr.String(); tail != "" { + msg += fmt.Sprintf(" (stderr: %s)", tail) + } + return nil, errors.New(msg) + } + if finalMsg == nil { + return nil, errors.New("engine: claude-code: turn ended with no assistant message") + } + return finalMsg, nil +} + +// lastUserMessageText returns the Text of the LAST message in history if +// it is a RoleUser message, or "" otherwise. runClaudeCodeTurn's one +// caller-contract requirement is that its caller has already appended +// (or, for the goal-loop directive-reuse retry, left in place) exactly one +// unanswered RoleUser message at the tail — this reads it back rather than +// threading the text through an extra parameter, so both call sites (a +// fresh append, and a retry that appends nothing new) share one path. +func lastUserMessageText(history []message.Message) string { + if len(history) == 0 { + return "" + } + last := history[len(history)-1] + if last.Role != message.RoleUser { + return "" + } + return last.Parts.Text() +} + +// consumeClaudeCodeStream reads newline-delimited stream-json events from +// r (the `claude` child's stdout) until EOF, appending/emitting each +// decoded event per this file's package-doc mapping, and returns the last +// assistant message.Message it appended (nil if none) plus any turn- +// ending error a "result" event's IsError reported. +func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) (*message.Message, error) { + scanner := bufio.NewScanner(r) + // A tool call's arguments or a large tool result can exceed + // bufio.Scanner's 64KiB default token size; 8MiB comfortably covers + // any realistic single stream-json line without an unbounded read. + scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024) + + var finalMsg *message.Message + var turnErr error + for scanner.Scan() { + line := bytes.TrimSpace(scanner.Bytes()) + if len(line) == 0 { + continue + } + var env claudeCodeEnvelope + if err := json.Unmarshal(line, &env); err != nil { + // A line this decoder cannot even parse as JSON: never crash a + // turn over one malformed/unexpected line from the child — + // see this file's package doc on permissive decoding. + continue + } + switch env.Type { + case "system": + if env.Subtype == "init" { + s.recordClaudeCodeSessionID(env.SessionID) + } + // Any other subtype (e.g. "api_retry") is observed but + // requires no action — see the package doc's event-mapping + // section. + case "assistant": + msg := claudeCodeAssistantMessage(env.Message, model) + if len(msg.Parts) == 0 { + continue + } + s.append(msg) + s.emit(Event{Type: EventMessage, Message: &msg}) + for _, p := range msg.Parts { + switch part := p.(type) { + case *message.Text: + if part.Text != "" { + s.emit(Event{Type: EventTextDelta, Text: part.Text}) + } + case *message.ToolCall: + s.emit(Event{Type: EventToolStart, ToolCall: part}) + } + } + finalMsg = &msg + case "user": + msg := claudeCodeToolResultMessage(env.Message) + if msg == nil { + continue + } + s.append(*msg) + s.emit(Event{Type: EventMessage, Message: msg}) + for _, p := range msg.Parts { + if tr, ok := p.(*message.ToolResult); ok { + s.emit(Event{ + Type: EventToolEnd, + ToolCall: &message.ToolCall{CallID: tr.CallID}, + Output: tr.Content, + IsError: tr.IsError, + }) + } + } + case "result": + s.applyClaudeCodeUsage(mapClaudeCodeUsage(env.Usage)) + if env.IsError { + turnErr = fmt.Errorf("engine: claude-code: turn ended in error (subtype %q): %s", env.Subtype, env.Result) + } + // TotalCostUSD is intentionally dropped here — see the + // package doc's "result" bullet: provider.Usage has no cost + // field for it to occupy. + } + // Any other top-level "type" (this driver has none documented + // beyond the four above) is inert activity, per the package doc. + } + return finalMsg, turnErr +} + +// claudeCodeEnvelope is the outer discriminator every line of `claude +// --output-format stream-json` decodes into. Fields are read permissively +// (encoding/json ignores JSON fields with no matching Go field, and every +// field here is optional so a line missing one just zero-values it) — +// see this file's package doc for the decoding philosophy. +type claudeCodeEnvelope struct { + Type string `json:"type"` + Subtype string `json:"subtype,omitempty"` + SessionID string `json:"session_id,omitempty"` + Message json.RawMessage `json:"message,omitempty"` + IsError bool `json:"is_error,omitempty"` + Result string `json:"result,omitempty"` + Usage *claudeCodeUsage `json:"usage,omitempty"` + // TotalCostUSD is Claude Code's own dollar-cost accounting for the + // whole delegated turn — read but deliberately never mapped onto + // anything (see this file's package doc, "result" bullet). + TotalCostUSD float64 `json:"total_cost_usd,omitempty"` +} + +// claudeCodeUsage is a "result" event's usage object. +// +// # Usage mapping +// +// InputTokens and OutputTokens map onto provider.Usage's identically-named +// fields directly. CacheReadInputTokens/CacheCreationInputTokens map onto +// provider.Usage.CacheReadTokens/CacheWriteTokens — the naming differs +// (Claude Code mirrors the raw Anthropic Messages API's own +// cache_read_input_tokens/cache_creation_input_tokens field names; harness's +// provider.Usage instead follows provider/anthropic's own CacheRead/ +// CacheWrite convention) but the ACCOUNTING is identical: both are token +// counts for a cache read and a cache write, respectively, over the same +// underlying Anthropic billing model. See mapClaudeCodeUsage. +type claudeCodeUsage struct { + InputTokens int `json:"input_tokens"` + OutputTokens int `json:"output_tokens"` + CacheCreationInputTokens int `json:"cache_creation_input_tokens"` + CacheReadInputTokens int `json:"cache_read_input_tokens"` +} + +// mapClaudeCodeUsage converts a "result" event's usage object to +// provider.Usage — see claudeCodeUsage's own doc comment for the field-by- +// field mapping and cache-naming reconciliation. A nil u (a result event +// with no usage object at all — not expected from a real `claude` binary, +// but this file decodes permissively throughout) yields the zero Usage +// rather than a nil-dereference panic. +func mapClaudeCodeUsage(u *claudeCodeUsage) provider.Usage { + if u == nil { + return provider.Usage{} + } + return provider.Usage{ + InputTokens: u.InputTokens, + OutputTokens: u.OutputTokens, + CacheReadTokens: u.CacheReadInputTokens, + CacheWriteTokens: u.CacheCreationInputTokens, + } +} + +// claudeCodeMessage is the "message" field an "assistant" or "user" +// stream-json event carries — the same shape the raw Anthropic Messages +// API uses for either role. Content may be a bare JSON string (a plain- +// text-only message, which some SDK code paths emit) or an array of +// claudeCodeContentBlock — see decodeClaudeCodeContentBlocks, which +// accepts both. +type claudeCodeMessage struct { + Role string `json:"role"` + Content json.RawMessage `json:"content"` +} + +// claudeCodeContentBlock is one content block of a claudeCodeMessage. Only +// the fields relevant to the block's own Type are ever populated; the rest +// zero-value harmlessly. +type claudeCodeContentBlock struct { + Type string `json:"type"` // "text" | "tool_use" | "tool_result" + // Text is set on a "text" block. + Text string `json:"text,omitempty"` + // ID and Name are set on a "tool_use" block; Input is that tool + // call's arguments object. + ID string `json:"id,omitempty"` + Name string `json:"name,omitempty"` + Input json.RawMessage `json:"input,omitempty"` + // ToolUseID, Content, and IsError are set on a "tool_result" block. + // Content, like claudeCodeMessage's own field, may be a bare string or + // an array of blocks — see claudeCodeContentText. + ToolUseID string `json:"tool_use_id,omitempty"` + Content json.RawMessage `json:"content,omitempty"` + IsError bool `json:"is_error,omitempty"` +} + +// decodeClaudeCodeContentBlocks decodes raw as either a JSON array of +// claudeCodeContentBlock or, if that fails, a bare JSON string (treated as +// a single text block) — see claudeCodeMessage.Content's own doc comment. +// Any other or empty shape yields nil, never an error: this file never +// fails a turn over one unrecognized content shape (see the package doc's +// permissive-decoding philosophy). +func decodeClaudeCodeContentBlocks(raw json.RawMessage) []claudeCodeContentBlock { + if len(raw) == 0 { + return nil + } + var blocks []claudeCodeContentBlock + if err := json.Unmarshal(raw, &blocks); err == nil { + return blocks + } + var text string + if err := json.Unmarshal(raw, &text); err == nil && text != "" { + return []claudeCodeContentBlock{{Type: "text", Text: text}} + } + return nil +} + +// claudeCodeContentText flattens a tool_result block's own Content field +// (bare string, or an array of blocks each contributing its own Text) into +// one string, newline-joining multiple blocks. An unrecognized shape falls +// back to the raw JSON bytes verbatim, rather than silently discarding +// content the CLI genuinely sent. +func claudeCodeContentText(raw json.RawMessage) string { + if len(raw) == 0 { + return "" + } + var s string + if err := json.Unmarshal(raw, &s); err == nil { + return s + } + if blocks := decodeClaudeCodeContentBlocks(raw); blocks != nil { + var sb strings.Builder + for _, b := range blocks { + if b.Text == "" { + continue + } + if sb.Len() > 0 { + sb.WriteByte('\n') + } + sb.WriteString(b.Text) + } + return sb.String() + } + return string(raw) +} + +// claudeCodeAssistantMessage decodes an "assistant" event's message field +// into a canonical message.Message: one Text part per non-empty "text" +// content block, one ToolCall part per "tool_use" block, in the CLI's own +// order. A decode failure or a message with no recognized blocks yields a +// Message with a nil Parts, which consumeClaudeCodeStream's caller treats +// as "nothing to append". +func claudeCodeAssistantMessage(raw json.RawMessage, model message.ModelRef) message.Message { + var cm claudeCodeMessage + _ = json.Unmarshal(raw, &cm) // best-effort; a failure just yields no blocks below + var parts message.Parts + for _, b := range decodeClaudeCodeContentBlocks(cm.Content) { + switch b.Type { + case "text": + if b.Text != "" { + parts = append(parts, &message.Text{Text: b.Text}) + } + case "tool_use": + args := b.Input + if len(args) == 0 { + args = json.RawMessage("{}") + } + parts = append(parts, &message.ToolCall{CallID: b.ID, Name: b.Name, Arguments: args}) + } + } + return message.Message{ + ID: newID("msg"), + Role: message.RoleAssistant, + Parts: parts, + Model: model, + Origin: message.OriginClaudeCode, + CreatedAt: time.Now().UTC(), + } +} + +// claudeCodeToolResultMessage decodes a "user" event's message field — +// Claude Code's own tool_result delivery, in the raw Anthropic API's +// "user"-role convention — into a canonical RoleTool message.Message: one +// ToolResult part per "tool_result" content block. Returns nil when the +// message decodes to no tool_result blocks at all (an ordinary human- +// authored "user" event never reaches this driver — Session.History's +// tail is the only user input a delegated turn ever sends, over stdin, not +// stdout — so an empty result here means an unrecognized shape, not a +// real turn boundary to silently drop). +func claudeCodeToolResultMessage(raw json.RawMessage) *message.Message { + var cm claudeCodeMessage + if err := json.Unmarshal(raw, &cm); err != nil { + return nil + } + var parts message.Parts + for _, b := range decodeClaudeCodeContentBlocks(cm.Content) { + if b.Type != "tool_result" { + continue + } + parts = append(parts, &message.ToolResult{ + CallID: b.ToolUseID, + Content: message.Parts{&message.Text{Text: claudeCodeContentText(b.Content)}}, + IsError: b.IsError, + }) + } + if len(parts) == 0 { + return nil + } + return &message.Message{ + ID: newID("msg"), + Role: message.RoleTool, + Parts: parts, + Origin: message.OriginClaudeCode, + CreatedAt: time.Now().UTC(), + } +} + +// claudeCodeInputMessage is the stdin stream-json shape this driver writes +// — one line, one turn (see runClaudeCodeTurn's own doc comment on why a +// child is spawned fresh per harness turn rather than kept alive across +// several). +type claudeCodeInputMessage struct { + Type string `json:"type"` + Message claudeCodeInputInnerMessage `json:"message"` +} + +type claudeCodeInputInnerMessage struct { + Role string `json:"role"` + Content string `json:"content"` +} + +// capBuffer is an io.Writer that retains at most cap bytes, silently +// dropping anything beyond that — used to bound how much of the `claude` +// child's stderr this file holds onto for a diagnostic message (see +// claudeCodeStderrCap), without letting a runaway or malicious child +// exhaust memory buffering an unbounded stream nothing ever reads back in +// full. +type capBuffer struct { + buf bytes.Buffer + cap int +} + +func (c *capBuffer) Write(p []byte) (int, error) { + if room := c.cap - c.buf.Len(); room > 0 { + if len(p) > room { + c.buf.Write(p[:room]) + } else { + c.buf.Write(p) + } + } + // Report the full length written, per io.Writer's contract (a short + // count would make the child's own stderr write fail) — the cap only + // bounds what this file RETAINS, never what the child is allowed to + // emit. + return len(p), nil +} + +func (c *capBuffer) String() string { return strings.TrimSpace(c.buf.String()) } diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go new file mode 100644 index 00000000..612b8d43 --- /dev/null +++ b/engine/claude_code_backend_test.go @@ -0,0 +1,408 @@ +package engine + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/modelmeta" + "github.com/majorcontext/harness/provider" +) + +// fakeClaudeBin is the path to the compiled fakeclaude stand-in (see +// engine/testdata/fakeclaude/main.go), built once for the whole package +// run by buildFakeClaude below. +var ( + fakeClaudeBin string + fakeClaudeBinOnce sync.Once + fakeClaudeBinErr error +) + +// buildFakeClaude compiles engine/testdata/fakeclaude into a temp binary a +// single time (sync.Once) for however many tests in this package need it — +// mirrors e2e/e2e_test.go's buildHarness precedent for the same reason: +// paying one `go build` up front is far cheaper and more deterministic +// than a shell-script stand-in with its own quoting/portability concerns. +func buildFakeClaude(t *testing.T) string { + t.Helper() + fakeClaudeBinOnce.Do(func() { + dir, err := os.MkdirTemp("", "harness-fakeclaude") + if err != nil { + fakeClaudeBinErr = err + return + } + bin := filepath.Join(dir, "fakeclaude") + cmd := exec.Command("go", "build", "-o", bin, "./testdata/fakeclaude") + if out, err := cmd.CombinedOutput(); err != nil { + fakeClaudeBinErr = fmt.Errorf("go build fakeclaude: %v\n%s", err, out) + return + } + fakeClaudeBin = bin + }) + if fakeClaudeBinErr != nil { + t.Fatalf("buildFakeClaude: %v", fakeClaudeBinErr) + } + return fakeClaudeBin +} + +// claudeCodeTestSession builds a session whose model routes to the +// delegated backend, with binary set to the fake stand-in and mode/session +// id/log path threaded through environment variables the child inherits +// (see fakeclaude's own doc comment) — t.Setenv so each test gets its own +// isolated values without racing another test's env. +func claudeCodeTestSession(t *testing.T, mode string) (*Session, string) { + t.Helper() + bin := buildFakeClaude(t) + t.Setenv("FAKE_CLAUDE_MODE", mode) + logPath := filepath.Join(t.TempDir(), "invocations.jsonl") + t.Setenv("FAKE_CLAUDE_LOG", logPath) + s := NewSession(Config{ + SessionDir: t.TempDir(), + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + ClaudeCode: ClaudeCodeConfig{BinaryPath: bin}, + }) + return s, logPath +} + +// readInvocations parses the fakeclaude invocation log (one JSON array of +// argv per line, per FAKE_CLAUDE_LOG's own doc comment) into a slice of +// argv slices, one per `claude` child spawned so far. +func readInvocations(t *testing.T, logPath string) [][]string { + t.Helper() + data, err := os.ReadFile(logPath) + if err != nil { + t.Fatalf("reading invocation log: %v", err) + } + dec := json.NewDecoder(bytes.NewReader(data)) + var out [][]string + for { + var argv []string + if err := dec.Decode(&argv); err != nil { + break + } + out = append(out, argv) + } + return out +} + +func argvContains(argv []string, want string) bool { + for _, a := range argv { + if a == want { + return true + } + } + return false +} + +func argvValueAfter(argv []string, flag string) (string, bool) { + for i, a := range argv { + if a == flag && i+1 < len(argv) { + return argv[i+1], true + } + } + return "", false +} + +// TestClaudeCodeDelegatedTurnMapsEventsAndUsage drives one full turn +// through fakeclaude's default ("normal") canned sequence and asserts +// every event->message mapping this backend documents: an assistant text +// message, a ToolCall-bearing assistant message, a ToolResult-bearing tool +// message, a final assistant text message, the right engine.Events, the +// captured Claude Code session id, and the mapped usage. +func TestClaudeCodeDelegatedTurnMapsEventsAndUsage(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + + var events []Event + s.cfg.OnEvent = func(ev Event) { events = append(events, ev) } + // OnEvent is read from s.cfg at emit time (see Session.emit); the + // field assignment above is safe pre-Prompt since nothing else + // touches the session yet. + + msg, err := s.Prompt(context.Background(), "please run echo hi") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if msg == nil { + t.Fatal("Prompt returned a nil final message") + } + if got := msg.Parts.Text(); got != "Done — it printed hi." { + t.Errorf("final message text = %q", got) + } + if msg.Origin != message.OriginClaudeCode { + t.Errorf("final message Origin = %q, want %q", msg.Origin, message.OriginClaudeCode) + } + + hist := s.History() + // user prompt, assistant("Let me check that."), assistant(tool_use), + // tool(tool_result), assistant("Done...") = 5 messages. + if len(hist) != 5 { + t.Fatalf("History() len = %d, want 5: %+v", len(hist), hist) + } + if hist[0].Role != message.RoleUser { + t.Errorf("hist[0].Role = %q, want user", hist[0].Role) + } + if hist[1].Role != message.RoleAssistant || hist[1].Parts.Text() != "Let me check that." { + t.Errorf("hist[1] = %+v", hist[1]) + } + tc, ok := hist[2].Parts[0].(*message.ToolCall) + if hist[2].Role != message.RoleAssistant || !ok || tc.Name != "Bash" || tc.CallID != "toolu_1" { + t.Errorf("hist[2] = %+v, want an assistant ToolCall(Bash, toolu_1)", hist[2]) + } + tr, ok := hist[3].Parts[0].(*message.ToolResult) + if hist[3].Role != message.RoleTool || !ok || tr.CallID != "toolu_1" || tr.Content.Text() != "hi\n" { + t.Errorf("hist[3] = %+v, want a tool ToolResult(toolu_1, \"hi\\n\")", hist[3]) + } + if hist[4].Role != message.RoleAssistant || hist[4].Parts.Text() != "Done — it printed hi." { + t.Errorf("hist[4] = %+v", hist[4]) + } + + // Event mapping: at least one ToolStart and one ToolEnd, matched by + // call id. + var sawStart, sawEnd bool + for _, ev := range events { + if ev.Type == EventToolStart && ev.ToolCall != nil && ev.ToolCall.CallID == "toolu_1" { + sawStart = true + } + if ev.Type == EventToolEnd && ev.ToolCall != nil && ev.ToolCall.CallID == "toolu_1" { + sawEnd = true + } + } + if !sawStart { + t.Error("no EventToolStart for toolu_1") + } + if !sawEnd { + t.Error("no EventToolEnd for toolu_1") + } + + // Usage mapping (mapClaudeCodeUsage): input/output/cache read/cache + // write map from fakeclaude's canned result usage object. + usage := s.Usage() + if usage.InputTokens != 101 || usage.OutputTokens != 42 || usage.CacheReadTokens != 7 || usage.CacheWriteTokens != 5 { + t.Errorf("Usage() = %+v, want {101 42 7 5}", usage) + } + + if s.claudeCodeSessionID() != "fake-session-1" { + t.Errorf("claudeCodeSessionID() = %q, want fake-session-1", s.claudeCodeSessionID()) + } +} + +// TestClaudeCodeSessionIDResumedAcrossTurns proves the SECOND delegated +// turn against the same session passes --resume naming the CLI session id +// captured from the FIRST turn's system/init event. +func TestClaudeCodeSessionIDResumedAcrossTurns(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + + if _, err := s.Prompt(context.Background(), "first turn"); err != nil { + t.Fatalf("first Prompt: %v", err) + } + if _, err := s.Prompt(context.Background(), "second turn"); err != nil { + t.Fatalf("second Prompt: %v", err) + } + + invocations := readInvocations(t, logPath) + if len(invocations) != 2 { + t.Fatalf("invocations = %d, want 2: %+v", len(invocations), invocations) + } + if argvContains(invocations[0], "--resume") { + t.Errorf("first invocation argv unexpectedly carries --resume: %v", invocations[0]) + } + resumeID, ok := argvValueAfter(invocations[1], "--resume") + if !ok { + t.Fatalf("second invocation argv has no --resume: %v", invocations[1]) + } + if resumeID != "fake-session-1" { + t.Errorf("--resume value = %q, want fake-session-1", resumeID) + } + // --model sonnet must also be passed through on both calls. + for i, argv := range invocations { + if v, ok := argvValueAfter(argv, "--model"); !ok || v != "sonnet" { + t.Errorf("invocation %d --model = %q, ok=%v, want sonnet", i, v, ok) + } + } + + // The session-id record survives a reload (LoadSession folds + // recClaudeCodeSessionID) — a third turn after a fresh load must + // still resume the same CLI session. + reloaded, err := LoadSession(Config{ + SessionDir: s.cfg.SessionDir, + ClaudeCode: s.cfg.ClaudeCode, + }, s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if reloaded.claudeCodeSessionID() != "fake-session-1" { + t.Errorf("reloaded claudeCodeSessionID() = %q, want fake-session-1", reloaded.claudeCodeSessionID()) + } + if _, err := reloaded.Prompt(context.Background(), "third turn"); err != nil { + t.Fatalf("third Prompt (post-reload): %v", err) + } + invocations = readInvocations(t, logPath) + if len(invocations) != 3 { + t.Fatalf("invocations after reload = %d, want 3", len(invocations)) + } + if v, ok := argvValueAfter(invocations[2], "--resume"); !ok || v != "fake-session-1" { + t.Errorf("post-reload --resume = %q, ok=%v, want fake-session-1", v, ok) + } +} + +// TestClaudeCodeErrorResultReturnsError proves an IsError "result" event +// surfaces as Prompt's returned error, and that Session.Usage() still +// reflects the failed attempt's own billed usage (Claude Code, like a +// native provider, bills a call whether or not it produced a usable +// outcome). +func TestClaudeCodeErrorResultReturnsError(t *testing.T) { + s, _ := claudeCodeTestSession(t, "error") + + msg, err := s.Prompt(context.Background(), "do something that fails") + if err == nil { + t.Fatal("Prompt returned no error for an is_error result") + } + if msg != nil { + t.Errorf("Prompt returned a non-nil message alongside an error: %+v", msg) + } + usage := s.Usage() + if usage.InputTokens != 11 || usage.OutputTokens != 3 { + t.Errorf("Usage() = %+v, want {11 3 0 0} (the failed call's own billed usage)", usage) + } +} + +// TestClaudeCodeAbortSignalsChild proves a canceled context makes +// runClaudeCodeTurn return promptly (bounded well under fakeclaude's own +// hour-long sleep) rather than hanging until the child exits on its own — +// the "hang" mode's child installs no signal handlers, so the driver's own +// SIGINT (Go's default disposition terminates an unhandled-signal process) +// is what has to end it. See claude_code_backend.go's signal-cascade +// goroutine. +func TestClaudeCodeAbortSignalsChild(t *testing.T) { + s, _ := claudeCodeTestSession(t, "hang") + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { + _, err := s.Prompt(ctx, "this will hang") + done <- err + }() + + // Give fakeclaude a moment to actually start and emit its init event + // before pulling the plug — proves the abort interrupts a GENUINELY + // in-flight child, not one that never started. + time.Sleep(200 * time.Millisecond) + cancel() + + select { + case err := <-done: + if !errors.Is(err, context.Canceled) { + t.Errorf("Prompt error = %v, want context.Canceled", err) + } + case <-time.After(10 * time.Second): + t.Fatal("Prompt did not return within 10s of cancellation — child was not interrupted") + } +} + +// TestClaudeCodeModelRefSelection proves selection is purely a function of +// the session's model ref: a claude-code ref dispatches to the delegated +// backend (never touching the registered native provider at all), and an +// ordinary ref keeps running the native provider-call path untouched — +// the same session type, same Prompt call, branching only on +// claudeCodeDelegated(). +func TestClaudeCodeModelRefSelection(t *testing.T) { + t.Run("claude-code ref bypasses the native provider entirely", func(t *testing.T) { + bin := buildFakeClaude(t) + t.Setenv("FAKE_CLAUDE_MODE", "normal") + t.Setenv("FAKE_CLAUDE_LOG", filepath.Join(t.TempDir(), "invocations.jsonl")) + prov := &scriptedProvider{name: "native-should-not-be-called"} + s := NewSession(Config{ + SessionDir: t.TempDir(), + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + ClaudeCode: ClaudeCodeConfig{BinaryPath: bin}, + Providers: provider.Registry{prov.name: prov}, + }) + if !s.claudeCodeDelegated() { + t.Fatal("claudeCodeDelegated() = false for a claude-code model ref") + } + if _, err := s.Prompt(context.Background(), "hello"); err != nil { + t.Fatalf("Prompt: %v", err) + } + if prov.call != 0 { + t.Errorf("native provider Stream called %d times, want 0", prov.call) + } + }) + + t.Run("a normal ref never touches the delegated backend", func(t *testing.T) { + prov := &scriptedProvider{name: "native", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "hi from native"}), + }} + s := NewSession(Config{ + SessionDir: t.TempDir(), + Model: message.ModelRef{Provider: "native", Model: "m1"}, + Providers: provider.Registry{prov.name: prov}, + // Deliberately no ClaudeCode.BinaryPath: if this session were + // ever (incorrectly) dispatched to the delegated backend, the + // exec would fail loudly (no such binary) rather than + // silently succeeding, making this a meaningful negative + // check. + }) + if s.claudeCodeDelegated() { + t.Fatal("claudeCodeDelegated() = true for an ordinary provider ref") + } + msg, err := s.Prompt(context.Background(), "hello") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if msg.Parts.Text() != "hi from native" { + t.Errorf("final text = %q, want the native provider's own reply", msg.Parts.Text()) + } + if prov.call != 1 { + t.Errorf("native provider Stream called %d times, want 1", prov.call) + } + }) +} + +// TestClaudeCodeContextWindowSatisfiesRequireContextWindow proves the +// modelmeta entry this backend relies on: a session whose model names +// ClaudeCodeProviderFamily must not be refused at create time even when +// Config.RequireContextWindow is set (the default — see +// config.ContextWindowRequiredValue) — see modelmeta.ContextWindow's own +// claudeCodeProvider case. +func TestClaudeCodeContextWindowSatisfiesRequireContextWindow(t *testing.T) { + s := NewSession(Config{ + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + RequireContextWindow: true, + }) + if err := s.ContextWindowErr(); err != nil { + t.Errorf("ContextWindowErr() = %v, want nil for a claude-code model ref", err) + } +} + +// TestClaudeCodeProviderFamilyMatchesModelmeta proves +// ClaudeCodeProviderFamily and modelmeta's own (unexported) duplicate of +// that string — see modelmeta.ContextWindow's claudeCodeProvider case — +// have not drifted apart: a claude-code model ref must resolve a KNOWN +// context window, or RequireContextWindow refuses session create for +// every real deployment (see modelmeta.go's own doc comment on why the +// string is duplicated there rather than imported). +func TestClaudeCodeProviderFamilyMatchesModelmeta(t *testing.T) { + ref := message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"} + if _, ok := modelmeta.ContextWindow(ref); !ok { + t.Errorf("modelmeta.ContextWindow(%s) reported unknown — modelmeta's claudeCodeProvider constant has drifted from engine.ClaudeCodeProviderFamily (%q)", ref, ClaudeCodeProviderFamily) + } +} + +// TestClaudeCodeDefaultBinaryPath proves newSession defaults an unset +// ClaudeCodeConfig.BinaryPath to "claude" — config.Provider.BinaryPath's +// own doc comment promises this. +func TestClaudeCodeDefaultBinaryPath(t *testing.T) { + s := NewSession(Config{Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}}) + if s.cfg.ClaudeCode.BinaryPath != defaultClaudeCodeBinaryPath { + t.Errorf("ClaudeCode.BinaryPath = %q, want %q", s.cfg.ClaudeCode.BinaryPath, defaultClaudeCodeBinaryPath) + } +} diff --git a/engine/engine.go b/engine/engine.go index 766de232..2e90a171 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -298,6 +298,16 @@ type Config struct { MaxTokens int // per-response cap; defaults to 8192 WorkDir string // working directory for built-in tools + // ClaudeCode configures the delegated-turn backend a session whose + // Model.Provider is ClaudeCodeProviderFamily is driven through (see + // engine/claude_code_backend.go) — the non-HTTP counterpart to + // Providers above for that one family. The zero value is usable: an + // empty BinaryPath defaults to "claude" (newSession), and empty + // ExtraArgs/PermissionMode simply add nothing to the child's argv. + // Irrelevant, and never read, for any session whose model names a + // different provider. + ClaudeCode ClaudeCodeConfig + // SessionDir is where session logs are persisted, one JSONL file per // session. Empty disables persistence entirely. SessionDir string @@ -1034,6 +1044,19 @@ type Session struct { // above has something to read. committedOutcome *taskNotification + // claudeCodeCLISessionID is the Claude Code CLI's OWN session id for a + // delegated session (see engine/claude_code_backend.go): captured from + // the CLI's `system`/`init` stream-json event on this session's FIRST + // delegated turn, and passed back as --resume on every later one so + // the CLI continues the SAME conversation it already holds — the CLI + // owns that history, not this package's s.history, which exists only + // so the console/read-path see a mirrored transcript. Empty until the + // first delegated turn completes an init event; irrelevant for a + // session never delegated. Persisted (recClaudeCodeSessionID, + // store.go) and restored by LoadSession so --resume survives a process + // restart. + claudeCodeCLISessionID string + // spawnedChildIDs is every child id this session has ever Spawn'd — // appended to live (Spawn, session_manager.go) and folded back from // the durable recTaskSpawned audit trail on reload (store.go's @@ -1417,6 +1440,9 @@ func newSession(cfg Config) *Session { if cfg.BashTimeout <= 0 { cfg.BashTimeout = 2 * time.Minute } + if cfg.ClaudeCode.BinaryPath == "" { + cfg.ClaudeCode.BinaryPath = defaultClaudeCodeBinaryPath + } if cfg.Now == nil { cfg.Now = time.Now } @@ -2313,6 +2339,29 @@ func (s *Session) PromptEngineResume(ctx context.Context, text string) (*message // the caller already made. One parameterized entry point for the value // means a future third Origin value is handled in exactly one place. func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string) (*message.Message, error) { + // A session delegated to the Claude Code CLI (ClaudeCodeProviderFamily + // — see engine/claude_code_backend.go) dispatches here, FIRST, before + // every check and assembly step below: ContextWindowErr, + // ensureInstructions, ensureSkills, and maybeAutoCompact are all + // native-loop-only concerns (harness's own context-window bookkeeping, + // AGENTS.md/Agent-Skills injection into a system prompt this path + // never sends, and harness's own auto-compaction) that make no sense + // for a turn Claude Code drives end to end with its own context + // management. Appending the user message and handing off to + // runAgenticLoop is the entire job here; runAgenticLoop's own + // identical claudeCodeDelegated check (its doc comment explains why + // BOTH checks exist) is what also catches the goal-loop's direct + // runAgenticLoop retry call, which never reaches this function at all. + if s.claudeCodeDelegated() { + s.append(message.Message{ + ID: newID("msg"), + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: text}}, + CreatedAt: time.Now().UTC(), + Origin: origin, + }) + return s.runAgenticLoop(ctx) + } // Refuse a model with no known context window, before anything else // happens: no history append, no provider call, no instructions read. // Running one anyway is running with NO context management at all, @@ -2375,7 +2424,37 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri // instead of Prompt, so the retried attempt answers the existing message // rather than duplicating it — see // docs/design/goal-retry-directive-reuse.md. +// +// A session whose model names ClaudeCodeProviderFamily dispatches to +// runClaudeCodeTurn (engine/claude_code_backend.go) instead, at the very +// top, before any of the native-loop machinery below runs: maxTokensUsed +// accounting, streamTurnWithRetry, runToolCalls, and +// drainQueuedPromptsIntoHistory's tool-call-boundary drain are ALL native- +// provider-call concepts that make no sense for a turn Claude Code itself +// is driving end to end (it runs its own tool loop, its own retries, and +// its own mid-turn steering). Checking here — not only in +// PromptWithOrigin's own early dispatch below — is what keeps +// promptTurnWithRetry's directive-reuse retry (the doc comment above) from +// falling through to the native path for a delegated session: that caller +// reaches runAgenticLoop directly, bypassing PromptWithOrigin's dispatch +// entirely, so THIS is the one choke point every route into the agentic +// loop — fresh Prompt call or goal-loop retry alike — actually shares. func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) { + if s.claudeCodeDelegated() { + s.emitStatus("busy") + defer s.emitStatus("idle") + defer s.snapshotOnIdle() + msg, err := s.runClaudeCodeTurn(ctx) + if err != nil { + // Mirrors the native path's own error handling below + // (emitSessionError before returning) — a plugin host + // watching for session errors must see a delegated turn's + // failure exactly like a native one's. + s.emitSessionError(err) + return nil, err + } + return msg, nil + } s.emitStatus("busy") defer s.emitStatus("idle") // The on-idle snapshot trigger (snapshot.go and docs/design/journal- diff --git a/engine/store.go b/engine/store.go index a56f9597..0b031f52 100644 --- a/engine/store.go +++ b/engine/store.go @@ -184,6 +184,25 @@ const ( // DIFFERENT payload than the one the parent may already have // received. recTaskOutcomeCommitted = "task.outcome_committed" + // recClaudeCodeSessionID records the Claude Code CLI's OWN session id + // for a delegated session (see engine/claude_code_backend.go and + // Session.claudeCodeCLISessionID's own doc comment) — written once, + // the first time a delegated turn's system/init event reports it, and + // folded back on LoadSession so --resume keeps naming the same CLI + // session across a process restart. Mirrors recModel/recEffort + // exactly: a no-op record type with no lifecycle meaning beyond "the + // value changed", replayed last-writer-wins. + recClaudeCodeSessionID = "claude_code.session_id" + // recClaudeCodeUsage carries one delegated turn's AGGREGATE Usage (see + // Session.applyClaudeCodeUsage's own doc comment for why this is a + // dedicated record rather than riding a recMessage the way a native + // turn's usage does) — one per completed delegated turn's "result" + // event. Mirrors recCompact's own "Usage independent of any single + // message" precedent (record.Usage's doc comment), except this DOES + // fold into lastUsage on replay, unlike recCompact — see + // applyClaudeCodeUsage's doc comment for why that divergence is safe + // here. + recClaudeCodeUsage = "claude_code.usage" ) // record is one line of a session log file. @@ -297,6 +316,10 @@ type record struct { // namespaced tool names this record adds to the session's selected set. // nil on every other record type. MCPTools []string `json:"mcp_tools,omitempty"` + // ClaudeCodeSessionID carries a recClaudeCodeSessionID record's + // payload: the Claude Code CLI's own session id (see + // Session.claudeCodeCLISessionID). Empty on every other record type. + ClaudeCodeSessionID string `json:"claude_code_session_id,omitempty"` } // applyGoalRecord folds one goal.* record into the durable goal state a @@ -637,6 +660,38 @@ func (s *Session) persistEffort(e message.Effort) { } } +// persistClaudeCodeSessionID appends a claude_code.session_id record to the +// session log. It mirrors persistModel/persistEffort exactly: a no-op +// until the log exists (lazy creation), caller holds s.mu. +func (s *Session) persistClaudeCodeSessionID(id string) { + if s.cfg.SessionDir == "" || !s.logStarted { + return + } + if err := s.ensureLog(); err != nil { + s.lastPersistErr = err + return + } + if err := s.writeRecord(record{Type: recClaudeCodeSessionID, ClaudeCodeSessionID: id}); err != nil { + s.lastPersistErr = err + } +} + +// persistClaudeCodeUsage appends a claude_code.usage record to the session +// log. It mirrors persistModel/persistEffort exactly: a no-op until the +// log exists (lazy creation), caller holds s.mu. +func (s *Session) persistClaudeCodeUsage(usage provider.Usage) { + if s.cfg.SessionDir == "" || !s.logStarted { + return + } + if err := s.ensureLog(); err != nil { + s.lastPersistErr = err + return + } + if err := s.writeRecord(record{Type: recClaudeCodeUsage, Usage: &usage}); err != nil { + s.lastPersistErr = err + } +} + // persistGoalLocked appends a goal.* record to the session log. It forces the // log to exist (a goal.set may be the first thing written to a fresh session). // Caller holds s.mu. @@ -1441,6 +1496,20 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.model = rec.Model case recEffort: s.effort = rec.Effort + case recClaudeCodeSessionID: + s.claudeCodeCLISessionID = rec.ClaudeCodeSessionID + case recClaudeCodeUsage: + // See Session.applyClaudeCodeUsage's own doc comment for why + // this folds into BOTH cumulative usage and lastUsage, unlike + // recCompact's cumulative-only replay. + if rec.Usage != nil { + s.usage.InputTokens += rec.Usage.InputTokens + s.usage.OutputTokens += rec.Usage.OutputTokens + s.usage.CacheReadTokens += rec.Usage.CacheReadTokens + s.usage.CacheWriteTokens += rec.Usage.CacheWriteTokens + s.lastUsage = *rec.Usage + s.haveLastUsage = true + } case recMCPToolsSelected: // Union every record, in log order, into the restored selected // set (see mcp_lazy.go). Replay is defensive, like diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go new file mode 100644 index 00000000..9388757a --- /dev/null +++ b/engine/testdata/fakeclaude/main.go @@ -0,0 +1,137 @@ +// Command fakeclaude is a stand-in for the real `claude` CLI, built once by +// engine's TestMain and pointed at via ClaudeCodeConfig.BinaryPath for +// engine/claude_code_backend_test.go. It emits a canned `--output-format +// stream-json` sequence (system/init, assistant text, a tool_use+ +// tool_result pair, a final result) selected by the FAKE_CLAUDE_MODE +// environment variable, so the driver's event-mapping, resume, and usage +// logic can be tested against a REAL child process and REAL pipes without +// depending on an actual Anthropic subscription or the real binary. +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "os" + "time" +) + +func main() { + if logPath := os.Getenv("FAKE_CLAUDE_LOG"); logPath != "" { + // Record this invocation's full argv for the test to inspect + // afterward (the resume test's only way to see whether --resume + // was passed on the SECOND call). + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err == nil { + enc := json.NewEncoder(f) + _ = enc.Encode(os.Args[1:]) + f.Close() + } + } + + // Drain (don't require) exactly one stdin line — the real driver + // writes one turn message and closes stdin; reading it keeps this + // stand-in honest about the protocol without validating its content. + go func() { + scanner := bufio.NewScanner(os.Stdin) + for scanner.Scan() { + // discard + } + }() + + sessionID := os.Getenv("FAKE_CLAUDE_SESSION_ID") + if sessionID == "" { + sessionID = "fake-session-1" + } + + out := bufio.NewWriter(os.Stdout) + emit := func(v any) { + b, _ := json.Marshal(v) + fmt.Fprintln(out, string(b)) + out.Flush() + } + + emit(map[string]any{ + "type": "system", + "subtype": "init", + "session_id": sessionID, + }) + + switch os.Getenv("FAKE_CLAUDE_MODE") { + case "hang": + // No signal handlers installed: an unhandled SIGINT/SIGTERM uses + // Go's own default disposition, which terminates the process — + // exactly the behavior the driver's abort/interrupt cascade + // depends on (see claude_code_backend.go's signal-cascade + // goroutine). Blocks far longer than any test's own timeout so a + // working cascade is what ends this process, not a wall-clock + // race. + time.Sleep(time.Hour) + return + case "error": + emit(map[string]any{ + "type": "result", + "subtype": "error_during_execution", + "is_error": true, + "result": "fake failure", + "usage": map[string]any{ + "input_tokens": 11, + "output_tokens": 3, + }, + }) + return + } + + // Default ("normal"): assistant text, a tool_use/tool_result pair, + // then a final assistant text and result — one of each event shape + // engine/claude_code_backend.go documents mapping. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Let me check that."}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_1", "name": "Bash", "input": map[string]any{"command": "echo hi"}}, + }, + }, + }) + emit(map[string]any{ + "type": "user", + "message": map[string]any{ + "role": "user", + "content": []map[string]any{ + {"type": "tool_result", "tool_use_id": "toolu_1", "content": "hi\n"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Done — it printed hi."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Done — it printed hi.", + "usage": map[string]any{ + "input_tokens": 101, + "output_tokens": 42, + "cache_read_input_tokens": 7, + "cache_creation_input_tokens": 5, + }, + "total_cost_usd": 0.0123, + }) +} diff --git a/message/message.go b/message/message.go index 0934f411..e4df19a7 100644 --- a/message/message.go +++ b/message/message.go @@ -42,6 +42,20 @@ const ( // notification is ready to deliver. const OriginEngine = "engine" +// OriginClaudeCode marks a Message produced by a turn delegated to the +// Claude Code CLI (see engine/claude_code_backend.go) rather than a +// message this engine's own native provider loop produced or a human +// typed. Every assistant and tool message a delegated turn appends carries +// this Origin. Unlike OriginEngine (defined only for a RoleUser trigger +// message), this applies to RoleAssistant and RoleTool messages — but the +// same rule OriginEngine's own doc comment establishes still holds: this +// is presentation metadata only, read by a client (boxes' console) +// choosing how to RENDER a message, and it must never change how a +// transcoder or a model treats it. A session that later switches back to +// a native provider still sends these messages exactly like any other +// history — no transcoder reads this field. +const OriginClaudeCode = "claude_code" + // Message is one entry in a session's history. // // The system prompt is deliberately not part of history: it is assembled per diff --git a/modelmeta/modelmeta.go b/modelmeta/modelmeta.go index 2831c786..ce894a34 100644 --- a/modelmeta/modelmeta.go +++ b/modelmeta/modelmeta.go @@ -209,10 +209,39 @@ func ContextWindow(ref message.ModelRef) (tokens int, ok bool) { if suffix, isAnthropic := stripBedrockAnthropicPrefix(model); isAnthropic { tokens, ok = bedrockAnthropicContextWindows[stripBedrockVersionSuffix(suffix)] } + case claudeCodeProvider: + // A turn delegated to the Claude Code CLI (see + // engine/claude_code_backend.go) is driven entirely by that CLI's + // OWN context management: it runs its own tool loop and its own + // compaction over its own history, never harness's. This entry + // exists ONLY so engine.Config.RequireContextWindow (default true + // — an unrecognized model is a hard session-create refusal, see + // engine/context_window.go) does not refuse a claude-code model + // ref outright; harness's OWN automatic-compaction threshold is + // unconditionally skipped for a delegated turn regardless of what + // this reports (see PromptWithOrigin's early dispatch), so the + // exact figure here drives no real behavior. claudeCodeContextWindow + // (200,000, Sonnet's advertised first-party window) is a stand-in + // chosen only to be an honest, plausible-sounding number rather + // than an arbitrary placeholder like 0 or MaxInt. + tokens, ok = claudeCodeContextWindow, true } return tokens, ok } +// claudeCodeProvider is the message.ModelRef.Provider value that selects +// the Claude Code CLI delegated-turn backend (engine/claude_code_backend.go +// and config.TypeClaudeCodeCLI) — duplicated here, rather than imported, +// because package modelmeta must not depend on package engine (engine +// already depends on modelmeta for this very function). Kept in sync by +// engine's TestClaudeCodeProviderFamilyMatchesModelmeta. +const claudeCodeProvider = "claude-code" + +// claudeCodeContextWindow is the stand-in context-window figure reported +// for claudeCodeProvider — see the ContextWindow case above for why its +// exact value carries no real weight. +const claudeCodeContextWindow = 200_000 + // lastPathSegment returns the substring of model after its last '/', or // model unchanged if it contains no '/'. message.ModelRef.Model may itself // contain slashes (see that type's doc comment), which is exactly what the diff --git a/provider/claudecode/claudecode.go b/provider/claudecode/claudecode.go new file mode 100644 index 00000000..daaeabaf --- /dev/null +++ b/provider/claudecode/claudecode.go @@ -0,0 +1,51 @@ +// Package claudecode registers the provider family key that routes a +// message.ModelRef to harness's Claude Code CLI delegated-turn backend +// (engine/claude_code_backend.go, config.TypeClaudeCodeCLI). +// +// Client exists ONLY to satisfy provider.Registry lookups — the same map +// every native HTTP adapter (provider/anthropic, provider/openai, ...) +// registers into, which server.handleSetModel, the `model` tool, and +// Spawn's model-override validation all consult via Session.ModelSupported +// (engine/engine.go) before allowing a swap to a new ref. A claude-code +// session's actual turns NEVER reach Client.Stream in normal operation: +// PromptWithOrigin/runAgenticLoop (engine/engine.go) detect a claude-code +// model ref and dispatch to the delegated CLI driver BEFORE the native +// provider-call machinery (streamTurn) is ever reached — see that +// package's own doc comment for the full seam. Client.Stream is therefore +// a defensive backstop, not a real code path: if it is ever invoked, some +// caller bypassed that dispatch (a manual compaction call, a goal +// evaluator misconfigured to this family, a future call site that forgets +// the check), and returning a descriptive error here is far safer than +// either panicking or silently attempting an HTTP-shaped request this +// family was never meant to make. +package claudecode + +import ( + "context" + "fmt" + + "github.com/majorcontext/harness/provider" +) + +// Family is the provider key clients register under and message.ModelRef.Provider +// values route by — "claude-code", matching config.TypeClaudeCodeCLI's +// conventional providers-map key and engine.ClaudeCodeProviderFamily. Kept +// as its own named constant (rather than importing engine, which would be +// a package-layering inversion: engine already imports provider) so +// cmd/harness's registry() has a single source for the string; a parity +// test in cmd/harness pins it against engine.ClaudeCodeProviderFamily. +const Family = "claude-code" + +// Client is a provider.Provider stand-in for the claude-code family — see +// the package doc for why Stream is never expected to run. +type Client struct{} + +// Name implements provider.Provider. +func (Client) Name() string { return Family } + +// Stream implements provider.Provider. It always fails — see the package +// doc comment for why reaching this at all is itself the bug to fix, not a +// case this adapter should try to serve. +func (Client) Stream(ctx context.Context, req *provider.Request) (provider.Stream, error) { + return nil, fmt.Errorf("provider/claudecode: Stream called directly for model %s — a claude-code session's turns must be dispatched through the delegated CLI backend (engine/claude_code_backend.go), not the native provider-call path; this indicates a caller bypassed that dispatch", req.Model) +} diff --git a/provider/claudecode/claudecode_test.go b/provider/claudecode/claudecode_test.go new file mode 100644 index 00000000..aea62969 --- /dev/null +++ b/provider/claudecode/claudecode_test.go @@ -0,0 +1,34 @@ +package claudecode + +import ( + "context" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// TestClientNameMatchesFamily proves Name() reports the same string other +// packages route by — a divergence here would silently break +// provider.Registry.For for every claude-code model ref. +func TestClientNameMatchesFamily(t *testing.T) { + if got := (Client{}).Name(); got != Family { + t.Errorf("Name() = %q, want %q", got, Family) + } +} + +// TestClientStreamAlwaysErrors pins the deliberate backstop behavior: see +// the package doc comment for why Client.Stream is never expected to run +// in normal operation, and must fail loudly (never panic, never silently +// no-op) if something ever calls it anyway. +func TestClientStreamAlwaysErrors(t *testing.T) { + ref := message.ModelRef{Provider: Family, Model: "sonnet"} + _, err := (Client{}).Stream(context.Background(), &provider.Request{Model: ref}) + if err == nil { + t.Fatal("Stream returned no error") + } + if !strings.Contains(err.Error(), ref.String()) { + t.Errorf("error %q does not name the offending model ref %q", err, ref.String()) + } +} From 960309dab30f3e7c275c3b8532bafd11ea50fc3e Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Sun, 30 Aug 2026 09:12:06 -0400 Subject: [PATCH 26/95] provider/openai,config: sanitize tool schemas for the Responses tool-schema validator (#215) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ChatGPT Codex backend (chatgpt.com/backend-api/codex/responses) rejects tool parameter schemas the OpenAI platform Responses API accepts without complaint. Confirmed on a live box: openai: Invalid JSON schema: regex lookaround is not supported. Found at $.properties.email.pattern. (invalid_request_error, HTTP 400) Its tool-schema validator is a stricter subset of JSON Schema than the Responses API documents, and harness forwards a tool's InputSchema to the wire unsanitized. harness#213 flagged this exact gap while adding OmitResponseParams for the same backend's rejected request params; this closes it for tool schemas, the remaining blocker on the ChatGPT subscription lane (credential/gatekeeper path already proven working). Design: an allowlist-rebuild sanitizer, ported from opencode's sanitizeOpenAISchema (codex-request-normalize.ts, v1.18.23) — the same approach a comparable client already ships for this backend. It rebuilds each schema node keeping only keywords the validator accepts ($ref, description, enum/const, properties, required, items, additionalProperties, anyOf/oneOf/allOf, $defs/definitions, and a type drawn from a fixed supported set or inferred from structure), dropping everything else outright — notably pattern, format (except as a string-type inference signal), minLength/maxLength, and numeric constraints beyond type inference. The sanitizer is opt-in per provider entry, config.Provider's new sanitize_tool_schemas bool, mirroring OmitResponseParams: validated only on an entry that builds the native OpenAI Responses adapter (the buildsResponsesAdapter guard #197/#213 introduced), non-clearable across config layers like NoPromptCacheKey, and threaded into provider/openai.Client the same way OmitResponseParams is. Default false leaves every tool schema byte-identical to req.Tools, so the normal openai/anthropic/bifrost providers — which accept the richer schema and would only lose expressiveness from the rewrite — are unaffected. transcodeRequestFamily applies it only to the tools placed on the wire request, never to the canonical provider.Request the engine holds. Verification: - New sanitizer unit tests covering the ported cases: pattern/ format/minLength dropped from a nested property while structure, types, required, and description survive; const converted to enum; array/anyOf/$defs recursion; an unsupported type collapsing to {}; a boolean subschema becoming {type: string}; empty/ unparseable input passed through unchanged. - Config tests: sanitize_tool_schemas accepted on the native "openai" key and a keyed type:"openai" entry, rejected on openai-compat and native anthropic entries, false always valid, and non-clearable merge semantics (a project layer can set it true but not clear an inherited true). - Request-build tests: with the flag off, an emitted tool's parameter schema is byte-identical to the input, pattern included; with it on, pattern is stripped and type/required survive. - End-to-end registry test: a type:"openai" provider configured with sanitize_tool_schemas:true, driven through Client.Stream against a real httptest server, produces a wire tools[] payload with pattern stripped and structure intact. - go build ./..., go vet ./..., go test ./... all green. --- cmd/harness/main.go | 34 ++- cmd/harness/sanitize_tool_schemas_test.go | 133 +++++++++ config/config.go | 44 +++ config/sanitize_tool_schemas_test.go | 122 +++++++++ provider/openai/omit_response_params_test.go | 8 +- provider/openai/openai.go | 12 +- provider/openai/sanitize_tool_schemas_test.go | 83 ++++++ provider/openai/schema_sanitize.go | 255 ++++++++++++++++++ provider/openai/schema_sanitize_test.go | 226 ++++++++++++++++ provider/openai/transcode.go | 16 +- 10 files changed, 916 insertions(+), 17 deletions(-) create mode 100644 cmd/harness/sanitize_tool_schemas_test.go create mode 100644 config/sanitize_tool_schemas_test.go create mode 100644 provider/openai/sanitize_tool_schemas_test.go create mode 100644 provider/openai/schema_sanitize.go create mode 100644 provider/openai/schema_sanitize_test.go diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 394f9a03..c5bb2e73 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -939,9 +939,9 @@ func registry(cfg *config.Config) provider.Registry { anthropic.Family: &anthropic.Client{APIKey: akey, BaseURL: abase, CacheTTL: anthropicCacheTTL(cfg)}, // The built-in openai entry deliberately leaves Family empty: it IS // the package default, and naming it here would only invite the two - // to drift. Its ResponsesPath/OmitResponseParams come from the - // native entry, if any. - openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg), OmitResponseParams: nativeOmitResponseParams(cfg)}, + // to drift. Its ResponsesPath/OmitResponseParams/SanitizeToolSchemas + // come from the native entry, if any. + openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg), OmitResponseParams: nativeOmitResponseParams(cfg), SanitizeToolSchemas: nativeSanitizeToolSchemas(cfg)}, } registerOpenAICompatProviders(reg, cfg) registerOpenAIProviders(reg, cfg) @@ -1046,11 +1046,12 @@ func registerOpenAIProviders(reg provider.Registry, cfg *config.Config) { } apiKey := os.Getenv(keyEnv) reg[name] = &openai.Client{ - Family: name, - APIKey: apiKey, - BaseURL: p.BaseURL, - ResponsesPath: p.ResponsesPath, - OmitResponseParams: p.OmitResponseParams, + Family: name, + APIKey: apiKey, + BaseURL: p.BaseURL, + ResponsesPath: p.ResponsesPath, + OmitResponseParams: p.OmitResponseParams, + SanitizeToolSchemas: p.SanitizeToolSchemas, } } } @@ -1088,6 +1089,23 @@ func nativeOmitResponseParams(cfg *config.Config) []string { return p.OmitResponseParams } +// nativeSanitizeToolSchemas reads sanitize_tool_schemas configured on the +// NATIVE "openai" entry (map key "openai", no type — the same shape +// config.validateProviders permits sanitize_tool_schemas on). false leaves +// the adapter sending every tool schema unchanged. A keyed type:"openai" +// entry carries its own value and is wired by registerOpenAIProviders +// instead. Mirrors nativeOmitResponseParams exactly. +func nativeSanitizeToolSchemas(cfg *config.Config) bool { + if cfg == nil { + return false + } + p := cfg.Providers[openai.Family] + if p.Type != "" { + return false + } + return p.SanitizeToolSchemas +} + // registerOpenAICompatProviders builds a provider/openaicompat client for // every config.Providers entry of config.TypeOpenAICompat, keyed by its // providers map name — that name is what routes "name/model" refs to it, diff --git a/cmd/harness/sanitize_tool_schemas_test.go b/cmd/harness/sanitize_tool_schemas_test.go new file mode 100644 index 00000000..e44b6b7a --- /dev/null +++ b/cmd/harness/sanitize_tool_schemas_test.go @@ -0,0 +1,133 @@ +package main + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/majorcontext/harness/config" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" + "github.com/majorcontext/harness/provider/openai" +) + +// TestRegistryTypeOpenAIThreadsSanitizeToolSchemas: a `type: "openai"` +// entry's sanitize_tool_schemas must reach the built *openai.Client, +// exactly as OmitResponseParams and ResponsesPath do. +func TestRegistryTypeOpenAIThreadsSanitizeToolSchemas(t *testing.T) { + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: "https://gateway.example", + APIKeyEnv: "SECONDARY_API_KEY", + SanitizeToolSchemas: true, + }, + }}) + c, ok := reg["secondary"].(*openai.Client) + if !ok { + t.Fatalf("secondary provider is %T, want *openai.Client", reg["secondary"]) + } + if !c.SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true") + } +} + +// TestRegistryNativeOpenAIHonorsSanitizeToolSchemas: the bare "openai" key +// builds the same adapter, so its sanitize_tool_schemas must reach the +// client too — mirrors TestRegistryNativeOpenAIHonorsOmitResponseParams. +func TestRegistryNativeOpenAIHonorsSanitizeToolSchemas(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-builtin") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "openai": {SanitizeToolSchemas: true}, + }}) + c := reg[openai.Family].(*openai.Client) + if !c.SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true") + } +} + +// TestRegistrySanitizeToolSchemasEmitsCleanSchemaOnWire drives the +// production path end to end: a provider configured with +// sanitize_tool_schemas:true must actually send a tool parameter schema +// with `pattern` stripped, over real HTTP through Client.Stream — the +// direct proof this feature fixes the Codex-backend 400 ("Invalid JSON +// schema: regex lookaround is not supported"), not just that config +// threads a value through. +func TestRegistrySanitizeToolSchemasEmitsCleanSchemaOnWire(t *testing.T) { + var gotBody map[string]json.RawMessage + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := json.NewDecoder(r.Body).Decode(&gotBody); err != nil { + t.Errorf("decoding request body: %v", err) + } + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, "event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_1\"}}\n\n") //nolint:errcheck + })) + defer srv.Close() + + t.Setenv("SECONDARY_API_KEY", "sk-secondary") + reg := registry(&config.Config{Providers: map[string]config.Provider{ + "secondary": { + Type: config.TypeOpenAI, + BaseURL: srv.URL, + APIKeyEnv: "SECONDARY_API_KEY", + SanitizeToolSchemas: true, + }, + }}) + ref, err := message.ParseModelRef("secondary/gpt-5") + if err != nil { + t.Fatalf("ParseModelRef: %v", err) + } + p, err := reg.For(ref) + if err != nil { + t.Fatalf("reg.For: %v", err) + } + stream, err := p.Stream(context.Background(), &provider.Request{ + Model: ref, + Messages: []message.Message{{ID: "msg_1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hello"}}}}, + Tools: []provider.ToolDef{{ + Name: "send_email", + InputSchema: json.RawMessage(`{ + "type": "object", + "properties": {"email": {"type": "string", "pattern": "^(?=.*@).+$"}}, + "required": ["email"] + }`), + }}, + MaxTokens: 4096, + }) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer stream.Close() + for { + if _, err := stream.Next(); err == io.EOF { + break + } else if err != nil { + t.Fatalf("Next: %v", err) + } + } + + toolsRaw, ok := gotBody["tools"] + if !ok { + t.Fatal("request body has no tools field") + } + if strings.Contains(string(toolsRaw), `"pattern"`) { + t.Errorf("wire tools body contains pattern, want stripped: %s", toolsRaw) + } + var tools []struct { + Parameters map[string]interface{} `json:"parameters"` + } + if err := json.Unmarshal(toolsRaw, &tools); err != nil { + t.Fatalf("unmarshal tools: %v", err) + } + if len(tools) != 1 { + t.Fatalf("tools = %#v, want 1 entry", tools) + } + if tools[0].Parameters["type"] != "object" { + t.Errorf("parameters.type = %v, want object preserved", tools[0].Parameters["type"]) + } +} diff --git a/config/config.go b/config/config.go index ff5c600c..fc67e76a 100644 --- a/config/config.go +++ b/config/config.go @@ -606,6 +606,29 @@ type Provider struct { // entry by entry (a project layer un-listing a param the user layer // added would otherwise be unrepresentable). OmitResponseParams []string `json:"omit_response_params,omitempty"` + // SanitizeToolSchemas rewrites every tool's JSON Schema parameters + // through an allowlist rebuild before this provider's Responses request + // is sent, dropping keywords the target's tool-schema validator does + // not support (notably "pattern", "format", and length/numeric + // constraints — see provider/openai's sanitizeToolSchema, ported from + // opencode's sanitizeOpenAISchema). + // + // It exists because the ChatGPT Codex backend's tool-schema validator + // is STRICTER than the OpenAI platform API: it 400s on a regex + // `pattern` using lookaround (confirmed on a live box; harness#213 + // flagged this exact gap), and harness forwards tool schemas + // unsanitized. The default (false) sends every schema unchanged, so a + // normal openai/anthropic/bifrost provider — which accepts richer + // schemas and would only lose expressiveness from the rewrite — is + // unaffected. + // + // Valid ONLY on an entry that builds the Responses adapter, exactly + // like ResponsesPath and OmitResponseParams — the same + // buildsResponsesAdapter identity check gates all three. Merge + // semantics are non-clearable like NoPromptCacheKey (see its doc + // comment): a project layer can set this to true, but cannot flip an + // inherited true back to false. + SanitizeToolSchemas bool `json:"sanitize_tool_schemas,omitempty"` // CacheTTL selects the Anthropic prompt-cache breakpoint lifetime: // "5m" (the Anthropic API default) or "1h" (the extended TTL, beta // extended-cache-ttl-2025-04-11). Empty (the default) leaves the @@ -782,6 +805,9 @@ func validateProviders(providers map[string]Provider) error { if err := validateOmitResponseParams(name, p); err != nil { return err } + if err := validateSanitizeToolSchemas(name, p); err != nil { + return err + } if err := validateClaudeCodeFields(name, p); err != nil { return err } @@ -872,6 +898,21 @@ func validateOmitResponseParams(name string, p Provider) error { return nil } +// validateSanitizeToolSchemas fails loudly on sanitize_tool_schemas set on +// any entry that does not build the Responses adapter — the same +// buildsResponsesAdapter identity check validateResponsesPath and +// validateOmitResponseParams use, since only that adapter reads the field. +// false (the default) is always valid. +func validateSanitizeToolSchemas(name string, p Provider) error { + if !p.SanitizeToolSchemas { + return nil + } + if !buildsResponsesAdapter(name, p) { + return fmt.Errorf("providers.%s: sanitize_tool_schemas is only valid on an entry that builds the OpenAI Responses adapter (map key %q with no type, or any key with type %q); no other adapter reads it", name, "openai", TypeOpenAI) + } + return nil +} + // ValidateProviderCacheTTL reports whether ttl is a value this package // accepts for Provider.CacheTTL. It is the exported seam cmd/harness's // parity test uses to prove this list and provider/anthropic's own list @@ -1305,6 +1346,9 @@ func merge(base, over *Config) *Config { if v.NoPromptCacheKey { ex.NoPromptCacheKey = true } + if v.SanitizeToolSchemas { + ex.SanitizeToolSchemas = true + } if v.BinaryPath != "" { ex.BinaryPath = v.BinaryPath } diff --git a/config/sanitize_tool_schemas_test.go b/config/sanitize_tool_schemas_test.go new file mode 100644 index 00000000..7477326f --- /dev/null +++ b/config/sanitize_tool_schemas_test.go @@ -0,0 +1,122 @@ +package config + +import ( + "strings" + "testing" +) + +// TestSanitizeToolSchemasOnNativeOpenAIKeyOK: the bare "openai" key (empty +// type) builds the Responses adapter, so it may carry +// sanitize_tool_schemas too — the field is gated on the ADAPTER an entry +// builds, not on its type string, exactly like ResponsesPath and +// OmitResponseParams. +func TestSanitizeToolSchemasOnNativeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "openai": {SanitizeToolSchemas: true}, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["openai"].SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true") + } +} + +// TestSanitizeToolSchemasOnTypeOpenAIKeyOK: a TypeOpenAI entry under an +// arbitrary key builds the same adapter, so it may carry the field too. +func TestSanitizeToolSchemasOnTypeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "secondary": { + Type: TypeOpenAI, + BaseURL: "https://gateway.example", + SanitizeToolSchemas: true, + }, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["secondary"].SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true") + } +} + +// TestSanitizeToolSchemasOnWrongAdapterFails: only the Responses adapter +// reads sanitize_tool_schemas. Set anywhere else it would vanish silently +// into a client that never looks at it, so it is rejected loudly instead — +// the same rule responses_path, omit_response_params, no_prompt_cache_key, +// and cache_ttl follow. +func TestSanitizeToolSchemasOnWrongAdapterFails(t *testing.T) { + for _, tc := range []struct { + name string + key string + p Provider + }{ + {"openai-compat", "mycompat", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", SanitizeToolSchemas: true}}, + {"native anthropic", "anthropic", Provider{SanitizeToolSchemas: true}}, + } { + t.Run(tc.name, func(t *testing.T) { + c := &Config{Providers: map[string]Provider{tc.key: tc.p}} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatalf("mergeAndValidate accepted sanitize_tool_schemas on a %s entry", tc.name) + } + if !strings.Contains(err.Error(), "sanitize_tool_schemas") { + t.Errorf("error %q does not name the offending field", err) + } + if !strings.Contains(err.Error(), tc.key) { + t.Errorf("error %q does not name the offending key", err) + } + }) + } +} + +// TestSanitizeToolSchemasFalseIsAlwaysValid: the default (false/absent) is +// valid on every provider type — the check only fires when the flag is set. +func TestSanitizeToolSchemasFalseIsAlwaysValid(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "mycompat": {Type: TypeOpenAICompat, BaseURL: "http://x"}, + }} + if _, err := mergeAndValidate(c, &Config{}); err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } +} + +// TestMergeProviderSanitizeToolSchemasNonClearable: sanitize_tool_schemas is +// non-clearable like NoPromptCacheKey — a project layer can set it to true, +// but an override layer that leaves it false/absent must not clear an +// inherited true. +func TestMergeProviderSanitizeToolSchemasNonClearable(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "openai": {SanitizeToolSchemas: true}, + }} + over := &Config{Providers: map[string]Provider{ + "openai": {}, + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["openai"].SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true inherited from base layer") + } +} + +// TestMergeProviderSanitizeToolSchemasProjectCanSetTrue: an override layer +// can flip an unset base to true. +func TestMergeProviderSanitizeToolSchemasProjectCanSetTrue(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "openai": {}, + }} + over := &Config{Providers: map[string]Provider{ + "openai": {SanitizeToolSchemas: true}, + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["openai"].SanitizeToolSchemas { + t.Error("SanitizeToolSchemas = false, want true set by override layer") + } +} diff --git a/provider/openai/omit_response_params_test.go b/provider/openai/omit_response_params_test.go index 97c71b60..69b231a9 100644 --- a/provider/openai/omit_response_params_test.go +++ b/provider/openai/omit_response_params_test.go @@ -20,7 +20,7 @@ func TestOmitResponseParamsNoneListedUnchanged(t *testing.T) { req.Temperature = float64Ptr(0.5) req.TopP = float64Ptr(0.9) - out, err := transcodeRequestFamily(req, Family, nil) + out, err := transcodeRequestFamily(req, Family, nil, false) if err != nil { t.Fatalf("transcodeRequestFamily: %v", err) } @@ -48,7 +48,7 @@ func TestOmitResponseParamsAllFourOmitsFromWire(t *testing.T) { req.TopP = float64Ptr(0.9) omit := []string{"max_output_tokens", "temperature", "top_p", "metadata"} - out, err := transcodeRequestFamily(req, Family, omit) + out, err := transcodeRequestFamily(req, Family, omit, false) if err != nil { t.Fatalf("transcodeRequestFamily: %v", err) } @@ -82,7 +82,7 @@ func TestOmitResponseParamsPartialList(t *testing.T) { req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) req.Temperature = float64Ptr(0.5) - out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}) + out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}, false) if err != nil { t.Fatalf("transcodeRequestFamily: %v", err) } @@ -102,7 +102,7 @@ func TestOmitResponseParamsPartialList(t *testing.T) { func TestOmitResponseParamsWinsOverReasoningFloor(t *testing.T) { req := reasoningRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) - out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}) + out, err := transcodeRequestFamily(req, Family, []string{"max_output_tokens"}, false) if err != nil { t.Fatalf("transcodeRequestFamily: %v", err) } diff --git a/provider/openai/openai.go b/provider/openai/openai.go index 6ff6e0d4..3e3660a5 100644 --- a/provider/openai/openai.go +++ b/provider/openai/openai.go @@ -62,6 +62,16 @@ type Client struct { // canonical provider.Request this client is handed, only the JSON body // transcodeRequestFamily builds from it. OmitResponseParams []string + // SanitizeToolSchemas rewrites every tool's JSON Schema parameters + // through sanitizeToolParameterSchema before this client sends a + // request — see config.Provider's field of the same name for the full + // rationale (the ChatGPT Codex backend's tool-schema validator rejects + // keywords the OpenAI platform API accepts, e.g. a regex `pattern` + // using lookaround). This is wire-only: it never affects the canonical + // provider.Request this client is handed, only the JSON body + // transcodeRequestFamily builds from it. Default false leaves every + // tool schema byte-identical to req.Tools. + SanitizeToolSchemas bool } // familyOrDefault resolves a configured family override to the family key @@ -86,7 +96,7 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St if c.APIKey == "" { return nil, fmt.Errorf("openai: no API key configured (set OPENAI_API_KEY)") } - wire, err := transcodeRequestFamily(req, c.family(), c.OmitResponseParams) + wire, err := transcodeRequestFamily(req, c.family(), c.OmitResponseParams, c.SanitizeToolSchemas) if err != nil { return nil, err } diff --git a/provider/openai/sanitize_tool_schemas_test.go b/provider/openai/sanitize_tool_schemas_test.go new file mode 100644 index 00000000..3641d99b --- /dev/null +++ b/provider/openai/sanitize_tool_schemas_test.go @@ -0,0 +1,83 @@ +package openai + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// toolRequestWithPattern is a request carrying one tool whose parameter +// schema has a `pattern` keyword — the exact keyword the ChatGPT Codex +// backend 400s on when it uses a regex lookaround +// ($.properties.email.pattern). +func toolRequestWithPattern() *provider.Request { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.Tools = []provider.ToolDef{{ + Name: "send_email", + Description: "sends an email", + InputSchema: json.RawMessage(`{ + "type": "object", + "properties": { + "email": {"type": "string", "pattern": "^(?=.*@).+$"} + }, + "required": ["email"] + }`), + }} + return req +} + +// TestTranscodeSanitizeToolSchemasOff: with the flag off (the default), +// transcodeRequestFamily must leave a tool's parameter schema — including +// an unsupported keyword like `pattern` — completely unchanged. This is the +// "normal openai/anthropic/bifrost providers are unaffected" half of the +// feature. +func TestTranscodeSanitizeToolSchemasOff(t *testing.T) { + req := toolRequestWithPattern() + out, err := transcodeRequestFamily(req, Family, nil, false) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if len(out.Tools) != 1 { + t.Fatalf("Tools = %#v, want 1 entry", out.Tools) + } + got := string(out.Tools[0].Parameters) + want := string(req.Tools[0].InputSchema) + if got != want { + t.Errorf("Parameters = %s, want unchanged %s", got, want) + } + if !strings.Contains(got, `"pattern"`) { + t.Errorf("Parameters = %s, want pattern to survive when sanitization is off", got) + } +} + +// TestTranscodeSanitizeToolSchemasOn is the request-build proof: with the +// flag on, the emitted tool's parameter schema has `pattern` stripped. +func TestTranscodeSanitizeToolSchemasOn(t *testing.T) { + req := toolRequestWithPattern() + out, err := transcodeRequestFamily(req, Family, nil, true) + if err != nil { + t.Fatalf("transcodeRequestFamily: %v", err) + } + if len(out.Tools) != 1 { + t.Fatalf("Tools = %#v, want 1 entry", out.Tools) + } + got := string(out.Tools[0].Parameters) + if strings.Contains(got, `"pattern"`) { + t.Errorf("Parameters = %s, want pattern stripped when sanitization is on", got) + } + + var schema map[string]interface{} + if err := json.Unmarshal(out.Tools[0].Parameters, &schema); err != nil { + t.Fatalf("unmarshal sanitized parameters: %v", err) + } + if schema["type"] != "object" { + t.Errorf("type = %v, want object preserved", schema["type"]) + } + required, ok := schema["required"].([]interface{}) + if !ok || len(required) != 1 || required[0] != "email" { + t.Errorf("required = %#v, want [email] preserved", schema["required"]) + } +} diff --git a/provider/openai/schema_sanitize.go b/provider/openai/schema_sanitize.go new file mode 100644 index 00000000..59a0f852 --- /dev/null +++ b/provider/openai/schema_sanitize.go @@ -0,0 +1,255 @@ +package openai + +import "encoding/json" + +// supportedSchemaTypes are the JSON Schema "type" values the ChatGPT Codex +// backend's tool-schema validator accepts. Anything else is dropped by +// sanitizeSchemaObject, either falling back to inference or vanishing +// entirely — see that function. +var supportedSchemaTypes = map[string]bool{ + "string": true, + "number": true, + "boolean": true, + "integer": true, + "object": true, + "array": true, + "null": true, +} + +// compositionKeys are the JSON Schema keywords that combine subschemas. +// Each is recursed into like any other subschema slot, never validated +// against supportedSchemaTypes itself. +var compositionKeys = [...]string{"anyOf", "oneOf", "allOf"} + +// sanitizeToolParameterSchema rewrites a tool's JSON Schema parameters for +// the ChatGPT Codex backend's stricter tool-schema validator, which rejects +// keywords the OpenAI platform API accepts without complaint — e.g. a +// regex `pattern` using lookaround, `format`, or `minLength` (confirmed +// live: "Invalid JSON schema: regex lookaround is not supported. Found at +// $.properties.email.pattern"). It rebuilds the schema by ALLOWLIST, +// keeping only the keywords sanitizeSchemaObject recognizes and dropping +// everything else, so an unsupported keyword cannot slip through by being +// merely unrecognized. +// +// Ported from opencode's sanitizeOpenAISchema (v1.18.23, +// codex-request-normalize.ts) — same allowlist, same type-inference +// fallback, same edge cases (boolean schema, const->enum, missing +// object/array defaults). +// +// An empty or unparseable schema passes through unchanged rather than risk +// corrupting a tool definition this code does not understand; only +// callers that have opted in via config.Provider.SanitizeToolSchemas +// invoke this at all (see Client.SanitizeToolSchemas / transcodeRequestFamily). +func sanitizeToolParameterSchema(raw json.RawMessage) json.RawMessage { + if len(raw) == 0 { + return raw + } + var parsed interface{} + if err := json.Unmarshal(raw, &parsed); err != nil { + return raw + } + sanitized, err := json.Marshal(sanitizeSchemaValue(parsed)) + if err != nil { + return raw + } + return sanitized +} + +// sanitizeSchemaValue is the recursive worker sanitizeToolParameterSchema +// drives, operating on values already decoded from JSON +// (map[string]interface{}, []interface{}, string, float64, bool, nil). +// +// A JSON Schema boolean value ("additionalProperties": true aside — that +// one is handled specially by its caller) such as a bare `true`/`false` +// subschema means "accept anything"/"accept nothing"; OpenCode's rebuild +// treats it as a permissive string schema rather than reproducing the +// boolean-schema form the Codex validator does not accept, so this mirrors +// that. +func sanitizeSchemaValue(value interface{}) interface{} { + switch v := value.(type) { + case bool: + return map[string]interface{}{"type": "string"} + case []interface{}: + out := make([]interface{}, len(v)) + for i, item := range v { + out[i] = sanitizeSchemaValue(item) + } + return out + case map[string]interface{}: + return sanitizeSchemaObject(v) + default: + return value + } +} + +// sanitizeSchemaObject rebuilds one JSON Schema object node by allowlist. +// Keys not named below are dropped unconditionally — notably `pattern`, +// `format` (except as a string-type inference signal), `minLength`/ +// `maxLength`, `default`, `examples`, and `title`. +func sanitizeSchemaObject(value map[string]interface{}) map[string]interface{} { + result := map[string]interface{}{} + + if ref, ok := value["$ref"].(string); ok { + result["$ref"] = ref + } + if desc, ok := value["description"].(string); ok { + result["description"] = desc + } + + if constVal, ok := value["const"]; ok { + result["enum"] = []interface{}{constVal} + } else if enumVal, ok := value["enum"].([]interface{}); ok { + result["enum"] = enumVal + } + + if props, ok := value["properties"].(map[string]interface{}); ok { + newProps := make(map[string]interface{}, len(props)) + for k, item := range props { + newProps[k] = sanitizeSchemaValue(item) + } + result["properties"] = newProps + } + + if reqVal, ok := value["required"].([]interface{}); ok { + filtered := make([]interface{}, 0, len(reqVal)) + for _, item := range reqVal { + if s, ok := item.(string); ok { + filtered = append(filtered, s) + } + } + result["required"] = filtered + } + + if items, ok := value["items"]; ok { + result["items"] = sanitizeSchemaValue(items) + } + + if ap, ok := value["additionalProperties"]; ok { + if b, isBool := ap.(bool); isBool { + result["additionalProperties"] = b + } else { + result["additionalProperties"] = sanitizeSchemaValue(ap) + } + } + + for _, key := range compositionKeys { + if arr, ok := value[key].([]interface{}); ok { + out := make([]interface{}, len(arr)) + for i, item := range arr { + out[i] = sanitizeSchemaValue(item) + } + result[key] = out + } + } + + for _, key := range [...]string{"$defs", "definitions"} { + if defs, ok := value[key].(map[string]interface{}); ok { + newDefs := make(map[string]interface{}, len(defs)) + for k, item := range defs { + newDefs[k] = sanitizeSchemaValue(item) + } + result[key] = newDefs + } + } + + schemaTypes := supportedTypesOf(value["type"]) + + if len(schemaTypes) == 0 && (hasStringKey(result, "$ref") || hasAnyCompositionKey(result)) { + return result + } + + inferredTypes := schemaTypes + switch { + case len(inferredTypes) > 0: + // already resolved from the explicit "type" + case hasAnyKey(value, "properties", "required", "additionalProperties"): + inferredTypes = []string{"object"} + case hasAnyKey(value, "items", "prefixItems"): + inferredTypes = []string{"array"} + case hasAnyKey(result, "enum") || hasAnyKey(value, "format"): + inferredTypes = []string{"string"} + case hasAnyKey(value, "minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum", "multipleOf"): + inferredTypes = []string{"number"} + } + + if len(inferredTypes) == 0 { + return map[string]interface{}{} + } + + if len(inferredTypes) == 1 { + result["type"] = inferredTypes[0] + } else { + typeArr := make([]interface{}, len(inferredTypes)) + for i, t := range inferredTypes { + typeArr[i] = t + } + result["type"] = typeArr + } + + if containsString(inferredTypes, "object") { + if _, ok := result["properties"]; !ok { + result["properties"] = map[string]interface{}{} + } + } + if containsString(inferredTypes, "array") { + if _, ok := result["items"]; !ok { + result["items"] = map[string]interface{}{"type": "string"} + } + } + + return result +} + +// supportedTypesOf resolves a decoded "type" value (a string, an array of +// strings, or anything else) to the subset of supportedSchemaTypes it +// names, in the same order, dropping unsupported entries rather than +// rejecting the whole schema. +func supportedTypesOf(rawType interface{}) []string { + switch t := rawType.(type) { + case string: + if supportedSchemaTypes[t] { + return []string{t} + } + case []interface{}: + var out []string + for _, item := range t { + if s, ok := item.(string); ok && supportedSchemaTypes[s] { + out = append(out, s) + } + } + return out + } + return nil +} + +func hasStringKey(m map[string]interface{}, key string) bool { + _, ok := m[key].(string) + return ok +} + +func hasAnyCompositionKey(m map[string]interface{}) bool { + for _, key := range compositionKeys { + if _, ok := m[key]; ok { + return true + } + } + return false +} + +func hasAnyKey(m map[string]interface{}, keys ...string) bool { + for _, key := range keys { + if _, ok := m[key]; ok { + return true + } + } + return false +} + +func containsString(list []string, s string) bool { + for _, item := range list { + if item == s { + return true + } + } + return false +} diff --git a/provider/openai/schema_sanitize_test.go b/provider/openai/schema_sanitize_test.go new file mode 100644 index 00000000..48e99256 --- /dev/null +++ b/provider/openai/schema_sanitize_test.go @@ -0,0 +1,226 @@ +package openai + +import ( + "encoding/json" + "testing" +) + +// mustSanitize round-trips a JSON Schema literal through +// sanitizeToolParameterSchema and unmarshals the result back into a generic +// map for assertion. +func mustSanitize(t *testing.T, schema string) map[string]interface{} { + t.Helper() + out := sanitizeToolParameterSchema(json.RawMessage(schema)) + var got map[string]interface{} + if err := json.Unmarshal(out, &got); err != nil { + t.Fatalf("unmarshal sanitized schema: %v\nraw: %s", err, out) + } + return got +} + +// TestSanitizeToolSchemaDropsUnsupportedKeywords is the direct regression +// case: the ChatGPT Codex backend 400s on a `pattern` using regex +// lookaround ("Invalid JSON schema: regex lookaround is not supported. +// Found at $.properties.email.pattern"). Sanitizing must strip `pattern`, +// `format`, and `minLength` while preserving the surrounding structure, +// types, and required list. +func TestSanitizeToolSchemaDropsUnsupportedKeywords(t *testing.T) { + got := mustSanitize(t, `{ + "type": "object", + "properties": { + "email": { + "type": "string", + "pattern": "^(?=.*@)[^\\s]+$", + "format": "email", + "minLength": 3, + "description": "the user's email" + }, + "name": {"type": "string"} + }, + "required": ["email"] + }`) + + props, ok := got["properties"].(map[string]interface{}) + if !ok { + t.Fatalf("properties missing or wrong type: %#v", got["properties"]) + } + email, ok := props["email"].(map[string]interface{}) + if !ok { + t.Fatalf("properties.email missing or wrong type: %#v", props["email"]) + } + for _, key := range []string{"pattern", "format", "minLength"} { + if _, present := email[key]; present { + t.Errorf("properties.email retained %q, want dropped: %#v", key, email) + } + } + if email["type"] != "string" { + t.Errorf("properties.email.type = %v, want string", email["type"]) + } + if email["description"] != "the user's email" { + t.Errorf("properties.email.description = %v, want preserved", email["description"]) + } + if got["type"] != "object" { + t.Errorf("type = %v, want object", got["type"]) + } + required, ok := got["required"].([]interface{}) + if !ok || len(required) != 1 || required[0] != "email" { + t.Errorf("required = %#v, want [email]", got["required"]) + } +} + +// TestSanitizeToolSchemaConstBecomesEnum mirrors opencode's const->enum +// rewrite: the Codex validator does not carry `const` through the same +// allowlist path as `enum`, so opencode always converts it. +func TestSanitizeToolSchemaConstBecomesEnum(t *testing.T) { + got := mustSanitize(t, `{"type": "string", "const": "fixed"}`) + enum, ok := got["enum"].([]interface{}) + if !ok || len(enum) != 1 || enum[0] != "fixed" { + t.Errorf("enum = %#v, want [fixed]", got["enum"]) + } + if _, present := got["const"]; present { + t.Errorf("const retained, want dropped: %#v", got) + } +} + +// TestSanitizeToolSchemaNestedItemsAndAnyOf exercises recursion through +// `items` and a composition keyword (`anyOf`), and confirms an unsupported +// type on one branch is dropped while a supported branch survives. +func TestSanitizeToolSchemaNestedItemsAndAnyOf(t *testing.T) { + got := mustSanitize(t, `{ + "type": "array", + "items": { + "anyOf": [ + {"type": "string", "pattern": "no"}, + {"type": "widget"} + ] + } + }`) + if got["type"] != "array" { + t.Fatalf("type = %v, want array", got["type"]) + } + items, ok := got["items"].(map[string]interface{}) + if !ok { + t.Fatalf("items missing or wrong type: %#v", got["items"]) + } + anyOf, ok := items["anyOf"].([]interface{}) + if !ok || len(anyOf) != 2 { + t.Fatalf("anyOf = %#v, want 2 entries", items["anyOf"]) + } + first, ok := anyOf[0].(map[string]interface{}) + if !ok || first["type"] != "string" { + t.Errorf("anyOf[0] = %#v, want type string", anyOf[0]) + } + if _, present := first["pattern"]; present { + t.Errorf("anyOf[0] retained pattern, want dropped: %#v", first) + } + second, ok := anyOf[1].(map[string]interface{}) + if !ok { + t.Fatalf("anyOf[1] wrong type: %#v", anyOf[1]) + } + if _, present := second["type"]; present { + t.Errorf("anyOf[1] retained unsupported type %q, want dropped entirely: %#v", second["type"], second) + } +} + +// TestSanitizeToolSchemaDefs exercises $defs recursion. +func TestSanitizeToolSchemaDefs(t *testing.T) { + got := mustSanitize(t, `{ + "type": "object", + "properties": {"thing": {"$ref": "#/$defs/Thing"}}, + "$defs": { + "Thing": {"type": "string", "format": "uri"} + } + }`) + defs, ok := got["$defs"].(map[string]interface{}) + if !ok { + t.Fatalf("$defs missing or wrong type: %#v", got["$defs"]) + } + thing, ok := defs["Thing"].(map[string]interface{}) + if !ok { + t.Fatalf("$defs.Thing wrong type: %#v", defs["Thing"]) + } + if thing["type"] != "string" { + t.Errorf("$defs.Thing.type = %v, want string", thing["type"]) + } + if _, present := thing["format"]; present { + t.Errorf("$defs.Thing retained format, want dropped: %#v", thing) + } + props, ok := got["properties"].(map[string]interface{}) + if !ok { + t.Fatalf("properties missing: %#v", got) + } + thingRef, ok := props["thing"].(map[string]interface{}) + if !ok || thingRef["$ref"] != "#/$defs/Thing" { + t.Errorf("properties.thing = %#v, want $ref preserved", props["thing"]) + } +} + +// TestSanitizeToolSchemaObjectGetsEmptyProperties: an object-typed node with +// no properties must gain an empty properties object, matching opencode's +// rebuild (some validators require it). +func TestSanitizeToolSchemaObjectGetsEmptyProperties(t *testing.T) { + got := mustSanitize(t, `{"type": "object"}`) + props, ok := got["properties"].(map[string]interface{}) + if !ok || len(props) != 0 { + t.Errorf("properties = %#v, want empty object", got["properties"]) + } +} + +// TestSanitizeToolSchemaArrayGetsStringItems: an array-typed node with no +// items must gain a default {"type":"string"} items schema. +func TestSanitizeToolSchemaArrayGetsStringItems(t *testing.T) { + got := mustSanitize(t, `{"type": "array"}`) + items, ok := got["items"].(map[string]interface{}) + if !ok || items["type"] != "string" { + t.Errorf("items = %#v, want {type: string}", got["items"]) + } +} + +// TestSanitizeToolSchemaInferredObjectFromProperties: no explicit "type", +// but "properties" is present — infer object, matching opencode. +func TestSanitizeToolSchemaInferredObjectFromProperties(t *testing.T) { + got := mustSanitize(t, `{"properties": {"x": {"type": "string"}}}`) + if got["type"] != "object" { + t.Errorf("type = %v, want inferred object", got["type"]) + } +} + +// TestSanitizeToolSchemaUnsupportedTypeDropsToEmpty: a node whose type is +// unsupported and that carries no inferable structure collapses to {} +// entirely, matching opencode's rebuild rather than emitting a +// half-populated node. +func TestSanitizeToolSchemaUnsupportedTypeDropsToEmpty(t *testing.T) { + got := mustSanitize(t, `{"type": "widget", "description": "should vanish too"}`) + if len(got) != 0 { + t.Errorf("got %#v, want empty object (node dropped)", got) + } +} + +// TestSanitizeToolSchemaBooleanValueBecomesStringSchema: a bare JSON Schema +// boolean value (used as a subschema, e.g. additionalProperties or an items +// entry) becomes a permissive {"type":"string"} node, matching opencode. +func TestSanitizeToolSchemaBooleanValueBecomesStringSchema(t *testing.T) { + got := mustSanitize(t, `{"type": "object", "additionalProperties": true}`) + if got["additionalProperties"] != true { + t.Errorf("additionalProperties = %v, want true preserved as-is", got["additionalProperties"]) + } + + got2 := mustSanitize(t, `{"type": "array", "items": false}`) + items, ok := got2["items"].(map[string]interface{}) + if !ok || items["type"] != "string" { + t.Errorf("items (from boolean false) = %#v, want {type: string}", got2["items"]) + } +} + +// TestSanitizeToolSchemaEmptyOrUnparseablePassesThrough: an empty schema and +// malformed JSON must pass through unchanged rather than risk corrupting a +// tool definition this code does not understand. +func TestSanitizeToolSchemaEmptyOrUnparseablePassesThrough(t *testing.T) { + if out := sanitizeToolParameterSchema(nil); out != nil { + t.Errorf("nil input: got %q, want nil", out) + } + bad := json.RawMessage(`{not valid json`) + if out := sanitizeToolParameterSchema(bad); string(out) != string(bad) { + t.Errorf("malformed input: got %q, want unchanged %q", out, bad) + } +} diff --git a/provider/openai/transcode.go b/provider/openai/transcode.go index 5d8222a2..5a1e6c33 100644 --- a/provider/openai/transcode.go +++ b/provider/openai/transcode.go @@ -154,7 +154,7 @@ func wireCallID(id string) string { // under the package Family constant — the default for a client that // configures no family of its own. func transcodeRequest(req *provider.Request) (*apiRequest, error) { - return transcodeRequestFamily(req, Family, nil) + return transcodeRequestFamily(req, Family, nil, false) } // transcodeRequestFamily is transcodeRequest with the ProviderData tag made @@ -167,8 +167,12 @@ func transcodeRequest(req *provider.Request) (*apiRequest, error) { // the wire entirely (Client.OmitResponseParams / config.Provider's field of // the same name) — applied last, after every other field of out is // computed, so it always wins over the reasoning-floor bump and every other -// adjustment above. -func transcodeRequestFamily(req *provider.Request, family string, omitParams []string) (*apiRequest, error) { +// adjustment above. sanitizeSchemas, when true, rewrites every tool's +// parameter schema through sanitizeToolParameterSchema before it is placed +// on out.Tools (Client.SanitizeToolSchemas / config.Provider's field of the +// same name) — false (the default) leaves every schema byte-identical to +// req.Tools, matching this adapter's behavior before the field existed. +func transcodeRequestFamily(req *provider.Request, family string, omitParams []string, sanitizeSchemas bool) (*apiRequest, error) { out := &apiRequest{ Model: req.Model.Model, Instructions: strings.Join(req.System, "\n\n"), @@ -221,11 +225,15 @@ func transcodeRequestFamily(req *provider.Request, family string, omitParams []s } for _, t := range req.Tools { + params := t.InputSchema + if sanitizeSchemas { + params = sanitizeToolParameterSchema(params) + } out.Tools = append(out.Tools, apiToolDef{ Type: "function", Name: t.Name, Description: t.Description, - Parameters: t.InputSchema, + Parameters: params, }) } From fbc371270f0096da14cda796989f27e1c992ccc9 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Sun, 30 Aug 2026 09:35:02 -0400 Subject: [PATCH 27/95] provider/openai: stream the Codex Responses API over websockets (#216) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ChatGPT Codex backend (wss://chatgpt.com/backend-api/codex/responses) speaks its own persistent websocket transport for the Responses API, alongside the HTTP POST + SSE stream every other Responses-compatible endpoint uses. harness's native openai adapter only ever POSTed, so a provider routed at the Codex backend could not use the transport its own reference client prefers. Port opencode's Codex websocket transport (packages/opencode/src/plugin/openai/{ws-pool,ws}.ts) to Go: - ws.go dials wss:// with the SAME headers the HTTP path sends (Authorization included), frames a request as {"type":"response.create", ...body-minus-stream}, and reads response.* frames back. The dial goes through the SAME *http.Client the HTTP path already uses (github.com/coder/ websocket's DialOptions.HTTPClient performs the handshake via that client's own Transport, per its doc comment and Go's own net/http Upgrade support since 1.12) — proxy (HTTPS_PROXY) and TLS trust are therefore identical between the two transports by construction, not by two configurations kept in sync by hand. - ws_stream.go adapts a websocket frame's "type" field to the same (name, data) shape stream.readSSE already produces from an SSE "event:"/"data:" pair, so stream.handle — the Responses event to provider.Event mapper — runs UNCHANGED for a websocket-delivered response. Only how a (name, data) pair is obtained differs; the event mapping itself is not duplicated. - ws_pool.go pools one persistent connection per harness session (provider.Request.SessionKey, already set to the session ID on every request — see engine.go/goal.go/compact.go), reused across turns: idle timeout 5m, max connection age 55m, up to 5 consecutive failures before falling back to HTTP for the rest of the session, and a MESSAGE_TOO_BIG (1009) close is an immediate permanent fallback. A concurrent request on a session whose socket is already mid-turn also falls back to HTTP rather than contend for the connection. Ported from opencode's ws-pool.ts createWebSocketFetch, adapted from its fetch()-interception shape to provider.Provider's Stream/Next contract. - Gating: config.Provider.UseWebSocketTransport (use_websocket_transport), valid only on an entry that builds the Responses adapter — the same buildsResponsesAdapter guard #213's OmitResponseParams and #215's SanitizeToolSchemas use — and non-clearable across config layers like NoPromptCacheKey. Default false is byte-identical to this adapter's pre-existing behavior; turning it on can only add a transport attempt in front of the existing HTTP path, never remove it. Any failure at all — no SessionKey, a busy or previously-broken session, dial/send/ first-frame failure — falls straight through to HTTP for that request. Sibling dependency (#215, sanitize_tool_schemas): that PR merged to main while this one was in flight. It rewrites each tool's schema inside transcodeRequestFamily, before the request is marshaled to the `body` bytes both the HTTP POST and this ws transport send. The ws path therefore inherits schema sanitization automatically — it sends the same already-sanitized body, wrapped as response.create — with no sanitizer logic duplicated here. Gatekeeper finding (load-bearing for whether this is usable at all): verified against gatekeeper's own source (proxy/proxy.go's handleConnectWithInterception) that it already injects the real OAuth bearer on a websocket Upgrade GET exactly as it does any other HTTPS request through it. gatekeeper MITM- terminates the CONNECT tunnel (when a CA is configured, which production is, since HTTP credential injection already works) and runs every decrypted request — Upgrade included — through a real httputil.ReverseProxy whose Rewrite hook injects credentials unconditionally, method- and Upgrade-agnostic. That ReverseProxy has built-in Upgrade support: it hijacks both connections after the 101 and pipes bytes bidirectionally, so a genuinely long-lived duplex websocket survives the proxy intact. There is already a passing gatekeeper test for exactly this path (TestIntercept_WebSocketUpgrade, proxy/intercept_test.go). No gatekeeper-side change is required for this transport to authenticate. Verification: - New fake-websocket-server tests (httptest + a real coder/ websocket Accept, not a mocked interface): connects and streams a full response.create -> response.completed turn end to end, reusing stream.handle's existing event assembly; asserts the exact response.create framing (stream field stripped); pool reuse across two turns on the same session (one dial, two response.create sends); a non-clean terminal (response.failed) drops its connection rather than pooling it; a busy session's concurrent second call falls back to HTTP; MESSAGE_TOO_BIG permanently falls back for that session; no SessionKey, dial failure, and UseWebSocketTransport left off all fall back to HTTP without attempting a dial. - Config tests mirroring OmitResponseParams'/NoPromptCacheKey's own shape: use_websocket_transport accepted on the native "openai" key and a keyed type:"openai" entry, rejected elsewhere, false always valid, non-clearable merge semantics. - go build ./..., go vet ./..., go test ./... (and -race on provider/openai, config, cmd/harness) all green; the existing SSE fuzz target and every pre-existing openai provider test pass unmodified. Dependency: github.com/coder/websocket (the actively maintained nhooyr.io/websocket successor), chosen for its dependency-free module, context-aware Read/Write, and dialing through a caller- supplied *http.Client rather than its own bespoke dialer — the property this port's proxy/CA reuse relies on. Follow-on (sequential, after merge, not part of this change): wire use_websocket_transport + sanitize_tool_schemas on the ChatGPT subscription lane's codex provider block, and bump boxes' HARNESS_REF. --- cmd/harness/main.go | 37 +- config/config.go | 48 +++ config/use_websocket_transport_test.go | 124 +++++++ go.mod | 2 + go.sum | 2 + provider/openai/openai.go | 116 +++++- provider/openai/ws.go | 161 ++++++++ provider/openai/ws_pool.go | 264 +++++++++++++ provider/openai/ws_stream.go | 101 +++++ provider/openai/ws_test.go | 490 +++++++++++++++++++++++++ 10 files changed, 1323 insertions(+), 22 deletions(-) create mode 100644 config/use_websocket_transport_test.go create mode 100644 provider/openai/ws.go create mode 100644 provider/openai/ws_pool.go create mode 100644 provider/openai/ws_stream.go create mode 100644 provider/openai/ws_test.go diff --git a/cmd/harness/main.go b/cmd/harness/main.go index c5bb2e73..9816df23 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -939,9 +939,10 @@ func registry(cfg *config.Config) provider.Registry { anthropic.Family: &anthropic.Client{APIKey: akey, BaseURL: abase, CacheTTL: anthropicCacheTTL(cfg)}, // The built-in openai entry deliberately leaves Family empty: it IS // the package default, and naming it here would only invite the two - // to drift. Its ResponsesPath/OmitResponseParams/SanitizeToolSchemas - // come from the native entry, if any. - openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg), OmitResponseParams: nativeOmitResponseParams(cfg), SanitizeToolSchemas: nativeSanitizeToolSchemas(cfg)}, + // to drift. Its ResponsesPath/OmitResponseParams/ + // SanitizeToolSchemas/UseWebSocketTransport come from the native + // entry, if any. + openai.Family: &openai.Client{APIKey: okey, BaseURL: obase, ResponsesPath: nativeResponsesPath(cfg), OmitResponseParams: nativeOmitResponseParams(cfg), SanitizeToolSchemas: nativeSanitizeToolSchemas(cfg), UseWebSocketTransport: nativeUseWebSocketTransport(cfg)}, } registerOpenAICompatProviders(reg, cfg) registerOpenAIProviders(reg, cfg) @@ -1046,12 +1047,13 @@ func registerOpenAIProviders(reg provider.Registry, cfg *config.Config) { } apiKey := os.Getenv(keyEnv) reg[name] = &openai.Client{ - Family: name, - APIKey: apiKey, - BaseURL: p.BaseURL, - ResponsesPath: p.ResponsesPath, - OmitResponseParams: p.OmitResponseParams, - SanitizeToolSchemas: p.SanitizeToolSchemas, + Family: name, + APIKey: apiKey, + BaseURL: p.BaseURL, + ResponsesPath: p.ResponsesPath, + OmitResponseParams: p.OmitResponseParams, + SanitizeToolSchemas: p.SanitizeToolSchemas, + UseWebSocketTransport: p.UseWebSocketTransport, } } } @@ -1106,6 +1108,23 @@ func nativeSanitizeToolSchemas(cfg *config.Config) bool { return p.SanitizeToolSchemas } +// nativeUseWebSocketTransport reads use_websocket_transport configured on +// the NATIVE "openai" entry (map key "openai", no type — the same shape +// config.validateProviders permits use_websocket_transport on). false +// leaves the adapter on the HTTP + SSE transport. A keyed type:"openai" +// entry carries its own value and is wired by registerOpenAIProviders +// instead. Mirrors nativeOmitResponseParams exactly. +func nativeUseWebSocketTransport(cfg *config.Config) bool { + if cfg == nil { + return false + } + p := cfg.Providers[openai.Family] + if p.Type != "" { + return false + } + return p.UseWebSocketTransport +} + // registerOpenAICompatProviders builds a provider/openaicompat client for // every config.Providers entry of config.TypeOpenAICompat, keyed by its // providers map name — that name is what routes "name/model" refs to it, diff --git a/config/config.go b/config/config.go index fc67e76a..9403422d 100644 --- a/config/config.go +++ b/config/config.go @@ -629,6 +629,33 @@ type Provider struct { // comment): a project layer can set this to true, but cannot flip an // inherited true back to false. SanitizeToolSchemas bool `json:"sanitize_tool_schemas,omitempty"` + // UseWebSocketTransport routes the native OpenAI Responses adapter's + // calls over a pooled wss:// connection (one persistent socket per + // harness session, see provider/openai's ws.go/ws_pool.go) instead of + // an HTTP POST + SSE stream per turn. + // + // It exists because the ChatGPT Codex backend speaks its OWN + // websocket transport for the Responses API — a per-session + // persistent connection the reference client (opencode) always + // prefers over HTTP for that backend. Any failure along the way + // (dial failure, a frame larger than the server will send over ws, + // too many consecutive stream failures, a concurrent request on a + // session whose socket is already busy) falls back to the existing + // HTTP path for that request, so turning this on can only add a + // transport, never remove the working one. Default false leaves + // every provider on HTTP exactly as before this field existed. Tool + // schemas sent over that transport are still whatever + // SanitizeToolSchemas above already made them — the wire body ws + // sends is the SAME transcoded body the HTTP path would send, so + // nothing here duplicates that sanitization. + // + // Valid ONLY on an entry that builds the Responses adapter, exactly + // like ResponsesPath and OmitResponseParams — the same + // buildsResponsesAdapter identity check gates all three. Merge + // semantics are non-clearable like NoPromptCacheKey (see its doc + // comment): a project layer can turn this on, but cannot flip an + // inherited true back to false. + UseWebSocketTransport bool `json:"use_websocket_transport,omitempty"` // CacheTTL selects the Anthropic prompt-cache breakpoint lifetime: // "5m" (the Anthropic API default) or "1h" (the extended TTL, beta // extended-cache-ttl-2025-04-11). Empty (the default) leaves the @@ -808,6 +835,9 @@ func validateProviders(providers map[string]Provider) error { if err := validateSanitizeToolSchemas(name, p); err != nil { return err } + if err := validateUseWebSocketTransport(name, p); err != nil { + return err + } if err := validateClaudeCodeFields(name, p); err != nil { return err } @@ -876,6 +906,21 @@ func validateResponsesPath(name string, p Provider) error { return nil } +// validateUseWebSocketTransport fails loudly on use_websocket_transport set +// on any entry that does not build the Responses adapter — the same +// buildsResponsesAdapter identity check validateResponsesPath and +// validateOmitResponseParams use, since only that adapter reads the field. +// false (the default) is always valid. +func validateUseWebSocketTransport(name string, p Provider) error { + if !p.UseWebSocketTransport { + return nil + } + if !buildsResponsesAdapter(name, p) { + return fmt.Errorf("providers.%s: use_websocket_transport is only valid on an entry that builds the OpenAI Responses adapter (map key %q with no type, or any key with type %q); no other adapter reads it", name, "openai", TypeOpenAI) + } + return nil +} + // validateOmitResponseParams fails loudly on an omit_response_params entry // that cannot possibly be honored: a name outside OmitResponseParamValues (a // typo that would otherwise silently keep sending the very param the @@ -1349,6 +1394,9 @@ func merge(base, over *Config) *Config { if v.SanitizeToolSchemas { ex.SanitizeToolSchemas = true } + if v.UseWebSocketTransport { + ex.UseWebSocketTransport = true + } if v.BinaryPath != "" { ex.BinaryPath = v.BinaryPath } diff --git a/config/use_websocket_transport_test.go b/config/use_websocket_transport_test.go new file mode 100644 index 00000000..73df1546 --- /dev/null +++ b/config/use_websocket_transport_test.go @@ -0,0 +1,124 @@ +package config + +import ( + "strings" + "testing" +) + +// UseWebSocketTransport follows the same buildsResponsesAdapter gating and +// non-clearable merge semantics as OmitResponseParams/NoPromptCacheKey — +// see omit_response_params_test.go and cache_ttl_test.go for the sibling +// fields this mirrors. + +func TestUseWebSocketTransportOnNativeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "openai": {UseWebSocketTransport: true}, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["openai"].UseWebSocketTransport { + t.Error("UseWebSocketTransport = false, want true") + } +} + +func TestUseWebSocketTransportOnTypeOpenAIKeyOK(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "secondary": { + Type: TypeOpenAI, + BaseURL: "https://gateway.example", + UseWebSocketTransport: true, + }, + }} + merged, err := mergeAndValidate(c, &Config{}) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["secondary"].UseWebSocketTransport { + t.Error("UseWebSocketTransport = false, want true") + } +} + +// TestUseWebSocketTransportOnWrongAdapterFails: only the Responses adapter +// reads use_websocket_transport — set anywhere else it would vanish +// silently into a client that never looks at it, the same rule +// omit_response_params and responses_path follow. +func TestUseWebSocketTransportOnWrongAdapterFails(t *testing.T) { + for _, tc := range []struct { + name string + key string + p Provider + }{ + {"openai-compat", "mycompat", Provider{Type: TypeOpenAICompat, BaseURL: "http://x", UseWebSocketTransport: true}}, + {"native anthropic", "anthropic", Provider{UseWebSocketTransport: true}}, + } { + t.Run(tc.name, func(t *testing.T) { + c := &Config{Providers: map[string]Provider{tc.key: tc.p}} + _, err := mergeAndValidate(c, &Config{}) + if err == nil { + t.Fatalf("mergeAndValidate accepted use_websocket_transport on a %s entry", tc.name) + } + if !strings.Contains(err.Error(), "use_websocket_transport") { + t.Errorf("error %q does not name the offending field", err) + } + if !strings.Contains(err.Error(), tc.key) { + t.Errorf("error %q does not name the offending key", err) + } + }) + } +} + +// TestUseWebSocketTransportFalseIsAlwaysValid: the default is never +// rejected on any adapter — only an explicit true is gated. +func TestUseWebSocketTransportFalseIsAlwaysValid(t *testing.T) { + c := &Config{Providers: map[string]Provider{ + "anthropic": {}, + "mycompat": {Type: TypeOpenAICompat, BaseURL: "http://x"}, + }} + if _, err := mergeAndValidate(c, &Config{}); err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } +} + +// TestUseWebSocketTransportMergesFromProject: a project override may turn +// the flag on without restating the entry, like no_prompt_cache_key. +func TestUseWebSocketTransportMergesFromProject(t *testing.T) { + user := &Config{Providers: map[string]Provider{ + "openai": {BaseURL: "https://api.example"}, + }} + proj := &Config{Providers: map[string]Provider{ + "openai": {UseWebSocketTransport: true}, + }} + got, err := mergeAndValidate(user, proj) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + p := got.Providers["openai"] + if !p.UseWebSocketTransport { + t.Error("UseWebSocketTransport = false, want true after merge") + } + if p.BaseURL != "https://api.example" { + t.Errorf("merge lost sibling fields: %+v", p) + } +} + +// TestUseWebSocketTransportNonClearable: like NoPromptCacheKey, a project +// layer can turn this on but a later layer cannot turn an inherited true +// back off — there is no *bool escape hatch for it, deliberately (see the +// field's doc comment). +func TestUseWebSocketTransportNonClearable(t *testing.T) { + base := &Config{Providers: map[string]Provider{ + "openai": {UseWebSocketTransport: true}, + }} + over := &Config{Providers: map[string]Provider{ + "openai": {BaseURL: "https://api.example"}, // does not restate the flag + }} + merged, err := mergeAndValidate(base, over) + if err != nil { + t.Fatalf("mergeAndValidate: %v", err) + } + if !merged.Providers["openai"].UseWebSocketTransport { + t.Error("UseWebSocketTransport = false, want true (inherited, non-clearable)") + } +} diff --git a/go.mod b/go.mod index b33addb0..f767e983 100644 --- a/go.mod +++ b/go.mod @@ -5,3 +5,5 @@ go 1.25.5 require pgregory.net/rapid v1.3.0 require golang.org/x/image v0.44.0 + +require github.com/coder/websocket v1.8.15 // indirect diff --git a/go.sum b/go.sum index 9b2243b1..1d2de6c2 100644 --- a/go.sum +++ b/go.sum @@ -1,3 +1,5 @@ +github.com/coder/websocket v1.8.15 h1:6B2JPeOGlpff2Uz6vOEH1Vzpi0iUz20A+lPVhPHtNUA= +github.com/coder/websocket v1.8.15/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg= golang.org/x/image v0.44.0 h1:+tDekMZED9+LrtB3G5xzRggpVh9CARjZqROla3R3R+I= golang.org/x/image v0.44.0/go.mod h1:V8K3KE9KKKE+pLpQDOeN18w9oacNSvy1tDOirTu4xtY= pgregory.net/rapid v1.3.0 h1:vBvO0VSqti75J1jjYqpgPNBLKMd1+gxa9fYo7vk/Exc= diff --git a/provider/openai/openai.go b/provider/openai/openai.go index 3e3660a5..4df4ec7e 100644 --- a/provider/openai/openai.go +++ b/provider/openai/openai.go @@ -10,6 +10,7 @@ import ( "io" "net/http" "strings" + "sync" "time" "github.com/majorcontext/harness/message" @@ -72,6 +73,30 @@ type Client struct { // transcodeRequestFamily builds from it. Default false leaves every // tool schema byte-identical to req.Tools. SanitizeToolSchemas bool + // UseWebSocketTransport routes this client's calls over a pooled + // wss:// connection (see ws.go/ws_pool.go) instead of HTTP POST + SSE + // — see config.Provider's field of the same name for the full + // rationale. Any failure along the way falls back to the HTTP path + // below for that request, so this can only add a transport, never + // remove the working one. Default false is byte-identical to this + // client's behavior before the field existed. Note this client + // already sanitizes tool schemas (SanitizeToolSchemas) and omits + // params (OmitResponseParams) BEFORE the wire body is built, so the + // websocket path — which sends that same already-transcoded body — + // inherits both with no duplicated logic. + UseWebSocketTransport bool + + wsPoolOnce sync.Once + wsPoolVal *wsPool +} + +// wsPoolFor lazily builds this client's websocket pool on first use, one +// per Client instance (not global) — a Client is already the unit main.go +// registers one per provider entry, so this scopes pooled connections to +// the same boundary API keys and base URLs are already scoped to. +func (c *Client) wsPoolFor() *wsPool { + c.wsPoolOnce.Do(func() { c.wsPoolVal = newWSPool() }) + return c.wsPoolVal } // familyOrDefault resolves a configured family override to the family key @@ -92,6 +117,18 @@ func (c *Client) family() string { return familyOrDefault(c.Family) } func (c *Client) Name() string { return c.family() } +// httpClient returns the *http.Client this adapter makes every request +// with — c.HTTPClient, or http.DefaultClient when unset. Both the HTTP POST +// path and the websocket dial (see Stream, wsPoolFor) use this exact value, +// which is what makes the two transports' proxy/TLS behavior identical by +// construction rather than by two configurations kept in sync by hand. +func (c *Client) httpClient() *http.Client { + if c.HTTPClient != nil { + return c.HTTPClient + } + return http.DefaultClient +} + func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.Stream, error) { if c.APIKey == "" { return nil, fmt.Errorf("openai: no API key configured (set OPENAI_API_KEY)") @@ -105,18 +142,43 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St return nil, err } - httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, responsesURL(c.BaseURL, c.ResponsesPath), bytes.NewReader(body)) + url := responsesURL(c.BaseURL, c.ResponsesPath) + headers := http.Header{ + "Content-Type": []string{"application/json"}, + "Accept": []string{"text/event-stream"}, + "Authorization": []string{"Bearer " + c.APIKey}, + } + hc := c.httpClient() + + // The websocket transport sends this SAME url/headers/body — the + // Authorization header included — so whatever credential injection + // applies to the HTTP path below (a box's gatekeeper proxy swapping + // this dummy bearer for a real OAuth one) applies identically to the + // ws dial, since both go through hc's own Transport (proxy + TLS + // trust store), and no separate code path could plausibly relay + // different credentials for the same client. Any failure at all — + // no SessionKey, a busy or previously-broken session, dial/send/ + // first-frame failure — falls through to the HTTP POST unchanged. + if c.UseWebSocketTransport && req.SessionKey != "" { + if st, ok := c.wsPoolFor().stream(ctx, wsStreamRequest{ + SessionKey: req.SessionKey, + URL: url, + Headers: headers, + Body: body, + Model: req.Model, + Family: c.family(), + HTTPClient: hc, + }); ok { + return st, nil + } + } + + httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) if err != nil { return nil, err } - httpReq.Header.Set("Content-Type", "application/json") - httpReq.Header.Set("Accept", "text/event-stream") - httpReq.Header.Set("Authorization", "Bearer "+c.APIKey) + httpReq.Header = headers.Clone() - hc := c.HTTPClient - if hc == nil { - hc = http.DefaultClient - } resp, err := hc.Do(httpReq) if err != nil { return nil, err @@ -231,9 +293,15 @@ type assembledItem struct { // forwards deltas as they arrive and assembles the canonical assistant // message, delivered with EventDone on response.completed. type stream struct { - body io.Closer - r *bufio.Reader - model message.ModelRef + body io.Closer + r *bufio.Reader + // wsConn is non-nil for a websocket-delivered response (see wsPool. + // stream) and nil for the HTTP+SSE path — exactly one of {r, wsConn} + // is set. Next/Close dispatch on it instead of duplicating the SSE + // decode loop and event mapping for the ws case: stream.handle below + // is shared, unmodified, between both transports. + wsConn *wsFrameSource + model message.ModelRef // family is the client's resolved provider family: the ProviderData tag // this stream writes reasoning attachments under. Empty means the // package Family constant (see familyOrDefault), which keeps a stream @@ -250,7 +318,29 @@ type stream struct { done bool } -func (s *stream) Close() error { return s.body.Close() } +// Close releases this stream's transport. For a websocket-delivered +// response, whether the underlying connection is actually torn down or +// kept pooled for the session's next turn was already decided when the +// terminal event was read (see wsFrameSource.close) — Close here just +// tells that decision whether it is being reached cleanly (s.done, the +// same flag Next uses to return io.EOF) or because the caller is +// abandoning the stream early. +func (s *stream) Close() error { + if s.wsConn != nil { + return s.wsConn.close(s.done) + } + return s.body.Close() +} + +// readEvent returns the next (name, data) pair from whichever transport +// this stream is reading — the ws frame source's buffered/live frames, or +// readSSE's HTTP+SSE decode — so handle below never needs to know which. +func (s *stream) readEvent() (string, []byte, error) { + if s.wsConn != nil { + return s.wsConn.next() + } + return s.readSSE() +} func (s *stream) Next() (provider.Event, error) { for { @@ -262,7 +352,7 @@ func (s *stream) Next() (provider.Event, error) { if s.done { return provider.Event{}, io.EOF } - name, data, err := s.readSSE() + name, data, err := s.readEvent() if err != nil { // Reaching this read at all means response.completed has not // been seen (s.done, checked above, would have returned the diff --git a/provider/openai/ws.go b/provider/openai/ws.go new file mode 100644 index 00000000..dfcb4214 --- /dev/null +++ b/provider/openai/ws.go @@ -0,0 +1,161 @@ +package openai + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "time" + + "github.com/coder/websocket" +) + +// wsProtocolHeader is the "openai-beta" header value this transport sends +// on the dial handshake unless the caller already set one. Ported verbatim +// from opencode's ws.ts PROTOCOL_HEADER — the Codex backend's websocket +// endpoint uses it to negotiate the response.create wire shape this file +// speaks. +const wsProtocolHeader = "responses_websockets=2026-02-06" + +// wsMessageTooBigCode is the WebSocket close code (RFC 6449) the Codex +// backend sends when a request or response frame exceeds its size limit — +// a permanent, this-session-can-never-work-over-ws condition, not a +// transient one. Ported from opencode's ws.ts MESSAGE_TOO_BIG_CLOSE_CODE. +const wsMessageTooBigCode = websocket.StatusMessageTooBig + +// toWebSocketURL rewrites an http(s):// Responses URL to its ws(s):// +// equivalent. Ported from opencode's ws.ts toWebSocketUrl. +func toWebSocketURL(rawURL string) string { + switch { + case strings.HasPrefix(rawURL, "https://"): + return "wss://" + strings.TrimPrefix(rawURL, "https://") + case strings.HasPrefix(rawURL, "http://"): + return "ws://" + strings.TrimPrefix(rawURL, "http://") + default: + return rawURL + } +} + +// dialResponsesWebSocket opens the persistent Codex Responses websocket: +// same URL/headers as the HTTP path (Authorization included), converted to +// ws(s)://, bounded by timeout. The dial goes through httpClient's own +// Transport — see the DialOptions.HTTPClient doc comment on +// github.com/coder/websocket — so it inherits whatever proxy (HTTPS_PROXY) +// and TLS trust store (SSL_CERT_FILE/SSL_CERT_DIR, the system pool) that +// client is already configured with, identically to every HTTP request +// this adapter makes. There is no separate proxy/CA plumbing to add here; +// reusing httpClient IS the proxy/CA support. +func dialResponsesWebSocket(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, error) { + dialCtx := ctx + var cancel context.CancelFunc + if timeout > 0 { + dialCtx, cancel = context.WithTimeout(ctx, timeout) + defer cancel() + } + hdr := headers.Clone() + if hdr == nil { + hdr = http.Header{} + } + if hdr.Get("openai-beta") == "" { + hdr.Set("openai-beta", wsProtocolHeader) + } + // Content-Length describes the (nonexistent) body of the HTTP POST this + // adapter would otherwise send; it has no meaning on a GET-style + // Upgrade handshake and coder/websocket's Dial already ignores it, but + // stripping it keeps the outgoing header set honest. + hdr.Del("Content-Length") + + conn, _, err := websocket.Dial(dialCtx, url, &websocket.DialOptions{ + HTTPClient: httpClient, + HTTPHeader: hdr, + }) + if err != nil { + return nil, fmt.Errorf("openai: websocket dial: %w", err) + } + // A single oversized frame (tool output, a huge pasted file) must not + // silently kill the connection: without raising this, coder/websocket's + // 32KiB default read limit would surface as an ordinary close, which + // this transport's caller (wsPool) cannot tell apart from the server's + // own MESSAGE_TOO_BIG close. 64MiB matches the Responses API's own + // documented per-request body cap, so nothing legitimate is still cut + // short. + conn.SetReadLimit(64 << 20) + return conn, nil +} + +// sendResponseCreate frames body (the same JSON the HTTP path POSTs, +// {"model":..., "input":..., "stream":true, ...}) as a Codex websocket +// request: {"type":"response.create", ...body-minus-stream}. "stream" is +// dropped because it is meaningless once the transport itself IS the +// streaming channel — ported from opencode's ws.ts streamResponsesWebSocket +// (the payload destructure that drops "stream"/"background"). +func sendResponseCreate(ctx context.Context, conn *websocket.Conn, body []byte) error { + var fields map[string]json.RawMessage + if err := json.Unmarshal(body, &fields); err != nil { + return fmt.Errorf("openai: websocket request.create: decoding request body: %w", err) + } + delete(fields, "stream") + fields["type"] = json.RawMessage(`"response.create"`) + payload, err := json.Marshal(fields) + if err != nil { + return fmt.Errorf("openai: websocket response.create: encoding request: %w", err) + } + if err := conn.Write(ctx, websocket.MessageText, payload); err != nil { + return fmt.Errorf("openai: websocket write response.create: %w", err) + } + return nil +} + +// wsFrameEnvelope reads only the "type" discriminator out of one websocket +// frame — every other field is left as raw JSON for stream.handle to decode +// itself, exactly as it already does for an SSE "data:" payload. +type wsFrameEnvelope struct { + Type string `json:"type"` +} + +// readResponsesFrame reads one text frame from conn and returns its "type" +// field (the ws-frame analog of an SSE "event:" line — see stream.handle, +// which this transport reuses unmodified) alongside the raw frame bytes. A +// binary frame is a protocol violation the Codex backend never sends in +// practice (ported from opencode's ws.ts onMessage, which invalidates the +// connection on isBinary). +func readResponsesFrame(ctx context.Context, conn *websocket.Conn) (name string, data []byte, err error) { + typ, data, err := conn.Read(ctx) + if err != nil { + return "", nil, err + } + if typ == websocket.MessageBinary { + return "", nil, errors.New("openai: unexpected binary websocket frame") + } + var env wsFrameEnvelope + if err := json.Unmarshal(data, &env); err != nil { + return "", nil, fmt.Errorf("openai: decoding websocket frame: %w", err) + } + return env.Type, data, nil +} + +// wsTerminalEventTypes are the Responses websocket event "type" values that +// end a response.create's stream — mirrors the case list in stream.handle +// (response.completed/response.incomplete/response.failed/error) plus +// opencode's defensive extra "response.done", which harness's SSE path has +// never seen from the real API but ws.ts treats as terminal too. +func isWSTerminalEvent(name string) bool { + switch name { + case "response.completed", "response.done", "response.incomplete", "response.failed", "error": + return true + default: + return false + } +} + +// isWSCleanTerminalEvent reports whether name is the terminal event kind +// that leaves the underlying connection reusable for the pool's next turn. +// Every other terminal (a model-level failure, not a transport failure) +// still ends the stream normally but the pool drops the socket rather than +// risk replaying against server-side state left by an abnormal end — ported +// from opencode's ws-pool.ts onTerminal. +func isWSCleanTerminalEvent(name string) bool { + return name == "response.completed" || name == "response.done" +} diff --git a/provider/openai/ws_pool.go b/provider/openai/ws_pool.go new file mode 100644 index 00000000..dfa89361 --- /dev/null +++ b/provider/openai/ws_pool.go @@ -0,0 +1,264 @@ +package openai + +import ( + "context" + "errors" + "net/http" + "sync" + "time" + + "github.com/coder/websocket" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// Default pool tunables, ported from opencode's ws-pool.ts +// (DEFAULT_CONNECT_TIMEOUT/DEFAULT_IDLE_TIMEOUT/DEFAULT_MAX_CONNECTION_AGE) +// and its streamRetries default. +const ( + wsDefaultConnectTimeout = 15 * time.Second + wsDefaultIdleTimeout = 5 * time.Minute + wsDefaultMaxConnectionAge = 55 * time.Minute + wsDefaultStreamRetries = 5 +) + +// errStreamClosedEarly marks a websocket stream torn down by its own +// caller (Close called before a terminal event) rather than by a wire-level +// failure. It is never returned to a caller of Client.Stream; it only +// drives wsPool's failure bookkeeping (see wsFrameSource.close). +var errStreamClosedEarly = errors.New("openai: websocket stream closed before a terminal event") + +// wsPoolEntry is one pooled session's websocket state — the Go analog of +// opencode's ws-pool.ts PoolEntry. This session's next request reuses conn +// as long as it is still open, younger than maxConnectionAge, and the +// session has not been marked fallback/busy. +type wsPoolEntry struct { + mu sync.Mutex + conn *websocket.Conn + connectedAt time.Time + lastUsedAt time.Time + busy bool + fallback bool // permanent: this session never uses ws again + streamFailures int +} + +// wsPool is a per-Client pool of persistent Codex Responses websocket +// connections, one per harness session (keyed by provider.Request. +// SessionKey), reused across turns. It is the Go port of opencode's +// ws-pool.ts createWebSocketFetch, adapted to provider.Provider's +// Stream/Next shape instead of a fetch() interception. +// +// Every failure mode falls back to the caller using HTTP for that request: +// wsPool.stream's second return value is false whenever the caller must not +// use the (nil) stream it returned — including "did not even attempt ws". +// Nothing here can make a request WORSE than the pre-existing HTTP path. +type wsPool struct { + connectTimeout time.Duration + idleTimeout time.Duration + maxConnectionAge time.Duration + streamRetries int + + // dial is overridden by tests that point it at an httptest server + // instead of the real chatgpt.com endpoint. Production always uses + // dialResponsesWebSocket via newWSPool's default assignment. + dial func(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, error) + + mu sync.Mutex + entries map[string]*wsPoolEntry +} + +func newWSPool() *wsPool { + return &wsPool{ + connectTimeout: wsDefaultConnectTimeout, + idleTimeout: wsDefaultIdleTimeout, + maxConnectionAge: wsDefaultMaxConnectionAge, + streamRetries: wsDefaultStreamRetries, + dial: dialResponsesWebSocket, + entries: make(map[string]*wsPoolEntry), + } +} + +// entryFor returns the pool entry for sessionKey, creating one on first +// use. The key mirrors opencode's `${sessionID}:conversation` — the suffix +// exists there to leave room for a second, differently-scoped key on the +// same session (e.g. title generation); harness has no such second use of +// this adapter yet, but the suffix costs nothing and keeps the port +// literal. +func (p *wsPool) entryFor(sessionKey string) *wsPoolEntry { + key := sessionKey + ":conversation" + p.mu.Lock() + defer p.mu.Unlock() + e, ok := p.entries[key] + if !ok { + e = &wsPoolEntry{} + p.entries[key] = e + } + return e +} + +// wsStreamRequest is the input wsPool.stream needs beyond the pool's own +// tunables — everything Client.Stream already computed for the HTTP path, +// so the ws path sends the byte-identical wire request. +type wsStreamRequest struct { + SessionKey string + URL string + Headers http.Header + Body []byte // the marshaled apiRequest, same bytes the HTTP path POSTs + Model message.ModelRef + Family string + HTTPClient *http.Client +} + +// stream attempts to serve req over this pool's session-affine websocket, +// returning (nil, false) whenever the caller must fall back to HTTP instead +// — a busy or permanently-fallen-back session, a dial failure, a send +// failure, or a failure to read even the first response frame. A non-nil +// stream is only ever returned once at least one real response frame has +// been read successfully, mirroring opencode's onFirstEvent gate: a socket +// that dies before producing anything must never be handed to the engine as +// if it were a working stream, since the engine has no transport-level +// retry of its own to fall back to HTTP with (Next() failing mid-response +// is reported as a truncated stream, not silently retried elsewhere — see +// provider.Stream's doc comment). +func (p *wsPool) stream(ctx context.Context, req wsStreamRequest) (provider.Stream, bool) { + entry := p.entryFor(req.SessionKey) + + entry.mu.Lock() + if entry.fallback || entry.busy { + entry.mu.Unlock() + return nil, false + } + entry.busy = true + now := time.Now() + reuse := entry.conn != nil && + !entry.connectedAt.IsZero() && + now.Sub(entry.connectedAt) < p.maxConnectionAge && + now.Sub(entry.lastUsedAt) < p.idleTimeout + entry.lastUsedAt = now + conn := entry.conn + entry.mu.Unlock() + + if !reuse { + p.invalidate(entry) + newConn, err := p.dial(ctx, req.URL, req.Headers, req.HTTPClient, p.connectTimeout) + if err != nil { + p.recordFailure(entry) + p.release(entry) + return nil, false + } + entry.mu.Lock() + entry.conn = newConn + entry.connectedAt = time.Now() + entry.mu.Unlock() + conn = newConn + } + + if err := sendResponseCreate(ctx, conn, req.Body); err != nil { + p.handleTransportError(entry, err) + p.release(entry) + return nil, false + } + + firstName, firstData, err := readFirstFrame(ctx, conn, p.idleTimeout) + if err != nil { + p.handleTransportError(entry, err) + p.release(entry) + return nil, false + } + + src := &wsFrameSource{ + conn: conn, + idleTimeout: p.idleTimeout, + buffered: &wsFrame{name: firstName, data: firstData}, + onTerminal: func(name string) { + entry.mu.Lock() + entry.busy = false + entry.lastUsedAt = time.Now() + entry.streamFailures = 0 + keep := isWSCleanTerminalEvent(name) + entry.mu.Unlock() + if !keep { + p.invalidate(entry) + } + }, + onBroken: func(err error) { + p.release(entry) + if errors.Is(err, errStreamClosedEarly) { + p.invalidate(entry) + return + } + p.handleTransportError(entry, err) + }, + } + + return &stream{ + wsConn: src, + model: req.Model, + family: req.Family, + }, true +} + +func readFirstFrame(ctx context.Context, conn *websocket.Conn, idleTimeout time.Duration) (string, []byte, error) { + if idleTimeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, idleTimeout) + defer cancel() + } + return readResponsesFrame(ctx, conn) +} + +// handleTransportError applies a real connection-level failure to entry: +// permanent fallback for a MESSAGE_TOO_BIG close (ported from opencode's +// ws-pool.ts onConnectionInvalid), otherwise a counted failure toward +// streamRetries. Either way the (now-suspect) connection is dropped. +func (p *wsPool) handleTransportError(entry *wsPoolEntry, err error) { + if websocket.CloseStatus(err) == wsMessageTooBigCode { + entry.mu.Lock() + entry.fallback = true + entry.mu.Unlock() + p.invalidate(entry) + return + } + p.recordFailure(entry) +} + +// recordFailure counts a connection failure toward this entry's +// permanent-fallback threshold and drops its (now-suspect) connection. +// Mirrors opencode's ws-pool.ts recordStreamFailure: Codex counts retries +// AFTER the initial failed attempt, so streamRetries+1 total attempts are +// allowed before an entry gives up on ws for the rest of the session. +func (p *wsPool) recordFailure(entry *wsPoolEntry) { + entry.mu.Lock() + entry.streamFailures++ + if entry.streamFailures > p.streamRetries { + entry.fallback = true + } + entry.mu.Unlock() + p.invalidate(entry) +} + +// release clears the busy flag a failed attempt set, without touching +// fallback/streamFailures (recordFailure/handleTransportError own those) — +// called on every exit path that must not leave an entry permanently +// marked busy. +func (p *wsPool) release(entry *wsPoolEntry) { + entry.mu.Lock() + entry.busy = false + entry.lastUsedAt = time.Now() + entry.mu.Unlock() +} + +// invalidate closes and clears entry's connection, if any, so the next +// stream() call for this session dials fresh. Closing is best-effort: the +// connection is already known bad, terminal, or about to be discarded +// either way. +func (p *wsPool) invalidate(entry *wsPoolEntry) { + entry.mu.Lock() + conn := entry.conn + entry.conn = nil + entry.connectedAt = time.Time{} + entry.mu.Unlock() + if conn != nil { + _ = conn.Close(websocket.StatusNormalClosure, "") + } +} diff --git a/provider/openai/ws_stream.go b/provider/openai/ws_stream.go new file mode 100644 index 00000000..6895eee5 --- /dev/null +++ b/provider/openai/ws_stream.go @@ -0,0 +1,101 @@ +package openai + +import ( + "context" + "time" + + "github.com/coder/websocket" +) + +// wsFrame is one already-read websocket frame, buffered so the pool can +// look at the FIRST event before handing a stream to the caller (see +// wsPool.stream) without that event being lost once it does. +type wsFrame struct { + name string + data []byte +} + +// wsFrameSource adapts a pooled *websocket.Conn to stream's frame-source +// contract (see stream.readEvent/Close in openai.go), so stream.handle — +// the Responses event-to-provider.Event mapper — runs UNCHANGED for a +// websocket-delivered response exactly as it does for an SSE-delivered one. +// Only how a (name, data) pair is obtained differs. +type wsFrameSource struct { + conn *websocket.Conn + idleTimeout time.Duration + buffered *wsFrame + + // onTerminal fires exactly once, the first time a terminal event type + // is observed (from the buffered first frame or a later read) — never + // from Close, so a stream that is Closed after reaching Next() io.EOF + // does not double-report. name is the event's wire type. + onTerminal func(name string) + // onBroken fires when the connection dies before a terminal event is + // observed: a read error, or Close() called while the stream is still + // mid-flight (context canceled, engine gave up on the turn). It never + // fires after onTerminal has already fired for this source. + onBroken func(err error) + + terminal bool +} + +// next returns the next (name, data) pair, buffered first-frame included. +// It satisfies the same shape stream.readSSE does. +func (w *wsFrameSource) next() (string, []byte, error) { + if w.buffered != nil { + f := w.buffered + w.buffered = nil + w.observe(f.name, nil) + return f.name, f.data, nil + } + ctx := context.Background() + if w.idleTimeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, w.idleTimeout) + defer cancel() + } + name, data, err := readResponsesFrame(ctx, w.conn) + if err != nil { + w.observe("", err) + return "", nil, err + } + w.observe(name, nil) + return name, data, nil +} + +// observe records a successfully read event's terminality, or a read +// failure — each reported to the pool at most once per source. +func (w *wsFrameSource) observe(name string, err error) { + if w.terminal { + return + } + switch { + case err != nil: + w.terminal = true + if w.onBroken != nil { + w.onBroken(err) + } + case isWSTerminalEvent(name): + w.terminal = true + if w.onTerminal != nil { + w.onTerminal(name) + } + } +} + +// close implements stream.Close for a websocket-backed stream. clean is +// true when the stream's own decode loop already reached its terminal +// event (stream.done) — in that case onTerminal has already told the pool +// whether to keep or drop the connection, and close must not additionally +// terminate a connection the pool chose to keep pooled. clean is false for +// every other reason Close is called (context canceled mid-turn, a decode +// error stream.handle returned, the caller simply giving up) — the +// connection cannot be trusted for reuse, so it is torn down and reported +// exactly like a read failure. +func (w *wsFrameSource) close(clean bool) error { + if clean { + return nil + } + w.observe("", errStreamClosedEarly) + return w.conn.Close(websocket.StatusNormalClosure, "") +} diff --git a/provider/openai/ws_test.go b/provider/openai/ws_test.go new file mode 100644 index 00000000..e6afbb7a --- /dev/null +++ b/provider/openai/ws_test.go @@ -0,0 +1,490 @@ +package openai + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" + + "github.com/coder/websocket" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// canned ws response.* frames for one complete, tool-free turn — the ws +// analog of streamFixture in stream_test.go, minus the SSE "event:" +// envelope (a ws frame carries only the JSON body; its own "type" field is +// the event name — see readResponsesFrame). +var wsCannedFrames = []string{ + `{"type":"response.created","response":{"id":"resp_ws_1"}}`, + `{"type":"response.output_text.delta","output_index":0,"delta":"hi"}`, + `{"type":"response.output_item.done","output_index":0,"item":{"id":"msg_ws_1","type":"message","role":"assistant","content":[{"type":"output_text","text":"hi"}]}}`, + `{"type":"response.completed","response":{"id":"resp_ws_1","usage":{"input_tokens":5,"output_tokens":2}}}`, +} + +// wsFrameEventName extracts a canned frame's "type" field so tests can +// serve the identical fixture over both transports: unwrapped over ws (see +// readResponsesFrame), and under an SSE "event:" line for the HTTP+SSE +// fallback path (see stream.readSSE), which key on it differently. +func wsFrameEventName(frame string) string { + var env wsFrameEnvelope + json.Unmarshal([]byte(frame), &env) //nolint:errcheck + return env.Type +} + +// wsTestServer is an httptest server that speaks BOTH halves of this +// adapter's wire: a normal HTTP+SSE Responses POST, and (on an Upgrade +// request) the Codex response.create websocket protocol, replaying +// wsCannedFrames for every response.create it reads on a connection — +// enough to prove pool reuse sends more than one turn over the same +// accepted connection. +type wsTestServer struct { + *httptest.Server + upgrades atomic.Int32 + lastAuth atomic.Value // string + rejectWS atomic.Bool + closeAfter atomic.Bool // close the connection after one response instead of looping +} + +func newWSTestServer(t *testing.T) *wsTestServer { + t.Helper() + ts := &wsTestServer{} + ts.lastAuth.Store("") + ts.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + ts.lastAuth.Store(r.Header.Get("Authorization")) + if r.Header.Get("Upgrade") == "" { + w.Header().Set("Content-Type", "text/event-stream") + for _, f := range wsCannedFrames { + io.WriteString(w, sse(wsFrameEventName(f), f)) //nolint:errcheck + } + return + } + if ts.rejectWS.Load() { + http.Error(w, "websocket disabled", http.StatusForbidden) + return + } + ts.upgrades.Add(1) + // InsecureSkipVerify here disables coder/websocket's Origin-header + // check for this loopback httptest server (plain HTTP, no TLS + // involved) — it is not a TLS setting and does not weaken + // certificate verification anywhere in this transport. + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.Close(websocket.StatusInternalError, "") + for { + ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) + _, _, err := conn.Read(ctx) + cancel() + if err != nil { + return + } + for _, f := range wsCannedFrames { + if err := conn.Write(context.Background(), websocket.MessageText, []byte(f)); err != nil { + return + } + } + if ts.closeAfter.Load() { + conn.Close(websocket.StatusNormalClosure, "") + return + } + } + })) + t.Cleanup(ts.Close) + return ts +} + +func wsRequest(sessionKey string) *provider.Request { + return &provider.Request{ + Model: message.ModelRef{Provider: Family, Model: "gpt-5"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + MaxTokens: 10, + SessionKey: sessionKey, + } +} + +// TestWebSocketTransportConnectsAndStreams: the happy path end to end — +// UseWebSocketTransport + a SessionKey routes Client.Stream over the ws +// server, sends response.create, and the resulting provider.Stream carries +// the same assembled message/usage the HTTP+SSE path would for identical +// wire events (stream.handle is shared code — see openai.go's readEvent). +func TestWebSocketTransportConnectsAndStreams(t *testing.T) { + ts := newWSTestServer(t) + c := &Client{APIKey: "test-key", BaseURL: ts.URL, UseWebSocketTransport: true} + + s, err := c.Stream(context.Background(), wsRequest("sess-1")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + events := collect(t, s) + + if ts.upgrades.Load() != 1 { + t.Fatalf("upgrades = %d, want 1 (must have used the websocket path)", ts.upgrades.Load()) + } + if got := ts.lastAuth.Load().(string); got != "Bearer test-key" { + t.Errorf("Authorization sent over ws = %q, want Bearer test-key", got) + } + + var text string + var done *provider.Event + for i := range events { + if events[i].Type == provider.EventTextDelta { + text += events[i].Text + } + if events[i].Type == provider.EventDone { + done = &events[i] + } + } + if text != "hi" { + t.Errorf("text = %q, want hi", text) + } + if done == nil { + t.Fatal("no done event") + } + if done.StopReason != provider.StopEndTurn { + t.Errorf("stop reason = %s, want end_turn", done.StopReason) + } + if done.Usage.InputTokens != 5 || done.Usage.OutputTokens != 2 { + t.Errorf("usage = %+v", done.Usage) + } + if done.Message == nil || done.Message.ID != "resp_ws_1" { + t.Errorf("message = %+v", done.Message) + } +} + +// TestWebSocketTransportSendsResponseCreate proves the frame this transport +// puts on the wire is {"type":"response.create", ...request-minus-stream} +// — the exact framing ws.ts's streamResponsesWebSocket uses, and NOT the +// bare Responses request body the HTTP path POSTs. +func TestWebSocketTransportSendsResponseCreate(t *testing.T) { + var gotType string + var gotStreamPresent bool + done := make(chan struct{}) + + inspecting := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.Close(websocket.StatusNormalClosure, "") + _, data, err := conn.Read(r.Context()) + if err != nil { + close(done) + return + } + var fields map[string]json.RawMessage + json.Unmarshal(data, &fields) //nolint:errcheck + if v, ok := fields["type"]; ok { + json.Unmarshal(v, &gotType) //nolint:errcheck + } + _, gotStreamPresent = fields["stream"] + conn.Write(context.Background(), websocket.MessageText, []byte(wsCannedFrames[len(wsCannedFrames)-1])) //nolint:errcheck + close(done) + })) + defer inspecting.Close() + c := &Client{APIKey: "k", BaseURL: inspecting.URL, UseWebSocketTransport: true} + + s, err := c.Stream(context.Background(), wsRequest("sess-frame")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + collect(t, s) + <-done + + if gotType != "response.create" { + t.Errorf("frame type = %q, want response.create", gotType) + } + if gotStreamPresent { + t.Error("response.create frame still carries \"stream\" — must be stripped (meaningless once ws IS the stream)") + } +} + +// TestWebSocketTransportFallbackOnNoSessionKey: UseWebSocketTransport is on +// but the request carries no SessionKey — there is no pool key, so the +// call must go straight to HTTP without even attempting a dial. +func TestWebSocketTransportFallbackOnNoSessionKey(t *testing.T) { + ts := newWSTestServer(t) + c := &Client{APIKey: "k", BaseURL: ts.URL, UseWebSocketTransport: true} + + s, err := c.Stream(context.Background(), wsRequest("")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + collect(t, s) + + if ts.upgrades.Load() != 0 { + t.Errorf("upgrades = %d, want 0 (must not attempt ws with no session key)", ts.upgrades.Load()) + } +} + +// TestWebSocketTransportFallbackOnDialFailure: a server that refuses the +// upgrade handshake (a proxy/box without ws support, or the backend +// rejecting it) must still let the turn complete over HTTP — the whole +// point of the fallback design. +func TestWebSocketTransportFallbackOnDialFailure(t *testing.T) { + ts := newWSTestServer(t) + ts.rejectWS.Store(true) + c := &Client{APIKey: "k", BaseURL: ts.URL, UseWebSocketTransport: true} + + s, err := c.Stream(context.Background(), wsRequest("sess-2")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + events := collect(t, s) + + if ts.upgrades.Load() != 0 { + t.Errorf("upgrades = %d, want 0", ts.upgrades.Load()) + } + var sawDone bool + for _, ev := range events { + if ev.Type == provider.EventDone { + sawDone = true + } + } + if !sawDone { + t.Error("no done event: HTTP fallback did not complete the turn") + } +} + +// TestWebSocketTransportDisabledUsesHTTP: the default (false) must never +// attempt a dial at all, byte-identical to this adapter's pre-existing +// behavior. +func TestWebSocketTransportDisabledUsesHTTP(t *testing.T) { + ts := newWSTestServer(t) + c := &Client{APIKey: "k", BaseURL: ts.URL} // UseWebSocketTransport left false + + s, err := c.Stream(context.Background(), wsRequest("sess-3")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + collect(t, s) + + if ts.upgrades.Load() != 0 { + t.Errorf("upgrades = %d, want 0 (transport is off)", ts.upgrades.Load()) + } +} + +// TestWebSocketTransportPoolReusesConnection: two turns on the same +// session must share one accepted websocket connection, not dial twice. +func TestWebSocketTransportPoolReusesConnection(t *testing.T) { + ts := newWSTestServer(t) + c := &Client{APIKey: "k", BaseURL: ts.URL, UseWebSocketTransport: true} + + for i := 0; i < 2; i++ { + s, err := c.Stream(context.Background(), wsRequest("sess-reuse")) + if err != nil { + t.Fatalf("Stream #%d: %v", i, err) + } + collect(t, s) + s.Close() + } + + if got := ts.upgrades.Load(); got != 1 { + t.Errorf("upgrades = %d, want 1 (second turn should reuse the pooled connection)", got) + } +} + +// TestWebSocketTransportPoolDropsConnectionAfterFailedResponse: a terminal +// event other than response.completed (here, response.failed) must not +// leave its connection pooled for reuse — ported from opencode's +// ws-pool.ts onTerminal, which invalidates on anything but completed/done. +func TestWebSocketTransportPoolDropsConnectionAfterFailedResponse(t *testing.T) { + failThenSucceed := []string{ + `{"type":"response.failed","response":{"error":{"code":"server_error","message":"boom"}}}`, + } + var upgrades atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + upgrades.Add(1) + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.Close(websocket.StatusNormalClosure, "") + if _, _, err := conn.Read(r.Context()); err != nil { + return + } + for _, f := range failThenSucceed { + conn.Write(context.Background(), websocket.MessageText, []byte(f)) //nolint:errcheck + } + })) + defer srv.Close() + c := &Client{APIKey: "k", BaseURL: srv.URL, UseWebSocketTransport: true} + + for i := 0; i < 2; i++ { + s, err := c.Stream(context.Background(), wsRequest("sess-drop")) + if err != nil { + t.Fatalf("Stream #%d: %v", i, err) + } + for { + _, err := s.Next() + if err != nil { + break // response.failed surfaces as a stream error, not io.EOF + } + } + s.Close() + } + + if got := upgrades.Load(); got != 2 { + t.Errorf("upgrades = %d, want 2 (a failed response must not be pooled for reuse)", got) + } +} + +// TestWebSocketTransportBusyFallsBackToHTTP: a second concurrent call on a +// session whose socket is already mid-turn must use HTTP rather than +// contend for the same connection. +func TestWebSocketTransportBusyFallsBackToHTTP(t *testing.T) { + release := make(chan struct{}) + var upgrades atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Upgrade") == "" { + w.Header().Set("Content-Type", "text/event-stream") + for _, f := range wsCannedFrames { + io.WriteString(w, sse(wsFrameEventName(f), f)) //nolint:errcheck + } + return + } + upgrades.Add(1) + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.Close(websocket.StatusNormalClosure, "") + if _, _, err := conn.Read(r.Context()); err != nil { + return + } + <-release // hold the connection "busy" until the test releases it + for _, f := range wsCannedFrames { + conn.Write(context.Background(), websocket.MessageText, []byte(f)) //nolint:errcheck + } + })) + defer srv.Close() + c := &Client{APIKey: "k", BaseURL: srv.URL, UseWebSocketTransport: true} + + firstDone := make(chan struct{}) + go func() { + defer close(firstDone) + s, err := c.Stream(context.Background(), wsRequest("sess-busy")) + if err != nil { + return + } + defer s.Close() + collect(t, s) + }() + + // Give the first call time to reach the pool's busy state before firing + // the second one — bounded by the "polling for a condition" pattern + // this codebase's own retry/backoff tests use rather than a raw sleep + // standing in for synchronization. + deadline := time.After(2 * time.Second) + for upgrades.Load() == 0 { + select { + case <-deadline: + t.Fatal("first call never reached the ws server") + case <-time.After(time.Millisecond): + } + } + + s, err := c.Stream(context.Background(), wsRequest("sess-busy")) + if err != nil { + t.Fatalf("second Stream: %v", err) + } + collect(t, s) + s.Close() + close(release) + <-firstDone + + if got := upgrades.Load(); got != 1 { + t.Errorf("upgrades = %d, want 1 (the busy session's second call must use HTTP)", got) + } +} + +// TestWebSocketTransportMessageTooBigPermanentFallback: a 1009 close marks +// the session permanently HTTP-only, not just for the failed attempt — +// ported from opencode's ws-pool.ts fallback-on-MESSAGE_TOO_BIG. +func TestWebSocketTransportMessageTooBigPermanentFallback(t *testing.T) { + var upgrades atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Upgrade") == "" { + w.Header().Set("Content-Type", "text/event-stream") + for _, f := range wsCannedFrames { + io.WriteString(w, sse(wsFrameEventName(f), f)) //nolint:errcheck + } + return + } + upgrades.Add(1) + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + if _, _, err := conn.Read(r.Context()); err != nil { + return + } + conn.Close(websocket.StatusMessageTooBig, "message too big") + })) + defer srv.Close() + c := &Client{APIKey: "k", BaseURL: srv.URL, UseWebSocketTransport: true} + + for i := 0; i < 2; i++ { + s, err := c.Stream(context.Background(), wsRequest("sess-too-big")) + if err != nil { + t.Fatalf("Stream #%d: %v", i, err) + } + collect(t, s) + s.Close() + } + + if got := upgrades.Load(); got != 1 { + t.Errorf("upgrades = %d, want 1 (MESSAGE_TOO_BIG must permanently fall back to HTTP for this session)", got) + } +} + +// TestToWebSocketURL checks the http(s)->ws(s) rewrite ws.go's dial uses. +func TestToWebSocketURL(t *testing.T) { + cases := map[string]string{ + "https://chatgpt.com/backend-api/codex/responses": "wss://chatgpt.com/backend-api/codex/responses", + "http://localhost:1234/v1/responses": "ws://localhost:1234/v1/responses", + } + for in, want := range cases { + if got := toWebSocketURL(in); got != want { + t.Errorf("toWebSocketURL(%q) = %q, want %q", in, got, want) + } + } +} + +// TestIsWSTerminalEvent/TestIsWSCleanTerminalEvent lock in the event-name +// classification wsPool's onTerminal/onBroken wiring depends on. +func TestIsWSTerminalEvent(t *testing.T) { + for _, name := range []string{"response.completed", "response.done", "response.incomplete", "response.failed", "error"} { + if !isWSTerminalEvent(name) { + t.Errorf("isWSTerminalEvent(%q) = false, want true", name) + } + } + for _, name := range []string{"response.created", "response.output_text.delta", ""} { + if isWSTerminalEvent(name) { + t.Errorf("isWSTerminalEvent(%q) = true, want false", name) + } + } +} + +func TestIsWSCleanTerminalEvent(t *testing.T) { + for _, name := range []string{"response.completed", "response.done"} { + if !isWSCleanTerminalEvent(name) { + t.Errorf("isWSCleanTerminalEvent(%q) = false, want true", name) + } + } + for _, name := range []string{"response.incomplete", "response.failed", "error"} { + if isWSCleanTerminalEvent(name) { + t.Errorf("isWSCleanTerminalEvent(%q) = true, want false", name) + } + } +} From 1c907edca48e16fea6532d4fa6e75a0268fd2a0e Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Sun, 30 Aug 2026 22:58:11 -0400 Subject: [PATCH 28/95] engine: forward MCP, effort, reasoning, subagent nesting, and metrics in the Claude Code backend (#217) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * message,engine: carry a delegated subagent's parent_tool_use_id onto Message The Claude Code CLI's stream-json events name parent_tool_use_id: null at the top level of a delegated turn, and the spawning tool_use id inside a subagent's own turn. engine/claude_code_backend.go dropped this field entirely, so every message a subagent produced looked identical to a top-level one once appended to session history, flattening the subagent hierarchy for any journal or console reader. Add message.Message.ParentToolUseID, presentation/lineage metadata in the same spirit as the existing Origin field: never sent to a model, never interpreted by a transcoder, read only by a client reconstructing a delegated turn's subagent nesting. Surface it on the read-only GET /session/{id}/journal projection too (JournalRecord.ParentToolUseID in engine/journal.go), alongside the existing MessageID/MessageRole identity fields, so a console can read a message's lineage without fetching its full content. The live/SSE journal (server/journal.go) needs no change: its Event already embeds the full *message.Message, so the new field rides through automatically once marshaled. Verified with go build ./... and go test ./message/... ./engine/... (this commit's own message/journal.go slice builds and tests green standalone, verified via git stash --keep-index before committing); the CLI-side wiring that actually populates the field lands in a following commit. * engine: expose MCPManager.Servers for the Claude Code delegated backend The delegated-turn backend (claude_code_backend.go) needs a session's RAW configured MCP server definitions (command, args, env, url, headers) to translate them into the `claude` CLI's own --mcp-config JSON shape. MCPManager keeps its per-server config unexported, and the existing MCPRegistry interface (Tools/CallTool/CallServerTool) only ever hands back post-connect tool schemas, never a server's own definition. Add MCPManager.Servers, mirroring ConfiguredNames' existing "m.servers is immutable after NewMCPManager, no lock needed" reasoning, returning a shallow copy so a caller can never mutate the manager's own configuration through the result. This is deliberately NOT added to the MCPRegistry interface itself: doing so would force every existing MCPRegistry fake across this package, cmd/harness, and server to grow a new method just to keep compiling, for a capability only the delegated backend needs. The next commit consumes it through a narrower, single-purpose interface instead. Verified with go build ./... and go test ./engine/... -run TestMCPManager (this commit's own slice builds and tests green standalone, verified via git stash --keep-index before committing). * engine: forward MCP, effort, reasoning, subagent nesting, and metrics in the Claude Code backend engine/claude_code_backend.go's own package doc flagged five gaps as "Deferred for v1": no --mcp-config passthrough, no --effort forwarding, a silently-dropped "thinking" content block, a dropped parent_tool_use_id, and no OnTurnMetrics for a delegated turn. A sixth, related gap sat next to the goal-loop note: a delegated turn's error was always a plain error, so goal.go's promptTurnWithRetry treated every failure as a deterministic stall, never provider weather worth a backoff-and-retry. This closes all six in the one file that owns them, since every one of them lives inside runClaudeCodeTurn or consumeClaudeCodeStream and touches the same envelope/content-block decoding this file already does. MCP passthrough: runClaudeCodeTurn now builds a --mcp-config JSON file from the session's configured MCP servers (via the new claudeCodeMCPServerLister seam onto MCPManager.Servers) and passes --strict-mcp-config alongside it, so the child sees exactly harness's own configured servers and nothing a project-local .mcp.json might otherwise add. The config rides a temp file, not an inline argv value, because a server's Headers/Env can carry real credential material that argv would expose via /proc or ps; the file is removed once the child has started. A session with no configured MCP servers gets no file and no flags at all — passthrough is opt-in. --effort forwarding: runClaudeCodeTurn reads s.Effort() at turn time, exactly like it already reads s.Model(), and appends --effort through claudeCodeEffortArg's mapping (off/minimal/low -> low, medium -> medium, high -> high; EffortUnset sends no flag). No new config field: this reads session state directly, the same way the native path's provider.Request.Effort does. Thinking-block decode: claudeCodeContentBlock now carries a "thinking" block's Thinking/Signature fields, and claudeCodeAssistantMessage maps one to a message.Reasoning part (Signature carried into ProviderData under the "anthropic" family key, mirroring provider/anthropic's own thinking-block shape) instead of silently dropping it. Each one is also emitted as EventReasoningDelta, pairing with --effort forwarding: a delegated turn that thinks harder now has something to show for it. parent_tool_use_id: the envelope's own parent_tool_use_id (null at top level, the spawning tool_use id inside a subagent's own turn) now rides onto every appended message.Message via the new ParentToolUseID field (prior commit), for both "assistant" and "user" (tool_result) events. turn_metrics: the "result" event's own ttft_ms/duration_ms (zero, never an error, on a CLI build that omits them) feed an OnTurnMetrics call built from the same usage applyClaudeCodeUsage already maps, mirroring the native streamTurn's own EventDone emission as closely as a delegated turn's available data allows. Attempt is always 1: Claude Code retries internally, invisible to harness. Goal-loop retry classification: an is_error "result" event is now run through claudeCodeRetryableClass, which wraps provider.RetryableError for a signal this file can actually name as transient (a rate-limit or overload mention, or the CLI's own "error_during_execution" catch-all) and leaves a genuinely deterministic outcome (max turns reached, a refusal) as a plain error. Separately, a child that exits nonzero WITHOUT ever emitting a clean result event at all (a crash, an OOM kill) is also wrapped RetryableError(RetryableServerError): that shape is non-deterministic process weather, not a domain-level failure the CLI reported, the same reasoning MarkStreamTruncated already applies to a native adapter's stream that dies mid-body. goal.go needed no change: provider.AsRetryable's errors.As walk already finds the wrapped error through runAgenticLoop's untouched pass-through. engine/testdata/fakeclaude/main.go gains five narrow modes ("thinking", "subagent", "rate_limit_error", "deterministic_error", "crash"), one per gap that needs a canned event shape "normal"/"error" don't already cover; "normal" itself gains ttft_ms/duration_ms so the existing event-mapping test's fixture also exercises turn_metrics. Verification: go build ./..., go vet ./..., and gofmt -l . are clean on every touched file. go test ./engine/... ./config/... ./message/... passes (one pre-existing, unrelated flake, TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhenLogCannotReconstruct, reproduces identically with this branch's changes stashed out, so it predates this change). go test ./engine/... -race -count=1 -skip TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhenLogCannotReconstruct is green. New tests: TestClaudeCodeMCPConfigFileWritesConfiguredServers, TestClaudeCodeMCPConfigFileEmptyWithNoServers, TestClaudeCodeMCPConfigForwardedToChildArgv, TestClaudeCodeEffortForwardedToChildArgv, TestClaudeCodeEffortUnsetOmitsFlag, TestClaudeCodeThinkingBlockDecodesToReasoningPart, TestClaudeCodeParentToolUseIDCarriedOntoMessage, TestClaudeCodeTurnMetricsEmittedForDelegatedTurn, TestClaudeCodeRetryableClassification, TestClaudeCodeChildCrashWithoutResultIsRetryable. * engine: fail fast on a delegated child that never started Review of PR #217 flagged that the child-exit branch in runClaudeCodeTurn marked ANY nonzero exit without a clean "result" event provider.RetryableError. That is right for a genuine mid-session crash or OOM kill, but it also caught a DETERMINISTIC startup failure — an unknown flag on an older `claude` build, a malformed --mcp-config command, an invalid --model value — which also exits nonzero with no result event. In a PursueGoal loop, marking those retryable meant burning the entire retryable budget (12 attempts, with backoff) before parking, delaying the surfacing of a config error that no amount of waiting will ever fix. consumeClaudeCodeStream now returns a third value, started, set true the first time it sees ANY "system" event (init is documented as the CLI's first event, so seeing one at all means the child's stream-json protocol actually came up). runClaudeCodeTurn's waitErr branch only wraps the exit error provider.RetryableError when started is true; otherwise it returns the plain error, exactly like every other deterministic failure this file returns. engine/testdata/fakeclaude gains "crash_before_init", which exits nonzero before emitting even a "system" event, alongside a new test (TestClaudeCodeChildExitBeforeAnySystemEventIsNotRetryable) proving that shape is NOT retryable, next to the existing TestClaudeCodeChildCrashWithoutResultIsRetryable (unchanged behavior: a child that starts normally and dies later still classifies retryable). Also from the same review round: - MCPManager.Servers' doc comment claimed a deep copy; the implementation is a shallow one (the nested Command/Env slices and Headers map are shared with m.servers). Fixed the comment to say so and to spell out why that is still safe today (its one caller only ever reads the returned config to build a --mcp-config JSON payload). - claudeCodeEffortArg's doc comment cited a nonexistent message.Effort.Valid; fixed it to name the real validator, message.ParseEffort. - Added TestClaudeCodeMCPConfigCredentialsNeverInChildArgv, driving one real turn with a bearer-token header and a secret-bearing env entry configured, asserting the secret never appears in any element of the child's own argv — the regression lock for why --mcp-config rides a temp file rather than an inline value. Verification: go build ./..., go vet ./..., and gofmt -l . are clean. go test ./engine/... ./config/... ./message/... -race -count=1 (with -skip on the one pre-existing, unrelated flaky test already identified in PR #217, TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhenLogCannotReconstruct) is green. * engine: tolerate a broken-pipe turn-input write when the child already finished PR #217's CI (go test -race ./...) failed: --- FAIL: TestClaudeCodeTurnMetricsEmittedForDelegatedTurn write turn input: write |1: broken pipe The delegated `claude` child can legitimately finish its ENTIRE result and exit — closing its own end of the stdin pipe — before runClaudeCodeTurn finishes writing and closing its side, for a fast or trivial turn. Before this change, a write or close error on that pipe killed the child and returned the write error immediately, discarding whatever the child had already produced on stdout, even a perfectly complete and valid result. That is a real product robustness bug, not a test-only artifact: any real `claude` binary that answers a trivial prompt fast enough can hit the identical race, particularly under load (the CI failure needed -race's overhead on harness's own write path to surface reliably; a real subscription-backed binary is not instrumented and could theoretically still race the same way on a slow enough write or fast enough answer). The write and close no longer kill the child or return early on failure. Both are folded into a single inputErr, checked only once the child's stdout has actually been drained: if consumeClaudeCodeStream still produced a usable finalMsg, inputErr is dropped entirely — the child got everything it needed regardless of what happened on its stdin descriptor. Only when there is NO usable result at all does inputErr get promoted to the actual returned error, in place of the generic "turn ended with no assistant message" (a write failure with no result at all really does mean the child likely never received the turn's own prompt). The end-of-turn decision (caller abort, classified result error, process-exit error with its started/retryable branch, and now this input-write tolerance) is extracted into its own pure function, claudeCodeTurnResult, so the exact precedence between these is directly unit-testable (TestClaudeCodeTurnResult) without needing to force each interleaving out of a real child process — the underlying race this bug came from is fundamentally difficult to force deterministically (the child's own process startup latency dominates, so the parent's write reliably wins outside of -race's slowdown). TestClaudeCodeSucceedsDespiteInputWriteBrokenPipe is the best-effort integration-level companion, using a new fakeclaude "fast_no_drain" mode (closes its own stdin immediately, before doing anything else, and runs at native speed since it is compiled without -race) to lean the race as far as possible toward reproducing the exact failure shape while still passing either way, since the fix makes the outcome correct regardless of which side of the race actually happens. Verification: go build ./..., go vet ./..., and gofmt -l . are clean. go test ./engine/ -run TestClaudeCode -race -count=5 is green. go test ./engine/... ./config/... ./message/... -race -count=1 (skipping the one pre-existing, unrelated flaky test already identified in PR #217, TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhenLogCannotReconstruct) is green. --- engine/claude_code_backend.go | 623 ++++++++++++++++++++++++----- engine/claude_code_backend_test.go | 497 +++++++++++++++++++++++ engine/journal.go | 6 + engine/mcp.go | 23 ++ engine/testdata/fakeclaude/main.go | 201 +++++++++- message/message.go | 11 + 6 files changed, 1251 insertions(+), 110 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 34446f9c..c60086cb 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -55,60 +55,83 @@ // (claudeCodeCLISessionID) for --resume on this harness session's next // delegated turn. Any other subtype (e.g. "api_retry") is activity // only — observed, never fatal. -// - "assistant": one COMPLETE API-level assistant message (text and/or -// tool_use content blocks together) — NOT a token-by-token delta; this -// driver does not pass --include-partial-messages, so there is nothing -// more granular to forward. Decoded into one message.Message (Text and -// ToolCall parts, in order), appended via plain Session.append (no -// usage — see the usage-mapping note below) and emitted as -// EventMessage, with one EventTextDelta per non-empty text part -// (folding the whole block's text into the message in a single -// "delta", the closest honest match to the native EventTextDelta -// contract given the CLI hands over complete blocks) and one -// EventToolStart per tool_use part. +// - "assistant": one COMPLETE API-level assistant message (text, thinking, +// and/or tool_use content blocks together) — NOT a token-by-token +// delta; this driver does not pass --include-partial-messages, so +// there is nothing more granular to forward. Decoded into one +// message.Message (Reasoning, Text, and ToolCall parts, in order), +// appended via plain Session.append (no usage — see the usage-mapping +// note below) and emitted as EventMessage, with one EventReasoningDelta +// per non-empty thinking part, one EventTextDelta per non-empty text +// part (folding the whole block's text into the message in a single +// "delta", the closest honest match to the native +// EventTextDelta/EventReasoningDelta contract given the CLI hands over +// complete blocks), and one EventToolStart per tool_use part. The +// envelope's own parent_tool_use_id (null at top level, the spawning +// tool_use id inside a subagent's own turn) rides onto the appended +// message as Message.ParentToolUseID, unmodified. // - "user": Claude Code's own tool execution results, arriving in the // Anthropic API's own convention of a "user"-role message carrying // tool_result content blocks (Claude Code executes its OWN tools here // — this package never calls runToolCalls for a delegated turn). // Decoded into one RoleTool message.Message (one ToolResult part per -// block), appended and emitted as EventMessage, with one EventToolEnd -// per part. +// block, Message.ParentToolUseID set the same way as the "assistant" +// case above), appended and emitted as EventMessage, with one +// EventToolEnd per part. // - "result": the turn's terminal event. Never itself appended as a // message (the assistant text it summarizes was already appended by // the last "assistant" event above) — it instead carries the turn's // AGGREGATE usage, applied once via applyClaudeCodeUsage (a durable, -// message-independent record — see recClaudeCodeUsage in store.go). -// An IsError result becomes this call's returned error; TotalCostUSD -// has no home in provider.Usage (no adapter carries a cost field — -// every consumer derives cost from token counts) and is deliberately +// message-independent record — see recClaudeCodeUsage in store.go), +// and this turn's timing, emitted once via emitTurnMetrics (ttft_ms/ +// duration_ms, permissively zero-valued if the CLI's own build does not +// send them). An IsError result becomes this call's returned error — +// wrapped provider.RetryableError for a known-transient shape (see +// claudeCodeRetryableClass), a plain error otherwise. TotalCostUSD has +// no home in provider.Usage (no adapter carries a cost field — every +// consumer derives cost from token counts) and is deliberately // dropped, not persisted. // -// # Deferred for v1 (flagged, not silently skipped) +// # Formerly deferred, now closed // -// - MCP passthrough (`--mcp-config`): a delegated turn does not forward -// harness's configured MCP servers to the `claude` child. Wiring this -// correctly means translating engine/mcp.go's server specs into the -// CLI's own --mcp-config JSON shape and reconciling two independent -// permission/tool-approval models — real, separable follow-on work. -// - `--append-system-prompt`: not auto-populated from s.cfg.System. -// Harness's system-prompt assembly (project instructions, Agent -// Skills, tool-batching guidance) is deliberately native-loop-only -// (see PromptWithOrigin's dispatch comment) — Claude Code already -// discovers its own CLAUDE.md/AGENTS.md in the box workspace, and -// re-injecting harness's OWN native-tool-shaped instructions into a -// CLI that has different tools would be actively misleading. An -// operator who wants extra injected wording can still reach the flag -// via config.Provider.ExtraArgs. -// - Full goal-loop support: the directive-reuse retry path (see above) -// IS dispatched correctly, so a goal loop driving a delegated session -// does not error out — but goal.go's retryable-error CLASSIFICATION -// (provider.RetryableError, promptTurnWithRetry's backoff/park -// decisions) is shaped entirely around native provider.Stream errors. -// An error this file returns is a plain error, never classified -// retryable, so a goal loop treats every delegated-turn failure as a -// deterministic stall rather than transient provider weather. Basic -// interactive Prompt-driven delegation is the verified, supported -// shape for v1. +// The v1 doc above (see git history for its original wording) flagged five +// gaps this package now closes: +// +// - MCP passthrough (`--mcp-config`): runClaudeCodeTurn now translates +// the session's configured MCP servers (via the mcpServerLister seam +// below) into the CLI's own --mcp-config JSON and passes +// --strict-mcp-config alongside it — see buildClaudeCodeMCPConfig. +// - `--effort`: runClaudeCodeTurn reads s.Effort() and forwards it as +// --effort, mapped through claudeCodeEffortArg. +// - "thinking" content blocks: claudeCodeAssistantMessage now decodes +// them into message.Reasoning parts instead of dropping them. +// - parent_tool_use_id: captured on the envelope and carried onto the +// appended message.Message (Message.ParentToolUseID) so a subagent +// turn's nesting survives into the journal. +// - turn_metrics: consumeClaudeCodeStream now emits an OnTurnMetrics +// record from the "result" event's own timing/usage fields. +// +// `--append-system-prompt` remains NOT auto-populated from s.cfg.System — +// see the original reasoning, unchanged: harness's system-prompt assembly +// is deliberately native-loop-only (PromptWithOrigin's dispatch comment), +// Claude Code already discovers its own CLAUDE.md/AGENTS.md in the box +// workspace, and re-injecting harness's own native-tool-shaped instructions +// into a CLI with different tools would be actively misleading. An operator +// who wants extra injected wording can still reach the flag via +// config.Provider.ExtraArgs. +// +// Full goal-loop support is now PARTIAL rather than absent: a delegated +// turn's error is wrapped provider.RetryableError (see +// claudeCodeRetryableClass and the process-exit branch in +// runClaudeCodeTurn) for the known-transient shapes a "result" event or a +// child crash can report — a rate-limit/overload signal, the CLI's own +// "error_during_execution" catch-all, or the child process exiting without +// ever emitting a clean result at all — so goal.go's promptTurnWithRetry +// gives those the same backoff-and-retry treatment a native provider's +// weather gets. A deterministic failure (max turns reached, a genuine +// refusal) still surfaces as a plain error, exactly as before, so the goal +// loop still fails fast on those rather than burning a retry budget on a +// request that will fail identically every time. package engine import ( @@ -119,6 +142,7 @@ import ( "errors" "fmt" "io" + "os" "os/exec" "strings" "syscall" @@ -266,6 +290,17 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro } model := s.Model() + // The --mcp-config file (if s.cfg.MCP has any servers to describe) is + // written before the args slice below so its path can be included, and + // removed unconditionally on every return path via cleanupMCPConfig — + // see claudeCodeMCPConfigFile's own doc comment for why this rides a + // temp file rather than an inline argv value. + mcpConfigPath, cleanupMCPConfig, err := s.claudeCodeMCPConfigFile() + if err != nil { + return nil, err + } + defer cleanupMCPConfig() + args := []string{ "--input-format", "stream-json", "--output-format", "stream-json", @@ -280,6 +315,18 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro if cfg.PermissionMode != "" { args = append(args, "--permission-mode", cfg.PermissionMode) } + if effort, ok := claudeCodeEffortArg(s.Effort()); ok { + args = append(args, "--effort", effort) + } + if mcpConfigPath != "" { + // --strict-mcp-config: only harness's own configured servers are + // visible to the child, never whatever a project-local .mcp.json or + // the operator's own ~/.claude.json might additionally define — + // harness's config is the single source of truth for what tools a + // delegated turn can reach, exactly like the native loop's own + // toolDefs assembly. + args = append(args, "--mcp-config", mcpConfigPath, "--strict-mcp-config") + } args = append(args, cfg.ExtraArgs...) cmd := exec.Command(binary, args...) //nolint:gosec // binary/args are operator config, not request input @@ -320,20 +367,34 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro _ = cmd.Wait() return nil, fmt.Errorf("engine: claude-code: encoding turn input: %w", err) } + // inputErr captures a failure writing or closing stdin WITHOUT killing + // the child or returning early: a fast/trivial turn's child can + // legitimately finish its whole result and exit (closing its own end + // of the pipe) before this call finishes writing/closing its side, + // which turns an otherwise-harmless race into a broken-pipe/closed- + // pipe error right here. That is not a real failure — the child still + // has a complete, valid result waiting on stdout — so this call must + // keep going and read it: only if the turn ends with NO usable result + // at all does inputErr get promoted to the actual returned error, + // below. Deliberately not stdin.Close() after a failed Write: closing + // an already-broken pipe has nothing useful to report, and calling it + // anyway would risk overwriting a meaningful inputErr with a second, + // less informative one. + var inputErr error if _, err := stdin.Write(append(inputLine, '\n')); err != nil { - _ = cmd.Process.Kill() - _ = cmd.Wait() - return nil, fmt.Errorf("engine: claude-code: writing turn input: %w", err) - } - // Close stdin: this driver sends exactly one turn per `claude` child - // (continuity across harness turns is --resume, not a long-lived - // child — see this file's package doc), and an unclosed stdin would - // leave the CLI waiting indefinitely for a second message that is - // never coming, wedging cmd.Wait() below forever. - if err := stdin.Close(); err != nil { - _ = cmd.Process.Kill() - _ = cmd.Wait() - return nil, fmt.Errorf("engine: claude-code: closing stdin: %w", err) + inputErr = fmt.Errorf("engine: claude-code: writing turn input: %w", err) + } else if err := stdin.Close(); err != nil { + // Close stdin: this driver sends exactly one turn per `claude` + // child (continuity across harness turns is --resume, not a long- + // lived child — see this file's package doc), and an unclosed + // stdin would leave the CLI waiting indefinitely for a second + // message that is never coming, wedging cmd.Wait() below forever — + // but a Close failing is itself just as benign as a Write failing, + // for the exact same reason (the read end may already be gone + // because the child already finished), so it gets the same + // deferred treatment as the Write error above rather than an + // immediate kill-and-return. + inputErr = fmt.Errorf("engine: claude-code: closing stdin: %w", err) } // The signal-abort cascade: SIGINT first (Claude Code's own docs @@ -371,32 +432,107 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro _ = proc.Kill() }() - finalMsg, turnErr := s.consumeClaudeCodeStream(stdout, model) + finalMsg, started, turnErr := s.consumeClaudeCodeStream(stdout, model) waitErr := cmd.Wait() - if ctx.Err() != nil { + return claudeCodeTurnResult(claudeCodeTurnOutcome{ + ctxErr: ctx.Err(), + turnErr: turnErr, + waitErr: waitErr, + inputErr: inputErr, + started: started, + finalMsg: finalMsg, + binary: binary, + stderr: stderr.String(), + }) +} + +// claudeCodeTurnOutcome collects everything runClaudeCodeTurn learns about +// one delegated turn's process/stream lifecycle — see claudeCodeTurnResult, +// the sole consumer, for what each field decides. +type claudeCodeTurnOutcome struct { + ctxErr error + turnErr error + waitErr error + inputErr error + started bool + finalMsg *message.Message + binary string + stderr string +} + +// claudeCodeTurnResult turns one claudeCodeTurnOutcome into runClaudeCodeTurn's +// own (*message.Message, error) return. Split out from runClaudeCodeTurn as +// its own pure function so the precedence between a caller abort, a +// classified result error, a process-exit error, and a benign input-write +// race is unit-testable directly (TestClaudeCodeTurnResult) without having +// to force each interleaving out of a real child process. +func claudeCodeTurnResult(o claudeCodeTurnOutcome) (*message.Message, error) { + if o.ctxErr != nil { // An abort/shutdown-driven cancellation always wins over whatever // the stream decoded — the same precedence the native path's // context.Canceled handling gives ctx (engine.go's // streamTurnWithRetry/runAgenticLoop treat a canceled context as a // deliberate stop, not an ordinary failure). - return nil, ctx.Err() + return nil, o.ctxErr } - if turnErr != nil { - return nil, turnErr + if o.turnErr != nil { + return nil, o.turnErr } - if waitErr != nil { - msg := fmt.Sprintf("engine: claude-code: %q exited with error: %v", binary, waitErr) - if tail := stderr.String(); tail != "" { - msg += fmt.Sprintf(" (stderr: %s)", tail) + if o.waitErr != nil { + msg := fmt.Sprintf("engine: claude-code: %q exited with error: %v", o.binary, o.waitErr) + if o.stderr != "" { + msg += fmt.Sprintf(" (stderr: %s)", o.stderr) + } + err := error(errors.New(msg)) + if o.started { + // The child got far enough to emit at least one "system" event + // — the CLI's own protocol came up, so a session genuinely + // started — and then exited without ever emitting a clean + // "result" event (o.turnErr == nil, so this is not the + // deterministic IsError shape claudeCodeRetryableClass + // classifies above): a crash, an OOM kill, a signal from + // something other than this call's own abort cascade (o.ctxErr + // was already checked nil above). That is exactly the same + // kind of non-deterministic, worth-a-retry provider weather + // MarkStreamTruncated marks for a native adapter's stream that + // dies mid-body, so goal.go's promptTurnWithRetry gives it the + // same backoff-and-retry treatment rather than parking on + // attempt 1. + err = provider.MarkRetryable(err, provider.RetryableServerError) } - return nil, errors.New(msg) + // !o.started means the child never even got its own protocol off + // the ground — an unknown flag on an older `claude` build, a + // malformed --mcp-config command, an invalid --model value, a + // missing binary's exec succeeding but the binary itself refusing + // to run — deterministic startup failures that will fail + // identically on every retry. Marking THOSE retryable would have a + // PursueGoal loop burn its entire retryable budget + // (goalRetryableMaxAttempts, goal.go) with backoff before parking, + // delaying the surfacing of what is really a config error nothing + // will fix by waiting. Left a plain error here, exactly like every + // other deterministic failure this file returns. + return nil, err } - if finalMsg == nil { + if o.finalMsg == nil { + if o.inputErr != nil { + // No usable result at all, AND writing/closing stdin itself + // failed: the write error is almost certainly the actual root + // cause here (the child never got the turn's own prompt to + // answer), so surface it in place of the generic "no assistant + // message" error below. + return nil, o.inputErr + } return nil, errors.New("engine: claude-code: turn ended with no assistant message") } - return finalMsg, nil + // o.finalMsg != nil: the child produced a complete, usable result + // despite any o.inputErr recorded above (see runClaudeCodeTurn's own + // inputErr doc comment). A benign broken-pipe/closed-pipe race + // resolves itself once the turn's own output proves the child got + // everything it needed; o.inputErr is deliberately dropped here, never + // surfaced once the turn otherwise succeeded. + return o.finalMsg, nil } // lastUserMessageText returns the Text of the LAST message in history if @@ -420,17 +556,19 @@ func lastUserMessageText(history []message.Message) string { // consumeClaudeCodeStream reads newline-delimited stream-json events from // r (the `claude` child's stdout) until EOF, appending/emitting each // decoded event per this file's package-doc mapping, and returns the last -// assistant message.Message it appended (nil if none) plus any turn- -// ending error a "result" event's IsError reported. -func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) (*message.Message, error) { +// assistant message.Message it appended (nil if none), whether the child +// got far enough to emit at least one "system" event (see the started +// return value's own use in runClaudeCodeTurn's waitErr branch: it is what +// tells a child that never even started apart from one that started and +// later crashed), and any turn-ending error a "result" event's IsError +// reported. +func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) (finalMsg *message.Message, started bool, turnErr error) { scanner := bufio.NewScanner(r) // A tool call's arguments or a large tool result can exceed // bufio.Scanner's 64KiB default token size; 8MiB comfortably covers // any realistic single stream-json line without an unbounded read. scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024) - var finalMsg *message.Message - var turnErr error for scanner.Scan() { line := bytes.TrimSpace(scanner.Bytes()) if len(line) == 0 { @@ -445,6 +583,13 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( } switch env.Type { case "system": + // ANY "system" event — not only subtype "init" — is proof the + // child's stream-json protocol actually came up: init is + // documented as the first event a real `claude` binary ever + // emits, so seeing one at all (whatever its subtype) means the + // session started. See the started return value's own doc + // comment above. + started = true if env.Subtype == "init" { s.recordClaudeCodeSessionID(env.SessionID) } @@ -452,7 +597,7 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // requires no action — see the package doc's event-mapping // section. case "assistant": - msg := claudeCodeAssistantMessage(env.Message, model) + msg := claudeCodeAssistantMessage(env.Message, model, env.ParentToolUseID) if len(msg.Parts) == 0 { continue } @@ -464,13 +609,17 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( if part.Text != "" { s.emit(Event{Type: EventTextDelta, Text: part.Text}) } + case *message.Reasoning: + if part.Text != "" { + s.emit(Event{Type: EventReasoningDelta, Text: part.Text}) + } case *message.ToolCall: s.emit(Event{Type: EventToolStart, ToolCall: part}) } } finalMsg = &msg case "user": - msg := claudeCodeToolResultMessage(env.Message) + msg := claudeCodeToolResultMessage(env.Message, env.ParentToolUseID) if msg == nil { continue } @@ -487,9 +636,34 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( } } case "result": - s.applyClaudeCodeUsage(mapClaudeCodeUsage(env.Usage)) + usage := mapClaudeCodeUsage(env.Usage) + s.applyClaudeCodeUsage(usage) + streamMillis := env.DurationMillis - env.TTFTMillis + if streamMillis < 0 { + // A CLI build that sends duration_ms but not ttft_ms (or + // vice versa) must never produce a negative StreamMillis — + // see TurnMetrics.StreamMillis's own "zero when EventDone + // was itself the first delta" precedent for the native + // path; here it means "no usable breakdown", not "the + // turn took negative time". + streamMillis = 0 + } + s.emitTurnMetrics(TurnMetrics{ + SessionID: s.ID, + Model: model, + Attempt: 1, // see this file's package doc: Claude Code retries internally, invisible to harness + TTFTMillis: env.TTFTMillis, + StreamMillis: streamMillis, + InputTokens: usage.InputTokens, + OutputTokens: usage.OutputTokens, + CacheReadTokens: usage.CacheReadTokens, + CacheWriteTokens: usage.CacheWriteTokens, + }) if env.IsError { turnErr = fmt.Errorf("engine: claude-code: turn ended in error (subtype %q): %s", env.Subtype, env.Result) + if class, ok := claudeCodeRetryableClass(env.Subtype, env.Result); ok { + turnErr = provider.MarkRetryable(turnErr, class) + } } // TotalCostUSD is intentionally dropped here — see the // package doc's "result" bullet: provider.Usage has no cost @@ -498,7 +672,7 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // Any other top-level "type" (this driver has none documented // beyond the four above) is inert activity, per the package doc. } - return finalMsg, turnErr + return finalMsg, started, turnErr } // claudeCodeEnvelope is the outer discriminator every line of `claude @@ -518,6 +692,23 @@ type claudeCodeEnvelope struct { // whole delegated turn — read but deliberately never mapped onto // anything (see this file's package doc, "result" bullet). TotalCostUSD float64 `json:"total_cost_usd,omitempty"` + // ParentToolUseID is null (so absent, or explicit JSON null — either + // decodes to "" for a plain string field, encoding/json's documented + // no-op-on-null behavior for a non-pointer target) at the top level of + // a delegated turn's own events, and set to the spawning tool_use id + // inside a subagent's own turn — see the package doc's event-mapping + // section. Carried verbatim onto the appended message.Message + // (Message.ParentToolUseID) by claudeCodeAssistantMessage/ + // claudeCodeToolResultMessage. + ParentToolUseID string `json:"parent_tool_use_id,omitempty"` + // TTFTMillis and DurationMillis are a "result" event's own timing + // fields (time to first token, and this turn's total wall time), + // forwarded into emitTurnMetrics's TTFTMillis/StreamMillis — see the + // package doc's "result" bullet. Zero, never an error, if a particular + // `claude` build does not send them (this file's usual permissive- + // decoding philosophy). + TTFTMillis int64 `json:"ttft_ms,omitempty"` + DurationMillis int64 `json:"duration_ms,omitempty"` } // claudeCodeUsage is a "result" event's usage object. @@ -587,6 +778,16 @@ type claudeCodeContentBlock struct { ToolUseID string `json:"tool_use_id,omitempty"` Content json.RawMessage `json:"content,omitempty"` IsError bool `json:"is_error,omitempty"` + // Thinking and Signature are set on a "thinking" block — the raw + // Anthropic Messages API shape Claude Code's own "assistant" events + // reuse verbatim (see provider/anthropic/anthropic.go's identical + // content_block_start/content_block_delta fields for the API this + // mirrors). Signature is opaque, provider-native reasoning state, + // carried into message.Reasoning.ProviderData rather than dropped — + // see claudeCodeAssistantMessage's "thinking" case and + // claudeCodeReasoningProviderData. + Thinking string `json:"thinking,omitempty"` + Signature string `json:"signature,omitempty"` } // decodeClaudeCodeContentBlocks decodes raw as either a JSON array of @@ -639,13 +840,38 @@ func claudeCodeContentText(raw json.RawMessage) string { return string(raw) } +// claudeCodeReasoningFamily tags a "thinking" block's opaque Signature under +// message.Reasoning.ProviderData, keyed the same way provider/anthropic's +// own Family constant names it ("anthropic") — a delegated turn's thinking +// blocks are Anthropic's own wire shape verbatim (see +// claudeCodeContentBlock's "thinking"/"signature" doc comment), so tagging +// them under the same family a future transcode of this history would +// expect is the honest key, even though this file cannot import +// provider/anthropic to reference its constant directly (package engine +// does not import a specific provider package — see +// ClaudeCodeProviderFamily's own doc comment for the same duplication-over- +// import precedent with provider/claudecode.Family). +const claudeCodeReasoningFamily = "anthropic" + +// claudeCodeReasoningData is the JSON shape stored under +// message.Reasoning.ProviderData[claudeCodeReasoningFamily] — mirrors +// provider/anthropic/transcode.go's anthropicReasoningData one-for-one +// (Signature only; a delegated turn never sees a "redacted_thinking" block +// on its own stream-json output, so there is no Redacted field to carry). +type claudeCodeReasoningData struct { + Signature string `json:"signature,omitempty"` +} + // claudeCodeAssistantMessage decodes an "assistant" event's message field -// into a canonical message.Message: one Text part per non-empty "text" -// content block, one ToolCall part per "tool_use" block, in the CLI's own -// order. A decode failure or a message with no recognized blocks yields a -// Message with a nil Parts, which consumeClaudeCodeStream's caller treats -// as "nothing to append". -func claudeCodeAssistantMessage(raw json.RawMessage, model message.ModelRef) message.Message { +// into a canonical message.Message: one Reasoning part per "thinking" +// block, one Text part per non-empty "text" content block, one ToolCall +// part per "tool_use" block, in the CLI's own order. parentToolUseID rides +// straight onto the returned Message (see claudeCodeEnvelope. +// ParentToolUseID's own doc comment) — empty for a top-level delegated +// turn's own messages. A decode failure or a message with no recognized +// blocks yields a Message with a nil Parts, which consumeClaudeCodeStream's +// caller treats as "nothing to append". +func claudeCodeAssistantMessage(raw json.RawMessage, model message.ModelRef, parentToolUseID string) message.Message { var cm claudeCodeMessage _ = json.Unmarshal(raw, &cm) // best-effort; a failure just yields no blocks below var parts message.Parts @@ -655,6 +881,12 @@ func claudeCodeAssistantMessage(raw json.RawMessage, model message.ModelRef) mes if b.Text != "" { parts = append(parts, &message.Text{Text: b.Text}) } + case "thinking": + data, _ := json.Marshal(claudeCodeReasoningData{Signature: b.Signature}) + parts = append(parts, &message.Reasoning{ + Text: b.Thinking, + ProviderData: message.ProviderData{claudeCodeReasoningFamily: data}, + }) case "tool_use": args := b.Input if len(args) == 0 { @@ -664,25 +896,27 @@ func claudeCodeAssistantMessage(raw json.RawMessage, model message.ModelRef) mes } } return message.Message{ - ID: newID("msg"), - Role: message.RoleAssistant, - Parts: parts, - Model: model, - Origin: message.OriginClaudeCode, - CreatedAt: time.Now().UTC(), + ID: newID("msg"), + Role: message.RoleAssistant, + Parts: parts, + Model: model, + Origin: message.OriginClaudeCode, + CreatedAt: time.Now().UTC(), + ParentToolUseID: parentToolUseID, } } // claudeCodeToolResultMessage decodes a "user" event's message field — // Claude Code's own tool_result delivery, in the raw Anthropic API's // "user"-role convention — into a canonical RoleTool message.Message: one -// ToolResult part per "tool_result" content block. Returns nil when the -// message decodes to no tool_result blocks at all (an ordinary human- -// authored "user" event never reaches this driver — Session.History's -// tail is the only user input a delegated turn ever sends, over stdin, not -// stdout — so an empty result here means an unrecognized shape, not a -// real turn boundary to silently drop). -func claudeCodeToolResultMessage(raw json.RawMessage) *message.Message { +// ToolResult part per "tool_result" content block. parentToolUseID rides +// onto the returned Message exactly like claudeCodeAssistantMessage's own +// parameter. Returns nil when the message decodes to no tool_result blocks +// at all (an ordinary human-authored "user" event never reaches this +// driver — Session.History's tail is the only user input a delegated turn +// ever sends, over stdin, not stdout — so an empty result here means an +// unrecognized shape, not a real turn boundary to silently drop). +func claudeCodeToolResultMessage(raw json.RawMessage, parentToolUseID string) *message.Message { var cm claudeCodeMessage if err := json.Unmarshal(raw, &cm); err != nil { return nil @@ -702,12 +936,201 @@ func claudeCodeToolResultMessage(raw json.RawMessage) *message.Message { return nil } return &message.Message{ - ID: newID("msg"), - Role: message.RoleTool, - Parts: parts, - Origin: message.OriginClaudeCode, - CreatedAt: time.Now().UTC(), + ID: newID("msg"), + Role: message.RoleTool, + Parts: parts, + Origin: message.OriginClaudeCode, + CreatedAt: time.Now().UTC(), + ParentToolUseID: parentToolUseID, + } +} + +// claudeCodeEffortArg maps harness's message.Effort to the `claude` CLI's +// own --effort values (low/medium/high — the CLI has no "off"/"minimal" +// level of its own). EffortOff and EffortMinimal both collapse onto the +// CLI's floor, "low" — the same "cap at the nearest coarser level the +// target enum actually offers" precedent provider/openai/transcode.go's +// reasoningEffort and provider/anthropic/transcode.go's thinkingBudget +// already follow for their own, differently-shaped target enums (xhigh/max +// are similarly unreachable through harness's four-level Effort enum, so +// there is no higher tier to cap at here). ok is false for +// message.EffortUnset — send no --effort flag at all, mirroring how an +// unset provider.Request.Effort sends no reasoning control to a native +// provider — and for any value message.ParseEffort would not recognize. +func claudeCodeEffortArg(e message.Effort) (string, bool) { + switch e { + case message.EffortOff, message.EffortMinimal, message.EffortLow: + return "low", true + case message.EffortMedium: + return "medium", true + case message.EffortHigh: + return "high", true + default: + return "", false + } +} + +// claudeCodeRetryableClass classifies a "result" event's own reported +// failure — subtype plus the human-readable result text — as provider- +// weather retryable, mirroring how a native adapter classifies an HTTP +// status or inline API-error event (see provider.RetryableClass). This is +// deliberately NOT "every is_error result is retryable": a genuine +// deterministic outcome (max turns reached, a refusal) must still fail +// fast so goal.go's promptTurnWithRetry does not burn its retry budget on +// a request that will fail identically every time — only a signal this +// file can actually name as transient provider weather gets wrapped. +func claudeCodeRetryableClass(subtype, result string) (provider.RetryableClass, bool) { + hay := strings.ToLower(subtype + " " + result) + switch { + case strings.Contains(hay, "rate_limit") || strings.Contains(hay, "rate limit"): + return provider.RetryableRateLimited, true + case strings.Contains(hay, "overloaded"): + return provider.RetryableOverloaded, true + case subtype == "error_during_execution": + // The CLI's own catch-all subtype for an infrastructure-side + // hiccup during its turn (e.g. a transient API error surfaced + // mid-execution, not a deterministic domain failure) — mirrors + // provider/anthropic's inline "error" SSE event mapping to + // RetryableServerError. + return provider.RetryableServerError, true + default: + return "", false + } +} + +// claudeCodeMCPServerLister is the seam runClaudeCodeTurn uses to read a +// session's configured MCP server definitions for --mcp-config. Kept +// separate from the MCPRegistry interface itself (Tools/CallTool/ +// CallServerTool) deliberately: extending MCPRegistry would force every +// existing fake implementation across this package, cmd/harness, and +// server to grow a new method just to keep compiling, for a capability +// only this one call site needs. Only *MCPManager (the production +// implementation, see its own Servers method in mcp.go) and any test fake +// that chooses to implement it need to; an s.cfg.MCP that does not (nil, or +// a fake with no reason to care) simply contributes no MCP passthrough, +// the same fail-open philosophy as an MCP server that never connects (see +// engine/mcp.go's package doc). +type claudeCodeMCPServerLister interface { + Servers() map[string]MCPServerConfig +} + +// claudeCodeMCPServers returns reg's configured MCP servers, or nil if reg +// is nil or does not implement claudeCodeMCPServerLister — see that +// interface's own doc comment. +func claudeCodeMCPServers(reg MCPRegistry) map[string]MCPServerConfig { + lister, ok := reg.(claudeCodeMCPServerLister) + if !ok { + return nil + } + return lister.Servers() +} + +// claudeCodeMCPConfig is the `claude` CLI's own --mcp-config JSON shape: a +// top-level "mcpServers" object of server definitions (see +// https://code.claude.com/docs's MCP configuration file contract) — a +// stdio server names a command/args/env, an HTTP server names a type/url +// and optional headers. +type claudeCodeMCPConfig struct { + MCPServers map[string]claudeCodeMCPServerSpec `json:"mcpServers"` +} + +// claudeCodeMCPServerSpec is one server entry of claudeCodeMCPConfig. Type +// is omitted for a stdio server (the CLI's own default) and "http" for a +// Streamable HTTP server, in which case Command/Args/Env are unset and +// URL/Headers carry the server's endpoint instead — see +// claudeCodeMCPServerSpecFor. +type claudeCodeMCPServerSpec struct { + Type string `json:"type,omitempty"` + Command string `json:"command,omitempty"` + Args []string `json:"args,omitempty"` + Env map[string]string `json:"env,omitempty"` + URL string `json:"url,omitempty"` + Headers map[string]string `json:"headers,omitempty"` +} + +// claudeCodeMCPServerSpecFor translates one engine MCPServerConfig into its +// --mcp-config wire shape. Exactly one of spec.Command/spec.URL is ever +// set — see MCPServerConfig's own doc comment; validateMCPServers enforces +// this at config-load time, well before a value can ever reach here — so +// checking URL first and falling through to the stdio shape otherwise +// never mismatches a server's actual kind. +func claudeCodeMCPServerSpecFor(spec MCPServerConfig) claudeCodeMCPServerSpec { + if spec.URL != "" { + return claudeCodeMCPServerSpec{Type: "http", URL: spec.URL, Headers: spec.Headers} + } + out := claudeCodeMCPServerSpec{Env: claudeCodeMCPServerEnv(spec.Env)} + if len(spec.Command) > 0 { + out.Command = spec.Command[0] + if len(spec.Command) > 1 { + out.Args = append([]string(nil), spec.Command[1:]...) + } + } + return out +} + +// claudeCodeMCPServerEnv converts MCPServerConfig.Env's "KEY=VALUE" argv- +// style entries into the map object --mcp-config's JSON shape expects. An +// entry with no "=" is skipped rather than failing the whole turn over one +// malformed entry — this file's usual permissive-decoding philosophy +// applied to config translation instead of CLI output. +func claudeCodeMCPServerEnv(env []string) map[string]string { + if len(env) == 0 { + return nil + } + out := make(map[string]string, len(env)) + for _, kv := range env { + k, v, ok := strings.Cut(kv, "=") + if !ok { + continue + } + out[k] = v + } + return out +} + +// claudeCodeMCPConfigFile writes s's configured MCP servers (if any) to a +// fresh temp file in the CLI's own --mcp-config JSON shape, returning its +// path plus a cleanup func that removes it (always non-nil, a no-op when +// path is ""). A temp FILE, not an inline JSON string on the command line, +// deliberately: an MCPServerConfig can carry Headers/Env holding real +// credential material, and argv is visible to any other process on the box +// (via /proc or ps) — writing to a file only this process's own return +// value names, then removing it once this call returns (the child has +// already read it by then; it only needs the file at startup), keeps that +// material out of the process list. len(servers) == 0 (MCP unconfigured, +// or s.cfg.MCP does not implement claudeCodeMCPServerLister — see +// claudeCodeMCPServers) returns "", a no-op cleanup, and a nil error: MCP +// passthrough is opt-in, never a hard requirement for a delegated turn to +// proceed. +func (s *Session) claudeCodeMCPConfigFile() (path string, cleanup func(), err error) { + noop := func() {} + servers := claudeCodeMCPServers(s.cfg.MCP) + if len(servers) == 0 { + return "", noop, nil + } + cfg := claudeCodeMCPConfig{MCPServers: make(map[string]claudeCodeMCPServerSpec, len(servers))} + for name, spec := range servers { + cfg.MCPServers[name] = claudeCodeMCPServerSpecFor(spec) + } + data, err := json.Marshal(cfg) + if err != nil { + return "", noop, fmt.Errorf("engine: claude-code: encoding --mcp-config: %w", err) + } + f, err := os.CreateTemp("", "harness-claude-code-mcp-*.json") + if err != nil { + return "", noop, fmt.Errorf("engine: claude-code: creating --mcp-config file: %w", err) + } + if _, err := f.Write(data); err != nil { + _ = f.Close() + _ = os.Remove(f.Name()) + return "", noop, fmt.Errorf("engine: claude-code: writing --mcp-config file: %w", err) + } + if err := f.Close(); err != nil { + _ = os.Remove(f.Name()) + return "", noop, fmt.Errorf("engine: claude-code: closing --mcp-config file: %w", err) } + name := f.Name() + return name, func() { _ = os.Remove(name) }, nil } // claudeCodeInputMessage is the stdin stream-json shape this driver writes diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 612b8d43..d42cf091 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -9,6 +9,7 @@ import ( "os" "os/exec" "path/filepath" + "strings" "sync" "testing" "time" @@ -406,3 +407,499 @@ func TestClaudeCodeDefaultBinaryPath(t *testing.T) { t.Errorf("ClaudeCode.BinaryPath = %q, want %q", s.cfg.ClaudeCode.BinaryPath, defaultClaudeCodeBinaryPath) } } + +// TestClaudeCodeMCPConfigFileWritesConfiguredServers proves +// Session.claudeCodeMCPConfigFile translates a session's configured MCP +// servers (a stdio server and an HTTP server) into the CLI's own +// --mcp-config JSON shape, and that its cleanup func actually removes the +// file. +func TestClaudeCodeMCPConfigFileWritesConfiguredServers(t *testing.T) { + mgr := NewMCPManager(map[string]MCPServerConfig{ + "fs": {Command: []string{"mcp-fs", "--root", "/work"}, Env: []string{"FOO=bar"}}, + "remote": {URL: "https://example.com/mcp", Headers: map[string]string{"Authorization": "Bearer tok"}}, + }) + s := NewSession(Config{ + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + MCP: mgr, + }) + + path, cleanup, err := s.claudeCodeMCPConfigFile() + if err != nil { + t.Fatalf("claudeCodeMCPConfigFile: %v", err) + } + if path == "" { + t.Fatal("claudeCodeMCPConfigFile returned an empty path for a session with configured MCP servers") + } + + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading mcp-config file: %v", err) + } + var got claudeCodeMCPConfig + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("decoding mcp-config file: %v", err) + } + + fs, ok := got.MCPServers["fs"] + if !ok { + t.Fatalf(`mcp-config missing "fs" server: %+v`, got.MCPServers) + } + if fs.Type != "" || fs.Command != "mcp-fs" || len(fs.Args) != 2 || fs.Args[0] != "--root" || fs.Args[1] != "/work" { + t.Errorf("fs server = %+v, want a stdio server: command mcp-fs, args [--root /work]", fs) + } + if fs.Env["FOO"] != "bar" { + t.Errorf("fs server Env = %+v, want FOO=bar", fs.Env) + } + + remote, ok := got.MCPServers["remote"] + if !ok { + t.Fatalf(`mcp-config missing "remote" server: %+v`, got.MCPServers) + } + if remote.Type != "http" || remote.URL != "https://example.com/mcp" || remote.Headers["Authorization"] != "Bearer tok" { + t.Errorf("remote server = %+v, want an http server naming example.com with its Authorization header", remote) + } + + cleanup() + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Errorf("mcp-config file %q still exists after cleanup", path) + } +} + +// TestClaudeCodeMCPConfigFileEmptyWithNoServers proves a session with no +// configured MCP servers (nil Config.MCP, the default) gets no +// --mcp-config file at all — MCP passthrough is opt-in, never a hard +// requirement for a delegated turn. +func TestClaudeCodeMCPConfigFileEmptyWithNoServers(t *testing.T) { + s := NewSession(Config{Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}}) + path, cleanup, err := s.claudeCodeMCPConfigFile() + if err != nil { + t.Fatalf("claudeCodeMCPConfigFile: %v", err) + } + defer cleanup() + if path != "" { + t.Errorf("claudeCodeMCPConfigFile path = %q, want empty for a session with no configured MCP servers", path) + } +} + +// TestClaudeCodeMCPConfigForwardedToChildArgv proves runClaudeCodeTurn +// actually appends --mcp-config (naming a real, readable file) and +// --strict-mcp-config to the child's argv when the session has configured +// MCP servers. +func TestClaudeCodeMCPConfigForwardedToChildArgv(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.cfg.MCP = NewMCPManager(map[string]MCPServerConfig{ + "fs": {Command: []string{"mcp-fs"}}, + }) + // s.cfg.MCP is read fresh by claudeCodeMCPConfigFile at turn time (see + // claudeCodeTestSession's OnEvent precedent above): safe to set + // directly here, pre-Prompt, before anything else touches the session. + + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if len(invocations) != 1 { + t.Fatalf("invocations = %d, want 1: %+v", len(invocations), invocations) + } + path, ok := argvValueAfter(invocations[0], "--mcp-config") + if !ok || path == "" { + t.Fatalf("argv has no non-empty --mcp-config value: %v", invocations[0]) + } + if !argvContains(invocations[0], "--strict-mcp-config") { + t.Errorf("argv missing --strict-mcp-config: %v", invocations[0]) + } +} + +// TestClaudeCodeMCPConfigCredentialsNeverInChildArgv locks in the reason +// claudeCodeMCPConfigFile writes a temp FILE rather than passing an inline +// --mcp-config JSON string: a server's Headers/Env can carry real +// credential material, and argv is visible to any other process on the +// box via /proc or ps. This configures a server with a bearer-token header +// and a secret-bearing env entry, drives one real turn, and asserts the +// secret value never appears in ANY element of the child's own argv — see +// TestClaudeCodeMCPConfigFileWritesConfiguredServers for the companion +// assertion that the same secret DOES reach the server correctly, via the +// (by-then-removed) config file's own JSON content. +func TestClaudeCodeMCPConfigCredentialsNeverInChildArgv(t *testing.T) { + const secret = "sk-super-secret-token-do-not-leak" + s, logPath := claudeCodeTestSession(t, "normal") + s.cfg.MCP = NewMCPManager(map[string]MCPServerConfig{ + "remote": {URL: "https://example.com/mcp", Headers: map[string]string{"Authorization": "Bearer " + secret}}, + "fs": {Command: []string{"mcp-fs"}, Env: []string{"MCP_FS_TOKEN=" + secret}}, + }) + + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if len(invocations) != 1 { + t.Fatalf("invocations = %d, want 1: %+v", len(invocations), invocations) + } + for _, arg := range invocations[0] { + if strings.Contains(arg, secret) { + t.Fatalf("child argv leaked the MCP credential (%q): %v", secret, invocations[0]) + } + } + if _, ok := argvValueAfter(invocations[0], "--mcp-config"); !ok { + t.Fatalf("argv has no --mcp-config at all: %v", invocations[0]) + } +} + +// TestClaudeCodeEffortForwardedToChildArgv proves runClaudeCodeTurn reads +// s.Effort() at turn time and forwards it as --effort, mapped through +// claudeCodeEffortArg exactly as that function's own doc comment promises. +func TestClaudeCodeEffortForwardedToChildArgv(t *testing.T) { + tests := []struct { + name string + effort message.Effort + want string + }{ + {"off maps to the CLI floor", message.EffortOff, "low"}, + {"minimal maps to the CLI floor", message.EffortMinimal, "low"}, + {"low maps to low", message.EffortLow, "low"}, + {"medium maps to medium", message.EffortMedium, "medium"}, + {"high maps to high", message.EffortHigh, "high"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.SetEffort(tt.effort) + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + got, ok := argvValueAfter(invocations[0], "--effort") + if !ok { + t.Fatalf("argv has no --effort: %v", invocations[0]) + } + if got != tt.want { + t.Errorf("--effort = %q, want %q", got, tt.want) + } + }) + } +} + +// TestClaudeCodeEffortUnsetOmitsFlag proves a session that never called +// SetEffort (message.EffortUnset, the zero value) sends no --effort flag at +// all, mirroring how an unset provider.Request.Effort sends no reasoning +// control to a native provider. +func TestClaudeCodeEffortUnsetOmitsFlag(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if argvContains(invocations[0], "--effort") { + t.Errorf("argv unexpectedly carries --effort for EffortUnset: %v", invocations[0]) + } +} + +// TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" +// content block — previously silently dropped (see claudeCodeContentBlock's +// switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning +// part, is appended to history, and is emitted as EventReasoningDelta. +func TestClaudeCodeThinkingBlockDecodesToReasoningPart(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking") + + var events []Event + s.cfg.OnEvent = func(ev Event) { events = append(events, ev) } + + msg, err := s.Prompt(context.Background(), "think about it") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if got := msg.Parts.Text(); got != "Here is my answer." { + t.Errorf("final message text = %q", got) + } + + hist := s.History() + // user prompt, assistant(thinking), assistant(text) = 3 messages. + if len(hist) != 3 { + t.Fatalf("History() len = %d, want 3: %+v", len(hist), hist) + } + reasoning, ok := hist[1].Parts[0].(*message.Reasoning) + if hist[1].Role != message.RoleAssistant || !ok || reasoning.Text != "Let me reason about this." { + t.Fatalf("hist[1] = %+v, want an assistant Reasoning(%q)", hist[1], "Let me reason about this.") + } + if len(reasoning.ProviderData) == 0 { + t.Error("Reasoning.ProviderData is empty, want the thinking block's signature carried through") + } + + var sawReasoningDelta bool + for _, ev := range events { + if ev.Type == EventReasoningDelta && ev.Text == "Let me reason about this." { + sawReasoningDelta = true + } + } + if !sawReasoningDelta { + t.Error("no EventReasoningDelta for the thinking block") + } +} + +// TestClaudeCodeParentToolUseIDCarriedOntoMessage proves the envelope's own +// parent_tool_use_id (null at top level, set to the spawning tool_use id +// inside a subagent's own turn) rides onto Message.ParentToolUseID for +// both an "assistant" and a "user" (tool_result) event. +func TestClaudeCodeParentToolUseIDCarriedOntoMessage(t *testing.T) { + s, _ := claudeCodeTestSession(t, "subagent") + + if _, err := s.Prompt(context.Background(), "spawn a subagent"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + hist := s.History() + // user, assistant(tool_use Task, top-level), assistant(text, nested), + // tool(tool_result, nested), assistant(text, top-level) = 5 messages. + if len(hist) != 5 { + t.Fatalf("History() len = %d, want 5: %+v", len(hist), hist) + } + if got := hist[1].ParentToolUseID; got != "" { + t.Errorf("hist[1] (top-level tool_use) ParentToolUseID = %q, want empty", got) + } + if got := hist[2].ParentToolUseID; got != "toolu_parent" { + t.Errorf("hist[2] (nested assistant text) ParentToolUseID = %q, want toolu_parent", got) + } + if got := hist[3].ParentToolUseID; got != "toolu_parent" { + t.Errorf("hist[3] (nested tool result) ParentToolUseID = %q, want toolu_parent", got) + } + if got := hist[4].ParentToolUseID; got != "" { + t.Errorf("hist[4] (final top-level text) ParentToolUseID = %q, want empty", got) + } +} + +// TestClaudeCodeTurnMetricsEmittedForDelegatedTurn proves runClaudeCodeTurn +// emits exactly one OnTurnMetrics record per delegated turn, built from the +// "result" event's own ttft_ms/duration_ms and the usage +// applyClaudeCodeUsage already maps — engine.go's native streamTurn emits +// one per completed turn too (see its own EventDone case), and a delegated +// turn getting none at all was one of this backend's v1 gaps. +func TestClaudeCodeTurnMetricsEmittedForDelegatedTurn(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + + var metrics []TurnMetrics + s.cfg.OnTurnMetrics = func(m TurnMetrics) { metrics = append(metrics, m) } + + if _, err := s.Prompt(context.Background(), "please run echo hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + if len(metrics) != 1 { + t.Fatalf("OnTurnMetrics called %d times, want 1: %+v", len(metrics), metrics) + } + m := metrics[0] + if m.SessionID != s.ID { + t.Errorf("SessionID = %q, want %q", m.SessionID, s.ID) + } + if m.TTFTMillis != 50 { + t.Errorf("TTFTMillis = %d, want 50 (fakeclaude's own canned ttft_ms)", m.TTFTMillis) + } + if m.StreamMillis != 350 { + t.Errorf("StreamMillis = %d, want 350 (duration_ms 400 - ttft_ms 50)", m.StreamMillis) + } + if m.InputTokens != 101 || m.OutputTokens != 42 { + t.Errorf("TurnMetrics usage = {input:%d output:%d}, want {101 42} (fakeclaude's own canned usage)", m.InputTokens, m.OutputTokens) + } +} + +// TestClaudeCodeRetryableClassification proves a "result" event this file +// can actually name as transient provider weather (a rate-limit signal) is +// wrapped provider.RetryableError, while a genuinely deterministic result +// (max turns reached) is not — goal.go's promptTurnWithRetry uses exactly +// this distinction (provider.AsRetryable) to decide whether to back off and +// retry or fail fast. +func TestClaudeCodeRetryableClassification(t *testing.T) { + t.Run("a rate-limit result is retryable", func(t *testing.T) { + s, _ := claudeCodeTestSession(t, "rate_limit_error") + _, err := s.Prompt(context.Background(), "do something") + if err == nil { + t.Fatal("Prompt returned no error for an is_error result") + } + class, ok := provider.AsRetryable(err) + if !ok { + t.Fatalf("provider.AsRetryable(%v) = false, want a retryable classification", err) + } + if class != provider.RetryableRateLimited { + t.Errorf("class = %q, want %q", class, provider.RetryableRateLimited) + } + }) + t.Run("a deterministic max-turns result is not retryable", func(t *testing.T) { + s, _ := claudeCodeTestSession(t, "deterministic_error") + _, err := s.Prompt(context.Background(), "do something") + if err == nil { + t.Fatal("Prompt returned no error for an is_error result") + } + if _, ok := provider.AsRetryable(err); ok { + t.Errorf("provider.AsRetryable(%v) = true, want false for a deterministic max-turns failure", err) + } + }) +} + +// TestClaudeCodeChildCrashWithoutResultIsRetryable proves a child that +// emitted at least one "system" event (so a session genuinely started) and +// THEN exits nonzero WITHOUT ever emitting a clean "result" event (a +// crash, an OOM kill — non-deterministic child-process weather, not a +// domain-level failure the CLI itself reported) is wrapped +// provider.RetryableError, via runClaudeCodeTurn's own waitErr branch +// rather than claudeCodeRetryableClass. See +// TestClaudeCodeChildExitBeforeAnySystemEventIsNotRetryable for the +// opposite case this same branch must get right. +func TestClaudeCodeChildCrashWithoutResultIsRetryable(t *testing.T) { + s, _ := claudeCodeTestSession(t, "crash") + _, err := s.Prompt(context.Background(), "do something") + if err == nil { + t.Fatal("Prompt returned no error for a child that exited without a result event") + } + class, ok := provider.AsRetryable(err) + if !ok { + t.Fatalf("provider.AsRetryable(%v) = false, want retryable for a non-deterministic child crash", err) + } + if class != provider.RetryableServerError { + t.Errorf("class = %q, want %q", class, provider.RetryableServerError) + } +} + +// TestClaudeCodeChildExitBeforeAnySystemEventIsNotRetryable proves a child +// that exits nonzero WITHOUT ever emitting even a "system" event — the +// deterministic-startup-failure shape (an unknown flag on an older +// `claude` build, a malformed --mcp-config command, an invalid --model +// value) — is NOT wrapped provider.RetryableError. Marking a deterministic +// startup failure retryable would have a PursueGoal loop burn its entire +// retryable budget with backoff before parking, delaying the surfacing of +// a config error no amount of waiting will fix. Contrast +// TestClaudeCodeChildCrashWithoutResultIsRetryable, whose child DOES get +// as far as "system" before dying. +func TestClaudeCodeChildExitBeforeAnySystemEventIsNotRetryable(t *testing.T) { + s, _ := claudeCodeTestSession(t, "crash_before_init") + _, err := s.Prompt(context.Background(), "do something") + if err == nil { + t.Fatal("Prompt returned no error for a child that exited before any system event") + } + if _, ok := provider.AsRetryable(err); ok { + t.Errorf("provider.AsRetryable(%v) = true, want false for a child that never started", err) + } +} + +// TestClaudeCodeTurnResult table-drives claudeCodeTurnResult directly — +// the exact precedence between a caller abort, a classified result error, +// a process-exit error (started vs. not), and a benign input-write race — +// without needing to force any of these interleavings out of a real child +// process. The last two cases are this test's whole reason to exist: they +// lock in the "input-write EPIPE + have-result -> ignore; input-write +// error + no-result -> real error" rule a real subprocess race can only +// exercise probabilistically (see TestClaudeCodeSucceedsDespiteInputWriteBrokenPipe +// below for that best-effort integration-level companion). +func TestClaudeCodeTurnResult(t *testing.T) { + ctxCanceled := context.Canceled + turnErr := errors.New("boom: turn error") + waitErr := errors.New("exit status 1") + inputErr := errors.New("engine: claude-code: writing turn input: write |1: broken pipe") + okMsg := &message.Message{ID: "msg_ok"} + + tests := []struct { + name string + outcome claudeCodeTurnOutcome + wantMsg *message.Message + wantErrIs error // set when the returned error must be exactly (or wrap) this value + wantErrText string // set when the returned error is freshly constructed; substring to require instead + wantRetry bool + }{ + { + name: "a caller abort wins over everything else", + outcome: claudeCodeTurnOutcome{ctxErr: ctxCanceled, turnErr: turnErr, waitErr: waitErr, inputErr: inputErr, finalMsg: okMsg}, + wantErrIs: ctxCanceled, + }, + { + name: "a classified result error returns as-is", + outcome: claudeCodeTurnOutcome{turnErr: turnErr}, + wantErrIs: turnErr, + }, + { + name: "a process exit after the child started is retryable", + outcome: claudeCodeTurnOutcome{waitErr: waitErr, started: true}, + wantErrText: waitErr.Error(), + wantRetry: true, + }, + { + name: "a process exit before the child ever started is NOT retryable", + outcome: claudeCodeTurnOutcome{waitErr: waitErr, started: false}, + wantErrText: waitErr.Error(), + }, + { + name: "no result and no input error is the generic no-assistant-message error", + outcome: claudeCodeTurnOutcome{}, + wantErrText: "turn ended with no assistant message", + }, + { + name: "no result AND an input-write error surfaces the input-write error", + outcome: claudeCodeTurnOutcome{inputErr: inputErr}, + wantErrIs: inputErr, + }, + { + name: "a usable result suppresses a benign input-write error", + outcome: claudeCodeTurnOutcome{inputErr: inputErr, finalMsg: okMsg}, + wantMsg: okMsg, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + msg, err := claudeCodeTurnResult(tt.outcome) + if msg != tt.wantMsg { + t.Errorf("msg = %v, want %v", msg, tt.wantMsg) + } + if tt.wantErrIs == nil && tt.wantErrText == "" { + if err != nil { + t.Fatalf("err = %v, want nil", err) + } + return + } + if err == nil { + t.Fatalf("err = nil, want an error") + } + if tt.wantErrIs != nil && !errors.Is(err, tt.wantErrIs) { + t.Errorf("err = %q, want one wrapping %q", err, tt.wantErrIs) + } + if tt.wantErrText != "" && !strings.Contains(err.Error(), tt.wantErrText) { + t.Errorf("err = %q, want it to contain %q", err, tt.wantErrText) + } + if _, ok := provider.AsRetryable(err); ok != tt.wantRetry { + t.Errorf("provider.AsRetryable(%v) = %v, want %v", err, ok, tt.wantRetry) + } + }) + } +} + +// TestClaudeCodeSucceedsDespiteInputWriteBrokenPipe proves runClaudeCodeTurn +// tolerates a broken-pipe/closed-pipe error writing (or closing) the +// turn's stdin input when the child has ALREADY produced — or is about +// to produce — a complete, valid result: a fast/trivial turn's child can +// legitimately finish and exit, closing its own end of the pipe, before +// this call finishes writing/closing its side. This reproduces a real CI +// failure (go test -race caught it; a non-race run is fast enough that +// the write usually wins the race instead) where a delegated turn failed +// with "writing turn input: write |1: broken pipe" even though the child +// had already produced a perfectly good result. +// +// fakeclaude's "fast_no_drain" mode reliably wins this race deliberately +// (see its own doc comment): it closes its own stdin immediately, before +// doing anything else, and runs at native speed since buildFakeClaude +// compiles it without -race, while harness's own -race-instrumented write +// path is comparatively slow. The turn must still succeed end to end: +// final message present, no error, and (since the fix's whole point is +// that a benign inputErr never surfaces once the turn has a usable +// result) OnTurnMetrics still fires exactly once. +func TestClaudeCodeSucceedsDespiteInputWriteBrokenPipe(t *testing.T) { + s, _ := claudeCodeTestSession(t, "fast_no_drain") + + var metrics []TurnMetrics + s.cfg.OnTurnMetrics = func(m TurnMetrics) { metrics = append(metrics, m) } + + msg, err := s.Prompt(context.Background(), "hi") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if msg == nil || msg.Parts.Text() != "Done before you finished writing." { + t.Fatalf("Prompt returned %+v, want fakeclaude's own canned final message", msg) + } + if len(metrics) != 1 { + t.Errorf("OnTurnMetrics called %d times, want 1: %+v", len(metrics), metrics) + } +} diff --git a/engine/journal.go b/engine/journal.go index 5d171cf1..c29a02ab 100644 --- a/engine/journal.go +++ b/engine/journal.go @@ -73,6 +73,11 @@ type JournalRecord struct { MessageID string `json:"message_id,omitempty"` MessageRole string `json:"message_role,omitempty"` RecoveryMarker bool `json:"recovery_marker,omitempty"` + // ParentToolUseID mirrors message.Message.ParentToolUseID (see its own + // doc comment): the spawning tool_use id of a Claude Code CLI subagent + // turn, letting a journal reader reconstruct subagent nesting for a + // delegated session without fetching the message's full content. + ParentToolUseID string `json:"parent_tool_use_id,omitempty"` // Model (Type == recSession or recModel) / effort (Type == recSession or // recEffort). @@ -199,6 +204,7 @@ func projectJournalRecord(seq int, rec record) JournalRecord { out.MessageRole = string(rec.Message.Role) out.CreatedAt = rec.Message.CreatedAt out.RecoveryMarker = isRecoverySyntheticCloser(*rec.Message) + out.ParentToolUseID = rec.Message.ParentToolUseID } case recModel: out.Model = rec.Model diff --git a/engine/mcp.go b/engine/mcp.go index 3a167e71..a1afa610 100644 --- a/engine/mcp.go +++ b/engine/mcp.go @@ -312,6 +312,29 @@ func (m *MCPManager) ConfiguredNames() []string { return names } +// Servers returns a SHALLOW copy of every server this manager was +// constructed with, WITHOUT connecting to any of them — same "m.servers is +// immutable after NewMCPManager, no lock needed" reasoning as +// ConfiguredNames. The returned map is a fresh map (a caller mutating it, +// or adding/removing an entry, never touches m.servers), but each +// MCPServerConfig value's own Command/Env slices and Headers map are +// shared with m.servers, not copied again. That is safe today because this +// method's one caller, claudeCodeMCPServerLister (claude_code_backend.go), +// only ever reads a returned MCPServerConfig to build a --mcp-config JSON +// payload — it never mutates Command/Env/Headers in place. A future +// caller that DOES need to mutate a server's own nested fields must deep- +// copy them itself; this method does not promise that. +func (m *MCPManager) Servers() map[string]MCPServerConfig { + if m == nil { + return nil + } + out := make(map[string]MCPServerConfig, len(m.servers)) + for name, spec := range m.servers { + out[name] = spec + } + return out +} + // mcpConnectFunc is the seam ensureConnected/retryServer call to perform // one connect attempt. Production always uses connectMCPServer; tests that // need to assert a backoff SCHEDULE inside a testing/synctest bubble diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 9388757a..6a0f7a93 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -6,6 +6,19 @@ // environment variable, so the driver's event-mapping, resume, and usage // logic can be tested against a REAL child process and REAL pipes without // depending on an actual Anthropic subscription or the real binary. +// +// Beyond "normal"/"hang"/"error", a handful of narrower modes each cover +// exactly one of the gaps claude_code_backend.go closes: "thinking" (a +// thinking content block plus the result event's own ttft_ms/duration_ms +// timing fields), "subagent" (a null-then-set parent_tool_use_id pair), +// "rate_limit_error"/"deterministic_error" (a result event this file's own +// claudeCodeRetryableClass must classify retryable/not-retryable, +// respectively), "crash"/"crash_before_init" (a child that exits nonzero +// with no "result" event, AFTER vs. BEFORE ever emitting a "system" event +// — runClaudeCodeTurn's waitErr branch must classify only the former +// retryable), and "fast_no_drain" (closes its own stdin immediately, +// without ever draining it, to reliably win the race against harness's +// own turn-input write — see runClaudeCodeTurn's inputErr handling). package main import ( @@ -17,6 +30,23 @@ import ( ) func main() { + mode := os.Getenv("FAKE_CLAUDE_MODE") + + if mode == "fast_no_drain" { + // Close our OWN stdin's read end IMMEDIATELY — before doing + // anything else, including the argv log write below — to + // deterministically win the race this mode's own test regresses: + // harness's own turn-input Write/Close must tolerate a broken-pipe/ + // closed-pipe error when the child has already (or is about to) + // exit with a complete, valid result, exactly the shape a fast/ + // trivial real turn can hit. fakeclaude runs at native speed + // (buildFakeClaude compiles it without -race); harness's own + // -race-instrumented write path is comparatively slow, so closing + // this early reliably beats it. Skips the shared stdin-drain + // goroutine below entirely — there is nothing left to drain. + _ = os.Stdin.Close() + } + if logPath := os.Getenv("FAKE_CLAUDE_LOG"); logPath != "" { // Record this invocation's full argv for the test to inspect // afterward (the resume test's only way to see whether --resume @@ -29,15 +59,18 @@ func main() { } } - // Drain (don't require) exactly one stdin line — the real driver - // writes one turn message and closes stdin; reading it keeps this - // stand-in honest about the protocol without validating its content. - go func() { - scanner := bufio.NewScanner(os.Stdin) - for scanner.Scan() { - // discard - } - }() + if mode != "fast_no_drain" { + // Drain (don't require) exactly one stdin line — the real driver + // writes one turn message and closes stdin; reading it keeps this + // stand-in honest about the protocol without validating its + // content. + go func() { + scanner := bufio.NewScanner(os.Stdin) + for scanner.Scan() { + // discard + } + }() + } sessionID := os.Getenv("FAKE_CLAUDE_SESSION_ID") if sessionID == "" { @@ -51,13 +84,43 @@ func main() { out.Flush() } + if mode == "crash_before_init" { + // Exits nonzero WITHOUT ever emitting so much as a "system" event — + // the deterministic-startup-failure shape (an unknown flag, a + // malformed --mcp-config command, an invalid --model) that + // runClaudeCodeTurn's waitErr branch must NOT classify retryable, + // unlike "crash" below (which starts normally and dies later). + os.Exit(1) + } + emit(map[string]any{ "type": "system", "subtype": "init", "session_id": sessionID, }) - switch os.Getenv("FAKE_CLAUDE_MODE") { + switch mode { + case "fast_no_drain": + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Done before you finished writing."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Done before you finished writing.", + "usage": map[string]any{ + "input_tokens": 4, + "output_tokens": 6, + }, + }) + return case "hang": // No signal handlers installed: an unhandled SIGINT/SIGTERM uses // Go's own default disposition, which terminates the process — @@ -68,6 +131,14 @@ func main() { // race. time.Sleep(time.Hour) return + case "crash": + // Emitted "system"/"init" above, THEN exits nonzero without ever + // emitting a "result" event — the non-deterministic mid-session + // child-failure shape runClaudeCodeTurn's own waitErr branch must + // classify retryable (a crash, not a deterministic domain failure + // the CLI itself reported). Contrast crash_before_init above, + // which never gets this far. + os.Exit(1) case "error": emit(map[string]any{ "type": "result", @@ -80,6 +151,114 @@ func main() { }, }) return + case "rate_limit_error": + emit(map[string]any{ + "type": "result", + "subtype": "error_during_execution", + "is_error": true, + "result": "rate_limit_error: please retry later", + "usage": map[string]any{ + "input_tokens": 6, + "output_tokens": 1, + }, + }) + return + case "deterministic_error": + emit(map[string]any{ + "type": "result", + "subtype": "error_max_turns", + "is_error": true, + "result": "exceeded maximum turns", + "usage": map[string]any{ + "input_tokens": 8, + "output_tokens": 2, + }, + }) + return + case "thinking": + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Let me reason about this.", "signature": "sig-abc"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Here is my answer."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Here is my answer.", + "usage": map[string]any{ + "input_tokens": 20, + "output_tokens": 10, + }, + "ttft_ms": 120, + "duration_ms": 800, + }) + return + case "subagent": + emit(map[string]any{ + "type": "assistant", + "parent_tool_use_id": nil, + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_parent", "name": "Task", "input": map[string]any{}}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "parent_tool_use_id": "toolu_parent", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Working inside the subagent."}, + }, + }, + }) + emit(map[string]any{ + "type": "user", + "parent_tool_use_id": "toolu_parent", + "message": map[string]any{ + "role": "user", + "content": []map[string]any{ + {"type": "tool_result", "tool_use_id": "toolu_parent", "content": "subagent done"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "parent_tool_use_id": nil, + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "All done."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "All done.", + "usage": map[string]any{ + "input_tokens": 30, + "output_tokens": 15, + }, + }) + return } // Default ("normal"): assistant text, a tool_use/tool_result pair, @@ -133,5 +312,7 @@ func main() { "cache_creation_input_tokens": 5, }, "total_cost_usd": 0.0123, + "ttft_ms": 50, + "duration_ms": 400, }) } diff --git a/message/message.go b/message/message.go index e4df19a7..2483d075 100644 --- a/message/message.go +++ b/message/message.go @@ -86,6 +86,17 @@ type Message struct { // like any other user message, Origin untouched and untransmitted (no // transcoder reads this field). Origin string `json:"origin,omitempty"` + // ParentToolUseID identifies the tool_use call that spawned the Claude + // Code CLI subagent turn which produced this message — set only on a + // message OriginClaudeCode appended from a stream-json event whose own + // parent_tool_use_id was non-null (see engine/claude_code_backend.go's + // package doc, "parent_tool_use_id"). Empty for a top-level delegated + // turn's own messages, and for every message no delegated turn + // produced. Like Origin, this is presentation/lineage metadata only — + // read by a client (boxes' console) to reconstruct subagent nesting in + // a delegated turn's transcript — never sent to a model and never + // interpreted by a transcoder. + ParentToolUseID string `json:"parent_tool_use_id,omitempty"` } // Normalize scrubs known encoding/json footguns from m's parts in place. It From e5b72a54c2b269a3fc0c558c82a0975e970beb48 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Mon, 31 Aug 2026 10:16:35 -0400 Subject: [PATCH 29/95] Capture subscription rate-limit usage on the session (#218) * message,provider: add a normalized subscription-usage snapshot type Two lanes run a turn against a user's own model subscription instead of metered API access: the Claude Code CLI delegated backend, and a native OpenAI Responses provider pointed at the ChatGPT Codex backend. Both providers already send a rate-limit/quota signal on every turn, but nothing in harness captures or exposes it, so an orchestrator (e.g. boxes) has no way to show a user how much of their subscription window remains. This adds the shared foundation both lanes build on: message. SubscriptionUsage (plus its Window and Overage sub-types), a normalized shape any capturing lane maps its own wire format onto, and a SubscriptionUsage field on provider.Event, set only on EventDone by an adapter that captured the signal on that particular response. No adapter populates it yet - that lands in the next two commits. Verified: go build ./... && go vet ./... && gofmt -l . clean. * engine: capture Claude Code's rate_limit_event as subscription usage The `claude` CLI emits a rate_limit_event as (typically) the second stream-json message of every delegated turn, carrying the user's Claude subscription rate-limit/quota state - unifiedWindows keyed by window name (five_hour, seven_day), plus overage state. Before this change consumeClaudeCodeStream had no case for "rate_limit_event", so it fell through to the inert unknown-type default and the signal was simply dropped. Add a case that decodes rate_limit_info and maps it to message.SubscriptionUsage via the new mapClaudeCodeRateLimit: one window per unifiedWindows entry (sorted by key for byte-stable output), labeled "5-hour"/"Weekly" for the two keys a real CLI sends today; overage mapped straight from isUsingOverage/overageStatus/ overageResetsAt. Plan is left "" - the event carries no plan field, and shelling out to `claude auth status` just to learn one is out of scope for a pure capture-and-map change. The mapped snapshot is recorded on the session via the new Session.applySubscriptionUsage/ SubscriptionUsage methods (engine.go), process-local like lastSystem, not persisted: the CLI resends the signal on the session's very next delegated turn regardless. fakeclaude gains a "rate_limit_event" mode emitting the documented event shape ahead of a turn's final assistant text. Verified: new TestClaudeCodeRateLimitEventCapturesSubscriptionUsage drives a turn through the new fakeclaude mode and asserts Session.SubscriptionUsage() carries the mapped provider, windows, and overage - the same accessor GET /session reads. go test ./engine/... -race green. * provider/openai: capture Codex subscription-usage response headers The ChatGPT Codex backend (chatgpt.com/backend-api/codex/responses) answers every request with x-codex-* response headers carrying the user's Codex subscription rate-limit/quota state: plan type, a primary/secondary window pair (used-percent, window-minutes, reset-at), and a second, separately-named "bengalfox" pair riding alongside it. Nothing read these headers before this change, on either of this adapter's two transports. Add codexSubscriptionUsageFromHeaders (subscription_usage.go), gated by the new CodexFamily constant so only a client configured under the conventional "codex" providers-map key reads them - an ordinary "openai" entry never does. It maps the primary window (labeled "Weekly"/"5-hour"/"-min" by its own window-minutes), the bengalfox pair's primary window as a "bengalfox_primary" 5-hour window, and the secondary window when its window-minutes is positive (the documented capture's secondary is unused: window-minutes 0, reset-at empty, so it is dropped rather than emitted as a hollow zero-value entry). No Overage: the codex lane's headers carry no overage concept. Both of this adapter's transports carry the same headers, on different responses, so both are wired: - HTTP+SSE (openai.go): read off resp.Header in Client.Stream, attached to the stream's EventDone. - websocket (ws.go, ws_pool.go): dialResponsesWebSocket now returns coder/websocket's own upgrade *http.Response alongside the connection (previously discarded) so wsPool.stream can read the same headers off it. A pooled connection is reused across several turns without re-dialing, so the pool entry caches the snapshot from its own last dial and reuses it for every stream built on that connection, refreshed only on the next redial - the Codex backend sends these headers on the websocket upgrade response only, never on any later frame. Verified: new TestCodexSubscriptionUsageFromHeaders(NoSignal) unit- tests the mapping directly against the documented header capture; TestStreamCapturesCodexSubscriptionUsageOverHTTP proves the HTTP path captures it for a CodexFamily client and not for a plain "openai" one against the identical response; TestWebSocketTransportCapturesCodex- SubscriptionUsage proves the same for the websocket path, reading the upgrade response header. go test ./provider/openai/... -race green. * server: expose subscription_usage on GET /session Both subscription lanes now capture a normalized rate-limit/quota snapshot on the session (message.SubscriptionUsage, engine.Session. SubscriptionUsage), but nothing served it - an orchestrator like boxes had no way to read it back. Add sessionJSON.SubscriptionUsage, read from sess.SubscriptionUsage() in buildSession (the live/resident path). Deliberately not omitempty: it serializes as explicit `null` until a turn in this process has carried the signal, rather than being omitted, so a caller can unmarshal into a fixed struct without special-casing key presence. buildSessionFromIndex (the cold, non-resident path) has no durable source for it and reports null unconditionally - persisting this snapshot is out of scope for a pure capture-and-expose change; the provider resends the signal on the session's very next subscription turn regardless. Documented in server/openapi.yaml alongside the new SubscriptionUsage/SubscriptionUsageWindow/SubscriptionOverage schemas. Verified: new TestSessionSubscriptionUsageSurfacedOnGet drives two turns through a scripted provider (one with no SubscriptionUsage on its EventDone, one with one) and asserts GET /session's subscription_usage is null after the first and the mapped snapshot after the second. go test ./server/... -race green. * engine,provider/openai: address review - gate overage, doc ws staleness Review of the subscription-usage capture PR found one low-severity correctness nit and one limitation that needed to be stated plainly rather than left implicit. mapClaudeCodeRateLimit unconditionally allocated an Overage object for every rate_limit_event, so a turn with no overage in play emitted the zero-value noise object {"in_use":false,"status":"","resets_at":0} - contradicting message.SubscriptionUsage.Overage's own doc comment (nil/omitted when not applicable). Only allocate it when the event actually carries an overage signal: IsUsingOverage true, or a non-empty OverageStatus (a status can describe an overage state, e.g. "approaching", even while IsUsingOverage is still false). fakeclaude gains a "rate_limit_event_no_overage" mode (overageStatus "", isUsingOverage false, overageResetsAt 0) to cover the case the existing fixture's overageStatus:"allowed" could not exercise. wsPoolEntry.subUsage's doc comment now states the known staleness bound precisely: the Codex backend sends x-codex-* headers on the websocket upgrade response only, so a pooled connection's snapshot is refreshed only on redial, not per turn, and can lag real usage by up to idleTimeout (5m) or maxConnectionAge (55m) on an actively-reused connection - unlike the HTTP path, which re-reads fresh headers every request. This is a deliberate limitation (no periodic redial or separate query added): SubscriptionUsage.CapturedAt makes the staleness visible to any caller rendering the value. Also fixed the asymmetry Session.SubscriptionUsage() had between its two pointer/slice fields: it deep-copied Windows but shared the Overage pointer via the struct copy, so a caller mutating the returned snapshot's Overage would mutate the session's own stored copy. Now allocates a fresh *SubscriptionOverage too. Verified: go build ./... && go vet ./... && gofmt -l . clean; go test ./engine/... ./provider/... ./server/... ./message/... -race green, including the new TestClaudeCodeRateLimitEventWithNoOverageOmitsOverage. --- engine/claude_code_backend.go | 110 ++++++++++++ engine/claude_code_backend_test.go | 74 ++++++++ engine/engine.go | 58 +++++++ engine/testdata/fakeclaude/main.go | 59 ++++++- message/subscription_usage.go | 68 ++++++++ provider/openai/openai.go | 37 +++- provider/openai/subscription_usage.go | 145 ++++++++++++++++ provider/openai/subscription_usage_test.go | 193 +++++++++++++++++++++ provider/openai/ws.go | 17 +- provider/openai/ws_pool.go | 44 ++++- provider/provider.go | 9 + server/handlers.go | 55 ++++-- server/openapi.yaml | 83 ++++++++- server/subscription_usage_test.go | 98 +++++++++++ server/usage_test.go | 9 +- 15 files changed, 1016 insertions(+), 43 deletions(-) create mode 100644 message/subscription_usage.go create mode 100644 provider/openai/subscription_usage.go create mode 100644 provider/openai/subscription_usage_test.go create mode 100644 server/subscription_usage_test.go diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index c60086cb..b8434e49 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -91,6 +91,14 @@ // no home in provider.Usage (no adapter carries a cost field — every // consumer derives cost from token counts) and is deliberately // dropped, not persisted. +// - "rate_limit_event": the CLI's own subscription rate-limit/quota +// signal, typically the SECOND event of a turn (right after +// "system"/"init"). Never appended as a message — it carries no +// conversational content — but mapped via mapClaudeCodeRateLimit and +// applied to the session via applySubscriptionUsage (engine.go), +// process-local only, surfaced on GET /session as +// subscription_usage. See mapClaudeCodeRateLimit's own doc comment +// for the field-by-field mapping. // // # Formerly deferred, now closed // @@ -144,6 +152,7 @@ import ( "io" "os" "os/exec" + "sort" "strings" "syscall" "time" @@ -668,6 +677,16 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // TotalCostUSD is intentionally dropped here — see the // package doc's "result" bullet: provider.Usage has no cost // field for it to occupy. + case "rate_limit_event": + // The CLI's own subscription rate-limit/quota signal — see + // mapClaudeCodeRateLimit's own doc comment for the wire shape + // and mapping. Typically the SECOND event of a turn (right + // after "system"/"init"), but this file reacts to it whenever + // it arrives, and to every occurrence, not only the first: a + // long-running turn can see its own limits shift mid-turn. + if usage, ok := mapClaudeCodeRateLimit(env.RateLimitInfo); ok { + s.applySubscriptionUsage(usage) + } } // Any other top-level "type" (this driver has none documented // beyond the four above) is inert activity, per the package doc. @@ -709,6 +728,97 @@ type claudeCodeEnvelope struct { // decoding philosophy). TTFTMillis int64 `json:"ttft_ms,omitempty"` DurationMillis int64 `json:"duration_ms,omitempty"` + // RateLimitInfo is a "rate_limit_event" envelope's own payload — see + // mapClaudeCodeRateLimit. nil for every other event type. + RateLimitInfo *claudeCodeRateLimitInfo `json:"rate_limit_info,omitempty"` +} + +// claudeCodeRateLimitInfo is a "rate_limit_event" envelope's own +// rate_limit_info object — the `claude` CLI's subscription rate-limit/quota +// signal, typically the SECOND stream-json message of every turn. See +// mapClaudeCodeRateLimit for how this becomes message.SubscriptionUsage. +type claudeCodeRateLimitInfo struct { + Status string `json:"status,omitempty"` + ResetsAt int64 `json:"resetsAt,omitempty"` + RateLimitType string `json:"rateLimitType,omitempty"` + OverageStatus string `json:"overageStatus,omitempty"` + OverageResetsAt int64 `json:"overageResetsAt,omitempty"` + IsUsingOverage bool `json:"isUsingOverage,omitempty"` + UnifiedWindows map[string]claudeCodeRateLimitWindow `json:"unifiedWindows,omitempty"` +} + +// claudeCodeRateLimitWindow is one entry of a claudeCodeRateLimitInfo's own +// unifiedWindows map — one rate-limit window (e.g. "five_hour", +// "seven_day"), keyed by the CLI's own window name. +type claudeCodeRateLimitWindow struct { + Utilization float64 `json:"utilization"` + ResetsAt int64 `json:"resetsAt"` +} + +// claudeCodeRateLimitWindowLabel maps a unifiedWindows key to the human +// label message.SubscriptionUsageWindow.Label reports — the two keys a real +// `claude` binary sends today. An unrecognized key (a future CLI addition +// this file has not seen) falls back to the key itself: an honest label +// beats a hardcoded guess for a window this file cannot yet name. +func claudeCodeRateLimitWindowLabel(key string) string { + switch key { + case "five_hour": + return "5-hour" + case "seven_day": + return "Weekly" + default: + return key + } +} + +// mapClaudeCodeRateLimit converts a "rate_limit_event" envelope's own +// rate_limit_info object into message.SubscriptionUsage: provider "claude"; +// Plan left "" (the CLI's event carries no plan field, and this file does +// not shell out to `claude auth status` just to learn one — see this +// package's own CONSTRAINTS); one window per unifiedWindows entry, sorted +// by key for byte-stable output across turns (map iteration order is not); +// Overage is set only when the event actually carries one — IsUsingOverage +// true, or a non-empty OverageStatus (a status can legitimately describe an +// overage state, e.g. "approaching", even while IsUsingOverage is still +// false) — so a turn with no overage in play leaves it nil, matching +// message.SubscriptionUsage.Overage's own doc comment (omitted on the +// wire, never a hollow zero-value object). CapturedAt is left zero — +// applySubscriptionUsage stamps it from s.cfg.Now(), the single clock +// every consumer of Session.SubscriptionUsage sees. ok is false for a nil +// info (a rate_limit_event with no rate_limit_info at all — not expected +// from a real `claude` binary, but this file decodes permissively +// throughout, per its own package doc). +func mapClaudeCodeRateLimit(info *claudeCodeRateLimitInfo) (message.SubscriptionUsage, bool) { + if info == nil { + return message.SubscriptionUsage{}, false + } + keys := make([]string, 0, len(info.UnifiedWindows)) + for k := range info.UnifiedWindows { + keys = append(keys, k) + } + sort.Strings(keys) + windows := make([]message.SubscriptionUsageWindow, 0, len(keys)) + for _, k := range keys { + w := info.UnifiedWindows[k] + windows = append(windows, message.SubscriptionUsageWindow{ + Key: k, + Label: claudeCodeRateLimitWindowLabel(k), + UsedPercent: w.Utilization * 100, + ResetsAt: w.ResetsAt, + }) + } + out := message.SubscriptionUsage{ + Provider: "claude", + Windows: windows, + } + if info.IsUsingOverage || info.OverageStatus != "" { + out.Overage = &message.SubscriptionOverage{ + InUse: info.IsUsingOverage, + Status: info.OverageStatus, + ResetsAt: info.OverageResetsAt, + } + } + return out, true } // claudeCodeUsage is a "result" event's usage object. diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index d42cf091..68629d07 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -700,6 +700,80 @@ func TestClaudeCodeTurnMetricsEmittedForDelegatedTurn(t *testing.T) { } } +// TestClaudeCodeRateLimitEventCapturesSubscriptionUsage drives a turn +// through fakeclaude's "rate_limit_event" mode (a rate_limit_event ahead of +// the turn's final assistant text — the CLI's own documented ordering, see +// this file's package doc) and asserts Session.SubscriptionUsage() — +// exactly what buildSession (server/handlers.go) reads for GET /session's +// subscription_usage field — carries the mapped provider, windows, and +// overage. +func TestClaudeCodeRateLimitEventCapturesSubscriptionUsage(t *testing.T) { + s, _ := claudeCodeTestSession(t, "rate_limit_event") + + if got := s.SubscriptionUsage(); got != nil { + t.Fatalf("SubscriptionUsage() before any turn = %+v, want nil", got) + } + + if _, err := s.Prompt(context.Background(), "how am I doing on quota?"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + usage := s.SubscriptionUsage() + if usage == nil { + t.Fatal("SubscriptionUsage() after the turn = nil, want a captured snapshot") + } + if usage.Provider != "claude" { + t.Errorf("Provider = %q, want claude", usage.Provider) + } + if usage.Plan != "" { + t.Errorf("Plan = %q, want \"\" (not cheaply available from rate_limit_event)", usage.Plan) + } + if usage.CapturedAt == 0 { + t.Error("CapturedAt = 0, want a stamped Unix timestamp") + } + if len(usage.Windows) != 2 { + t.Fatalf("Windows = %+v, want 2 entries", usage.Windows) + } + // Sorted by key (mapClaudeCodeRateLimit): "five_hour" before "seven_day". + fh, sd := usage.Windows[0], usage.Windows[1] + if fh.Key != "five_hour" || fh.Label != "5-hour" || fh.UsedPercent != 2 || fh.ResetsAt != 1788785267 { + t.Errorf("Windows[0] = %+v, want {five_hour 5-hour 2 1788785267}", fh) + } + if sd.Key != "seven_day" || sd.Label != "Weekly" || sd.UsedPercent != 13 || sd.ResetsAt != 1789200000 { + t.Errorf("Windows[1] = %+v, want {seven_day Weekly 13 1789200000}", sd) + } + if usage.Overage == nil { + t.Fatal("Overage = nil, want a mapped overage object") + } + if usage.Overage.InUse || usage.Overage.Status != "allowed" || usage.Overage.ResetsAt != 1789000000 { + t.Errorf("Overage = %+v, want {false allowed 1789000000}", usage.Overage) + } +} + +// TestClaudeCodeRateLimitEventWithNoOverageOmitsOverage proves +// mapClaudeCodeRateLimit leaves SubscriptionUsage.Overage nil — omitted on +// the wire, per message.SubscriptionUsage.Overage's own doc comment — +// when a rate_limit_event's own overage fields report no overage in play +// at all (overageStatus "", isUsingOverage false, overageResetsAt 0), the +// counterpart to TestClaudeCodeRateLimitEventCapturesSubscriptionUsage +// above, whose fixture's overageStatus "allowed" is itself a real (if +// benign) overage signal and so is a case that test cannot cover. +func TestClaudeCodeRateLimitEventWithNoOverageOmitsOverage(t *testing.T) { + s, _ := claudeCodeTestSession(t, "rate_limit_event_no_overage") + + if _, err := s.Prompt(context.Background(), "how am I doing on quota?"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + usage := s.SubscriptionUsage() + if usage == nil { + t.Fatal("SubscriptionUsage() after the turn = nil, want a captured snapshot") + } + if usage.Overage != nil { + t.Errorf("Overage = %+v, want nil (no overage in play)", usage.Overage) + } +} + // TestClaudeCodeRetryableClassification proves a "result" event this file // can actually name as transient provider weather (a rate-limit signal) is // wrapped provider.RetryableError, while a genuinely deterministic result diff --git a/engine/engine.go b/engine/engine.go index 2e90a171..3635f30f 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -967,6 +967,18 @@ type Session struct { lastUsage provider.Usage haveLastUsage bool + // subscriptionUsage is this session's most recently captured + // subscription-lane rate-limit/quota snapshot (see + // message.SubscriptionUsage's own doc comment for what captures one + // and why) — nil until a turn in THIS process has carried the signal. + // Deliberately process-local only, like lastSystem/committedOutcome + // below: a per-process latest-value cache, not folded into cumulative + // state and not replayed by LoadSession. GET /session reports null + // rather than a stale value from a prior process, which is honest — + // the provider will resend the signal on this session's very next + // delegated/subscription turn regardless. + subscriptionUsage *message.SubscriptionUsage + // turnUnsettled is SessionManager.recoverInterruptedTurnLocked's // restart-recovery signal, replacing an earlier, unreliable // heuristic (hasUnansweredTurn, since removed) that tried to infer @@ -2002,6 +2014,45 @@ func (s *Session) LastUsage() (usage provider.Usage, ok bool) { return s.lastUsage, s.haveLastUsage } +// applySubscriptionUsage records u as this session's latest subscription- +// usage snapshot (see message.SubscriptionUsage's own doc comment) — the +// single choke point both subscription lanes go through: engine/ +// claude_code_backend.go's consumeClaudeCodeStream for a "rate_limit_event", +// and streamTurn's EventDone case for a codex-family provider.Event that +// carried one. CapturedAt is always stamped here from s.cfg.Now(), not +// trusted from the caller, so every consumer sees a harness-clock +// timestamp regardless of which lane captured the underlying signal. +func (s *Session) applySubscriptionUsage(u message.SubscriptionUsage) { + u.CapturedAt = s.cfg.Now().Unix() + s.mu.Lock() + defer s.mu.Unlock() + s.subscriptionUsage = &u +} + +// SubscriptionUsage returns this session's most recently captured +// subscription-usage snapshot (see applySubscriptionUsage), or nil if no +// turn in this process has carried the signal yet — see +// subscriptionUsage's own doc comment for why this never falls back to a +// durable source. +func (s *Session) SubscriptionUsage() *message.SubscriptionUsage { + s.mu.Lock() + defer s.mu.Unlock() + if s.subscriptionUsage == nil { + return nil + } + cp := *s.subscriptionUsage + cp.Windows = append([]message.SubscriptionUsageWindow(nil), s.subscriptionUsage.Windows...) + if s.subscriptionUsage.Overage != nil { + // Deep-copy Overage too, not just Windows: the struct copy above + // (cp := *s.subscriptionUsage) only copies the pointer value, so + // without this a caller mutating the returned snapshot's Overage + // would mutate this session's own stored one. + overage := *s.subscriptionUsage.Overage + cp.Overage = &overage + } + return &cp +} + // LastActivityAt returns the timestamp of the most recently appended // message (user, assistant, or tool), or CreatedAt if no message has been // appended yet. @@ -2932,6 +2983,13 @@ func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message SystemLen: len(strings.Join(system, "\n")), ToolsCount: len(tools), }) + if ev.SubscriptionUsage != nil { + // See provider.Event.SubscriptionUsage's own doc comment: + // only a subscription-lane adapter (provider/openai's codex + // family today) ever sets this, and only when its own + // response actually carried the signal. + s.applySubscriptionUsage(*ev.SubscriptionUsage) + } return ev.Message, ev.StopReason, ev.Usage, nil } } diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 6a0f7a93..99bdf853 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -16,9 +16,15 @@ // respectively), "crash"/"crash_before_init" (a child that exits nonzero // with no "result" event, AFTER vs. BEFORE ever emitting a "system" event // — runClaudeCodeTurn's waitErr branch must classify only the former -// retryable), and "fast_no_drain" (closes its own stdin immediately, +// retryable), "fast_no_drain" (closes its own stdin immediately, // without ever draining it, to reliably win the race against harness's -// own turn-input write — see runClaudeCodeTurn's inputErr handling). +// own turn-input write — see runClaudeCodeTurn's inputErr handling), and +// "rate_limit_event" (a subscription rate-limit/quota event ahead of the +// final assistant text, mapped by mapClaudeCodeRateLimit onto +// Session.SubscriptionUsage), and "rate_limit_event_no_overage" (the same +// event with no overage in play at all — overageStatus "", isUsingOverage +// false, overageResetsAt 0 — proving mapClaudeCodeRateLimit leaves +// SubscriptionUsage.Overage nil rather than a hollow zero-value object). package main import ( @@ -261,6 +267,55 @@ func main() { return } + if mode == "rate_limit_event" || mode == "rate_limit_event_no_overage" { + info := map[string]any{ + "status": "allowed", + "resetsAt": 1788785267, + "rateLimitType": "five_hour", + "overageStatus": "allowed", + "overageResetsAt": 1789000000, + "isUsingOverage": false, + "unifiedWindows": map[string]any{ + "five_hour": map[string]any{"utilization": 0.02, "resetsAt": 1788785267}, + "seven_day": map[string]any{"utilization": 0.13, "resetsAt": 1789200000}, + }, + } + if mode == "rate_limit_event_no_overage" { + // No overage in play at all — overageStatus empty, + // isUsingOverage false, overageResetsAt 0 — the shape + // mapClaudeCodeRateLimit must leave Overage nil for (see + // TestClaudeCodeRateLimitEventWithNoOverageOmitsOverage), unlike + // the "rate_limit_event" mode above, whose overageStatus: + // "allowed" is itself a real (if benign) overage signal. + info["overageStatus"] = "" + info["overageResetsAt"] = 0 + } + emit(map[string]any{ + "type": "rate_limit_event", + "rate_limit_info": info, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Here is my answer."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Here is my answer.", + "usage": map[string]any{ + "input_tokens": 9, + "output_tokens": 4, + }, + }) + return + } + // Default ("normal"): assistant text, a tool_use/tool_result pair, // then a final assistant text and result — one of each event shape // engine/claude_code_backend.go documents mapping. diff --git a/message/subscription_usage.go b/message/subscription_usage.go new file mode 100644 index 00000000..92335c3d --- /dev/null +++ b/message/subscription_usage.go @@ -0,0 +1,68 @@ +package message + +// SubscriptionUsage is a normalized snapshot of a subscription-backed +// session's provider-reported rate-limit/quota signal, captured — not +// polled: nothing that produces one makes an extra outbound request for it. +// It exists for the two lanes that run a turn against a user's own model +// subscription rather than metered API access: +// +// - "claude": engine/claude_code_backend.go decodes it from the `claude` +// CLI's own rate_limit_event stream-json message. +// - "codex": provider/openai decodes it from the ChatGPT Codex backend's +// x-codex-* response headers (HTTP and websocket transports alike). +// +// A session that has never delegated a turn through either lane, or has +// but whose first turn has not yet completed in this process, has no +// SubscriptionUsage — see engine.Session.SubscriptionUsage's own doc +// comment for why this is a process-local snapshot, not a durable record. +type SubscriptionUsage struct { + // Provider names which lane captured this snapshot: "claude" or + // "codex" — see this type's own doc comment. + Provider string `json:"provider"` + // Plan is the subscription tier ("max"/"pro" for codex, from its own + // x-codex-plan-type header). Empty when the capturing lane has no + // cheap source for it — the claude lane's rate_limit_event carries no + // plan field of its own, and this package does not shell out to + // `claude auth status` just to learn one. + Plan string `json:"plan"` + // Windows is one entry per rate-limit window the provider reported on + // this turn, in the provider's own order. Never nil when a + // SubscriptionUsage exists, even if empty. + Windows []SubscriptionUsageWindow `json:"windows"` + // Overage describes a pay-as-you-go overage state riding on top of the + // subscription (claude's rate_limit_event only — the codex lane's + // x-codex-* headers carry no overage concept, so this is always nil + // for provider "codex"). nil (omitted on the wire) when not + // applicable. + Overage *SubscriptionOverage `json:"overage,omitempty"` + // CapturedAt is when this snapshot was captured — harness's own clock + // (Config.Now), not a provider-reported time — Unix seconds. + CapturedAt int64 `json:"captured_at"` +} + +// SubscriptionUsageWindow is one rate-limit window inside a +// SubscriptionUsage snapshot — e.g. claude's "five_hour"/"seven_day", or +// codex's "primary"/"secondary"/"bengalfox_primary". +type SubscriptionUsageWindow struct { + // Key is the capturing lane's own stable window identifier. A caller + // tracking one particular window turn over turn keys off this, not + // Label. + Key string `json:"key"` + // Label is a short human-readable name for the window (e.g. "5-hour", + // "Weekly"), derived by the capturing lane — neither provider sends a + // label on the wire. + Label string `json:"label"` + // UsedPercent is this window's utilization, 0-100. + UsedPercent float64 `json:"used_percent"` + // ResetsAt is when this window resets, Unix seconds. + ResetsAt int64 `json:"resets_at"` +} + +// SubscriptionOverage describes a subscription's pay-as-you-go overage +// state — see SubscriptionUsage.Overage's own doc comment for why this is +// claude-only today. +type SubscriptionOverage struct { + InUse bool `json:"in_use"` + Status string `json:"status"` + ResetsAt int64 `json:"resets_at"` +} diff --git a/provider/openai/openai.go b/provider/openai/openai.go index 4df4ec7e..80f5207b 100644 --- a/provider/openai/openai.go +++ b/provider/openai/openai.go @@ -188,13 +188,27 @@ func (c *Client) Stream(ctx context.Context, req *provider.Request) (provider.St return nil, apiError(resp) } return &stream{ - body: resp.Body, - r: bufio.NewReader(resp.Body), - model: req.Model, - family: c.family(), + body: resp.Body, + r: bufio.NewReader(resp.Body), + model: req.Model, + family: c.family(), + subUsage: c.codexSubscriptionUsage(resp.Header), }, nil } +// codexSubscriptionUsage reads h for the x-codex-* subscription-usage +// headers (see codexSubscriptionUsageFromHeaders), but only for a client +// configured under CodexFamily — see that constant's own doc comment for +// why family is the gate: an ordinary "openai" entry never even looks at +// these headers, whether or not a proxy in front of it happens to echo +// some of the same header names. +func (c *Client) codexSubscriptionUsage(h http.Header) *message.SubscriptionUsage { + if c.family() != CodexFamily { + return nil + } + return codexSubscriptionUsageFromHeaders(h) +} + // responsesURL joins a base URL and a request path, applying each field's // default and normalizing the separator between them to exactly one slash. // @@ -313,6 +327,12 @@ type stream struct { items []*assembledItem usage provider.Usage hasToolCall bool + // subUsage is this response's captured subscription-usage snapshot — + // set only for a CodexFamily client (see Client.codexSubscriptionUsage + // and wsPool.stream, the two sources), nil otherwise. Carried onto the + // EventDone event queued in the "response.completed"/"response. + // incomplete" case below. + subUsage *message.SubscriptionUsage queue []provider.Event done bool @@ -576,10 +596,11 @@ func (s *stream) handle(name string, data []byte) error { stop = provider.StopEndTurn } s.queue = append(s.queue, provider.Event{ - Type: provider.EventDone, - Message: s.assemble(), - StopReason: stop, - Usage: s.usage, + Type: provider.EventDone, + Message: s.assemble(), + StopReason: stop, + Usage: s.usage, + SubscriptionUsage: s.subUsage, }) s.done = true diff --git a/provider/openai/subscription_usage.go b/provider/openai/subscription_usage.go new file mode 100644 index 00000000..50945ecb --- /dev/null +++ b/provider/openai/subscription_usage.go @@ -0,0 +1,145 @@ +package openai + +import ( + "net/http" + "strconv" + + "github.com/majorcontext/harness/message" +) + +// CodexFamily is the conventional provider/openai Client.Family value for a +// providers-map entry that speaks the ChatGPT Codex backend's Responses +// wire (chatgpt.com/backend-api/codex/responses) — see cmd/harness's +// registerOpenAIProviders, where a config.TypeOpenAI entry's Family is set +// to its own providers-map key. Only a client whose resolved family equals +// this constant captures the x-codex-* response headers below (see +// Client.Stream and wsPool.stream); an ordinary "openai" entry never reads +// or reports them. +// +// This is a naming convention, not something buildsResponsesAdapter or any +// other config validation enforces — the same "the operator's own key IS +// the signal" precedent engine.ClaudeCodeProviderFamily documents for the +// Claude Code delegated backend, applied here because nothing in a +// provider.Request or an HTTP response can otherwise tell this package +// "this endpoint is the ChatGPT Codex backend" without adding a dedicated +// config field for a single conventionally-named deployment. +const CodexFamily = "codex" + +// codexWindowLabel maps an x-codex-*-window-minutes value to the human +// label message.SubscriptionUsageWindow.Label reports. 10080 (7 days) and +// 300 (5 hours) are the two windows a real Codex backend sends today; +// anything else falls back to "-min" rather than a hardcoded +// guess for a window this file has not seen. +func codexWindowLabel(minutes int64) string { + switch minutes { + case 10080: + return "Weekly" + case 300: + return "5-hour" + default: + return strconv.FormatInt(minutes, 10) + "-min" + } +} + +// codexHeaderFloat parses header key h.Get(key) as a float64, returning +// ok=false for an absent or unparseable value — the same permissive- +// decoding posture engine/claude_code_backend.go takes with the sibling +// subscription lane: a header this file cannot parse is treated as absent, +// never a hard failure. +func codexHeaderFloat(h http.Header, key string) (float64, bool) { + v := h.Get(key) + if v == "" { + return 0, false + } + f, err := strconv.ParseFloat(v, 64) + if err != nil { + return 0, false + } + return f, true +} + +// codexHeaderInt is codexHeaderFloat's integer twin, used for the +// window-minutes and reset-at (Unix seconds) headers. +func codexHeaderInt(h http.Header, key string) (int64, bool) { + v := h.Get(key) + if v == "" { + return 0, false + } + n, err := strconv.ParseInt(v, 10, 64) + if err != nil { + return 0, false + } + return n, true +} + +// codexWindow reads one x-codex--{used-percent,window-minutes, +// reset-at} header trio into a message.SubscriptionUsageWindow keyed +// exactly as prefix. ok is false when window-minutes is absent, unparsable, +// or not positive — the same "window-minutes>0" presence gate for every +// window this file reads, primary/secondary/bengalfox-primary alike: a +// window a real Codex response reports as 0-minute-wide (e.g. the example +// capture's unused x-codex-secondary-window-minutes: 0) is not in use and +// must not appear as a hollow zero-value entry. +func codexWindow(prefix string, h http.Header) (message.SubscriptionUsageWindow, bool) { + minutes, ok := codexHeaderInt(h, "x-codex-"+prefix+"-window-minutes") + if !ok || minutes <= 0 { + return message.SubscriptionUsageWindow{}, false + } + used, _ := codexHeaderFloat(h, "x-codex-"+prefix+"-used-percent") + resetsAt, _ := codexHeaderInt(h, "x-codex-"+prefix+"-reset-at") + return message.SubscriptionUsageWindow{ + Key: prefix, + Label: codexWindowLabel(minutes), + UsedPercent: used, + ResetsAt: resetsAt, + }, true +} + +// codexSubscriptionUsageFromHeaders maps the ChatGPT Codex backend's +// x-codex-* response headers into message.SubscriptionUsage — present on +// every chatgpt.com/backend-api/codex/responses reply, HTTP response +// headers or the websocket upgrade response's own header alike (see +// Client.Stream and ws_pool.go's stream, the two callers). Windows, in +// order: "primary" (the plan's own primary window — Weekly at 10080 +// minutes in the documented capture); "bengalfox_primary" (a second, +// separately-named 5-hour+weekly bucket riding alongside the plan windows — +// only its primary/5-hour window is captured); "secondary", when its own +// window-minutes is positive (the documented capture's secondary is +// unused: window-minutes 0, reset-at empty). +// +// Overage is never set: the codex lane's headers carry no overage concept +// (credits are a separate, out-of-scope system — see this file's own +// CONSTRAINTS). Returns nil when the response carries neither a plan nor +// any window at all — a "codex"-family client that reached a plain, +// non-Codex OpenAI-compatible endpoint by misconfiguration, or an older +// backend build that has not shipped these headers yet — so a caller only +// ever applies a genuinely captured signal, never a hollow zero-value one. +// +// Freshness differs by caller: the HTTP path (Client.codexSubscriptionUsage) +// calls this on every single request, so its result is always current as +// of that turn. The websocket path (wsPool.stream) calls this only when +// its pooled connection is dialed, NOT on every turn the connection then +// serves — see wsPoolEntry.subUsage's own doc comment for the resulting +// staleness bound on an actively-reused connection. +func codexSubscriptionUsageFromHeaders(h http.Header) *message.SubscriptionUsage { + plan := h.Get("x-codex-plan-type") + windows := []message.SubscriptionUsageWindow{} + if w, ok := codexWindow("primary", h); ok { + windows = append(windows, w) + } + if w, ok := codexWindow("bengalfox-primary", h); ok { + w.Key = "bengalfox_primary" + windows = append(windows, w) + } + if w, ok := codexWindow("secondary", h); ok { + windows = append(windows, w) + } + if plan == "" && len(windows) == 0 { + return nil + } + return &message.SubscriptionUsage{ + Provider: "codex", + Plan: plan, + Windows: windows, + } +} diff --git a/provider/openai/subscription_usage_test.go b/provider/openai/subscription_usage_test.go new file mode 100644 index 00000000..73f32e5b --- /dev/null +++ b/provider/openai/subscription_usage_test.go @@ -0,0 +1,193 @@ +package openai + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "testing" + + "github.com/coder/websocket" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// codexHeaders is the documented capture this feature is built against: +// the ChatGPT Codex backend's x-codex-* response headers on +// chatgpt.com/backend-api/codex/responses, plan windows plus an unused +// secondary window plus a bengalfox 5-hour+weekly pair. +func codexHeaders() http.Header { + h := http.Header{} + h.Set("x-codex-plan-type", "pro") + h.Set("x-codex-primary-used-percent", "0") + h.Set("x-codex-primary-window-minutes", "10080") + h.Set("x-codex-primary-reset-at", "1788785267") + h.Set("x-codex-secondary-used-percent", "0") + h.Set("x-codex-secondary-window-minutes", "0") + h.Set("x-codex-secondary-reset-at", "") + h.Set("x-codex-bengalfox-primary-used-percent", "12.5") + h.Set("x-codex-bengalfox-primary-window-minutes", "300") + h.Set("x-codex-bengalfox-primary-reset-at", "1788700000") + h.Set("x-codex-bengalfox-secondary-used-percent", "3") + h.Set("x-codex-bengalfox-secondary-window-minutes", "10080") + h.Set("x-codex-bengalfox-secondary-reset-at", "1789200000") + return h +} + +// TestCodexSubscriptionUsageFromHeaders proves the header-> +// message.SubscriptionUsage mapping the documented capture demands: +// plan from x-codex-plan-type; a "primary" window labeled "Weekly" (10080 +// minutes); a "bengalfox_primary" window labeled "5-hour" (300 minutes); +// the unused "secondary" window (window-minutes 0) dropped, not emitted as +// a hollow zero-value entry; no Overage (codex has none). +func TestCodexSubscriptionUsageFromHeaders(t *testing.T) { + got := codexSubscriptionUsageFromHeaders(codexHeaders()) + if got == nil { + t.Fatal("codexSubscriptionUsageFromHeaders = nil, want a captured snapshot") + } + if got.Provider != "codex" { + t.Errorf("Provider = %q, want codex", got.Provider) + } + if got.Plan != "pro" { + t.Errorf("Plan = %q, want pro", got.Plan) + } + if got.Overage != nil { + t.Errorf("Overage = %+v, want nil (codex has no overage concept)", got.Overage) + } + if len(got.Windows) != 2 { + t.Fatalf("Windows = %+v, want 2 entries (secondary must be dropped)", got.Windows) + } + primary, bengal := got.Windows[0], got.Windows[1] + if primary.Key != "primary" || primary.Label != "Weekly" || primary.UsedPercent != 0 || primary.ResetsAt != 1788785267 { + t.Errorf("Windows[0] = %+v, want {primary Weekly 0 1788785267}", primary) + } + if bengal.Key != "bengalfox_primary" || bengal.Label != "5-hour" || bengal.UsedPercent != 12.5 || bengal.ResetsAt != 1788700000 { + t.Errorf("Windows[1] = %+v, want {bengalfox_primary 5-hour 12.5 1788700000}", bengal) + } +} + +// TestCodexSubscriptionUsageFromHeadersNoSignal proves an ordinary, +// non-Codex response (no x-codex-* headers at all) maps to nil, not a +// hollow zero-value snapshot. +func TestCodexSubscriptionUsageFromHeadersNoSignal(t *testing.T) { + if got := codexSubscriptionUsageFromHeaders(http.Header{}); got != nil { + t.Errorf("codexSubscriptionUsageFromHeaders(no headers) = %+v, want nil", got) + } +} + +// codexStreamFixture is a minimal complete Responses SSE turn — enough to +// reach response.completed and queue EventDone, the only event this +// feature attaches SubscriptionUsage to. +var codexStreamFixture = sse("response.completed", `{"type":"response.completed","response":{"id":"resp_codex_1","usage":{"input_tokens":5,"output_tokens":2}}}`) + +// TestStreamCapturesCodexSubscriptionUsageOverHTTP proves the HTTP+SSE path +// (Client.Stream, openai.go) reads x-codex-* response headers and attaches +// the mapped message.SubscriptionUsage to the turn's EventDone — but ONLY +// for a client configured under CodexFamily; an ordinary "openai"-family +// client talking to the exact same headers must not. +func TestStreamCapturesCodexSubscriptionUsageOverHTTP(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + for k, vs := range codexHeaders() { + for _, v := range vs { + w.Header().Add(k, v) + } + } + w.Header().Set("Content-Type", "text/event-stream") + io.WriteString(w, codexStreamFixture) //nolint:errcheck + })) + t.Cleanup(srv.Close) + + t.Run("codex family captures it", func(t *testing.T) { + c := &Client{APIKey: "k", BaseURL: srv.URL, Family: CodexFamily} + s, err := c.Stream(context.Background(), &provider.Request{ + Model: message.ModelRef{Provider: CodexFamily, Model: "gpt-5"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + }) + if err != nil { + t.Fatal(err) + } + defer s.Close() + done := lastDoneEvent(t, s) + if done.SubscriptionUsage == nil { + t.Fatal("SubscriptionUsage = nil, want a captured snapshot") + } + if done.SubscriptionUsage.Provider != "codex" || done.SubscriptionUsage.Plan != "pro" { + t.Errorf("SubscriptionUsage = %+v", done.SubscriptionUsage) + } + }) + + t.Run("plain openai family does not capture it", func(t *testing.T) { + c := &Client{APIKey: "k", BaseURL: srv.URL} + s, err := c.Stream(context.Background(), &provider.Request{ + Model: message.ModelRef{Provider: Family, Model: "gpt-5"}, + Messages: []message.Message{{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}}, + }) + if err != nil { + t.Fatal(err) + } + defer s.Close() + done := lastDoneEvent(t, s) + if done.SubscriptionUsage != nil { + t.Errorf("SubscriptionUsage = %+v, want nil (not a codex-family client)", done.SubscriptionUsage) + } + }) +} + +// TestWebSocketTransportCapturesCodexSubscriptionUsage proves the websocket +// path (ws.go/ws_pool.go) reads the SAME x-codex-* headers off the upgrade +// RESPONSE (coder/websocket's own Dial return, not any frame on the wire) +// and attaches them to EventDone identically to the HTTP path above. +func TestWebSocketTransportCapturesCodexSubscriptionUsage(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + for k, vs := range codexHeaders() { + for _, v := range vs { + w.Header().Add(k, v) + } + } + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.Close(websocket.StatusNormalClosure, "") + if _, _, err := conn.Read(r.Context()); err != nil { + return + } + for _, f := range wsCannedFrames { + if err := conn.Write(context.Background(), websocket.MessageText, []byte(f)); err != nil { + return + } + } + })) + t.Cleanup(srv.Close) + + c := &Client{APIKey: "k", BaseURL: srv.URL, Family: CodexFamily, UseWebSocketTransport: true} + s, err := c.Stream(context.Background(), wsRequest("sess-subusage")) + if err != nil { + t.Fatalf("Stream: %v", err) + } + defer s.Close() + done := lastDoneEvent(t, s) + if done.SubscriptionUsage == nil { + t.Fatal("SubscriptionUsage = nil, want a captured snapshot from the upgrade response header") + } + if done.SubscriptionUsage.Provider != "codex" || done.SubscriptionUsage.Plan != "pro" { + t.Errorf("SubscriptionUsage = %+v", done.SubscriptionUsage) + } + if len(done.SubscriptionUsage.Windows) != 2 { + t.Errorf("Windows = %+v, want 2 entries", done.SubscriptionUsage.Windows) + } +} + +// lastDoneEvent drains s and returns its EventDone, failing the test if +// none arrived. +func lastDoneEvent(t *testing.T, s provider.Stream) *provider.Event { + t.Helper() + for _, ev := range collect(t, s) { + if ev.Type == provider.EventDone { + cp := ev + return &cp + } + } + t.Fatal("no EventDone") + return nil +} diff --git a/provider/openai/ws.go b/provider/openai/ws.go index dfcb4214..e8cfa366 100644 --- a/provider/openai/ws.go +++ b/provider/openai/ws.go @@ -47,7 +47,16 @@ func toWebSocketURL(rawURL string) string { // client is already configured with, identically to every HTTP request // this adapter makes. There is no separate proxy/CA plumbing to add here; // reusing httpClient IS the proxy/CA support. -func dialResponsesWebSocket(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, error) { +// dialResponsesWebSocket's second return value is the raw HTTP upgrade +// response — coder/websocket's own Dial return, non-nil on a successful +// upgrade (a normal 101 Switching Protocols) and often non-nil even on a +// failed one (a rejected upgrade the server answered with an ordinary HTTP +// error). wsPool.stream reads its Header off this for the x-codex-* +// subscription-usage headers (see codexSubscriptionUsageFromHeaders) — +// the Codex backend sends them on this same upgrade response, not inside +// any websocket frame, so there is no other point in this transport where +// they are ever visible. +func dialResponsesWebSocket(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, *http.Response, error) { dialCtx := ctx var cancel context.CancelFunc if timeout > 0 { @@ -67,12 +76,12 @@ func dialResponsesWebSocket(ctx context.Context, url string, headers http.Header // stripping it keeps the outgoing header set honest. hdr.Del("Content-Length") - conn, _, err := websocket.Dial(dialCtx, url, &websocket.DialOptions{ + conn, resp, err := websocket.Dial(dialCtx, url, &websocket.DialOptions{ HTTPClient: httpClient, HTTPHeader: hdr, }) if err != nil { - return nil, fmt.Errorf("openai: websocket dial: %w", err) + return nil, resp, fmt.Errorf("openai: websocket dial: %w", err) } // A single oversized frame (tool output, a huge pasted file) must not // silently kill the connection: without raising this, coder/websocket's @@ -82,7 +91,7 @@ func dialResponsesWebSocket(ctx context.Context, url string, headers http.Header // documented per-request body cap, so nothing legitimate is still cut // short. conn.SetReadLimit(64 << 20) - return conn, nil + return conn, resp, nil } // sendResponseCreate frames body (the same JSON the HTTP path POSTs, diff --git a/provider/openai/ws_pool.go b/provider/openai/ws_pool.go index dfa89361..3b64fa07 100644 --- a/provider/openai/ws_pool.go +++ b/provider/openai/ws_pool.go @@ -40,6 +40,26 @@ type wsPoolEntry struct { busy bool fallback bool // permanent: this session never uses ws again streamFailures int + // subUsage is the subscription-usage snapshot captured off conn's own + // dial (upgrade response) headers — see codexSubscriptionUsageFromHeaders + // and dialResponsesWebSocket's doc comment. Only ever set for a + // CodexFamily request (see stream below); nil otherwise. + // + // KNOWN STALENESS: the Codex backend sends these headers on the + // websocket upgrade response only, never on any later frame, so this + // is refreshed exclusively when conn itself is re-dialed — NOT on + // every turn a reused connection serves. On an actively-reused + // connection, the value can therefore lag the account's real usage by + // up to the connection's own reuse window: idleTimeout (5 minutes of + // no traffic invalidates it) or maxConnectionAge (55 minutes, whichever + // comes first — see wsPool's own tunables above). Contrast the HTTP + // path (Client.codexSubscriptionUsage), which re-reads fresh headers + // on every single request with no such lag. This is a deliberate, + // accepted limitation, not a bug: a caller showing this value (e.g. + // boxes rendering it in a UI) also has SubscriptionUsage.CapturedAt, + // so staleness is always visible rather than silently assumed live. + // No periodic redial or separate query exists to force a refresh. + subUsage *message.SubscriptionUsage } // wsPool is a per-Client pool of persistent Codex Responses websocket @@ -61,7 +81,7 @@ type wsPool struct { // dial is overridden by tests that point it at an httptest server // instead of the real chatgpt.com endpoint. Production always uses // dialResponsesWebSocket via newWSPool's default assignment. - dial func(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, error) + dial func(ctx context.Context, url string, headers http.Header, httpClient *http.Client, timeout time.Duration) (*websocket.Conn, *http.Response, error) mu sync.Mutex entries map[string]*wsPoolEntry @@ -138,19 +158,32 @@ func (p *wsPool) stream(ctx context.Context, req wsStreamRequest) (provider.Stre conn := entry.conn entry.mu.Unlock() + var subUsage *message.SubscriptionUsage if !reuse { p.invalidate(entry) - newConn, err := p.dial(ctx, req.URL, req.Headers, req.HTTPClient, p.connectTimeout) + newConn, resp, err := p.dial(ctx, req.URL, req.Headers, req.HTTPClient, p.connectTimeout) if err != nil { p.recordFailure(entry) p.release(entry) return nil, false } + // Only a CodexFamily request captures the x-codex-* subscription- + // usage headers off the upgrade response — see CodexFamily's own + // doc comment for why family is the gate, mirroring Client. + // codexSubscriptionUsage's identical check on the HTTP path. + if req.Family == CodexFamily && resp != nil { + subUsage = codexSubscriptionUsageFromHeaders(resp.Header) + } entry.mu.Lock() entry.conn = newConn entry.connectedAt = time.Now() + entry.subUsage = subUsage entry.mu.Unlock() conn = newConn + } else { + entry.mu.Lock() + subUsage = entry.subUsage + entry.mu.Unlock() } if err := sendResponseCreate(ctx, conn, req.Body); err != nil { @@ -192,9 +225,10 @@ func (p *wsPool) stream(ctx context.Context, req wsStreamRequest) (provider.Stre } return &stream{ - wsConn: src, - model: req.Model, - family: req.Family, + wsConn: src, + model: req.Model, + family: req.Family, + subUsage: subUsage, }, true } diff --git a/provider/provider.go b/provider/provider.go index 2a86c81b..566bb7d1 100644 --- a/provider/provider.go +++ b/provider/provider.go @@ -122,6 +122,15 @@ type Event struct { Message *message.Message StopReason StopReason Usage Usage + // SubscriptionUsage carries a subscription lane's captured rate-limit/ + // quota snapshot (see message.SubscriptionUsage's own doc comment), + // set only on EventDone by an adapter that captured one on THIS call — + // nil for every other event, and nil on EventDone itself unless the + // adapter is a subscription lane that found the signal on this + // response (provider/openai's codex family reads it from x-codex-* + // response headers; see engine.streamTurn's EventDone case for where + // this rides onto Session.SubscriptionUsage). + SubscriptionUsage *message.SubscriptionUsage } // Stream yields events for one model call. Next returns io.EOF after the diff --git a/server/handlers.go b/server/handlers.go index 962f1847..b5d31722 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -125,6 +125,19 @@ type sessionJSON struct { // cancellation, and completion delivery. The two are unrelated // concepts that happen to share the word "parent." Lineage *lineageJSON `json:"lineage,omitempty"` + // SubscriptionUsage is this session's most recently captured + // subscription-lane rate-limit/quota snapshot (see + // engine.Session.SubscriptionUsage / message.SubscriptionUsage's own + // doc comments for the two lanes that capture one and the exact field + // mapping). Deliberately NOT omitempty: null on the wire until a turn + // in THIS process has carried the signal — a session that has never + // delegated a turn through either lane, or has but this process + // hasn't seen its first one yet — rather than omitted, so a caller can + // unmarshal into a fixed struct without special-casing key presence. + // Process-local only, like LastTurn/Goal above: never re-derived from + // a durable source on a cold read (buildSessionFromIndex), since + // persisting it is not required for GET /session's own contract here. + SubscriptionUsage *message.SubscriptionUsage `json:"subscription_usage"` } // lineageJSON is sessionJSON's subagent-sessions extension, sourced from @@ -3826,25 +3839,26 @@ func (s *Server) buildSession(lv liveSession) sessionJSON { lastTurn := s.lastTurnJSONLocked(id) s.mu.Unlock() return sessionJSON{ - ID: id, - CreatedAt: sess.CreatedAt(), - Model: sess.Model(), - Effort: sess.Effort(), - Status: status, - State: compositeState(status == "busy", goal != nil && goal.Active, forcesIdlePause(goal)), - Messages: len(sess.History()), - Seq: seq, - Goal: goal, - WorkDir: sess.WorkDir(), - LastTurn: lastTurn, - Usage: usageJSONForSession(sess), - LastActivityAt: sess.LastActivityAt(), - ParentSession: sess.ParentSession(), - CompactionCount: sess.CompactionCount(), - LastCompactedAt: sess.LastCompactedAt(), - Plugins: sess.Plugins(), - Queued: len(sess.QueuedPrompts()), - Lineage: lineageJSONFor(lv), + ID: id, + CreatedAt: sess.CreatedAt(), + Model: sess.Model(), + Effort: sess.Effort(), + Status: status, + State: compositeState(status == "busy", goal != nil && goal.Active, forcesIdlePause(goal)), + Messages: len(sess.History()), + Seq: seq, + Goal: goal, + WorkDir: sess.WorkDir(), + LastTurn: lastTurn, + Usage: usageJSONForSession(sess), + LastActivityAt: sess.LastActivityAt(), + ParentSession: sess.ParentSession(), + CompactionCount: sess.CompactionCount(), + LastCompactedAt: sess.LastCompactedAt(), + Plugins: sess.Plugins(), + Queued: len(sess.QueuedPrompts()), + Lineage: lineageJSONFor(lv), + SubscriptionUsage: sess.SubscriptionUsage(), } } @@ -3901,6 +3915,9 @@ func (s *Server) buildSessionFromIndex(ix engine.SessionIndex) sessionJSON { Plugins: s.pluginInfo(ix.ID), Queued: ix.Queued, Lineage: coldLineageJSON(ix.TaskParentID, ix.TaskAgentType, ix.TaskDepth, ix.SpawnedChildIDs), + // SubscriptionUsage has no durable source (see sessionJSON's own + // field doc comment) — null on every cold read, by construction. + SubscriptionUsage: nil, } } diff --git a/server/openapi.yaml b/server/openapi.yaml index 29669a2c..1eb0d24a 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -99,7 +99,7 @@ components: Session: type: object - required: [id, created_at, model, status, state, messages, workdir, usage, last_activity_at, plugins, queued] + required: [id, created_at, model, status, state, messages, workdir, usage, last_activity_at, plugins, queued, subscription_usage] properties: id: type: string @@ -238,6 +238,87 @@ components: session's own prompt.queued/prompt.dequeued log records), but never dispatches on its own — the same "boot never auto-runs" rule goals follow. + subscription_usage: + $ref: "#/components/schemas/SubscriptionUsage" + nullable: true + description: > + The session's most recently captured subscription-lane rate- + limit/quota snapshot, for a session whose turns run against a + user's own model subscription rather than metered API access + (the Claude Code CLI delegated backend, or a "codex"-family + native OpenAI Responses provider). Captured from a signal the + provider already sends on every turn — never an extra outbound + request. null until a turn has carried the signal in THIS + process (never re-derived from a durable source on a cold + read), and always null for a session that never uses either + lane. + + SubscriptionUsage: + type: object + description: > + A normalized subscription rate-limit/quota snapshot — see + Session.subscription_usage. + required: [provider, plan, windows, captured_at] + properties: + provider: + type: string + enum: [claude, codex] + description: Which subscription lane captured this snapshot. + plan: + type: string + description: > + The subscription tier ("max"/"pro" for codex, from its own + plan-type response header). Empty when the capturing lane has + no cheap source for it (the claude lane today). + windows: + type: array + items: + $ref: "#/components/schemas/SubscriptionUsageWindow" + description: > + One entry per rate-limit window the provider reported on this + turn, in the provider's own order. Empty, never null. + overage: + $ref: "#/components/schemas/SubscriptionOverage" + description: > + A pay-as-you-go overage state riding on top of the + subscription. Absent when not applicable — always absent for + provider "codex", present only when the claude lane's own + signal carried one. + captured_at: + type: integer + description: When this snapshot was captured, Unix seconds. + + SubscriptionUsageWindow: + type: object + required: [key, label, used_percent, resets_at] + properties: + key: + type: string + description: > + The capturing lane's own stable window identifier (e.g. + "five_hour", "primary"). Track one window turn over turn by + this, not label. + label: + type: string + description: A short human-readable window name (e.g. "5-hour", "Weekly"). + used_percent: + type: number + description: This window's utilization, 0-100. + resets_at: + type: integer + description: When this window resets, Unix seconds. + + SubscriptionOverage: + type: object + required: [in_use, status, resets_at] + properties: + in_use: + type: boolean + status: + type: string + resets_at: + type: integer + description: Unix seconds. PluginInfo: type: object diff --git a/server/subscription_usage_test.go b/server/subscription_usage_test.go new file mode 100644 index 00000000..89b4e603 --- /dev/null +++ b/server/subscription_usage_test.go @@ -0,0 +1,98 @@ +package server + +import ( + "encoding/json" + "net/http" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// withSubscriptionUsageTurn is withUsageTurn's (usage_test.go) sibling: a +// completed turn whose EventDone also carries a captured +// message.SubscriptionUsage — the shape provider/openai's codex family +// attaches (see provider/openai/subscription_usage_test.go for the header- +// capture half; this exercises only the engine->server wiring GET /session +// reads). +func withSubscriptionUsageTurn(text string, usage message.SubscriptionUsage) []provider.Event { + msg := &message.Message{ID: message.ProviderCallID("m", text, 12), Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: text}}} + return []provider.Event{{ + Type: provider.EventDone, + Message: msg, + StopReason: provider.StopEndTurn, + Usage: provider.Usage{InputTokens: 5, OutputTokens: 2}, + SubscriptionUsage: &usage, + }} +} + +// TestSessionSubscriptionUsageSurfacedOnGet proves GET /session/{id} +// carries subscription_usage: null before any turn has carried the +// signal, then the mapped snapshot once one has — the exact field +// buildSession (handlers.go) reads from engine.Session.SubscriptionUsage. +func TestSessionSubscriptionUsageSurfacedOnGet(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + {{Type: provider.EventDone, Message: &message.Message{ID: message.ProviderCallID("m", "no signal", 12), Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "no signal"}}}, StopReason: provider.StopEndTurn}}, + withSubscriptionUsageTurn("here's your usage", message.SubscriptionUsage{ + Provider: "codex", + Plan: "pro", + Windows: []message.SubscriptionUsageWindow{ + {Key: "primary", Label: "Weekly", UsedPercent: 12.5, ResetsAt: 1788785267}, + }, + }), + }} + h := newHarness(t, prov) + id := h.createSession("test/m1") + + sse := h.openSSE("?from=0", "") + + // Turn 1: no SubscriptionUsage on this turn's EventDone at all — + // subscription_usage must stay null (not, say, a zero-value object). + h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "hi"}}, + }) + sse.collectUntilIdle(t) + + resp, data := h.do("GET", "/session/"+id, nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET session status %d: %s", resp.StatusCode, data) + } + var sess sessionJSONForTest + if err := json.Unmarshal(data, &sess); err != nil { + t.Fatal(err) + } + if sess.SubscriptionUsage != nil { + t.Fatalf("subscription_usage before any turn carried the signal = %+v, want null", sess.SubscriptionUsage) + } + + // Turn 2: this turn's EventDone carries a captured snapshot. + h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "again"}}, + }) + sse.collectUntilIdle(t) + + resp, data = h.do("GET", "/session/"+id, nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET session status %d: %s", resp.StatusCode, data) + } + if err := json.Unmarshal(data, &sess); err != nil { + t.Fatal(err) + } + if sess.SubscriptionUsage == nil { + t.Fatal("subscription_usage after a turn carried the signal = null, want the captured snapshot") + } + su := sess.SubscriptionUsage + if su.Provider != "codex" || su.Plan != "pro" { + t.Errorf("subscription_usage = %+v, want provider=codex plan=pro", su) + } + if len(su.Windows) != 1 || su.Windows[0].Key != "primary" || su.Windows[0].Label != "Weekly" || + su.Windows[0].UsedPercent != 12.5 || su.Windows[0].ResetsAt != 1788785267 { + t.Errorf("subscription_usage.windows = %+v", su.Windows) + } + if su.CapturedAt == 0 { + t.Error("subscription_usage.captured_at = 0, want a stamped Unix timestamp") + } + if su.Overage != nil { + t.Errorf("subscription_usage.overage = %+v, want nil (this turn's usage carried none)", su.Overage) + } +} diff --git a/server/usage_test.go b/server/usage_test.go index fb1ef655..ead3e32f 100644 --- a/server/usage_test.go +++ b/server/usage_test.go @@ -21,10 +21,11 @@ type usageJSONForTest struct { } type sessionJSONForTest struct { - ID string `json:"id"` - Messages int `json:"messages"` - Usage usageJSONForTest `json:"usage"` - LastActivityAt time.Time `json:"last_activity_at"` + ID string `json:"id"` + Messages int `json:"messages"` + Usage usageJSONForTest `json:"usage"` + LastActivityAt time.Time `json:"last_activity_at"` + SubscriptionUsage *message.SubscriptionUsage `json:"subscription_usage"` } func withUsageTurn(text string, in, out int) []provider.Event { From a4da2ca25ec22325eaeac1492218a3612fd6e8cb Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Mon, 31 Aug 2026 13:45:10 -0400 Subject: [PATCH 30/95] fix(engine): pass --forward-subagent-text to the claude-code CLI (#219) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-code-lane subagent (Task) work rendered inline in the boxes console instead of nested under its spawning Task. The console nests a frame under its parent only when the frame carries a non-null top-level parent_tool_use_id. The claude CLI (v2.1.251) only forwards a subagent's own assistant text/thinking content as parented assistant/user frames when passed --forward-subagent-text; this defaults off. Without it, tool_use and tool_result plumbing frames still carry parent_tool_use_id, but the subagent's own generated text is never emitted as its own parented message — so the console has no parented content to nest, and the Task's aggregate result renders as ordinary top-level conversation instead. runClaudeCodeTurn built its argv without this flag. It belongs beside --verbose: both are always safe here because this driver always spawns the CLI with -p and --output-format=stream-json, which is what the flag is gated on. Harness's downstream plumbing already reads parent_tool_use_id onto Message.ParentToolUseID correctly (see TestClaudeCodeParentToolUseIDCarriedOntoMessage); only the flag was missing. Add --forward-subagent-text unconditionally to the argv build and update the package doc's parent_tool_use_id note. Add TestClaudeCodeForwardSubagentTextAlwaysSet asserting the flag appears in the child argv. Verified live against the real claude 2.1.251 binary in a boxes sandbox: a Task-spawning turn run with and without the flag shows the subagent's own final assistant text frame appears with parent_tool_use_id set to the Task's tool_use id only when the flag is passed; without it, that frame is absent and only the Task's aggregate tool_result reaches the outer conversation. --- engine/claude_code_backend.go | 17 ++++++++++++++++- engine/claude_code_backend_test.go | 16 ++++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index b8434e49..6d0bf1a4 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -115,7 +115,11 @@ // them into message.Reasoning parts instead of dropping them. // - parent_tool_use_id: captured on the envelope and carried onto the // appended message.Message (Message.ParentToolUseID) so a subagent -// turn's nesting survives into the journal. +// turn's nesting survives into the journal. The CLI only sets this +// field on subagent assistant/user frames when told to with +// --forward-subagent-text (default off); runClaudeCodeTurn now +// always passes that flag alongside --verbose, so this mapping has +// real subagent frames to read. // - turn_metrics: consumeClaudeCodeStream now emits an OnTurnMetrics // record from the "result" event's own timing/usage fields. // @@ -314,6 +318,17 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro "--input-format", "stream-json", "--output-format", "stream-json", "--verbose", + // The CLI defaults subagent (Task) assistant/user frames to a flat, + // unparented stream unless told otherwise. Without this flag, a + // subagent's activity carries no parent_tool_use_id, so a consumer + // like the boxes console can't nest it under its spawning Task and + // renders it inline instead. --forward-subagent-text makes the CLI + // set parent_tool_use_id on those frames, which this driver already + // reads onto Message.ParentToolUseID (see the package doc above). + // It is gated only on --print/--output-format=stream-json, both of + // which this driver always sets, so it is safe to pass + // unconditionally. + "--forward-subagent-text", } if model.Model != "" { args = append(args, "--model", model.Model) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 68629d07..4c2a9caa 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -594,6 +594,22 @@ func TestClaudeCodeEffortUnsetOmitsFlag(t *testing.T) { } } +// TestClaudeCodeForwardSubagentTextAlwaysSet proves runClaudeCodeTurn always +// sends --forward-subagent-text. The CLI defaults this off, so a subagent's +// assistant/user frames would otherwise carry no parent_tool_use_id, and a +// consumer like the boxes console would render subagent work inline instead +// of nested under its spawning Task. +func TestClaudeCodeForwardSubagentTextAlwaysSet(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if !argvContains(invocations[0], "--forward-subagent-text") { + t.Errorf("argv missing --forward-subagent-text: %v", invocations[0]) + } +} + // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning From 64b0ff6be43287ba7a42f0bc2e97ccbb85fc8973 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Mon, 31 Aug 2026 17:22:28 -0400 Subject: [PATCH 31/95] engine: return on the claude-code result event, not stdout EOF (#220) * fix(engine): return a claude-code turn on result, not stdout EOF consumeClaudeCodeStream kept calling scanner.Scan() after decoding a "result" event, even though the package doc already documents result as the turn's terminal event. The loop only exited on the child's stdout reaching EOF, and EOF on a pipe only arrives once EVERY process holding the write end open has exited. `claude --bg` is the CLI's own sanctioned pattern for a long-running background task: it spawns a daemon that reparents to PID 1 and is meant to keep running after the turn ends. That daemon (or a further child of its own, e.g. a dev server) can inherit the turn's stdout fd unless it explicitly redirects it. The direct `claude` child still prints "result" and exits on schedule, but the leaked descendant keeps the pipe open, so the old loop blocked on scanner.Scan() forever, wedging the whole harness turn - and the box's conversation with it - unkillably, even though a complete result was already in hand. Return finalMsg/started/turnErr as soon as the "result" event is handled, instead of falling through to another Scan(). This decouples turn completion from any descendant's fd lifetime and changes nothing for a normal turn, where the direct child has no more events to send after result anyway. The subsequent cmd.Wait() in runClaudeCodeTurn does not reintroduce the wait: it calls waitpid on the direct child's own PID, which has already exited by the time consumeClaudeCodeStream returns, and Go's os/exec closes our own read end of the pipe right after (StdoutPipe's reader lives in Cmd.parentIOPipes, which Wait() closes unconditionally once the process exits - see os/exec/exec.go). Wait() never signals or otherwise touches the leaked grandchild, so a surviving `--bg` daemon keeps running exactly as designed; only our own read end of the pipe closes. Added fakeclaude's "bg_leak" mode: it emits a normal assistant/result sequence, then spawns a grandchild that inherits its stdout and sleeps for an hour before the direct child exits, standing in for the leaked --bg daemon. TestClaudeCodeReturnsOnResultDespiteLeakedDescendantFD drives a turn against it with a 10s hard timeout and asserts the turn still returns the expected final message and usage; temporarily reverting the fix makes this test fail as a timeout, reproducing the wedge directly (see the fix's own comment for the reasoning this red run confirmed). Verified: gofmt -l . clean; go vet ./engine/... clean; go build ./... clean; go test -race ./engine/... passes (82.9s, no hang), including the new test and every existing TestClaudeCode* case. * fix(engine): drain claude-code stderr via a pipe, not a Wait-blocking writer Adversarial review of the prior commit (fixing stdout's EOF wedge) found the wedge survives through stderr. runClaudeCodeTurn set cmd.Stderr to a capBuffer, a plain io.Writer. Per os/exec, assigning a non-*os.File io.Writer makes Cmd allocate its own internal pipe and a copying goroutine, and cmd.Wait() blocks (awaitGoroutines) until that goroutine's io.Copy sees EOF. A leaked `claude --bg` descendant (a dev server, say) commonly inherits its parent's whole stdio, not just fd 1, so it could still wedge the turn through Wait() on stderr alone, even after the previous commit decoupled turn completion from stdout. Replace cmd.Stderr = &capBuffer with cmd.StderrPipe(), whose read end os/exec records in Cmd.parentIOPipes - the same place StdoutPipe's read end already lives, which Wait() only closes, never waits on. A new goroutine drains that pipe into capBuffer for as long as it stays open, so a real crash's stderr text is still captured faithfully; the goroutine is deliberately never joined, since Wait() closing the pipe already guarantees its blocked Read ends promptly regardless of whether a leaked descendant is still holding the write end open. capBuffer gained a mutex: it is now written from that goroutine and read from the main goroutine without a happens-before edge between them, so both need their own synchronization. Updated the cmd.Wait() comment, which previously (and incorrectly, as of this fix) claimed Cmd started no internal copying goroutine at all; it now explains why that is true for both stdout and stderr only after this change. Also live-verified the remaining open question from the first fix: against a real `claude` 2.1.252 binary, "result" was confirmed the last line of the stream in two separate turns (a plain reply and a tool-calling one) - rate_limit_event, when present, always arrived before it, never after. Noted next to the early return. fakeclaude's "bg_leak" mode now leaks the grandchild's stderr as well as its stdout (leaker.Stderr = os.Stderr), so TestClaudeCodeReturnsOnResultDespiteLeakedDescendantFD exercises both fds. Red-verified by reverting only the stderr change (StderrPipe back to cmd.Stderr = &capBuffer) with the stdout fix and the two-fd test both left in place: the test fails as a 10s timeout, reproducing the exact wedge this commit closes; restoring the fix turns it green again under -race. Verified: gofmt -l . clean; go vet ./engine/... clean; go build ./... clean; go test -race ./engine/... passes (97.1s, no hang), including the updated test and every existing TestClaudeCode* case. --- engine/claude_code_backend.go | 180 +++++++++++++++++++++++++---- engine/claude_code_backend_test.go | 114 ++++++++++++++++++ engine/testdata/fakeclaude/main.go | 75 +++++++++++- 3 files changed, 343 insertions(+), 26 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 6d0bf1a4..d3e2d0de 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -78,19 +78,21 @@ // block, Message.ParentToolUseID set the same way as the "assistant" // case above), appended and emitted as EventMessage, with one // EventToolEnd per part. -// - "result": the turn's terminal event. Never itself appended as a -// message (the assistant text it summarizes was already appended by -// the last "assistant" event above) — it instead carries the turn's -// AGGREGATE usage, applied once via applyClaudeCodeUsage (a durable, -// message-independent record — see recClaudeCodeUsage in store.go), -// and this turn's timing, emitted once via emitTurnMetrics (ttft_ms/ -// duration_ms, permissively zero-valued if the CLI's own build does not -// send them). An IsError result becomes this call's returned error — -// wrapped provider.RetryableError for a known-transient shape (see -// claudeCodeRetryableClass), a plain error otherwise. TotalCostUSD has -// no home in provider.Usage (no adapter carries a cost field — every -// consumer derives cost from token counts) and is deliberately -// dropped, not persisted. +// - "result": the turn's terminal event. consumeClaudeCodeStream RETURNS +// as soon as this event is handled — it does not keep scanning for +// stdout EOF (see that function's own doc comment for why this +// matters). Never itself appended as a message (the assistant text it +// summarizes was already appended by the last "assistant" event +// above) — it instead carries the turn's AGGREGATE usage, applied once +// via applyClaudeCodeUsage (a durable, message-independent record — +// see recClaudeCodeUsage in store.go), and this turn's timing, emitted +// once via emitTurnMetrics (ttft_ms/duration_ms, permissively zero- +// valued if the CLI's own build does not send them). An IsError result +// becomes this call's returned error — wrapped provider.RetryableError +// for a known-transient shape (see claudeCodeRetryableClass), a plain +// error otherwise. TotalCostUSD has no home in provider.Usage (no +// adapter carries a cost field — every consumer derives cost from +// token counts) and is deliberately dropped, not persisted. // - "rate_limit_event": the CLI's own subscription rate-limit/quota // signal, typically the SECOND event of a turn (right after // "system"/"init"). Never appended as a message — it carries no @@ -158,6 +160,7 @@ import ( "os/exec" "sort" "strings" + "sync" "syscall" "time" @@ -371,14 +374,47 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro if err != nil { return nil, fmt.Errorf("engine: claude-code: creating stdout pipe: %w", err) } + // stderr is deliberately read via cmd.StderrPipe(), NOT handed to Cmd + // as a plain cmd.Stderr = io.Writer, even though the latter reads + // simpler. Assigning a non-*os.File io.Writer makes Cmd allocate its + // OWN internal pipe and copying goroutine (os/exec's writerDescriptor) + // and register that goroutine so cmd.Wait() BLOCKS on it via + // awaitGoroutines — i.e. Wait() would not return until stderr's pipe + // sees EOF, which reintroduces exactly the hazard this file's fix to + // consumeClaudeCodeStream just removed from stdout: a `claude --bg` + // turn's leaked descendant (a dev server, say) commonly inherits BOTH + // fd 1 AND fd 2, so it can wedge Wait() through stderr even once + // stdout no longer can. StderrPipe's read end instead lands in + // Cmd.parentIOPipes, which Wait() only CLOSES, never waits on (see + // this call's own comment on cmd.Wait(), below) — so this file must + // drain it itself, in the anonymous goroutine started right after + // Start() below. + stderrPipe, err := cmd.StderrPipe() + if err != nil { + return nil, fmt.Errorf("engine: claude-code: creating stderr pipe: %w", err) + } var stderr capBuffer stderr.cap = claudeCodeStderrCap - cmd.Stderr = &stderr if err := cmd.Start(); err != nil { return nil, fmt.Errorf("engine: claude-code: starting %q: %w", binary, err) } + // Drain stderrPipe into stderr for as long as it stays open, same as + // Cmd's own now-avoided internal copying goroutine would have — + // capturing a real crash's full stderr text (bounded by + // claudeCodeStderrCap) for the ordinary case where the direct child is + // the pipe's only writer and closes it on exit. This goroutine is + // deliberately NEVER joined by the rest of this call (see cmd.Wait()'s + // own comment on why that is safe rather than a leak): io.Copy's + // blocked Read ends on its own, promptly, the moment cmd.Wait() closes + // stderrPipe's read end below — whether that close finds EOF already + // pending (the ordinary case) or a leaked descendant still holding the + // write end open (the `--bg` case this file's package doc describes). + go func() { + _, _ = io.Copy(&stderr, stderrPipe) + }() + inputLine, err := json.Marshal(claudeCodeInputMessage{ Type: "user", Message: claudeCodeInputInnerMessage{ @@ -458,6 +494,48 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro finalMsg, started, turnErr := s.consumeClaudeCodeStream(stdout, model) + // cmd.Wait() below does NOT reintroduce the EOF wait that + // consumeClaudeCodeStream's early return on "result" just avoided, on + // EITHER stdout or stderr, and it does not touch (let alone kill) a + // surviving `claude --bg` daemon: + // + // - Wait() waits on c.Process.Wait(), i.e. waitpid(2) on the DIRECT + // `claude` child's own PID. That is a wait for one specific + // process's exit status, never a wait for a pipe's write end to + // see every holder close it. The direct child has already printed + // its "result" and exited by the time consumeClaudeCodeStream + // returns, so this waitpid returns immediately. + // - Both stdout and stderr here came from Cmd's own *Pipe methods + // (cmd.StdoutPipe() above, cmd.StderrPipe() further above) rather + // than a plain cmd.Stdout/cmd.Stderr io.Writer. That distinction is + // load-bearing: os/exec's writerDescriptor allocates an internal + // pipe AND a copying goroutine ONLY for a plain io.Writer target, + // and registers that goroutine in Cmd.goroutineErr, which Wait()'s + // own awaitGoroutines step explicitly BLOCKS on until it finishes + // (i.e. until that pipe sees EOF). This file used to hand stderr to + // Cmd exactly that way (cmd.Stderr = &capBuffer), which — even + // after stdout's own fix above — still let a leaked `--bg` + // descendant that inherits fd 2 (a dev server commonly inherits + // BOTH fd 1 and fd 2, not just fd 1) wedge Wait() through stderr + // alone. Both pipes are now read by THIS file's own code instead + // (consumeClaudeCodeStream for stdout, the anonymous drain + // goroutine started right after cmd.Start() above for stderr), so + // Cmd.goroutineErr stays nil and awaitGoroutines has nothing to + // block on for either fd. + // - Go's os/exec source (os/exec/exec.go, StdoutPipe/StderrPipe) + // records each pipe's READ end (the `stdout` and `stderrPipe` + // variables this call reads) in Cmd.parentIOPipes, and Wait() + // closes every entry of parentIOPipes itself right after the + // waitpid above returns. So Wait() actively closes both of our + // read ends for us; it never blocks on either. + // - The leaked grandchild only ever held the pipes' WRITE ends (fds + // duplicated across fork/exec). Wait() closing our read ends does + // not signal or touch that process at all — the background daemon + // keeps running untouched, exactly as `claude --bg` intends. A + // later write of its own may see EPIPE/SIGPIPE once nothing is + // left to read that fd, which is the same fate any well-behaved + // detached daemon should already tolerate by redirecting its own + // stdio, not a signal this call sends it. waitErr := cmd.Wait() return claudeCodeTurnResult(claudeCodeTurnOutcome{ @@ -577,15 +655,35 @@ func lastUserMessageText(history []message.Message) string { return last.Parts.Text() } -// consumeClaudeCodeStream reads newline-delimited stream-json events from -// r (the `claude` child's stdout) until EOF, appending/emitting each -// decoded event per this file's package-doc mapping, and returns the last -// assistant message.Message it appended (nil if none), whether the child -// got far enough to emit at least one "system" event (see the started -// return value's own use in runClaudeCodeTurn's waitErr branch: it is what -// tells a child that never even started apart from one that started and -// later crashed), and any turn-ending error a "result" event's IsError -// reported. +// consumeClaudeCodeStream reads newline-delimited stream-json events from r +// (the `claude` child's stdout), appending/emitting each decoded event per +// this file's package-doc mapping, and returns the last assistant +// message.Message it appended (nil if none), whether the child got far +// enough to emit at least one "system" event (see the started return +// value's own use in runClaudeCodeTurn's waitErr branch: it is what tells a +// child that never even started apart from one that started and later +// crashed), and any turn-ending error a "result" event's IsError reported. +// +// It returns as soon as it handles the "result" event rather than reading +// on to stdout's EOF, because "result" IS the turn's documented terminal +// event (see this file's package doc) and EOF is not a safe thing to wait +// for: EOF on a pipe only arrives once EVERY process holding the write end +// open has exited, and the direct `claude` child is not the only process +// that can hold it. `claude --bg` — the CLI's own sanctioned pattern for a +// long-running background task — spawns a daemon that reparents to PID 1 +// and is meant to keep running after the turn ends; that daemon (or a +// further child of its own, e.g. a dev server it starts) inherits this +// process's file descriptors, including stdout, unless it explicitly +// closes or redirects them. The direct `claude` child still prints its own +// "result" and exits right on schedule, but the leaked descendant keeps +// the pipe's write end open indefinitely, so a scan loop that waits for +// EOF blocks forever — wedging the harness turn (and the whole session) +// while the surviving daemon keeps running exactly as designed. Returning +// on "result" decouples turn completion from every descendant's fd +// lifetime, which is also why this must never be "solved" by killing the +// child's process group: that would kill the very background session +// --bg exists to keep alive. See runClaudeCodeTurn's own comment on why +// its subsequent cmd.Wait() does not reintroduce this wait. func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) (finalMsg *message.Message, started bool, turnErr error) { scanner := bufio.NewScanner(r) // A tool call's arguments or a large tool result can exceed @@ -692,6 +790,25 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // TotalCostUSD is intentionally dropped here — see the // package doc's "result" bullet: provider.Usage has no cost // field for it to occupy. + // + // Return NOW rather than falling through to another + // scanner.Scan(): "result" is the documented terminal event + // (package doc above), and continuing to scan would instead + // wait for stdout's EOF — which a `claude --bg` turn's leaked + // descendant fd can withhold forever. See this function's own + // doc comment for the full reasoning. Any bytes a lingering + // writer emits after this point are simply never read by this + // call; they are not folded into the turn in any way. + // + // This is safe for "rate_limit_event" specifically, not just + // documented that way: live-verified against a real `claude` + // 2.1.252 binary (two separate turns, one plain and one with a + // tool call) that "result" is the LAST line the direct child + // ever emits — rate_limit_event, when present, arrived before + // it both times, never after. Nothing observed contradicts the + // package doc's "result" bullet calling it the turn's terminal + // event. + return finalMsg, started, turnErr case "rate_limit_event": // The CLI's own subscription rate-limit/quota signal — see // mapClaudeCodeRateLimit's own doc comment for the wire shape @@ -1278,12 +1395,23 @@ type claudeCodeInputInnerMessage struct { // claudeCodeStderrCap), without letting a runaway or malicious child // exhaust memory buffering an unbounded stream nothing ever reads back in // full. +// +// mu guards buf because runClaudeCodeTurn's stderr drain goroutine (see +// its own comment on why stderr is read via cmd.StderrPipe rather than +// handed to Cmd as a plain io.Writer) writes here concurrently with the +// main goroutine's eventual String() call once the turn ends — that call +// is deliberately NOT sequenced after the drain goroutine's own exit (see +// runClaudeCodeTurn), so both sides need their own synchronization rather +// than relying on happens-before through some other event. type capBuffer struct { + mu sync.Mutex buf bytes.Buffer cap int } func (c *capBuffer) Write(p []byte) (int, error) { + c.mu.Lock() + defer c.mu.Unlock() if room := c.cap - c.buf.Len(); room > 0 { if len(p) > room { c.buf.Write(p[:room]) @@ -1298,4 +1426,8 @@ func (c *capBuffer) Write(p []byte) (int, error) { return len(p), nil } -func (c *capBuffer) String() string { return strings.TrimSpace(c.buf.String()) } +func (c *capBuffer) String() string { + c.mu.Lock() + defer c.mu.Unlock() + return strings.TrimSpace(c.buf.String()) +} diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 4c2a9caa..a44b2921 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -9,6 +9,7 @@ import ( "os" "os/exec" "path/filepath" + "strconv" "strings" "sync" "testing" @@ -993,3 +994,116 @@ func TestClaudeCodeSucceedsDespiteInputWriteBrokenPipe(t *testing.T) { t.Errorf("OnTurnMetrics called %d times, want 1: %+v", len(metrics), metrics) } } + +// TestClaudeCodeReturnsOnResultDespiteLeakedDescendantFD reproduces the +// `claude --bg` wedge that consumeClaudeCodeStream's early return on +// "result" PLUS runClaudeCodeTurn's stderr handling both have to fix: a +// delegated turn must complete as soon as the direct `claude` child prints +// its terminal "result" event, even when some OTHER process still holds +// stdout OR stderr's pipe write end open. Before the stdout fix, +// consumeClaudeCodeStream kept calling scanner.Scan() looking for EOF, +// which only arrives once EVERY holder of the write end closes it. Before +// the stderr fix, that same class of bug survived one layer down: Cmd's +// own internal copying goroutine for a plain cmd.Stderr = io.Writer target +// (this file used cmd.Stderr = &capBuffer) makes cmd.Wait() block until +// STDERR's pipe sees EOF too — so a leaked descendant that inherits fd 2 +// (a real dev server commonly inherits its parent's whole stdio, not just +// fd 1) could still wedge the turn through Wait() even once stdout no +// longer could. `claude --bg`'s own detached daemon (or a further child of +// its own) is designed to keep running after the direct child exits, and +// can inherit either or both fds — that wedges the whole harness turn +// forever, unkillably, even though the turn's own result was already in +// hand. +// +// fakeclaude's "bg_leak" mode stands in for exactly that: it emits a +// normal assistant/result pair, THEN spawns a grandchild that inherits +// BOTH its stdout and stderr and sleeps for an hour (never emitting +// anything, never exiting on its own within any sane test bound) before +// the direct child itself returns. The select below is this test's hard +// timeout guard: if a regression reintroduces either EOF-wait, this test +// fails loudly as a TIMEOUT rather than hanging the suite. +func TestClaudeCodeReturnsOnResultDespiteLeakedDescendantFD(t *testing.T) { + s, _ := claudeCodeTestSession(t, "bg_leak") + pidFile := filepath.Join(t.TempDir(), "leaked.pid") + t.Setenv("FAKE_CLAUDE_LEAK_PID_FILE", pidFile) + + var metrics []TurnMetrics + s.cfg.OnTurnMetrics = func(m TurnMetrics) { metrics = append(metrics, m) } + + type outcome struct { + msg *message.Message + err error + } + done := make(chan outcome, 1) + go func() { + msg, err := s.Prompt(context.Background(), "start a background job") + done <- outcome{msg, err} + }() + + select { + case res := <-done: + if res.err != nil { + t.Fatalf("Prompt: %v", res.err) + } + if res.msg == nil || res.msg.Parts.Text() != "Starting a background job." { + t.Fatalf("Prompt returned %+v, want fakeclaude's own canned final message", res.msg) + } + case <-time.After(10 * time.Second): + // Cleaning up here too: if this branch ever fires, the leaked + // grandchild would otherwise outlive the test. + killLeakedFakeClaude(t, pidFile) + t.Fatal("Prompt did not return within 10s of the child's \"result\" event — " + + "consumeClaudeCodeStream is waiting for stdout EOF instead of returning " + + "on \"result\" (the claude --bg wedge this test guards against)") + } + + // The lingering grandchild's own stdout never got read as turn stream: + // it emits nothing at all, so any observable content past "result" + // (there is none here) would have to come from the direct child, and + // finalMsg/History already prove that ended cleanly at "Starting a + // background job." — see the assertions above and below. + usage := s.Usage() + if usage.InputTokens != 12 || usage.OutputTokens != 5 { + t.Errorf("Usage() = %+v, want {12 5 0 0} (the result event's own usage)", usage) + } + if len(metrics) != 1 { + t.Errorf("OnTurnMetrics called %d times, want 1: %+v", len(metrics), metrics) + } + hist := s.History() + if len(hist) != 2 { + t.Fatalf("History() len = %d, want 2 (user prompt + one assistant message): %+v", len(hist), hist) + } + + // cmd.Wait() (runClaudeCodeTurn) only returns once the DIRECT + // fakeclaude child has fully exited — which happens after its own + // "bg_leak" case has already spawned the grandchild and written its + // pid to pidFile, both of which run before that case's own return + // statement. So by the time s.Prompt() above has returned, pidFile is + // guaranteed to exist already: no polling or sleep needed to read it. + killLeakedFakeClaude(t, pidFile) +} + +// killLeakedFakeClaude reads the pid fakeclaude's "bg_leak" mode wrote to +// pidFile and kills that process, so this test does not leave its stand-in +// background daemon running after the test exits. +func killLeakedFakeClaude(t *testing.T, pidFile string) { + t.Helper() + data, err := os.ReadFile(pidFile) + if err != nil { + t.Logf("killLeakedFakeClaude: reading %s: %v (leaked grandchild not cleaned up)", pidFile, err) + return + } + pid, err := strconv.Atoi(strings.TrimSpace(string(data))) + if err != nil { + t.Logf("killLeakedFakeClaude: parsing pid %q: %v", data, err) + return + } + proc, err := os.FindProcess(pid) + if err != nil { + t.Logf("killLeakedFakeClaude: FindProcess(%d): %v", pid, err) + return + } + if err := proc.Kill(); err != nil { + t.Logf("killLeakedFakeClaude: Kill(%d): %v (may have already exited)", pid, err) + } +} diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 99bdf853..8713b983 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -21,10 +21,14 @@ // own turn-input write — see runClaudeCodeTurn's inputErr handling), and // "rate_limit_event" (a subscription rate-limit/quota event ahead of the // final assistant text, mapped by mapClaudeCodeRateLimit onto -// Session.SubscriptionUsage), and "rate_limit_event_no_overage" (the same +// Session.SubscriptionUsage), "rate_limit_event_no_overage" (the same // event with no overage in play at all — overageStatus "", isUsingOverage // false, overageResetsAt 0 — proving mapClaudeCodeRateLimit leaves -// SubscriptionUsage.Overage nil rather than a hollow zero-value object). +// SubscriptionUsage.Overage nil rather than a hollow zero-value object), +// and "bg_leak" (reproduces a `claude --bg` turn whose detached child +// inherits this process's stdout AND stderr and outlives it — see its own +// comment below and claude_code_backend.go's consumeClaudeCodeStream/ +// runClaudeCodeTurn doc comments for the wedge this proves fixed). package main import ( @@ -32,12 +36,29 @@ import ( "encoding/json" "fmt" "os" + "os/exec" + "strconv" "time" ) func main() { mode := os.Getenv("FAKE_CLAUDE_MODE") + if mode == "bg_leak_child" { + // The detached grandchild "bg_leak" mode spawns below, standing in + // for `claude --bg`'s own daemon (or a further child of its own, + // e.g. a dev server) that inherits the direct `claude` child's + // stdout AND stderr and outlives it. This process emits nothing at + // all — it exists solely to hold both inherited fds (this + // process's own os.Stdout/os.Stderr, which ARE the harness-owned + // pipes' write ends) open far past any sane test timeout, so a + // test can prove the driver returns without ever waiting for + // either fd's EOF. The test kills it by PID once it has proved + // that. + time.Sleep(time.Hour) + return + } + if mode == "fast_no_drain" { // Close our OWN stdin's read end IMMEDIATELY — before doing // anything else, including the argv log write below — to @@ -265,6 +286,56 @@ func main() { }, }) return + case "bg_leak": + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Starting a background job."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Starting a background job.", + "usage": map[string]any{ + "input_tokens": 12, + "output_tokens": 5, + }, + }) + // Simulate `claude --bg`'s detached child inheriting this + // process's stdout AND stderr fds (the harness-owned pipes' write + // ends) and outliving THIS direct child — see + // claude_code_backend.go's consumeClaudeCodeStream and + // runClaudeCodeTurn doc comments for the wedge this reproduces. + // BOTH fds matter here, not just stdout: a real leaked descendant + // (a dev server started by a background task, say) commonly + // inherits its parent's whole stdio, not a hand-picked subset, so + // a test that only leaks stdout would miss a driver that still + // wedges through stderr alone. Spawn a grandchild that inherits + // os.Stdout AND os.Stderr verbatim and sleeps well past any sane + // test timeout, holding both pipes' write ends open long after + // this process (the direct `claude` child harness itself manages) + // exits right below. A minimal, explicit Env — not this process's + // own os.Environ() — keeps the grandchild out of FAKE_CLAUDE_LOG's + // invocation log, since it is not a `claude` invocation a test + // should count. The test kills the grandchild by PID (recorded via + // FAKE_CLAUDE_LEAK_PID_FILE) once it has proved the driver did not + // wait for it. + leaker := exec.Command(os.Args[0]) + leaker.Env = []string{"FAKE_CLAUDE_MODE=bg_leak_child"} + leaker.Stdout = os.Stdout + leaker.Stderr = os.Stderr + if err := leaker.Start(); err == nil { + if pidFile := os.Getenv("FAKE_CLAUDE_LEAK_PID_FILE"); pidFile != "" { + _ = os.WriteFile(pidFile, []byte(strconv.Itoa(leaker.Process.Pid)), 0o644) + } + _ = leaker.Process.Release() + } + return } if mode == "rate_limit_event" || mode == "rate_limit_event_no_overage" { From e6a0487ba92a72512b32af71b8a0491ba544d22a Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Mon, 31 Aug 2026 21:51:19 -0400 Subject: [PATCH 32/95] feat(config): add append_system_prompt (#221) Add an additive append_system_prompt config key for operator-owned environment facts. Wire it through run and serve, and forward it to delegated Claude Code turns as one blank-line-joined --append-system-prompt value. Reject conflicting Claude Code append-prompt ExtraArgs so a project cannot replace the managed platform value. Keep Config.System native-only. Test the two-layer merge, run composition, serve request assembly, Claude Code argv, and conflicting option forms. Co-authored-by: andybons --- cmd/harness/append_system_prompt_test.go | 29 ++++++++++ cmd/harness/main.go | 42 +++++++++----- config/AGENTS.md | 9 +++ config/append_system_prompt_test.go | 57 ++++++++++++++++++ config/claude_code_cli_test.go | 41 +++++++++++++ config/config.go | 73 ++++++++++++++++++++---- docs/deploy-modal.md | 42 +++++++++----- docs/engine-request-cycle.md | 31 ++++++++++ e2e/e2e_test.go | 71 +++++++++++++++++++++++ engine/append_system_prompt_test.go | 72 +++++++++++++++++++++++ engine/claude_code_backend.go | 42 +++++++++++--- engine/engine.go | 20 +++++++ 12 files changed, 485 insertions(+), 44 deletions(-) create mode 100644 cmd/harness/append_system_prompt_test.go create mode 100644 config/append_system_prompt_test.go create mode 100644 engine/append_system_prompt_test.go diff --git a/cmd/harness/append_system_prompt_test.go b/cmd/harness/append_system_prompt_test.go new file mode 100644 index 00000000..afc71cd4 --- /dev/null +++ b/cmd/harness/append_system_prompt_test.go @@ -0,0 +1,29 @@ +package main + +import ( + "reflect" + "testing" + + "github.com/majorcontext/harness/config" +) + +func TestAppendSystemSegments(t *testing.T) { + tests := []struct { + name string + cfg *config.Config + flag string + want []string + }{ + {"unset", nil, "", nil}, + {"config", &config.Config{AppendSystemPrompt: []string{"one", "two"}}, "", []string{"one", "two"}}, + {"flag", nil, "flag", []string{"flag"}}, + {"both", &config.Config{AppendSystemPrompt: []string{"one", "two"}}, "flag", []string{"one", "two", "flag"}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := appendSystemSegments(tt.cfg, tt.flag); !reflect.DeepEqual(got, tt.want) { + t.Errorf("appendSystemSegments = %v, want %v", got, tt.want) + } + }) + } +} diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 9816df23..4cec960c 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -270,7 +270,7 @@ func runFlags(opts *runOptions) *flag.FlagSet { fs.StringVar(&opts.goal, "goal", "", "pursue a goal: prompt this condition, then re-prompt with evaluator feedback until an independent evaluator judges it met (requires config goal_evaluator_model)") fs.IntVar(&opts.goalMaxTurns, "goal-max-turns", 0, "maximum turns for -goal (0 = unlimited)") fs.StringVar(&opts.model, "model", "", "model ref (provider/model) or alias; overrides the persisted model when resuming; default from config, else "+config.DefaultModel) - fs.StringVar(&opts.system, "system", "", "extra system prompt segment") + fs.StringVar(&opts.system, "system", "", "extra system prompt segment; appended after any config append_system_prompt segments") fs.IntVar(&opts.maxTokens, "max-tokens", 0, "per-response output token cap") fs.BoolVar(&opts.jsonOut, "json", false, "emit the event stream as JSON lines instead of text") fs.BoolVar(&opts.noSave, "no-save", false, "disable session persistence") @@ -730,9 +730,11 @@ func runCmd(args []string) error { sessMgr.SetMaxTreeTokens(envInt("HARNESS_MAX_TREE_TOKENS")) s, err := resolveSession(engine.Config{ - Providers: registry(cfg), - Model: model, - System: systemPrompt(workDir, opts.system), + Providers: registry(cfg), + Model: model, + System: systemPrompt(workDir, ""), + // Config comes first; the per-run flag is the final refinement. + AppendSystemPrompt: appendSystemSegments(cfg, opts.system), MaxTokens: opts.maxTokens, WorkDir: workDir, SessionDir: sesDir, @@ -1638,15 +1640,16 @@ func serveCmd(args []string) error { }() mkCfg := func(model message.ModelRef) engine.Config { return engine.Config{ - Providers: reg, - Model: model, - System: systemPrompt(workDir, ""), - WorkDir: workDir, - SessionDir: sesDir, - SessionSync: cfg.SessionSync, - EngineVersion: version, - StartedAt: startedAt, - OnEvent: func(ev engine.Event) { srv.Publish(ev) }, + Providers: reg, + Model: model, + System: systemPrompt(workDir, ""), + AppendSystemPrompt: appendSystemSegments(cfg, ""), + WorkDir: workDir, + SessionDir: sesDir, + SessionSync: cfg.SessionSync, + EngineVersion: version, + StartedAt: startedAt, + OnEvent: func(ev engine.Event) { srv.Publish(ev) }, // The actual node registration (depth, lineage) happens // separately, in handleCreate, right after NewSession returns // (see AdoptRoot's call site there); wiring it here too means a @@ -2010,6 +2013,19 @@ func agentDefsDirs(cfg *config.Config, flagDirs []string, workDir string) []stri return out } +// appendSystemSegments returns config segments followed by the per-run flag. +// The result does not alias the config slice. +func appendSystemSegments(cfg *config.Config, extra string) []string { + var segs []string + if cfg != nil && len(cfg.AppendSystemPrompt) > 0 { + segs = append(segs, cfg.AppendSystemPrompt...) + } + if extra != "" { + segs = append(segs, extra) + } + return segs +} + func systemPrompt(workDir, extra string) []string { system := []string{ "You are harness, a fast coding agent. You execute tasks directly " + diff --git a/config/AGENTS.md b/config/AGENTS.md index 7eaf6732..9beb8945 100644 --- a/config/AGENTS.md +++ b/config/AGENTS.md @@ -22,6 +22,15 @@ value. Load user config first and project config second. A project value overrides the user value according to the field's documented merge rule. +`append_system_prompt` is the one additive key: the merged value is the user +segments followed by the project segments. Keep it additive. The user layer is +the platform's own config and the project layer is a cloned repository's file, +so an override rule would let a repository delete a platform segment. Do not +copy this shape to another key without the same argument. + +Reject both Claude Code append-prompt options in `extra_args` when this key is +non-empty. The CLI would replace the managed value or reject the invocation. + Validate providers after merge and native-default application. A partial project entry can become valid through its inherited fields. diff --git a/config/append_system_prompt_test.go b/config/append_system_prompt_test.go new file mode 100644 index 00000000..ec6945fc --- /dev/null +++ b/config/append_system_prompt_test.go @@ -0,0 +1,57 @@ +package config + +import ( + "path/filepath" + "reflect" + "strings" + "testing" +) + +func TestLoadProjectAppendSystemPrompt(t *testing.T) { + user := filepath.Join(t.TempDir(), "config.json") + writeFile(t, user, `{"append_system_prompt":["platform"]}`) + t.Setenv("HARNESS_CONFIG", user) + dir := t.TempDir() + writeFile(t, filepath.Join(dir, ".harness.json"), `{"append_system_prompt":["project"]}`) + + cfg, err := LoadProject(dir) + if err != nil { + t.Fatal(err) + } + if want := []string{"platform", "project"}; !reflect.DeepEqual(cfg.AppendSystemPrompt, want) { + t.Errorf("AppendSystemPrompt = %v, want %v", cfg.AppendSystemPrompt, want) + } +} + +func TestMergeAppendSystemPromptDoesNotAlias(t *testing.T) { + base := &Config{AppendSystemPrompt: []string{"platform"}} + over := &Config{AppendSystemPrompt: []string{"project"}} + got := merge(base, over) + got.AppendSystemPrompt[0], got.AppendSystemPrompt[1] = "x", "y" + if base.AppendSystemPrompt[0] != "platform" || over.AppendSystemPrompt[0] != "project" { + t.Fatalf("merge aliased its inputs: base=%v over=%v", base.AppendSystemPrompt, over.AppendSystemPrompt) + } +} + +func TestLoadProjectRejectsPromptReplacementExtraArg(t *testing.T) { + user := filepath.Join(t.TempDir(), "config.json") + writeFile(t, user, `{ + "append_system_prompt":["platform"], + "providers":{"claude-code":{"type":"claude-code-cli"}} + }`) + t.Setenv("HARNESS_CONFIG", user) + dir := t.TempDir() + writeFile(t, filepath.Join(dir, ".harness.json"), `{ + "providers":{"claude-code":{"extra_args":["--append-system-prompt","project"]}} + }`) + + _, err := LoadProject(dir) + if err == nil { + t.Fatal("LoadProject accepted conflicting extra_args") + } + for _, want := range []string{"append_system_prompt", "providers.claude-code.extra_args"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q does not contain %q", err, want) + } + } +} diff --git a/config/claude_code_cli_test.go b/config/claude_code_cli_test.go index 26a6abef..f5433376 100644 --- a/config/claude_code_cli_test.go +++ b/config/claude_code_cli_test.go @@ -130,6 +130,47 @@ func TestClaudeCodeCLIUnknownTypeErrorListsIt(t *testing.T) { } } +func TestAppendSystemPromptValidation(t *testing.T) { + for _, arg := range []string{ + "--append-system-prompt", + "--append-system-prompt=x", + "--append-system-prompt-file", + "--append-system-prompt-file=x", + } { + t.Run(arg, func(t *testing.T) { + cfg := &Config{ + AppendSystemPrompt: []string{"platform"}, + Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI, ExtraArgs: []string{arg}}, + }, + } + if _, err := mergeAndValidate(cfg, &Config{}); err == nil { + t.Fatal("accepted conflicting extra_args") + } + }) + } +} + +func TestAppendSystemPromptAllowsCompatibleExtraArgs(t *testing.T) { + cfg := &Config{ + AppendSystemPrompt: []string{"platform"}, + Providers: map[string]Provider{ + "claude-code": {Type: TypeClaudeCodeCLI, ExtraArgs: []string{"--dangerously-skip-permissions"}}, + }, + } + if _, err := mergeAndValidate(cfg, &Config{}); err != nil { + t.Fatal(err) + } + + cfg.AppendSystemPrompt = nil + cfg.Providers["claude-code"] = Provider{ + Type: TypeClaudeCodeCLI, ExtraArgs: []string{"--append-system-prompt", "legacy"}, + } + if _, err := mergeAndValidate(cfg, &Config{}); err != nil { + t.Fatal(err) + } +} + // TestMergeClaudeCodeExtraArgsNotAliased mirrors // TestMergeProviderExtraHeadersBaseOnlyKeyNotAliased for the new slice // field: a base-only key's ExtraArgs must not alias the base config's own diff --git a/config/config.go b/config/config.go index 9403422d..b9136c4a 100644 --- a/config/config.go +++ b/config/config.go @@ -58,6 +58,18 @@ type Config struct { // rendering with no outline. HARNESS_INSTRUCTIONS_MODE overrides this key // (see cmd/harness). See engine.InstructionsMode. InstructionsMode string `json:"instructions_mode,omitempty"` + // AppendSystemPrompt lists operator-supplied environment facts that the + // agent cannot discover. Do not use it for tool instructions or project + // instructions. The engine places entries after System and before its own + // generated segments. Claude Code receives one blank-line-joined + // --append-system-prompt value. + // + // Merge is additive: base segments come first, then project segments. + // This rule differs from every other slice field. In box deployments, the + // base file belongs to the platform and the project file belongs to the + // cloned repository. Override semantics would let the repository remove a + // platform environment fact. Keep this field additive. + AppendSystemPrompt []string `json:"append_system_prompt,omitempty"` // SkillsDirs lists directories scanned for Agent Skills (agentskills.io). // A nil (omitted) value leaves the engine default in place: use // /.agents/skills when it exists. In the project-config merge a @@ -673,17 +685,14 @@ type Provider struct { // so validateProviders rejects it elsewhere, the same rule every other // type-scoped field in this struct follows. BinaryPath string `json:"binary_path,omitempty"` - // ExtraArgs are appended verbatim to the `claude` invocation, after - // every flag the engine itself constructs (--input-format, - // --output-format, --verbose, --model, --resume, --permission-mode — - // see engine/claude_code_backend.go). This is the escape hatch for a - // flag this Provider struct has no dedicated field for (e.g. - // --append-system-prompt, --mcp-config, --allowedTools) — the engine - // deliberately does not auto-populate either of those two itself for - // v1 (see that file's package doc for why). Valid ONLY on a - // TypeClaudeCodeCLI entry. Merge semantics are additive like every - // other Provider slice field (see NoPromptCacheKey's doc comment): a - // non-empty project-layer list replaces the user-layer list wholesale. + // ExtraArgs are appended after the flags the engine constructs. Use this + // escape hatch only for flags without a dedicated Provider field, such as + // --allowedTools. When append_system_prompt is non-empty, validation + // rejects --append-system-prompt and --append-system-prompt-file here. + // Either option would replace the managed prompt or make Claude Code reject + // the invocation. Without append_system_prompt, the prompt option remains a + // supported legacy escape hatch. Valid only on TypeClaudeCodeCLI entries. + // A non-empty project list replaces the base list wholesale. ExtraArgs []string `json:"extra_args,omitempty"` // PermissionMode selects the `claude` child's --permission-mode flag // (one of ClaudeCodePermissionModeValues — "default", "acceptEdits", @@ -1132,6 +1141,9 @@ func Path() string { // - SkillsDirs, AgentDefsDirs: a non-empty project slice replaces the user // slice entirely (arrays override, they do not concatenate); an // empty/omitted project value inherits the user value. +// - AppendSystemPrompt: the ONE additive key. The user (platform) segments +// come first, then the project segments; neither layer can drop the +// other's. See the field's own doc comment. // - Aliases, Providers: maps are merged key by key — project keys are added // and override user keys of the same name. Within a Provider, a non-empty // project field (APIKeyEnv, BaseURL) overrides the user field. @@ -1241,9 +1253,38 @@ func mergeAndValidate(base, over *Config) (*Config, error) { if err := validateProviders(out.Providers); err != nil { return nil, fmt.Errorf("config: %w", err) } + if err := validateAppendSystemPromptArgs(out); err != nil { + return nil, fmt.Errorf("config: %w", err) + } return out, nil } +// validateAppendSystemPromptArgs prevents ExtraArgs from replacing the managed +// value or selecting the mutually exclusive file option. +func validateAppendSystemPromptArgs(cfg *Config) error { + if len(cfg.AppendSystemPrompt) == 0 { + return nil + } + for name, p := range cfg.Providers { + if p.Type != TypeClaudeCodeCLI { + continue + } + for _, arg := range p.ExtraArgs { + if claudeCodeAppendPromptArg(arg) { + return fmt.Errorf("providers.%s.extra_args: %q conflicts with append_system_prompt", name, arg) + } + } + } + return nil +} + +func claudeCodeAppendPromptArg(arg string) bool { + return arg == "--append-system-prompt" || + strings.HasPrefix(arg, "--append-system-prompt=") || + arg == "--append-system-prompt-file" || + strings.HasPrefix(arg, "--append-system-prompt-file=") +} + // merge returns base overlaid with the non-zero fields of over. The result // never aliases either input's maps: fresh maps are always built, even when // over contributes no entries, so mutating the merged config cannot corrupt @@ -1332,6 +1373,16 @@ func merge(base, over *Config) *Config { if len(agentDefsSrc) > 0 { out.AgentDefsDirs = append([]string(nil), agentDefsSrc...) } + // AppendSystemPrompt CONCATENATES, base first — the one additive slice + // rule in this function. See the field's own doc comment for why a + // project layer must not be able to drop a platform-supplied segment. + // A fresh slice, so the merged config aliases neither input. + if n := len(base.AppendSystemPrompt) + len(over.AppendSystemPrompt); n > 0 { + segs := make([]string, 0, n) + segs = append(segs, base.AppendSystemPrompt...) + segs = append(segs, over.AppendSystemPrompt...) + out.AppendSystemPrompt = segs + } if n := len(base.Aliases) + len(over.Aliases); n > 0 { m := make(map[string]string, n) for k, v := range base.Aliases { diff --git a/docs/deploy-modal.md b/docs/deploy-modal.md index 8d9d4c4c..98b72978 100644 --- a/docs/deploy-modal.md +++ b/docs/deploy-modal.md @@ -198,21 +198,37 @@ Agent Skills discovered under `/.agents/skills` (or config `skills_dirs` / the repeatable `-skills-dir` flag) are advertised the same way, so a cloned repo's skills are offered to box sessions automatically. +### Telling box sessions about the environment + +Config `append_system_prompt` is an array of environment facts for every +session created by `serve` or `run`. Use it for facts an agent cannot discover, +such as a gateway URL template or a required `0.0.0.0` bind address. + +The key merges additively. Platform entries come first, then entries from the +cloned repository's `.harness.json`. A repository can add facts but cannot +remove platform entries through this key. A `claude-code/*` session sends the +entries as one `--append-system-prompt` value. Do not also put either Claude +Code append-prompt option in provider `extra_args`; Harness rejects that +conflict. See [engine-request-cycle.md](engine-request-cycle.md). + ### Verifying what reaches the model -Because the box injects instructions and skills silently, you sometimes want to -confirm they actually landed in the prompt. Three surfaces answer that without -guesswork: `GET /session/{id}/request` returns the exact request the process -most recently assembled for a session — the ordered system segments, offered -tool names, message count, and sampling params — read from memory (full requests -are never persisted, so a session that has not prompted this process is `404`). -Every turn also journals a durable `request.meta` event carrying the system -hash, segment/tool/message counts, and (only when the hash changes) the full -system, so a `from=0` replay reconstructs exactly what each turn sent. And the -built-in `session_info` tool lets the model itself report the instructions -provenance, discovered skills, and system segments it received this turn. The -`e2e/` suite asserts a real repo's `AGENTS.md` body and skill catalog line reach -the assembled system, in that order, so this contract is CI-enforced. +For native providers, three surfaces show what Harness assembled: +`GET /session/{id}/request`, durable `request.meta` events, and the built-in +`session_info` tool. The request endpoint contains the ordered system segments, +tool names, message count, and sampling parameters. Harness stores full requests +in memory only, so a session that has not prompted in this process returns +`404`. A `request.meta` event includes the system hash and counts. It includes +the full system only when the hash changes. + +Delegated Claude Code turns do not use Harness request assembly. They therefore +do not populate these three native-request surfaces. Use child-process argv +logging or Claude Code diagnostics to verify its effective appended prompt. +The engine tests assert the exact managed CLI option. + +The `e2e/` suite verifies that a native request contains the exact configured +segments in their expected positions. It also verifies project instructions +and the skill catalog in their expected order. ## Inspecting sessions diff --git a/docs/engine-request-cycle.md b/docs/engine-request-cycle.md index c6c4ee87..2ec65bb1 100644 --- a/docs/engine-request-cycle.md +++ b/docs/engine-request-cycle.md @@ -75,6 +75,37 @@ part, and never make the guidance trust bracketed text syntax again. (Fix: the NEP ambient trust-spoofing finding; superseded PR #113's prose-only stopgap.) +## Appended system prompt (`append_system_prompt`) + +Config key `append_system_prompt` is an array of environment facts. Use it for +facts an agent cannot discover, such as a gateway URL or required bind address. +Do not use it for tool instructions or project instructions. + +The engine places entries after `Config.System` and before its generated +segments. `serve` and `run` both set `Config.AppendSystemPrompt`. For `run`, +configured entries come before the `-system` value. + +The merge is additive. User-config entries come first, then project-config +entries. Other config slices replace the user value. This field differs because +the user file can belong to the platform while `.harness.json` belongs to the +cloned repository. Replacement would remove platform environment facts. + +`serve` loads config once from its process working directory. A session with a +different working directory still receives that process config. Harness does +not load another `.harness.json` for each session. + +Claude Code receives one blank-line-joined `--append-system-prompt` value. +`Config.System` is not forwarded because it describes native Harness tools. +Claude Code keeps only the last repeated prompt option. Therefore, config +validation and the engine reject `--append-system-prompt` and +`--append-system-prompt-file` in `ExtraArgs` when this key is present. Without +this key, `--append-system-prompt` remains a legacy `ExtraArgs` escape hatch. + +The joined value crosses the operating system argument boundary. Keep these +environment facts short. The operating system can reject an unusually large +argument; Harness cannot derive one portable limit because the environment and +other arguments share that limit. + ## Project instructions (AGENTS.md) The engine auto-injects a project's `AGENTS.md` into the system prompt. On the diff --git a/e2e/e2e_test.go b/e2e/e2e_test.go index c34070de..7fc2eb96 100644 --- a/e2e/e2e_test.go +++ b/e2e/e2e_test.go @@ -1208,3 +1208,74 @@ func TestServeLogsConfigSummary(t *testing.T) { } }) } + +// TestAppendSystemPromptReachesModelOnServe drives the real serve path with +// platform and project config layers. It asserts the complete system slice so +// missing, reordered, duplicated, and surplus segments fail. +func TestAppendSystemPromptReachesModelOnServe(t *testing.T) { + skipShort(t) + + fake := newFakeAnthropic(0) + srv := httptest.NewServer(fake) + t.Cleanup(srv.Close) + t.Cleanup(fake.close) + + const platformSeg = "PLATFORM: bind 0.0.0.0, never 127.0.0.1." + const repoSeg = "REPO: the dev server runs on port 5173." + + cfg := map[string]any{ + "model": "anthropic/claude-fable-5", + "providers": map[string]any{ + "anthropic": map[string]any{ + "api_key_env": "ANTHROPIC_API_KEY", + "base_url": srv.URL, + }, + }, + "append_system_prompt": []string{platformSeg}, + } + b, err := json.Marshal(cfg) + if err != nil { + t.Fatalf("marshal config: %v", err) + } + cfgPath := filepath.Join(t.TempDir(), "config.json") + if err := os.WriteFile(cfgPath, b, 0o644); err != nil { + t.Fatalf("write config: %v", err) + } + + workDir := t.TempDir() + projCfg := []byte(`{"append_system_prompt": ["` + repoSeg + `"]}`) + if err := os.WriteFile(filepath.Join(workDir, ".harness.json"), projCfg, 0o644); err != nil { + t.Fatalf("write project config: %v", err) + } + + p := startServeIn(t, t.TempDir(), cfgPath, workDir) + id := p.createSession() + p.prompt(id, "hello") + p.waitMessages(id, 2) + + resp, data := p.do(http.MethodGet, "/session/"+id+"/request", nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET /request status %d: %s", resp.StatusCode, data) + } + var rq struct { + System []string `json:"system"` + } + if err := json.Unmarshal(data, &rq); err != nil { + t.Fatalf("decode /request: %v (%s)", err, data) + } + if len(rq.System) != 4 { + t.Fatalf("assembled system has %d segments, want exactly 4: %q", len(rq.System), rq.System) + } + if rq.System[1] != platformSeg { + t.Errorf("system[1] = %q, want exact platform segment %q", rq.System[1], platformSeg) + } + if rq.System[2] != repoSeg { + t.Errorf("system[2] = %q, want exact repository segment %q", rq.System[2], repoSeg) + } + if !strings.Contains(rq.System[0], "Working directory: "+workDir) { + t.Errorf("system[0] is not the base prompt for %q: %q", workDir, rq.System[0]) + } + if !strings.HasPrefix(rq.System[3], "If you intend to call multiple tools") { + t.Errorf("system[3] is not the tool-batching segment: %q", rq.System[3]) + } +} diff --git a/engine/append_system_prompt_test.go b/engine/append_system_prompt_test.go new file mode 100644 index 00000000..411feca5 --- /dev/null +++ b/engine/append_system_prompt_test.go @@ -0,0 +1,72 @@ +package engine + +import ( + "context" + "strings" + "testing" +) + +func TestAppendSystemPromptNativeOrder(t *testing.T) { + system := batchingSystem(t, Config{ + System: []string{"base"}, + AppendSystemPrompt: []string{"platform", "project"}, + }) + if len(system) != 4 { + t.Fatalf("system = %v, want four segments", system) + } + if system[0] != "base" || system[1] != "platform" || system[2] != "project" || !isBatchingSegment(system[3]) { + t.Errorf("unexpected system order: %v", system) + } +} + +func TestClaudeCodeAppendSystemPromptArgv(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.cfg.System = []string{"native-only"} + s.cfg.AppendSystemPrompt = []string{" first ", "gateway → --append-system-prompt"} + + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatal(err) + } + argv := readInvocations(t, logPath)[0] + got, ok := argvValueAfter(argv, "--append-system-prompt") + if !ok || got != " first \n\ngateway → --append-system-prompt" { + t.Errorf("append arg = %q, %v; argv=%v", got, ok, argv) + } + var count int + for _, arg := range argv { + if arg == "--append-system-prompt" { + count++ + } + if strings.Contains(arg, "native-only") { + t.Errorf("Config.System reached Claude Code argv: %q", arg) + } + } + if count != 1 { + t.Errorf("append flag count = %d, want 1", count) + } +} + +func TestClaudeCodeAppendSystemPromptValidation(t *testing.T) { + tests := []struct { + name string + segs []string + args []string + want string + }{ + {"prompt", []string{"platform"}, []string{"--append-system-prompt", "project"}, "ExtraArgs"}, + {"prompt equals", []string{"platform"}, []string{"--append-system-prompt=project"}, "ExtraArgs"}, + {"prompt file", []string{"platform"}, []string{"--append-system-prompt-file", "file"}, "ExtraArgs"}, + {"prompt file equals", []string{"platform"}, []string{"--append-system-prompt-file=file"}, "ExtraArgs"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + s.cfg.AppendSystemPrompt = tt.segs + s.cfg.ClaudeCode.ExtraArgs = tt.args + _, err := s.Prompt(context.Background(), "hi") + if err == nil || !strings.Contains(err.Error(), tt.want) { + t.Fatalf("Prompt error = %v, want text %q", err, tt.want) + } + }) + } +} diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index d3e2d0de..abcca8f1 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -130,9 +130,11 @@ // is deliberately native-loop-only (PromptWithOrigin's dispatch comment), // Claude Code already discovers its own CLAUDE.md/AGENTS.md in the box // workspace, and re-injecting harness's own native-tool-shaped instructions -// into a CLI with different tools would be actively misleading. An operator -// who wants extra injected wording can still reach the flag via -// config.Provider.ExtraArgs. +// into a CLI with different tools would be actively misleading. +// +// AppendSystemPrompt is forwarded because it contains environment facts, not +// native tool instructions. The engine sends one joined option and rejects +// conflicting append-prompt options in ExtraArgs. // // Full goal-loop support is now PARTIAL rather than absent: a delegated // turn's error is wrapped provider.RetryableError (see @@ -205,10 +207,8 @@ type ClaudeCodeConfig struct { // BinaryPath is the `claude` executable to spawn, resolved via PATH // like any exec. Empty defaults to "claude" (newSession). BinaryPath string - // ExtraArgs are appended verbatim after every flag this file - // constructs itself. See config.Provider.ExtraArgs's doc comment for - // the escape-hatch flags this is for (--append-system-prompt, - // --mcp-config, --allowedTools, ...). + // ExtraArgs follow engine-owned flags. Append-prompt options conflict with + // AppendSystemPrompt and are rejected when that field is set. ExtraArgs []string // PermissionMode, if non-empty, becomes --permission-mode . PermissionMode string @@ -306,6 +306,15 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro } model := s.Model() + appendPrompt, haveAppendPrompt := claudeCodeAppendSystemPrompt(s.cfg.AppendSystemPrompt) + if haveAppendPrompt { + for _, arg := range cfg.ExtraArgs { + if claudeCodeAppendPromptArg(arg) { + return nil, fmt.Errorf("engine: claude-code: Config.ClaudeCode.ExtraArgs contains %q, which conflicts with Config.AppendSystemPrompt; remove the extra arg and put the text in AppendSystemPrompt", arg) + } + } + } + // The --mcp-config file (if s.cfg.MCP has any servers to describe) is // written before the args slice below so its path can be included, and // removed unconditionally on every return path via cleanupMCPConfig — @@ -345,6 +354,9 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro if effort, ok := claudeCodeEffortArg(s.Effort()); ok { args = append(args, "--effort", effort) } + if haveAppendPrompt { + args = append(args, "--append-system-prompt", appendPrompt) + } if mcpConfigPath != "" { // --strict-mcp-config: only harness's own configured servers are // visible to the child, never whatever a project-local .mcp.json or @@ -1212,6 +1224,22 @@ func claudeCodeEffortArg(e message.Effort) (string, bool) { } } +// claudeCodeAppendSystemPrompt joins segments for the CLI's single-value +// option. Repeating the option would make Claude Code keep only the last value. +func claudeCodeAppendSystemPrompt(segments []string) (string, bool) { + if len(segments) == 0 { + return "", false + } + return strings.Join(segments, "\n\n"), true +} + +func claudeCodeAppendPromptArg(arg string) bool { + return arg == "--append-system-prompt" || + strings.HasPrefix(arg, "--append-system-prompt=") || + arg == "--append-system-prompt-file" || + strings.HasPrefix(arg, "--append-system-prompt-file=") +} + // claudeCodeRetryableClass classifies a "result" event's own reported // failure — subtype plus the human-readable result text — as provider- // weather retryable, mirroring how a native adapter classifies an HTTP diff --git a/engine/engine.go b/engine/engine.go index 3635f30f..d63be3fd 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -298,6 +298,21 @@ type Config struct { MaxTokens int // per-response cap; defaults to 8192 WorkDir string // working directory for built-in tools + // AppendSystemPrompt carries OPERATOR-supplied system prompt segments — + // environment truth the agent cannot otherwise discover (a gateway URL + // template, a required bind address), not tool shape. Each entry becomes + // one system segment, in order, directly after System and before every + // engine-assembled segment (tool batching, instructions, skills, the MCP + // catalog, plugin transforms). Config key `append_system_prompt` (see + // config.Config, whose merge for this key is additive so a project layer + // cannot drop a platform segment). + // + // Unlike System, these segments DO travel to the delegated Claude Code + // CLI, as one blank-line-joined --append-system-prompt value — see + // runClaudeCodeTurn and claude_code_backend.go's package doc for why the + // two are treated differently. + AppendSystemPrompt []string + // ClaudeCode configures the delegated-turn backend a session whose // Model.Provider is ClaudeCodeProviderFamily is driven through (see // engine/claude_code_backend.go) — the non-HTTP counterpart to @@ -2762,6 +2777,11 @@ func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message tools, mcpCatalog := s.toolDefsWithCatalog(ctx) system := append([]string(nil), s.cfg.System...) + // Operator-supplied environment segments (config `append_system_prompt`) + // sit with the base prompt, ahead of every engine-assembled segment: + // they describe the environment the session runs in, which the model + // needs before it reads anything about tools, the project, or skills. + system = append(system, s.cfg.AppendSystemPrompt...) // Tool-batching guidance sits with the base system prompt, ahead of // project instructions: it describes how this engine executes tools, // not anything about the project. Empty for a session that runs tools From 296f99b150970b78485c5ca164c925a679fd4adb Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Mon, 31 Aug 2026 21:57:27 -0400 Subject: [PATCH 33/95] feat(engine,server): harness-hosted MCP server for the claude-code lane (#222) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): give a delegated claude-code turn its prior history back A session that switches to the claude-code lane mid-conversation, or reaches its first-ever claude-code turn after native-loop history already exists, started blind: runClaudeCodeTurn sends the `claude` CLI only the single pending user message over stdin and otherwise relies on --resume, which is empty on that first turn. s.History() was never conveyed, so the CLI had no way to see what already happened. Seeding history over stdin does not work: a stream-json "user" input line gets live re-executed by the CLI instead of replayed, and an "assistant" input line is either dropped or crashes the CLI outright (confirmed by empirical spike, not assumed). The fix is pull, not push: harness now hosts an MCP tool, get_conversation_history, that a delegated `claude` process calls back into over HTTP. Once it does, the tool_result lands in the CLI's own session, so --resume carries it forward on every later turn for free — one pull is enough. This adds: - package mcpserver: a small, engine/server-independent MCP SERVER-role implementation (initialize, tools/list, tools/call) over the Streamable HTTP transport, mirroring package mcp's existing client-only role. Every response is a single JSON object; this transport issues no session ID, since the caller's own URL already carries session identity one layer up. - server: POST /session/{id}/mcp (server/mcp_history.go) resolves the session the same way every other {id} route does and serves an mcpserver.Registry wired with get_conversation_history. flattenHistory renders a session's message.Message history into readable "User:"/"Assistant:" lines plus one-line tool call/result summaries (truncated past historyContentTruncateBytes), with offset/limit pagination so a long history can be pulled in chunks; historyResultText wraps a page with an explicit PRIOR-context label and a next_offset hint. The route carries the same bearer-token gate as every other session route. - engine: ClaudeCodeConfig gains HTTPBaseURL/HTTPAuthToken, naming the harness HTTP server's own loopback base URL and bearer token. claudeCodeMCPConfigFile adds a synthetic "harness-history" --mcp-config entry pointing at /session//mcp whenever HTTPBaseURL is set, alongside whatever servers Config.MCP itself configures — never as an inline argv value, so the bearer token never appears in argv. cmd/harness's serveCmd wires HTTPBaseURL from serveURLForAddr(addr) (the same loopback-rewritten URL already used for the plugin host) and HTTPAuthToken from its own RunToken; `harness run` sets neither, since it serves no HTTP API for a delegated turn to call back into. - engine: runClaudeCodeTurn appends --append-system-prompt with a short catch-up directive (claudeCodeHistoryDirectiveArgs) exactly when this is a session's first-ever claude-code turn AND it already has prior history — resumeID == "" and len(history) > 1. A later, --resume'd turn never repeats it: the CLI's own resumed session already carries the earlier tool_result forward. Verified: `go build ./...`, `gofmt -l .` (empty), `go vet ./...`, and `go test -race ./...` all pass repository-wide. New tests were red-verified against a real `claude` stand-in (engine's fakeclaude) driving actual argv construction, and against real *engine.Session history over the actual HTTP route (server's httptest harness) — not just the isolated pure functions. * fix(engine): re-fire the history directive after a switch back from native Review of the get_conversation_history shim found the first-turn directive missed history on a model switch-back. claudeCodeCLISessionID is never cleared when a session leaves the claude-code lane (a later switch back still --resumes the same CLI session), so gating the directive on resumeID == "" only worked for a session's FIRST claude stretch: native -> claude-code (directive fires, history pulled, CLI session id recorded) -> native again (more turns) -> claude-code again (resumeID is still set, so no directive fires) left the CLI resuming a session that never saw the intervening native turns, with nothing left to prompt a re-pull. The fix replaces resumeID's emptiness with a watermark: a new Session.claudeCodeHistoryWatermark field (persisted as claude_code.history_watermark, restored by LoadSession exactly like claudeCodeCLISessionID) records len(s.History()) at the end of every delegated turn that actually started. claudeCodeHistoryDirectiveArgs now fires whenever the pending turn's PRIOR history (len(history)-1) exceeds that watermark, and stays silent when it is caught up — covering the original first-turn case (watermark 0, prior history exists), consecutive claude-code turns (the last turn's own watermark already accounts for everything now in history), and the switch-back case (intervening native turns grow history past the watermark even though --resume still names the stale, never-cleared CLI session). Also, from the same review: - server/mcp_history.go: historyResultText printed "Showing messages X-Y of N" before checking Returned == 0, so a clamped out-of-range offset rendered the nonsensical "Showing messages 5-4 of 4" ahead of "(no messages in the requested range)". The range line now only prints when Returned > 0. - mcpserver.go: initialize echoed the client's requested protocolVersion verbatim, including one this server does not implement. It now always reports its own single supported revision (protocolVersion), never the client's; the now-dead initializeParams/listToolsParams types (this server never reads the initialize request body or a tools/list cursor) are removed. Verification: `go build ./...`, `gofmt -l .` (empty), `go vet ./...`, and `go test -race ./...` all pass repository-wide. Each of the three fixes was red-verified individually by temporarily reintroducing the exact old mechanism and confirming the new test fails against it, then restoring the fix: - TestClaudeCodeHistoryDirectiveRefiresAfterSwitchBackFromNative (and TestClaudeCodeHistoryDirectiveAbsentOnConsecutiveClaudeTurns, which also caught the watermark-update being skipped) failed against a reinstated resumeID-only gate. - TestHistoryResultTextClampedOffsetOmitsNonsensicalShowingLine failed against the original line ordering. - TestRegistryInitializeReportsOwnVersionNotClientsUnsupportedOne failed against the original echo-the-client's-version logic. TestClaudeCodeSessionIDResumedAcrossTurns was extended to also assert the watermark survives a LoadSession reload at the exact pre-reload value, and that a post-reload consecutive turn still carries no directive. * feat(engine,server): generalize the history MCP shim into a tools server The harness-hosted MCP endpoint (POST /session/{id}/mcp) served exactly one tool, get_conversation_history, for a delegated Claude Code CLI turn to pull prior conversation history through. That turn has no other way to reach any NATIVE harness tool either: it drives its own tool loop entirely inside the `claude` binary, never through this package's own runToolCall path, so a box's managed long-lived processes (the `process` session tool, engine/process.go — start a `pnpm dev` server, check its status, read its logs) were simply unreachable from a delegated turn. This adds a generic external-dispatch seam and uses it to expose the `process` tool alongside history, without duplicating either the process tool's own schema or its execution logic: - engine.Session.RunTool(ctx, name, args) synthesizes a *message.ToolCall and drives it through the existing runToolCall path — the same hook (ToolExecuteBefore/After), event (tool.execute.start/end), and panic-recovery machinery a native-loop call gets. Its ToolCall/ToolResult are never appended to session history (there is no assistant message in THIS session's transcript to pair them with), mirroring MCPCall's identical "pass through, do not persist" contract. - engine.Session.ToolDef(name) returns a registered tool's Description/InputSchema, so a caller outside the native loop can advertise a native tool's REAL schema instead of a hand-copied one that could drift. - engine.ProcessToolName exports the process tool's name for callers outside package engine. server/mcp_history.go's registry now registers get_conversation_history (annotations: readOnlyHint) unconditionally, and, whenever Config.Processes is configured, `process` (annotations: destructiveHint — it can stop/kill a running process) via ToolDef + RunTool. A session with no Processes configured advertises only history, matching the native loop's own "process tool absent when unconfigured" rule. The --mcp-config server-name constant is renamed claudeCodeHistoryServerName -> claudeCodeToolsServerName ("harness-tools") to match the endpoint's now-broader scope; this is a pure identifier rename with no behavior change. Deliberately NOT exposed: the redundant file tools (read/write/edit/ glob/grep/ls — a delegated `claude` process already has its own native equivalents) and the loop-internal ones (session_info/goal/model/mcp/ task/read_tool_result — meaningless outside the native agentic loop this MCP surface exists to route around). Also from the same design pass: - mcpserver.go's protocolVersion doc comment described the OLD echo-the-client's-version behavior (already replaced in a prior fix) and referenced a handleInitialize function that does not exist; rewritten to match the current, actual behavior — this server always reports its own single supported revision (2025-11-25, matching the repo's own MCP CLIENT package so the two stay in lockstep), and documents why (the sole client, a delegated Claude Code CLI turn, is documented to fall back gracefully to an older server-reported revision). - Registry.ServeHTTP now validates the Origin header per the transport spec's DNS-rebinding security warning: absent (this server's real consumer, a subprocess's own HTTP client, sends no Origin at all) or loopback (localhost/127.0.0.1/::1) passes; a present cross-origin value gets 403. The harness HTTP server binds 127.0.0.1-only, but Origin is the one thing a browser-driven same-machine request could still carry that a genuine local caller would not, so validating it is cheap insurance beyond the loopback bind alone. - GET /process/{name}/logs (server/process_handlers.go) exposes process.Manager.Logs as {content, status} for a processes panel, behind the same auth as every other process route; 404s exactly like start/stop/restart when Processes is unconfigured or name is unknown. - server_test.go's shared newServer test harness now threads Options.Processes into every session's own engine.Config.Processes (mirroring cmd/harness/main.go's real wiring), closing a gap where no session built by the test harness could ever see the process tool at all, live or cold-loaded. Verification: `go build ./...`, `gofmt -l .` (empty), `go vet ./...`, and `go test -race ./...` all pass repository-wide. The Origin check and the doc-comment fix were each red-verified by temporarily reverting to the old behavior and confirming the corresponding new test fails against it, then restoring the fix. --- AGENTS.md | 1 + cmd/harness/main.go | 20 +- engine/claude_code_backend.go | 172 ++++++++++- engine/claude_code_backend_test.go | 335 ++++++++++++++++++++ engine/engine.go | 80 +++++ engine/process.go | 7 + engine/store.go | 36 +++ engine/tool_dispatch_test.go | 154 ++++++++++ mcpserver/AGENTS.md | 48 +++ mcpserver/doc.go | 46 +++ mcpserver/mcpserver.go | 322 ++++++++++++++++++++ mcpserver/mcpserver_test.go | 320 +++++++++++++++++++ server/mcp_history.go | 419 +++++++++++++++++++++++++ server/mcp_history_test.go | 472 +++++++++++++++++++++++++++++ server/process_handlers.go | 44 +++ server/process_handlers_test.go | 52 +++- server/server.go | 12 + server/server_test.go | 7 + 18 files changed, 2536 insertions(+), 11 deletions(-) create mode 100644 engine/tool_dispatch_test.go create mode 100644 mcpserver/AGENTS.md create mode 100644 mcpserver/doc.go create mode 100644 mcpserver/mcpserver.go create mode 100644 mcpserver/mcpserver_test.go create mode 100644 server/mcp_history.go create mode 100644 server/mcp_history_test.go diff --git a/AGENTS.md b/AGENTS.md index 159bb526..e7898741 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,7 @@ table to load scoped instructions before it edits a subsystem. | `engine/` | `engine/AGENTS.md` | | `imageclamp/` | `imageclamp/AGENTS.md` | | `mcp/` | `mcp/AGENTS.md` | +| `mcpserver/` | `mcpserver/AGENTS.md` | | `message/` | `message/AGENTS.md` | | `modelmeta/` | `modelmeta/AGENTS.md` | | `provider/` | `provider/AGENTS.md` | diff --git a/cmd/harness/main.go b/cmd/harness/main.go index 4cec960c..eba5c7e1 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -1702,7 +1702,25 @@ func serveCmd(args []string) error { // config.TypeClaudeCodeCLI entry configures (see // claudeCodeConfigFor); zero value when none is configured, // which engine.newSession defaults BinaryPath from ("claude"). - ClaudeCode: claudeCodeConfigFor(cfg, claudecode.Family), + // HTTPBaseURL/HTTPAuthToken are added on top, not part of + // claudeCodeConfigFor itself: they name THIS process's own + // `harness serve` HTTP API (serveURLForAddr(addr), the same + // loopback-rewritten URL the plugin host above already uses, + // and token, this process's own RunToken — "" when + // unauthenticated) so a delegated turn's synthetic + // get_conversation_history MCP server entry + // (claudeCodeMCPConfigFile) can reach back into this same + // process. claudeCodeConfigFor has no addr/token to work with + // (cmd/harness's `run` subcommand shares it but serves no HTTP + // API at all — see engine.ClaudeCodeConfig.HTTPBaseURL's own + // doc comment for why an empty value there is correct, not a + // gap), so this is the one place that adds them. + ClaudeCode: func() engine.ClaudeCodeConfig { + cc := claudeCodeConfigFor(cfg, claudecode.Family) + cc.HTTPBaseURL = serveURLForAddr(addr) + cc.HTTPAuthToken = token + return cc + }(), } } // monitorPage is a named local (rather than inlining monitor.Page below) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index abcca8f1..bd82b05a 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -212,6 +212,48 @@ type ClaudeCodeConfig struct { ExtraArgs []string // PermissionMode, if non-empty, becomes --permission-mode . PermissionMode string + // HTTPBaseURL is the harness HTTP server's own loopback base URL (e.g. + // "http://127.0.0.1:4096" — cmd/harness's serveCmd derives it from its + // own -addr the same way it already does for the plugin host, via + // serveURLForAddr), or "" when this session is not being served over + // HTTP at all (e.g. a one-shot `harness run`). It is the ONLY thing + // this package needs to reach the harness-hosted get_conversation_history + // MCP tool at /session/{id}/mcp (server/server.go) — see + // claudeCodeMCPConfigFile, which appends a synthetic "http" server + // entry naming /session//mcp whenever this is + // non-empty, regardless of whether Config.MCP configures any servers + // of its own. Empty disables the synthetic entry entirely: there is no + // endpoint for a delegated turn to call in that mode, so advertising + // one would just be a dead tool. + HTTPBaseURL string + // HTTPAuthToken, when non-empty, is sent as an "Authorization: Bearer + // " header on the synthetic history-server entry above — the + // same bearer scheme server.Server.authorized checks for every other + // route. Empty (e.g. a loopback-only Unauthenticated serve, per + // server.Options.Unauthenticated) omits the header entirely rather + // than sending an empty bearer value. + HTTPAuthToken string +} + +// claudeCodeToolsServerName is the synthetic --mcp-config server name +// claudeCodeMCPConfigFile registers for the harness-hosted MCP server +// (server/mcp_history.go's POST /session/{id}/mcp — get_conversation_history +// plus, when configured, the native `process` tool) — see +// ClaudeCodeConfig.HTTPBaseURL's own doc comment. A fixed, harness- +// namespaced name (never an operator-configured Config.MCP key) so it can +// never collide with one. +const claudeCodeToolsServerName = "harness-tools" + +// claudeCodeHistoryServerURL returns the synthetic history-server's own +// per-session URL — /session//mcp — or "" when +// ClaudeCodeConfig.HTTPBaseURL is unset (see its own doc comment for why +// that disables the entry entirely). +func (s *Session) claudeCodeHistoryServerURL() string { + base := strings.TrimRight(s.cfg.ClaudeCode.HTTPBaseURL, "/") + if base == "" { + return "" + } + return base + "/session/" + s.ID + "/mcp" } // claudeCodeDelegated reports whether s's CURRENT model routes to this @@ -249,6 +291,28 @@ func (s *Session) recordClaudeCodeSessionID(id string) { s.persistClaudeCodeSessionID(id) } +// claudeCodeHistoryWatermarkCount returns Session.claudeCodeHistoryWatermark +// — see its own doc comment. +func (s *Session) claudeCodeHistoryWatermarkCount() int { + s.mu.Lock() + defer s.mu.Unlock() + return s.claudeCodeHistoryWatermark +} + +// recordClaudeCodeHistoryWatermark durably records n as this session's +// claudeCodeHistoryWatermark — see that field's own doc comment. A no-op +// when n already matches the recorded value, so a turn that leaves +// s.History()'s length unchanged never writes a redundant journal record. +func (s *Session) recordClaudeCodeHistoryWatermark(n int) { + s.mu.Lock() + defer s.mu.Unlock() + if s.claudeCodeHistoryWatermark == n { + return + } + s.claudeCodeHistoryWatermark = n + s.persistClaudeCodeHistoryWatermark(n) +} + // applyClaudeCodeUsage folds a delegated turn's AGGREGATE usage (the // "result" event's own usage field, covering every internal API call // Claude Code made across the whole turn — not just the closing one) into @@ -294,7 +358,8 @@ func (s *Session) applyClaudeCodeUsage(usage provider.Usage) { // reports the failure, exactly like the native path's // interruptedTurnError partial-append behavior (engine.go). func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, error) { - text := lastUserMessageText(s.History()) + history := s.History() + text := lastUserMessageText(history) if text == "" { return nil, errors.New("engine: claude-code delegated turn found no pending user message to answer") } @@ -348,6 +413,13 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro if resumeID := s.claudeCodeSessionID(); resumeID != "" { args = append(args, "--resume", resumeID) } + // See claudeCodeHistoryDirectiveArgs's own doc comment: nil (a no-op + // append) unless history holds conversation the CLI's own resumed + // session (if any) has not already incorporated — deliberately + // independent of resumeID above, since a model switch away from + // claude-code and back leaves the CLI session id in place but can + // still leave it stale relative to history. + args = append(args, claudeCodeHistoryDirectiveArgs(history, s.claudeCodeHistoryWatermarkCount())...) if cfg.PermissionMode != "" { args = append(args, "--permission-mode", cfg.PermissionMode) } @@ -505,6 +577,20 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro }() finalMsg, started, turnErr := s.consumeClaudeCodeStream(stdout, model) + if started { + // The CLI's own session actually came up for this turn — whether + // or not turnErr is set (see claudeCodeTurnResult's own o.started + // branch for the identical "started but errored is still real + // activity" reasoning) — so by now it has incorporated everything + // currently in s.History(): either it produced those messages + // itself this turn, or an earlier turn's get_conversation_history + // pull already covered the rest. Recording that here, not only on + // a successful result, is what lets claudeCodeHistoryDirectiveArgs + // tell a genuinely stale resumed session (history grew via an + // intervening native-provider turn) apart from one that is merely + // mid-turn. + s.recordClaudeCodeHistoryWatermark(len(s.History())) + } // cmd.Wait() below does NOT reintroduce the EOF wait that // consumeClaudeCodeStream's early return on "result" just avoided, on @@ -667,6 +753,60 @@ func lastUserMessageText(history []message.Message) string { return last.Parts.Text() } +// claudeCodeHistoryDirective is the --append-system-prompt text +// runClaudeCodeTurn appends exactly once per delegated session — see +// claudeCodeHistoryDirectiveArgs. It tells the CLI to call the +// harness-hosted get_conversation_history tool (server/mcp_history.go's +// POST /session/{id}/mcp, advertised via the claudeCodeToolsServerName +// entry claudeCodeMCPConfigFile writes) before answering, so a session that +// switches to claude-code mid-conversation (or on its first-ever +// claude-code turn) does not start blind to everything that already +// happened: stream-json INPUT cannot seed prior history (a "user" line +// gets live re-executed; an "assistant" line is dropped or crashes the +// CLI — see this file's package doc for why this pull-based tool exists +// instead), so this is the one nudge that gets the CLI to pull it itself. +const claudeCodeHistoryDirective = "You are continuing a conversation that happened on another model. Before responding, call the get_conversation_history tool to read what happened so far." + +// claudeCodeHistoryDirectiveArgs returns the --append-system-prompt argv +// pair (flag plus value) whenever history holds conversation the CLI's own +// resumed session (if any) has NOT already incorporated — priorCount > +// watermark, where priorCount is len(history) minus the single pending +// trigger message runClaudeCodeTurn is about to answer (lastUserMessageText's +// own caller contract: history's last element is always that pending +// message) and watermark is Session.claudeCodeHistoryWatermarkCount(), the +// message count as of the end of whichever delegated turn last ran. nil +// (priorCount <= watermark) in two cases: +// +// - A session's genuine first-ever message: watermark is 0 (no delegated +// turn has ever run) and priorCount is also 0 (no prior history at +// all). Calling get_conversation_history would return nothing useful. +// - Consecutive claude-code turns with nothing new in between: the +// previous turn's own watermark update already accounts for +// everything currently in history, including that turn's own answer. +// The CLI's --resume'd session already carries forward whichever +// earlier turn's own get_conversation_history tool_result — re-pulling +// here would waste a call the CLI has no reason to repeat (see this +// file's package doc, "one call suffices"). +// +// This is deliberately NOT gated on Session.claudeCodeSessionID() (whether a +// CLI session id is recorded at all): a model switch away from claude-code +// and back leaves claudeCodeCLISessionID untouched (see its own doc +// comment), so a resumed CLI session can still be stale relative to +// s.history even though resumeID != "" — exactly the case an intervening +// native-provider turn (or several) produces. watermark, not resumeID's +// emptiness, is what actually answers "has the CLI's session seen +// everything currently in history". +func claudeCodeHistoryDirectiveArgs(history []message.Message, watermark int) []string { + priorCount := len(history) - 1 + if priorCount < 0 { + priorCount = 0 + } + if priorCount <= watermark { + return nil + } + return []string{"--append-system-prompt", claudeCodeHistoryDirective} +} + // consumeClaudeCodeStream reads newline-delimited stream-json events from r // (the `claude` child's stdout), appending/emitting each decoded event per // this file's package-doc mapping, and returns the last assistant @@ -1367,21 +1507,37 @@ func claudeCodeMCPServerEnv(env []string) map[string]string { // (via /proc or ps) — writing to a file only this process's own return // value names, then removing it once this call returns (the child has // already read it by then; it only needs the file at startup), keeps that -// material out of the process list. len(servers) == 0 (MCP unconfigured, -// or s.cfg.MCP does not implement claudeCodeMCPServerLister — see -// claudeCodeMCPServers) returns "", a no-op cleanup, and a nil error: MCP -// passthrough is opt-in, never a hard requirement for a delegated turn to -// proceed. +// material out of the process list. len(servers) == 0 AND no synthetic +// history-server entry (MCP unconfigured, or s.cfg.MCP does not implement +// claudeCodeMCPServerLister — see claudeCodeMCPServers — and +// ClaudeCodeConfig.HTTPBaseURL unset, e.g. a one-shot `harness run`) +// returns "", a no-op cleanup, and a nil error: MCP passthrough is +// opt-in, never a hard requirement for a delegated turn to proceed. func (s *Session) claudeCodeMCPConfigFile() (path string, cleanup func(), err error) { noop := func() {} servers := claudeCodeMCPServers(s.cfg.MCP) - if len(servers) == 0 { + historyURL := s.claudeCodeHistoryServerURL() + if len(servers) == 0 && historyURL == "" { return "", noop, nil } - cfg := claudeCodeMCPConfig{MCPServers: make(map[string]claudeCodeMCPServerSpec, len(servers))} + cfg := claudeCodeMCPConfig{MCPServers: make(map[string]claudeCodeMCPServerSpec, len(servers)+1)} for name, spec := range servers { cfg.MCPServers[name] = claudeCodeMCPServerSpecFor(spec) } + if historyURL != "" { + // A synthetic entry, not one of Config.MCP's own servers — see + // ClaudeCodeConfig.HTTPBaseURL's own doc comment. This rides the + // same temp-file mechanism as every other server here (never an + // inline argv value), so HTTPAuthToken's bearer value gets the + // same argv-visibility protection + // TestClaudeCodeMCPConfigCredentialsNeverInChildArgv already locks + // in for an operator-configured server's own credentials. + spec := claudeCodeMCPServerSpec{Type: "http", URL: historyURL} + if tok := s.cfg.ClaudeCode.HTTPAuthToken; tok != "" { + spec.Headers = map[string]string{"Authorization": "Bearer " + tok} + } + cfg.MCPServers[claudeCodeToolsServerName] = spec + } data, err := json.Marshal(cfg) if err != nil { return "", noop, fmt.Errorf("engine: claude-code: encoding --mcp-config: %w", err) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index a44b2921..2e70c626 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -244,6 +244,15 @@ func TestClaudeCodeSessionIDResumedAcrossTurns(t *testing.T) { if reloaded.claudeCodeSessionID() != "fake-session-1" { t.Errorf("reloaded claudeCodeSessionID() = %q, want fake-session-1", reloaded.claudeCodeSessionID()) } + // claudeCodeHistoryWatermark (recClaudeCodeHistoryWatermark) survives + // the same reload, alongside the session id, at the exact value the + // live session recorded — a process restart must not make the + // directive re-fire on the very next turn just because the watermark + // reset to 0. + wantWatermark := len(s.History()) + if got := reloaded.claudeCodeHistoryWatermarkCount(); got != wantWatermark { + t.Errorf("reloaded claudeCodeHistoryWatermarkCount() = %d, want %d (len(s.History()) before reload)", got, wantWatermark) + } if _, err := reloaded.Prompt(context.Background(), "third turn"); err != nil { t.Fatalf("third Prompt (post-reload): %v", err) } @@ -254,6 +263,9 @@ func TestClaudeCodeSessionIDResumedAcrossTurns(t *testing.T) { if v, ok := argvValueAfter(invocations[2], "--resume"); !ok || v != "fake-session-1" { t.Errorf("post-reload --resume = %q, ok=%v, want fake-session-1", v, ok) } + if argvContains(invocations[2], "--append-system-prompt") { + t.Errorf("post-reload invocation unexpectedly carries --append-system-prompt: %v", invocations[2]) + } } // TestClaudeCodeErrorResultReturnsError proves an IsError "result" event @@ -546,6 +558,329 @@ func TestClaudeCodeMCPConfigCredentialsNeverInChildArgv(t *testing.T) { } } +// TestClaudeCodeMCPConfigFileIncludesHistoryServer proves a session +// configured with ClaudeCodeConfig.HTTPBaseURL (the harness HTTP server's +// own loopback base URL — see cmd/harness's serve wiring) gets a synthetic +// "harness-history" entry in --mcp-config naming this session's own +// /session/{id}/mcp endpoint, alongside whatever servers Config.MCP itself +// configured. This is the fix for the bug this change closes: without this +// entry, a delegated turn has no way to reach get_conversation_history at +// all, so a session that switches to claude-code mid-conversation (or on +// its first-ever claude-code turn) starts blind. +func TestClaudeCodeMCPConfigFileIncludesHistoryServer(t *testing.T) { + mgr := NewMCPManager(map[string]MCPServerConfig{ + "fs": {Command: []string{"mcp-fs"}}, + }) + s := NewSession(Config{ + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + MCP: mgr, + ClaudeCode: ClaudeCodeConfig{ + HTTPBaseURL: "http://127.0.0.1:4096", + HTTPAuthToken: "run-token-123", + }, + }) + + path, cleanup, err := s.claudeCodeMCPConfigFile() + if err != nil { + t.Fatalf("claudeCodeMCPConfigFile: %v", err) + } + defer cleanup() + if path == "" { + t.Fatal("claudeCodeMCPConfigFile returned an empty path with HTTPBaseURL set") + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading mcp-config file: %v", err) + } + var got claudeCodeMCPConfig + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("decoding mcp-config file: %v", err) + } + + // The pre-existing configured server must still be present. + if _, ok := got.MCPServers["fs"]; !ok { + t.Errorf(`mcp-config missing "fs" server: %+v`, got.MCPServers) + } + + hist, ok := got.MCPServers[claudeCodeToolsServerName] + if !ok { + t.Fatalf("mcp-config missing %q server: %+v", claudeCodeToolsServerName, got.MCPServers) + } + wantURL := "http://127.0.0.1:4096/session/" + s.ID + "/mcp" + if hist.Type != "http" || hist.URL != wantURL { + t.Errorf("history server = %+v, want an http server naming %s", hist, wantURL) + } + if hist.Headers["Authorization"] != "Bearer run-token-123" { + t.Errorf("history server Headers = %+v, want Authorization: Bearer run-token-123", hist.Headers) + } +} + +// TestClaudeCodeMCPConfigFileHistoryServerWithNoConfiguredServers proves the +// synthetic history server entry is written even when Config.MCP has no +// servers of its own configured — HTTPBaseURL alone is enough to trigger a +// --mcp-config file, unlike len(servers)==0 with no HTTPBaseURL (see +// TestClaudeCodeMCPConfigFileEmptyWithNoServers, unchanged by this). +func TestClaudeCodeMCPConfigFileHistoryServerWithNoConfiguredServers(t *testing.T) { + s := NewSession(Config{ + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + ClaudeCode: ClaudeCodeConfig{HTTPBaseURL: "http://127.0.0.1:4096"}, + }) + path, cleanup, err := s.claudeCodeMCPConfigFile() + if err != nil { + t.Fatalf("claudeCodeMCPConfigFile: %v", err) + } + defer cleanup() + if path == "" { + t.Fatal("claudeCodeMCPConfigFile returned an empty path with HTTPBaseURL set and no other servers") + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading mcp-config file: %v", err) + } + var got claudeCodeMCPConfig + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("decoding mcp-config file: %v", err) + } + if len(got.MCPServers) != 1 { + t.Fatalf("mcp-config servers = %+v, want exactly the history server", got.MCPServers) + } + if _, ok := got.MCPServers[claudeCodeToolsServerName]; !ok { + t.Errorf("mcp-config missing %q server: %+v", claudeCodeToolsServerName, got.MCPServers) + } +} + +// TestClaudeCodeMCPConfigFileHistoryServerNoAuthHeaderWhenTokenEmpty proves +// an unset HTTPAuthToken (e.g. an Unauthenticated loopback-only serve, per +// server.Options.Unauthenticated) omits the Authorization header entirely +// rather than sending an empty bearer value. +func TestClaudeCodeMCPConfigFileHistoryServerNoAuthHeaderWhenTokenEmpty(t *testing.T) { + s := NewSession(Config{ + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + ClaudeCode: ClaudeCodeConfig{HTTPBaseURL: "http://127.0.0.1:4096"}, + }) + path, cleanup, err := s.claudeCodeMCPConfigFile() + if err != nil { + t.Fatalf("claudeCodeMCPConfigFile: %v", err) + } + defer cleanup() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("reading mcp-config file: %v", err) + } + var got claudeCodeMCPConfig + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("decoding mcp-config file: %v", err) + } + if _, ok := got.MCPServers[claudeCodeToolsServerName].Headers["Authorization"]; ok { + t.Errorf("history server Headers = %+v, want no Authorization header", got.MCPServers[claudeCodeToolsServerName].Headers) + } +} + +// TestClaudeCodeHistoryDirectiveArgs is a pure-function table test of +// claudeCodeHistoryDirectiveArgs's own watermark-based gate: the directive +// fires exactly when history holds more than `watermark` messages before +// the pending trigger message, regardless of whether a CLI session id +// happens to be recorded — see the function's own doc comment for why this +// is deliberately NOT keyed on resumeID's emptiness. +func TestClaudeCodeHistoryDirectiveArgs(t *testing.T) { + priorHistory := []message.Message{ + {ID: "1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "earlier question"}}}, + {ID: "2", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "earlier answer"}}}, + {ID: "3", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "the pending message"}}}, + } + onlyPending := []message.Message{ + {ID: "3", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "the pending message"}}}, + } + + tests := []struct { + name string + history []message.Message + watermark int + want []string + }{ + {"first turn with prior history and a zero watermark gets the directive", priorHistory, 0, []string{"--append-system-prompt", claudeCodeHistoryDirective}}, + {"first turn with no prior history gets nothing", onlyPending, 0, nil}, + {"watermark caught up to prior history gets nothing (consecutive claude turns)", priorHistory, 2, nil}, + {"watermark ahead of prior history is treated the same as caught up", priorHistory, 5, nil}, + {"watermark behind prior history gets the directive even though it is non-zero (switch-back)", priorHistory, 1, []string{"--append-system-prompt", claudeCodeHistoryDirective}}, + {"a non-zero watermark equal to the (empty) prior history gets nothing", onlyPending, 0, nil}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := claudeCodeHistoryDirectiveArgs(tt.history, tt.watermark) + if !slicesEqual(got, tt.want) { + t.Errorf("claudeCodeHistoryDirectiveArgs(len=%d, watermark=%d) = %v, want %v", len(tt.history), tt.watermark, got, tt.want) + } + }) + } +} + +func slicesEqual(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +// TestClaudeCodeHistoryDirectiveForwardedOnFirstTurnWithPriorHistory drives +// a REAL delegated turn (via fakeclaude) on a session that already carries +// prior conversation history — as if it had just switched from a native +// provider to claude-code, or was reloaded mid-conversation — and asserts +// the child's argv carries --append-system-prompt with the catch-up +// directive text. +func TestClaudeCodeHistoryDirectiveForwardedOnFirstTurnWithPriorHistory(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.append(message.Message{ + ID: "msg_prior_user", + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: "earlier question"}}, + }) + s.append(message.Message{ + ID: "msg_prior_assistant", + Role: message.RoleAssistant, + Parts: message.Parts{&message.Text{Text: "earlier answer"}}, + }) + + if _, err := s.Prompt(context.Background(), "follow up"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if len(invocations) != 1 { + t.Fatalf("invocations = %d, want 1: %+v", len(invocations), invocations) + } + got, ok := argvValueAfter(invocations[0], "--append-system-prompt") + if !ok || got != claudeCodeHistoryDirective { + t.Errorf("--append-system-prompt = %q, ok=%v, want %q", got, ok, claudeCodeHistoryDirective) + } +} + +// TestClaudeCodeHistoryDirectiveAbsentWithNoPriorHistory proves a session's +// very first message ever (nothing precedes the pending trigger message) +// gets no directive at all — there is no prior conversation to catch up +// on. TestClaudeCodeSessionIDResumedAcrossTurns already proves the second +// turn (resumeID != "") carries no --append-system-prompt; this covers the +// first turn's own "no prior history" branch explicitly. +func TestClaudeCodeHistoryDirectiveAbsentWithNoPriorHistory(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if argvContains(invocations[0], "--append-system-prompt") { + t.Errorf("argv unexpectedly carries --append-system-prompt on a session's first-ever message: %v", invocations[0]) + } +} + +// TestClaudeCodeHistoryDirectiveAbsentOnConsecutiveClaudeTurns proves two +// back-to-back claude-code turns with no intervening history growth get +// the directive only on the first one: the second turn's own watermark +// check sees priorCount == watermark (the first turn's own watermark +// update already accounted for everything now in history, including that +// turn's own answer), not priorCount > watermark. +func TestClaudeCodeHistoryDirectiveAbsentOnConsecutiveClaudeTurns(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.append(message.Message{ + ID: "msg_prior_user", + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: "earlier question"}}, + }) + s.append(message.Message{ + ID: "msg_prior_assistant", + Role: message.RoleAssistant, + Parts: message.Parts{&message.Text{Text: "earlier answer"}}, + }) + + if _, err := s.Prompt(context.Background(), "first claude turn"); err != nil { + t.Fatalf("first Prompt: %v", err) + } + if _, err := s.Prompt(context.Background(), "second claude turn"); err != nil { + t.Fatalf("second Prompt: %v", err) + } + + invocations := readInvocations(t, logPath) + if len(invocations) != 2 { + t.Fatalf("invocations = %d, want 2: %+v", len(invocations), invocations) + } + if got, ok := argvValueAfter(invocations[0], "--append-system-prompt"); !ok || got != claudeCodeHistoryDirective { + t.Errorf("first invocation --append-system-prompt = %q, ok=%v, want the directive (prior history existed)", got, ok) + } + if argvContains(invocations[1], "--append-system-prompt") { + t.Errorf("second (consecutive claude-code) invocation unexpectedly carries --append-system-prompt: %v", invocations[1]) + } + if resumeID, ok := argvValueAfter(invocations[1], "--resume"); !ok || resumeID != "fake-session-1" { + t.Errorf("second invocation --resume = %q, ok=%v, want fake-session-1", resumeID, ok) + } +} + +// TestClaudeCodeHistoryDirectiveRefiresAfterSwitchBackFromNative is the +// regression test for the bug this watermark mechanism fixes: switching +// from claude-code to a native provider and back must re-fire the +// directive, even though claudeCodeCLISessionID is never cleared by a +// model switch (see its own doc comment) and so --resume still names the +// STALE CLI session that never saw the intervening native turn. +func TestClaudeCodeHistoryDirectiveRefiresAfterSwitchBackFromNative(t *testing.T) { + bin := buildFakeClaude(t) + t.Setenv("FAKE_CLAUDE_MODE", "normal") + logPath := filepath.Join(t.TempDir(), "invocations.jsonl") + t.Setenv("FAKE_CLAUDE_LOG", logPath) + + nativeProv := &scriptedProvider{name: "native", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "native answer"}), + }} + s := NewSession(Config{ + SessionDir: t.TempDir(), + Providers: provider.Registry{nativeProv.name: nativeProv}, + Model: message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}, + ClaudeCode: ClaudeCodeConfig{BinaryPath: bin}, + }) + + // Turn 1: the session's genuine first-ever message — no prior history, + // so no directive, but the turn still records a CLI session id and a + // watermark covering this turn's own two messages (the prompt and the + // reply). + if _, err := s.Prompt(context.Background(), "first"); err != nil { + t.Fatalf("first (claude-code) Prompt: %v", err) + } + + // Switch to native and take a turn: this grows s.History() past the + // watermark the claude-code turn just recorded, WITHOUT touching + // claudeCodeCLISessionID at all. + s.SetModel(message.ModelRef{Provider: "native", Model: "m1"}) + if _, err := s.Prompt(context.Background(), "native turn"); err != nil { + t.Fatalf("native Prompt: %v", err) + } + if s.claudeCodeSessionID() != "fake-session-1" { + t.Fatalf("claudeCodeSessionID() = %q after a native turn, want it left untouched at fake-session-1", s.claudeCodeSessionID()) + } + + // Switch back to claude-code: --resume must still name the stale + // session (it was never cleared), but the directive must ALSO fire, + // since the native turn left history ahead of the watermark. + s.SetModel(message.ModelRef{Provider: ClaudeCodeProviderFamily, Model: "sonnet"}) + if _, err := s.Prompt(context.Background(), "back to claude"); err != nil { + t.Fatalf("second (claude-code) Prompt: %v", err) + } + + invocations := readInvocations(t, logPath) + if len(invocations) != 2 { + t.Fatalf("invocations = %d, want 2 (the native turn never spawns claude): %+v", len(invocations), invocations) + } + if argvContains(invocations[0], "--append-system-prompt") { + t.Errorf("first invocation unexpectedly carries --append-system-prompt (no prior history yet): %v", invocations[0]) + } + if resumeID, ok := argvValueAfter(invocations[1], "--resume"); !ok || resumeID != "fake-session-1" { + t.Errorf("second invocation --resume = %q, ok=%v, want the stale, never-cleared fake-session-1", resumeID, ok) + } + if got, ok := argvValueAfter(invocations[1], "--append-system-prompt"); !ok || got != claudeCodeHistoryDirective { + t.Errorf("second invocation --append-system-prompt = %q, ok=%v, want the catch-up directive (native turn grew history past the watermark)", got, ok) + } +} + // TestClaudeCodeEffortForwardedToChildArgv proves runClaudeCodeTurn reads // s.Effort() at turn time and forwards it as --effort, mapped through // claudeCodeEffortArg exactly as that function's own doc comment promises. diff --git a/engine/engine.go b/engine/engine.go index d63be3fd..a1680820 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -1084,6 +1084,26 @@ type Session struct { // restart. claudeCodeCLISessionID string + // claudeCodeHistoryWatermark is len(s.history) as of the end of the + // most recent delegated turn that actually started (see + // runClaudeCodeTurn's own watermark-update call and + // claudeCodeHistoryDirectiveArgs, engine/claude_code_backend.go) — + // i.e. how many of s.history's messages the CLI's OWN session + // (claudeCodeCLISessionID) has already incorporated, either by + // producing them itself or by pulling them via + // get_conversation_history. claudeCodeCLISessionID is NEVER cleared + // on a model switch away from claude-code (a later switch BACK still + // --resumes the same CLI session), so this watermark — not + // claudeCodeCLISessionID's own emptiness — is what tells + // runClaudeCodeTurn whether that resumed session is stale relative to + // s.history: a switch to a native provider and back can grow + // s.history past this watermark without ever clearing + // claudeCodeCLISessionID, and the directive must re-fire in exactly + // that case. Zero until the first delegated turn that starts + // completes. Persisted (recClaudeCodeHistoryWatermark, store.go) and + // restored by LoadSession, alongside claudeCodeCLISessionID. + claudeCodeHistoryWatermark int + // spawnedChildIDs is every child id this session has ever Spawn'd — // appended to live (Spawn, session_manager.go) and folded back from // the durable recTaskSpawned audit trail on reload (store.go's @@ -3680,6 +3700,66 @@ func (s *Session) MCPCall(ctx context.Context, server, tool string, args json.Ra return s.cfg.MCP.CallServerTool(ctx, server, tool, args) } +// ToolDef returns the provider.ToolDef (Name, Description, InputSchema) of +// the native session tool registered under name — e.g. "process" — or +// ok=false if no such tool is registered on this session (its owning +// Config field unset, e.g. Config.Processes nil for "process"; or name +// simply unrecognized). s.tools is populated once, at session construction +// (newSession), and never mutated afterward — see executeTool's own +// unsynchronized read of the same map — so this needs no locking either. +// +// This is the read half of RunTool's generic external-dispatch seam: a +// caller outside the native agentic loop (the harness-hosted MCP server, +// server/mcp_history.go) uses it to advertise a native tool's OWN +// Description/InputSchema in tools/list, rather than hand-duplicating a +// second copy of the schema that could silently drift from the real one. +func (s *Session) ToolDef(name string) (def provider.ToolDef, ok bool) { + t, ok := s.tools[name] + if !ok { + return provider.ToolDef{}, false + } + return t.Def, true +} + +// RunTool synthesizes a *message.ToolCall for name/args and drives it +// through runToolCall — the same hook (ToolExecuteBefore/ToolExecuteAfter), +// event (EventToolStart/EventToolEnd, tool.execute.start/end), and +// panic-recovery path every native-loop tool call goes through (see +// runToolCall's own doc comment). It is RunTool's write half of the same +// generic external-dispatch seam ToolDef reads from: the harness-hosted MCP +// server's `process` tool (server/mcp_history.go) is the first caller +// outside the native agentic loop, invoking a native harness tool by name +// on behalf of a REMOTE MCP client (a delegated Claude Code CLI turn) +// rather than this session's own model. +// +// Unlike a native-loop tool call, the synthesized ToolCall/its ToolResult +// are never appended to s.History(): there is no corresponding assistant +// message in THIS session's own transcript to pair them with, mirroring +// MCPCall's identical "pass through, do not persist" contract for an +// MCP-registry-routed call, just above. +// +// The returned error collapses runToolCall's separate isErr flag into one +// Go error (out's own text, wrapped) — matching mcpserver.ToolHandler's +// contract exactly (server/mcp_history.go's process-tool handler passes +// RunTool's own return straight through: any error becomes +// CallToolResult.IsError, never a protocol-level failure) — so a genuine +// TOOL failure (e.g. "no such process") and this call's own inability to +// dispatch at all (there isn't one today; s.tools falls back to an +// "unknown tool" text result, not a panic or an error return) are both +// reported the same, simple way. +func (s *Session) RunTool(ctx context.Context, name string, args json.RawMessage) (message.Parts, error) { + tc := &message.ToolCall{ + CallID: newID("call"), + Name: name, + Arguments: args, + } + out, isErr := s.runToolCall(ctx, tc) + if isErr { + return nil, fmt.Errorf("engine: tool %q: %s", name, out.Text()) + } + return out, nil +} + // shellEnv collects env additions from the shell.env hook chain. func (s *Session) shellEnv(ctx context.Context, tool, command string) map[string]string { if s.cfg.Hooks == nil { diff --git a/engine/process.go b/engine/process.go index c1e6306c..8b90fb81 100644 --- a/engine/process.go +++ b/engine/process.go @@ -49,6 +49,13 @@ type ProcessRegistry interface { // processToolName is the session tool's fixed name. const processToolName = "process" +// ProcessToolName exports processToolName for a caller outside this +// package that needs to name the SAME tool RunTool/ToolDef dispatch by — +// server/mcp_history.go's harness-hosted MCP `process` tool entry, +// notably — without hand-duplicating the literal "process" and risking it +// silently drifting from this package's own internal name. +const ProcessToolName = processToolName + // defaultLogTail is the logs action's default tail line count when the // caller omits it. const defaultLogTail = 50 diff --git a/engine/store.go b/engine/store.go index 0b031f52..0d5021b1 100644 --- a/engine/store.go +++ b/engine/store.go @@ -193,6 +193,16 @@ const ( // exactly: a no-op record type with no lifecycle meaning beyond "the // value changed", replayed last-writer-wins. recClaudeCodeSessionID = "claude_code.session_id" + // recClaudeCodeHistoryWatermark records + // Session.claudeCodeHistoryWatermark (see its own doc comment) — + // written at the end of every delegated turn that actually started, + // folded back on LoadSession alongside recClaudeCodeSessionID so a + // process restart does not lose track of how much history the CLI's + // resumed session has already incorporated. Mirrors + // recClaudeCodeSessionID exactly: a no-op record type with no + // lifecycle meaning beyond "the value changed", replayed + // last-writer-wins. + recClaudeCodeHistoryWatermark = "claude_code.history_watermark" // recClaudeCodeUsage carries one delegated turn's AGGREGATE Usage (see // Session.applyClaudeCodeUsage's own doc comment for why this is a // dedicated record rather than riding a recMessage the way a native @@ -320,6 +330,13 @@ type record struct { // payload: the Claude Code CLI's own session id (see // Session.claudeCodeCLISessionID). Empty on every other record type. ClaudeCodeSessionID string `json:"claude_code_session_id,omitempty"` + // ClaudeCodeHistoryWatermark carries a recClaudeCodeHistoryWatermark + // record's payload (see Session.claudeCodeHistoryWatermark). Zero on + // every other record type; also indistinguishable from an explicit + // watermark of 0, which is harmless — persistClaudeCodeHistoryWatermark + // is never called with 0 in practice (a delegated turn always appends + // at least the pending trigger message before this is recorded). + ClaudeCodeHistoryWatermark int `json:"claude_code_history_watermark,omitempty"` } // applyGoalRecord folds one goal.* record into the durable goal state a @@ -676,6 +693,23 @@ func (s *Session) persistClaudeCodeSessionID(id string) { } } +// persistClaudeCodeHistoryWatermark appends a +// claude_code.history_watermark record to the session log. It mirrors +// persistClaudeCodeSessionID exactly: a no-op until the log exists (lazy +// creation), caller holds s.mu. +func (s *Session) persistClaudeCodeHistoryWatermark(n int) { + if s.cfg.SessionDir == "" || !s.logStarted { + return + } + if err := s.ensureLog(); err != nil { + s.lastPersistErr = err + return + } + if err := s.writeRecord(record{Type: recClaudeCodeHistoryWatermark, ClaudeCodeHistoryWatermark: n}); err != nil { + s.lastPersistErr = err + } +} + // persistClaudeCodeUsage appends a claude_code.usage record to the session // log. It mirrors persistModel/persistEffort exactly: a no-op until the // log exists (lazy creation), caller holds s.mu. @@ -1498,6 +1532,8 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.effort = rec.Effort case recClaudeCodeSessionID: s.claudeCodeCLISessionID = rec.ClaudeCodeSessionID + case recClaudeCodeHistoryWatermark: + s.claudeCodeHistoryWatermark = rec.ClaudeCodeHistoryWatermark case recClaudeCodeUsage: // See Session.applyClaudeCodeUsage's own doc comment for why // this folds into BOTH cumulative usage and lastUsage, unlike diff --git a/engine/tool_dispatch_test.go b/engine/tool_dispatch_test.go new file mode 100644 index 00000000..5f7ee8d9 --- /dev/null +++ b/engine/tool_dispatch_test.go @@ -0,0 +1,154 @@ +package engine + +import ( + "context" + "encoding/json" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/plugin" + "github.com/majorcontext/harness/process" +) + +// TestRunToolDispatchesToRegisteredTool proves RunTool actually drives a +// real native tool (process) rather than merely validating its arguments: +// a "start" call through RunTool must leave the process running in the +// SAME Manager a native-loop call would have used. +func TestRunToolDispatchesToRegisteredTool(t *testing.T) { + dir := t.TempDir() + s, mgr := newProcessSession(t, dir, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", `echo "Ready in 5ms"; sleep 100`}, ReadyRegex: "Ready in .*ms"}, + }) + + parts, err := s.RunTool(context.Background(), processToolName, json.RawMessage(`{"action":"start","name":"dev"}`)) + if err != nil { + t.Fatalf("RunTool: %v", err) + } + text, ok := parts[0].(*message.Text) + if !ok { + t.Fatalf("RunTool result is not text: %#v", parts[0]) + } + var res processResult + if err := json.Unmarshal([]byte(text.Text), &res); err != nil { + t.Fatalf("RunTool result not valid JSON: %v (%s)", err, text.Text) + } + if res.State != string(process.StateReady) { + t.Fatalf("RunTool start result = %+v, want ready", res) + } + + // The SAME Manager sees it running — RunTool went through the real + // process tool, not a stand-in. + st, err := mgr.Status("dev") + if err != nil { + t.Fatalf("mgr.Status: %v", err) + } + if st.State != process.StateReady { + t.Fatalf("mgr.Status(dev) = %+v, want ready", st) + } +} + +// TestRunToolReusesHookPath proves RunTool does not bypass runToolCall's +// hook integration: ToolExecuteBefore/ToolExecuteAfter both fire, and the +// tool.execute.start/end plugin events are emitted, exactly like a +// native-loop tool call (see TestHooksIntegration for the native-loop +// equivalent this mirrors). +func TestRunToolReusesHookPath(t *testing.T) { + dir := t.TempDir() + hooks := &fakeHooks{afterSuffix: "[annotated]"} + mgr := process.NewManager(dir, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", "true"}}, + }) + s := NewSession(Config{ + WorkDir: dir, + Processes: mgr, + Hooks: hooks, + }) + + parts, err := s.RunTool(context.Background(), processToolName, json.RawMessage(`{"action":"status","name":"dev"}`)) + if err != nil { + t.Fatalf("RunTool: %v", err) + } + text, ok := parts[len(parts)-1].(*message.Text) + if !ok || text.Text != "[annotated]" { + t.Fatalf("RunTool result = %#v, want ToolExecuteAfter's own annotation appended", parts) + } + + wantTypes := []string{plugin.EventToolExecuteStart, plugin.EventToolExecuteEnd} + if len(hooks.events) != len(wantTypes) { + t.Fatalf("hook events = %+v, want types %v", hooks.events, wantTypes) + } + for i, want := range wantTypes { + if hooks.events[i].Type != want { + t.Errorf("events[%d].Type = %q, want %q", i, hooks.events[i].Type, want) + } + } +} + +// TestRunToolDeniedByToolExecuteBeforeHook proves a ToolExecuteBefore hook +// denial short-circuits RunTool exactly like it does a native-loop call: +// the process never actually starts, and the denial reason comes back as +// RunTool's own error text. +func TestRunToolDeniedByToolExecuteBeforeHook(t *testing.T) { + dir := t.TempDir() + hooks := &fakeHooks{deny: "blocked by policy"} + mgr := process.NewManager(dir, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", "sleep 100"}}, + }) + s := NewSession(Config{ + WorkDir: dir, + Processes: mgr, + Hooks: hooks, + }) + + _, err := s.RunTool(context.Background(), processToolName, json.RawMessage(`{"action":"start","name":"dev"}`)) + if err == nil || !strings.Contains(err.Error(), "blocked by policy") { + t.Fatalf("RunTool err = %v, want it to carry the hook's denial reason", err) + } + if st, statusErr := mgr.Status("dev"); statusErr == nil && st.State != "" { + t.Errorf("mgr.Status(dev) = %+v, want the process never started (denied before dispatch)", st) + } +} + +// TestRunToolUnknownToolReturnsCleanError proves RunTool never panics on an +// unrecognized name — it returns the same "unknown tool" text executeTool +// already produces for a native-loop call, wrapped as a plain error. +func TestRunToolUnknownToolReturnsCleanError(t *testing.T) { + s := NewSession(Config{}) + _, err := s.RunTool(context.Background(), "does_not_exist", json.RawMessage(`{}`)) + if err == nil { + t.Fatal("RunTool err = nil, want an error for an unrecognized tool name") + } + if !strings.Contains(err.Error(), "does_not_exist") { + t.Errorf("RunTool err = %v, want it to name the unrecognized tool", err) + } +} + +// TestToolDefReturnsRegisteredToolSchema proves ToolDef hands back the +// SAME Description/InputSchema the native loop advertises to a provider — +// server/mcp_history.go relies on this to avoid hand-duplicating the +// process tool's schema for its MCP tools/list entry. +func TestToolDefReturnsRegisteredToolSchema(t *testing.T) { + dir := t.TempDir() + s, _ := newProcessSession(t, dir, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", "true"}}, + }) + def, ok := s.ToolDef(processToolName) + if !ok { + t.Fatal("ToolDef(process) ok = false, want true with Config.Processes set") + } + want := s.tools[processToolName].Def + if def.Name != want.Name || def.Description != want.Description || string(def.InputSchema) != string(want.InputSchema) { + t.Errorf("ToolDef(process) = %+v, want it to match the registered tool's own Def exactly", def) + } +} + +// TestToolDefUnknownToolNotOK proves ToolDef reports ok=false, not a +// zero-valued lie, for a tool this session never registered (e.g. +// "process" with Config.Processes left nil). +func TestToolDefUnknownToolNotOK(t *testing.T) { + s := NewSession(Config{}) + if _, ok := s.ToolDef(processToolName); ok { + t.Error("ToolDef(process) ok = true with no Config.Processes configured, want false") + } +} diff --git a/mcpserver/AGENTS.md b/mcpserver/AGENTS.md new file mode 100644 index 00000000..e333ce52 --- /dev/null +++ b/mcpserver/AGENTS.md @@ -0,0 +1,48 @@ +# MCP server-role instructions + +These rules apply to `mcpserver/`. Harness does not merge ancestor files. If +root guidance is not active, locate the Git root and read +`/AGENTS.md`. Resolve repository paths from that root. +Read `mcp/AGENTS.md` for the client-role counterpart this package mirrors. + +## Package boundary + +Keep this package independent from `engine` and `server`. It implements the +MCP server-role JSON-RPC dispatch and the Streamable HTTP transport only, over +a caller-supplied set of tools. A concrete tool that needs `engine.Session` (or +any other harness type) is registered by its own caller (see +`server/mcp_history.go`), never added to this package. + +## Scope + +Implement initialize, notifications/initialized, tools/list, and tools/call +only. Do not add prompts, resources, roots, sampling, elicitation, or +resumable SSE streams without an explicit scope change. + +Every response is a single JSON object. Do not add a `text/event-stream` +response path unless a registered tool needs to push a server-initiated +message ahead of its own result — none does today. + +Do not add `Mcp-Session-Id` issuance or enforcement. This transport's session +identity is stateless by design; a caller that needs identity carries it in +its own URL, one layer above this package. + +`ServeHTTP` validates the `Origin` header (the transport spec's DNS-rebinding +MUST) before parsing a request body: absent or loopback passes, a present +cross-origin value is rejected with 403. Do not remove this check or relax it +to accept an arbitrary origin. + +## Errors + +Return a JSON-RPC `RPCError` (`mcp.RPCError`) for a protocol-level failure: an +unknown method, an unknown tool name, or malformed params. Return a successful +`CallToolResult` with `IsError` set for a tool-level failure (a registered +handler's own returned error). Keep this distinction — do not fold one into +the other. + +## Tests + +Use `httptest` and drive `Registry.ServeHTTP` directly. Cover initialize, +tools/list, tools/call (success, handler error, and unknown tool), unknown +method, and the notification (no-response-body) path. Do not depend on +`engine` or `server` in this package's own test suite. diff --git a/mcpserver/doc.go b/mcpserver/doc.go new file mode 100644 index 00000000..fbc07c5a --- /dev/null +++ b/mcpserver/doc.go @@ -0,0 +1,46 @@ +// Package mcpserver implements the MCP (Model Context Protocol, +// https://modelcontextprotocol.io) SERVER role, over the Streamable HTTP +// transport, for a fixed in-process set of tools. +// +// This is the mirror image of package mcp, which implements only the +// CLIENT role (mcp/doc.go) — harness had no server-role code at all before +// this package: every existing MCP server harness talks to (mcp.Client) +// runs out-of-process. This package exists so harness itself can host a +// tool a delegated Claude Code CLI turn calls back into — see +// engine/claude_code_backend.go's package doc for why that seam exists +// (get_conversation_history, the fix for a delegated turn otherwise +// starting blind to prior conversation history) and server/mcp_history.go +// for the concrete tool this package serves at POST /session/{id}/mcp. +// +// # Scope +// +// Implemented: the initialize/notifications-initialized lifecycle, +// tools/list, and tools/call — the same subset package mcp's client +// implements, mirrored server-side. Deliberately out of scope, exactly +// like package mcp's own client (see its doc comment for the identical +// list applied to the other role): OAuth 2.1 authorization (this package +// trusts whatever authenticated this HTTP request), prompts, resources, +// completion, logging subscriptions, roots, sampling, elicitation, and +// resumable SSE streams. +// +// # Session identity +// +// This transport's own optional Mcp-Session-Id concept +// (https://modelcontextprotocol.io/specification/2025-11-25/basic/transports#session-management) +// is not used: a Registry issues no session ID and enforces none on +// incoming requests. The spec allows this for a server with no +// transport-level state of its own to track, and this one has none — the +// identity a caller like harness's own /session/{id}/mcp route cares +// about (which harness session a request is for) already lives in the +// URL path one layer up, in server/server.go, not in this package. +// +// # Transport shape +// +// Every response is a single JSON object (Content-Type: application/json), +// never text/event-stream: this server has no server-initiated request or +// notification to push ahead of its own response, which is SSE's only +// advantage over a plain JSON body in this transport. A JSON-RPC +// notification (no "id" — notifications/initialized, or any other +// notification a future client version might send) gets HTTP 202 Accepted +// with an empty body, per the transport spec. +package mcpserver diff --git a/mcpserver/mcpserver.go b/mcpserver/mcpserver.go new file mode 100644 index 00000000..c41e4775 --- /dev/null +++ b/mcpserver/mcpserver.go @@ -0,0 +1,322 @@ +package mcpserver + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + + "github.com/majorcontext/harness/mcp" +) + +// protocolVersion is the ONE MCP protocol revision this server speaks — +// matches mcp.LatestProtocolVersion (package mcp's own client), the +// revision this repository has standardized on, and initialize always +// reports it verbatim (see dispatch's methodInitialize case), regardless +// of whatever protocolVersion a client's own initialize request asks for. +// This is deliberate, not a placeholder: this server implements exactly +// this one revision, so claiming support for a client-requested version it +// does not actually speak would be dishonest, and the transport spec's own +// negotiation contract expects a server to report a version it genuinely +// supports, not to echo the request. The sole intended client (a +// delegated Claude Code CLI turn, see engine/claude_code_backend.go) is +// documented to fall back gracefully when a server reports an older +// revision than it asked for, so pinning this — rather than tracking +// whatever the newest spec revision becomes — is the safer, simpler +// choice for as long as this server's own tool surface has no need for +// anything a newer revision adds. +const protocolVersion = "2025-11-25" + +// JSON-RPC 2.0 method and error-code constants — mirrors +// mcp/protocol.go's identically-named, unexported constants for the +// client role. Duplicated rather than imported: those are unexported +// package mcp internals, and importing package mcp is only for its +// EXPORTED wire types (Implementation, Tool, CallToolResult, ...) — see +// this package's own doc comment. +const ( + methodInitialize = "initialize" + methodToolsList = "tools/list" + methodToolsCall = "tools/call" + notificationInitialized = "notifications/initialized" + notificationCancelled = "notifications/cancelled" + codeParseError = -32700 + codeInvalidRequest = -32600 + codeMethodNotFound = -32601 + codeInvalidParams = -32602 + codeInternalError = -32603 +) + +// rpcMessage is the JSON-RPC 2.0 envelope this server reads and writes — +// the server-role mirror of mcp/protocol.go's unexported "message" type. +// ID is raw JSON (not a fixed Go type) because JSON-RPC permits either a +// string or a number, and a response must echo a request's ID verbatim, +// whichever shape it came in as. +type rpcMessage struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id,omitempty"` + Method string `json:"method,omitempty"` + Params json.RawMessage `json:"params,omitempty"` + Result json.RawMessage `json:"result,omitempty"` + Error *mcp.RPCError `json:"error,omitempty"` +} + +func (m rpcMessage) isNotification() bool { return m.Method != "" && len(m.ID) == 0 } + +// callToolParams is this server's own tools/call request-payload shape — +// the server-role mirror of mcp/types.go's identically-shaped, unexported +// client-role callToolParams. initialize's own request body is never +// decoded at all (see the methodInitialize case below) and tools/list +// takes no params this server reads (no pagination — see dispatch's own +// methodToolsList case), so neither gets a params struct of its own. +type callToolParams struct { + Name string `json:"name"` + Arguments json.RawMessage `json:"arguments,omitempty"` +} + +// ToolHandler executes one tools/call request for a registered tool. args +// is the request's raw "arguments" object — nil when the caller sent none +// at all, a zero-length-but-valid object ("{}") when it sent an empty one. +// +// Returning an error produces a CallToolResult with IsError set and the +// error's own message as its sole text content item — a TOOL-level +// failure (see mcp.CallToolResult's own doc comment for how this differs +// from a protocol-level RPCError, which Registry.ServeHTTP reserves for +// things like an unknown tool name) — so a handler never needs to +// construct a CallToolResult by hand just to report its own failure. +type ToolHandler func(ctx context.Context, args json.RawMessage) (mcp.CallToolResult, error) + +// Registry serves the MCP server role's Streamable HTTP transport +// (https://modelcontextprotocol.io/specification/2025-11-25/basic/transports#streamable-http) +// over a fixed, in-process set of tools — see this package's own doc +// comment for the transport shape and scope this implements. The zero +// value is not usable; construct with NewRegistry. +type Registry struct { + serverInfo mcp.Implementation + instructions string + + tools []mcp.Tool + handlers map[string]ToolHandler +} + +// NewRegistry returns an empty Registry that identifies itself as name at +// version version during initialize (mcp.InitializeResult.ServerInfo). +// RegisterTool adds tools before the first request is served — Registry +// has no locking of its own, so registration must complete before +// ServeHTTP is reachable by any client (every call site in this repo +// registers once, synchronously, right after construction — see +// server/mcp_history.go). +func NewRegistry(name, version string) *Registry { + return &Registry{ + serverInfo: mcp.Implementation{Name: name, Version: version}, + handlers: make(map[string]ToolHandler), + } +} + +// SetInstructions sets the free-text guidance returned in +// InitializeResult.Instructions — see that field's own doc comment in +// package mcp. Optional; the zero value (unset) omits the field entirely. +func (reg *Registry) SetInstructions(s string) { + reg.instructions = s +} + +// RegisterTool adds one tool to the registry: tool is advertised verbatim +// by tools/list, and a tools/call naming tool.Name dispatches to handler. +// Registering the same Name twice replaces the earlier tool/handler pair +// (last write wins) rather than duplicating an entry in tools/list. +func (reg *Registry) RegisterTool(tool mcp.Tool, handler ToolHandler) { + for i, existing := range reg.tools { + if existing.Name == tool.Name { + reg.tools[i] = tool + reg.handlers[tool.Name] = handler + return + } + } + reg.tools = append(reg.tools, tool) + reg.handlers[tool.Name] = handler +} + +// ServeHTTP implements the Streamable HTTP transport's single POST +// endpoint. Only POST is accepted (this server issues no session ID for a +// client to DELETE, and opens no independent GET listening stream — see +// this package's own doc comment) — any other method gets 405. +func (reg *Registry) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + w.WriteHeader(http.StatusMethodNotAllowed) + return + } + if !validOrigin(r) { + // The transport spec's own DNS-rebinding security warning + // (https://modelcontextprotocol.io/specification/2025-11-25/basic/transports#security-warning) + // makes Origin validation a MUST: a page loaded from an + // attacker's own site, opened in a victim's browser, can still + // issue same-machine requests to a server bound to 127.0.0.1 (the + // browser resolves and connects; the attacker's page just names + // the URL) — the Origin header is the one thing that request + // carries that a same-machine, non-browser caller's does not. + // See validOrigin's own doc comment for the accept/reject rule. + w.WriteHeader(http.StatusForbidden) + return + } + + var msg rpcMessage + if err := json.NewDecoder(r.Body).Decode(&msg); err != nil { + reg.writeError(w, nil, codeParseError, fmt.Sprintf("parse error: %v", err)) + return + } + + if msg.isNotification() { + // notifications/initialized, notifications/cancelled, or any other + // notification a future client sends: a JSON-RPC notification + // gets no response body at all, per the transport spec — this + // server has nothing to acknowledge or clean up for either one + // (RegisterTool state is fixed at construction, and a canceled + // tools/call is not tracked as a separate in-flight operation this + // stateless server could abort). + w.WriteHeader(http.StatusAccepted) + return + } + + if msg.Method == "" || len(msg.ID) == 0 { + reg.writeError(w, msg.ID, codeInvalidRequest, "invalid request: missing method or id") + return + } + + result, rerr := reg.dispatch(r.Context(), msg.Method, msg.Params) + if rerr != nil { + reg.writeError(w, msg.ID, rerr.Code, rerr.Message) + return + } + reg.writeResult(w, msg.ID, result) +} + +// validOrigin reports whether r's Origin header is safe to serve, per the +// transport spec's DNS-rebinding security warning (see ServeHTTP's own +// comment at its call site). Two cases pass: +// +// - No Origin header at all. A browser attaches Origin to every +// fetch/XHR; a plain HTTP client (net/http, or whatever the `claude` +// CLI's own MCP client uses) ordinarily does not, unless a caller +// explicitly sets it. This server's sole documented consumer — a +// delegated Claude Code CLI subprocess calling back over loopback, +// see engine/claude_code_backend.go's ClaudeCodeConfig.HTTPBaseURL — +// is exactly that case, so requiring an Origin header at all would +// break the one real client this server has. +// - An Origin naming a loopback host (localhost, 127.0.0.1, or ::1), +// any port. A same-machine tool genuinely running on the user's own +// box is the only thing that can claim this truthfully; a remote +// attacker's page cannot forge the browser's own Origin header to a +// value other than the page's real origin. +// +// Anything else — a parseable Origin naming a non-loopback host, or a +// value that fails to parse as a URL at all — is rejected: exactly the +// shape a cross-origin browser page (the DNS-rebinding attack the spec +// warns about) would send. +func validOrigin(r *http.Request) bool { + origin := r.Header.Get("Origin") + if origin == "" { + return true + } + u, err := url.Parse(origin) + if err != nil { + return false + } + switch u.Hostname() { + case "localhost", "127.0.0.1", "::1": + return true + default: + return false + } +} + +// dispatch routes one request method to its handler, returning either a +// result to marshal into the response's "result" field or a protocol-level +// *mcp.RPCError (an unknown method or tool name, or malformed params) — +// see ToolHandler's own doc comment for how this differs from a TOOL +// execution failure, which becomes a successful response carrying +// CallToolResult.IsError instead. +func (reg *Registry) dispatch(ctx context.Context, method string, params json.RawMessage) (any, *mcp.RPCError) { + switch method { + case methodInitialize: + // The request body (initializeParams) is intentionally not even + // decoded: this server implements exactly one protocol revision + // (protocolVersion) and always reports THAT version, never the + // client's own requested one. Per the transport spec, a server + // that does not support the client's requested version responds + // with a version it DOES support so the client can decide whether + // to proceed — echoing the client's request back unconditionally + // would claim support for a revision this server may not actually + // implement. + return mcp.InitializeResult{ + ProtocolVersion: protocolVersion, + Capabilities: mcp.ServerCapabilities{Tools: &mcp.ToolsCapability{}}, + ServerInfo: reg.serverInfo, + Instructions: reg.instructions, + }, nil + + case methodToolsList: + // No pagination: every Registry in this repo holds a small, fixed + // tool set (see this package's own doc comment), so there is + // nothing to page through and NextCursor is always left empty. + return mcp.ListToolsResult{Tools: append([]mcp.Tool(nil), reg.tools...)}, nil + + case methodToolsCall: + var req callToolParams + if err := json.Unmarshal(params, &req); err != nil { + return nil, &mcp.RPCError{Code: codeInvalidParams, Message: fmt.Sprintf("invalid tools/call params: %v", err)} + } + handler, ok := reg.handlers[req.Name] + if !ok { + return nil, &mcp.RPCError{Code: codeInvalidParams, Message: fmt.Sprintf("unknown tool %q", req.Name)} + } + res, err := handler(ctx, req.Arguments) + if err != nil { + return mcp.CallToolResult{ + Content: []mcp.Content{{Type: mcp.ContentTypeText, Text: err.Error()}}, + IsError: true, + }, nil + } + return res, nil + + default: + return nil, &mcp.RPCError{Code: codeMethodNotFound, Message: fmt.Sprintf("unknown method %q", method)} + } +} + +func (reg *Registry) writeResult(w http.ResponseWriter, id json.RawMessage, result any) { + raw, err := json.Marshal(result) + if err != nil { + reg.writeError(w, id, codeInternalError, fmt.Sprintf("encoding result: %v", err)) + return + } + body, err := json.Marshal(rpcMessage{JSONRPC: "2.0", ID: id, Result: raw}) + if err != nil { + w.WriteHeader(http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + _, _ = w.Write(body) +} + +// writeError writes a JSON-RPC error response. A nil id (a request this +// server could not even parse an id out of, e.g. a parse-error body) is +// written as JSON null, per the spec's guidance for that case. +func (reg *Registry) writeError(w http.ResponseWriter, id json.RawMessage, code int, message string) { + if id == nil { + id = json.RawMessage("null") + } + body, err := json.Marshal(rpcMessage{JSONRPC: "2.0", ID: id, Error: &mcp.RPCError{Code: code, Message: message}}) + if err != nil { + w.WriteHeader(http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/json") + // JSON-RPC errors still ride an HTTP 200: the error is at the JSON-RPC + // protocol layer, not the HTTP transport layer (the Streamable HTTP + // spec reserves non-2xx status codes for transport-level failures, + // e.g. an unrecognized session ID) — mirrors package mcp's own client, + // which reads msg.Error regardless of a 200 status. + w.WriteHeader(http.StatusOK) + _, _ = w.Write(body) +} diff --git a/mcpserver/mcpserver_test.go b/mcpserver/mcpserver_test.go new file mode 100644 index 00000000..6f65b602 --- /dev/null +++ b/mcpserver/mcpserver_test.go @@ -0,0 +1,320 @@ +package mcpserver + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/majorcontext/harness/mcp" +) + +// rpcResponse decodes one JSON-RPC 2.0 response body for assertions below. +type rpcResponse struct { + JSONRPC string `json:"jsonrpc"` + ID json.RawMessage `json:"id"` + Result json.RawMessage `json:"result"` + Error *mcp.RPCError `json:"error"` +} + +func post(t *testing.T, reg *Registry, method string, id string, params any) (int, rpcResponse) { + t.Helper() + body := map[string]any{"jsonrpc": "2.0", "method": method} + if id != "" { + body["id"] = id + } + if params != nil { + body["params"] = params + } + raw, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshaling request: %v", err) + } + req := httptest.NewRequest(http.MethodPost, "/mcp", strings.NewReader(string(raw))) + rec := httptest.NewRecorder() + reg.ServeHTTP(rec, req) + if id == "" { + return rec.Code, rpcResponse{} + } + var resp rpcResponse + if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil { + t.Fatalf("decoding response %s: %v", rec.Body.String(), err) + } + return rec.Code, resp +} + +// TestRegistryInitializeReturnsCapabilities proves initialize answers with +// the server's own identity and a tools capability — the minimum an MCP +// client needs to know it can proceed to tools/list. +func TestRegistryInitializeReturnsCapabilities(t *testing.T) { + reg := NewRegistry("test-server", "1.2.3") + code, resp := post(t, reg, methodInitialize, "1", map[string]any{"protocolVersion": "2025-11-25"}) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } + if resp.Error != nil { + t.Fatalf("initialize returned an error: %+v", resp.Error) + } + var result mcp.InitializeResult + if err := json.Unmarshal(resp.Result, &result); err != nil { + t.Fatalf("decoding InitializeResult: %v", err) + } + if result.ServerInfo.Name != "test-server" || result.ServerInfo.Version != "1.2.3" { + t.Errorf("ServerInfo = %+v, want Name test-server, Version 1.2.3", result.ServerInfo) + } + if result.Capabilities.Tools == nil { + t.Error("Capabilities.Tools is nil, want a non-nil tools capability") + } + if result.ProtocolVersion != protocolVersion { + t.Errorf("ProtocolVersion = %q, want this server's own supported version %q", result.ProtocolVersion, protocolVersion) + } +} + +// TestRegistryInitializeReportsOwnVersionNotClientsUnsupportedOne proves +// initialize never echoes back a client-requested protocolVersion this +// server does not actually implement — it always reports its own single +// supported revision (protocolVersion), regardless of what the client +// asked for. Echoing an arbitrary client value would falsely claim +// support for a revision this server may not speak at all. +func TestRegistryInitializeReportsOwnVersionNotClientsUnsupportedOne(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + _, resp := post(t, reg, methodInitialize, "1", map[string]any{"protocolVersion": "1999-01-01"}) + var result mcp.InitializeResult + if err := json.Unmarshal(resp.Result, &result); err != nil { + t.Fatalf("decoding InitializeResult: %v", err) + } + if result.ProtocolVersion != protocolVersion { + t.Errorf("ProtocolVersion = %q, want this server's own supported version %q, not the client's unsupported request", result.ProtocolVersion, protocolVersion) + } +} + +// TestRegistryToolsListIncludesRegisteredTools proves every RegisterTool +// call is reflected verbatim in tools/list. +func TestRegistryToolsListIncludesRegisteredTools(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + reg.RegisterTool(mcp.Tool{Name: "get_conversation_history", Description: "reads prior history"}, func(context.Context, json.RawMessage) (mcp.CallToolResult, error) { + return mcp.CallToolResult{}, nil + }) + + code, resp := post(t, reg, methodToolsList, "1", nil) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } + var result mcp.ListToolsResult + if err := json.Unmarshal(resp.Result, &result); err != nil { + t.Fatalf("decoding ListToolsResult: %v", err) + } + if len(result.Tools) != 1 || result.Tools[0].Name != "get_conversation_history" { + t.Errorf("Tools = %+v, want exactly [get_conversation_history]", result.Tools) + } +} + +// TestRegistryToolsCallDispatchesToHandlerWithArguments proves tools/call +// routes to the registered handler by name and hands it the raw arguments +// object byte-for-byte (the concrete get_conversation_history tool's own +// pagination args, e.g., ride this path unmodified). +func TestRegistryToolsCallDispatchesToHandlerWithArguments(t *testing.T) { + var gotArgs json.RawMessage + reg := NewRegistry("test-server", "1.0.0") + reg.RegisterTool(mcp.Tool{Name: "echo"}, func(_ context.Context, args json.RawMessage) (mcp.CallToolResult, error) { + gotArgs = args + return mcp.CallToolResult{Content: []mcp.Content{{Type: mcp.ContentTypeText, Text: "echoed"}}}, nil + }) + + code, resp := post(t, reg, methodToolsCall, "1", map[string]any{ + "name": "echo", + "arguments": map[string]any{"offset": 5, "limit": 10}, + }) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } + if resp.Error != nil { + t.Fatalf("tools/call returned an error: %+v", resp.Error) + } + var result mcp.CallToolResult + if err := json.Unmarshal(resp.Result, &result); err != nil { + t.Fatalf("decoding CallToolResult: %v", err) + } + if result.IsError { + t.Errorf("CallToolResult.IsError = true, want false") + } + if len(result.Content) != 1 || result.Content[0].Text != "echoed" { + t.Errorf("Content = %+v, want one text item %q", result.Content, "echoed") + } + + var args struct { + Offset int `json:"offset"` + Limit int `json:"limit"` + } + if err := json.Unmarshal(gotArgs, &args); err != nil { + t.Fatalf("decoding arguments the handler received: %v", err) + } + if args.Offset != 5 || args.Limit != 10 { + t.Errorf("handler received offset=%d limit=%d, want 5 and 10", args.Offset, args.Limit) + } +} + +// TestRegistryToolsCallHandlerErrorBecomesIsErrorResult proves a handler +// error is reported as a successful JSON-RPC response carrying +// CallToolResult.IsError — a TOOL-level failure, distinct from the +// protocol-level RPCError an unknown tool name gets (see the next test). +func TestRegistryToolsCallHandlerErrorBecomesIsErrorResult(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + reg.RegisterTool(mcp.Tool{Name: "fails"}, func(context.Context, json.RawMessage) (mcp.CallToolResult, error) { + return mcp.CallToolResult{}, errors.New("boom") + }) + + code, resp := post(t, reg, methodToolsCall, "1", map[string]any{"name": "fails"}) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } + if resp.Error != nil { + t.Fatalf("tools/call returned a protocol-level error for a tool-level failure: %+v", resp.Error) + } + var result mcp.CallToolResult + if err := json.Unmarshal(resp.Result, &result); err != nil { + t.Fatalf("decoding CallToolResult: %v", err) + } + if !result.IsError { + t.Error("CallToolResult.IsError = false, want true") + } + if len(result.Content) != 1 || result.Content[0].Text != "boom" { + t.Errorf("Content = %+v, want one text item %q", result.Content, "boom") + } +} + +// TestRegistryToolsCallUnknownToolReturnsRPCError proves an unregistered +// tool name fails cleanly as a JSON-RPC protocol error, not a panic or a +// silently empty result. +func TestRegistryToolsCallUnknownToolReturnsRPCError(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + code, resp := post(t, reg, methodToolsCall, "1", map[string]any{"name": "does_not_exist"}) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200 (JSON-RPC errors ride HTTP 200)", code) + } + if resp.Error == nil { + t.Fatal("tools/call for an unknown tool returned no error") + } + if resp.Error.Code != codeInvalidParams { + t.Errorf("error code = %d, want %d (invalid params)", resp.Error.Code, codeInvalidParams) + } +} + +// TestRegistryUnknownMethodReturnsRPCError proves an unrecognized +// top-level method fails cleanly as a JSON-RPC "method not found" error. +func TestRegistryUnknownMethodReturnsRPCError(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + code, resp := post(t, reg, "prompts/list", "1", nil) + if code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } + if resp.Error == nil { + t.Fatal("unknown method returned no error") + } + if resp.Error.Code != codeMethodNotFound { + t.Errorf("error code = %d, want %d (method not found)", resp.Error.Code, codeMethodNotFound) + } +} + +// TestRegistryNotificationGetsNoResponseBody proves a JSON-RPC +// notification (notifications/initialized, notably — the lifecycle step +// every MCP client sends right after a successful initialize) gets HTTP +// 202 with an empty body, per the Streamable HTTP transport spec, rather +// than a JSON-RPC response no one asked for. +func TestRegistryNotificationGetsNoResponseBody(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + code, _ := post(t, reg, notificationInitialized, "", nil) + if code != http.StatusAccepted { + t.Errorf("status = %d, want 202", code) + } +} + +// TestRegistryRejectsNonPOST proves this server's single endpoint refuses +// any method other than POST — it issues no session ID for a client to +// DELETE and opens no independent GET listening stream. +func TestRegistryRejectsNonPOST(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + req := httptest.NewRequest(http.MethodGet, "/mcp", nil) + rec := httptest.NewRecorder() + reg.ServeHTTP(rec, req) + if rec.Code != http.StatusMethodNotAllowed { + t.Errorf("status = %d, want 405", rec.Code) + } +} + +// postWithOrigin is post's counterpart for a caller that needs to set (or +// deliberately omit) the Origin header itself. +func postWithOrigin(t *testing.T, reg *Registry, origin string) int { + t.Helper() + body, err := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": "1", "method": methodInitialize}) + if err != nil { + t.Fatalf("marshaling request: %v", err) + } + req := httptest.NewRequest(http.MethodPost, "/mcp", strings.NewReader(string(body))) + if origin != "" { + req.Header.Set("Origin", origin) + } + rec := httptest.NewRecorder() + reg.ServeHTTP(rec, req) + return rec.Code +} + +// TestRegistryRejectsCrossOriginRequest proves a request carrying an +// Origin header naming a non-loopback host — the shape a DNS-rebinding +// attack (a page loaded from an attacker's own site, run in a victim's +// browser, issuing a same-machine request to this server's loopback bind) +// would produce — is rejected, per the transport spec's own security +// warning (see validOrigin's doc comment). +func TestRegistryRejectsCrossOriginRequest(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + if code := postWithOrigin(t, reg, "https://evil.example.com"); code != http.StatusForbidden { + t.Errorf("status = %d, want 403 for a cross-origin request", code) + } +} + +// TestRegistryAcceptsAbsentOrLoopbackOrigin proves the two safe cases +// validOrigin accepts: no Origin header at all (this server's real +// consumer, a delegated Claude Code CLI subprocess's own HTTP client — +// see validOrigin's doc comment), and an explicit loopback Origin. +func TestRegistryAcceptsAbsentOrLoopbackOrigin(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + for _, origin := range []string{"", "http://127.0.0.1:4096", "http://localhost:4096", "http://[::1]:4096"} { + if code := postWithOrigin(t, reg, origin); code != http.StatusOK { + t.Errorf("origin %q: status = %d, want 200", origin, code) + } + } +} + +// TestRegistryRegisterToolReplacesExistingByName proves registering the +// same tool name twice replaces the earlier entry rather than duplicating +// it in tools/list. +func TestRegistryRegisterToolReplacesExistingByName(t *testing.T) { + reg := NewRegistry("test-server", "1.0.0") + reg.RegisterTool(mcp.Tool{Name: "t", Description: "first"}, func(context.Context, json.RawMessage) (mcp.CallToolResult, error) { + return mcp.CallToolResult{}, nil + }) + reg.RegisterTool(mcp.Tool{Name: "t", Description: "second"}, func(context.Context, json.RawMessage) (mcp.CallToolResult, error) { + return mcp.CallToolResult{Content: []mcp.Content{{Type: mcp.ContentTypeText, Text: "second handler"}}}, nil + }) + + _, resp := post(t, reg, methodToolsList, "1", nil) + var listResult mcp.ListToolsResult + if err := json.Unmarshal(resp.Result, &listResult); err != nil { + t.Fatalf("decoding ListToolsResult: %v", err) + } + if len(listResult.Tools) != 1 || listResult.Tools[0].Description != "second" { + t.Errorf("Tools = %+v, want exactly one tool with Description \"second\"", listResult.Tools) + } + + _, callResp := post(t, reg, methodToolsCall, "1", map[string]any{"name": "t"}) + var callResult mcp.CallToolResult + if err := json.Unmarshal(callResp.Result, &callResult); err != nil { + t.Fatalf("decoding CallToolResult: %v", err) + } + if len(callResult.Content) != 1 || callResult.Content[0].Text != "second handler" { + t.Errorf("tools/call dispatched to the first handler, not the replacement: %+v", callResult.Content) + } +} diff --git a/server/mcp_history.go b/server/mcp_history.go new file mode 100644 index 00000000..8d8e529c --- /dev/null +++ b/server/mcp_history.go @@ -0,0 +1,419 @@ +// This file implements harness's own hosted MCP tools: get_conversation_history +// and, when configured, the native `process` tool. +// +// # Why this exists +// +// A session delegated to the Claude Code CLI (engine.ClaudeCodeProviderFamily +// — see engine/claude_code_backend.go) drives its turn entirely inside the +// `claude` binary's own stream-json protocol, which has no way to SEED prior +// conversation history: a stream-json "user" input line gets live +// re-executed by the CLI (expensive and nondeterministic, not a history +// replay), and an "assistant" input line is either silently dropped or +// crashes the CLI outright. So a session that switches to claude-code +// mid-conversation — or reaches its first-ever claude-code turn with +// native-loop history already behind it — has no way to hand that history +// to the CLI on stdin. +// +// The fix is pull, not push: this file registers get_conversation_history +// on the per-session Streamable HTTP endpoint POST /session/{id}/mcp (see +// handleSessionMCP and its routes() entry), backed by package mcpserver's +// generic JSON-RPC dispatch. The delegated `claude` process is handed this +// endpoint in its own --mcp-config (see engine.ClaudeCodeConfig.HTTPBaseURL +// and claudeCodeMCPConfigFile) and, on its first delegated turn with prior +// history to catch up on, a short --append-system-prompt directive +// (engine's claudeCodeHistoryDirective) tells it to call the tool before +// answering. Once it does, the tool_result lands in the CLI's OWN session +// — --resume then carries it forward on every later turn for free, so this +// tool needs to be called at most once per delegated session, not once per +// turn. +// +// # Beyond history: a generalized harness-tools server +// +// The same endpoint also advertises harness's native `process` tool +// (engine/process.go) when the session has one configured (Config.Processes +// non-nil) — the tool a box's `pnpm dev`-style long-lived processes are +// started, stopped, and inspected through. A delegated claude-code turn +// otherwise has NO way to reach it: it drives its own tool loop entirely +// inside the `claude` binary, never through this package's native +// runToolCall path, so without an MCP entry a delegated turn simply could +// not start/stop/check a managed process at all. tools/call routes through +// engine.Session.RunTool (see its own doc comment), the SAME generic +// external-dispatch seam a future harness-hosted tool would use — this file +// deliberately exposes ONLY these two tools, not the redundant file tools +// (read/write/edit/glob/grep/ls — a delegated `claude` process already has +// its own, native equivalents) or the loop-internal ones (session_info, +// goal, model, mcp, task, read_tool_result — meaningless outside the +// native agentic loop this MCP surface exists to route AROUND). + +package server + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "strings" + + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/mcp" + "github.com/majorcontext/harness/mcpserver" + "github.com/majorcontext/harness/message" +) + +const ( + // historyToolName is the MCP tool name advertised by tools/list and + // matched by tools/call — also the exact string engine's + // claudeCodeHistoryDirective tells the CLI to call. + historyToolName = "get_conversation_history" + + // historyDefaultLimit and historyMaxLimit bound how many messages one + // tools/call answers with — see flattenHistory. Default is generous + // enough that an ordinary session's whole history fits in one call; + // Max prevents a single call from building an unbounded response for + // a pathologically long session (the caller pages with offset + // instead — see historyResultText's "more history available" hint). + historyDefaultLimit = 500 + historyMaxLimit = 2000 + + // historyContentTruncateBytes bounds how much of any single tool + // call's arguments or a single tool result's content this file + // inlines per line — mirrors claude_code_backend.go's + // claudeCodeStderrCap precedent: a useful summary without letting one + // oversized tool call/result dominate (or blow up) the whole + // response. + historyContentTruncateBytes = 2000 +) + +// historyToolInputSchema is get_conversation_history's tools/list +// InputSchema: two optional integers, offset/limit, paginating over the +// session's message history oldest-first — see flattenHistory's own doc +// comment for the exact semantics. +var historyToolInputSchema = json.RawMessage(`{ + "type": "object", + "properties": { + "offset": { + "type": "integer", + "minimum": 0, + "description": "Number of messages (oldest first) to skip before the returned page. Defaults to 0. Use the next_offset value from a prior call's response to continue reading a long history." + }, + "limit": { + "type": "integer", + "minimum": 1, + "description": "Maximum number of messages to return in this call. Defaults to 500." + } + } +}`) + +// historyToolDescription is shown to the model in tools/list — it must +// stand on its own even without the --append-system-prompt directive +// (engine's claudeCodeHistoryDirective), since a later turn on a resumed +// CLI session never gets that directive again but can still see this tool +// in its own tools/list cache. +const historyToolDescription = "Read the PRIOR conversation history for this session: messages that already happened before this turn, either on a different model or in a part of this conversation you have not seen. Call this once, before responding, whenever you are continuing a conversation you have not already read. Supports offset/limit pagination for long histories." + +// historyToolAnnotations and processToolAnnotations are each tool's +// mcp.Tool.Annotations object (the spec's ToolAnnotations hints, +// https://modelcontextprotocol.io/specification/2025-11-25/server/tools#annotations) +// — 2026-era MCP client UIs surface these to a human (or gate a +// destructive call behind confirmation), so an honest hint here is worth +// setting even though this server does not enforce either one itself. +// get_conversation_history is read-only by construction (flattenHistory +// never mutates sess); the `process` tool is the opposite — its +// start/stop/restart actions can kill a running process — so it gets +// destructiveHint instead, never readOnlyHint. +var ( + historyToolAnnotations = json.RawMessage(`{"readOnlyHint": true}`) + processToolAnnotations = json.RawMessage(`{"destructiveHint": true}`) +) + +// newSessionMCPRegistry builds the per-request mcpserver.Registry for +// sess's own /session/{id}/mcp endpoint (handleSessionMCP): always +// get_conversation_history, plus the native `process` tool whenever sess +// has one configured (Config.Processes non-nil — see +// engine.Session.ToolDef) — see this file's package doc for why only +// these two. version is reported as the MCP server's own implementation +// version (Options.Version — the same harness build version every other +// endpoint already reports, not a version of the tool's own wire shape). +func newSessionMCPRegistry(sess *engine.Session, version string) *mcpserver.Registry { + // "harness-tools" is this server's own self-reported Implementation.Name + // (initialize's ServerInfo) — cosmetic identification only, unrelated + // to (if conveniently matching) engine's claudeCodeToolsServerName, + // the --mcp-config MAP KEY the delegated `claude` process's own client + // uses to reach this endpoint at all. + reg := mcpserver.NewRegistry("harness-tools", version) + reg.SetInstructions("Call get_conversation_history before responding if you have not already read this session's prior conversation history.") + reg.RegisterTool(mcp.Tool{ + Name: historyToolName, + Description: historyToolDescription, + InputSchema: historyToolInputSchema, + Annotations: historyToolAnnotations, + }, historyToolHandler(sess)) + + // def comes from the engine's OWN process tool registration + // (engine/process.go's processTool) via ToolDef, not a second, + // hand-duplicated copy of its Description/InputSchema — the two would + // otherwise be free to silently drift apart. ok is false exactly when + // Config.Processes is nil (see ToolDef's own doc comment), the same + // condition that hides the native `process` tool from the model + // entirely — this MCP surface must not advertise a tool a delegated + // turn could call and get "unknown tool" back from RunTool. + if def, ok := sess.ToolDef(engine.ProcessToolName); ok { + reg.RegisterTool(mcp.Tool{ + Name: def.Name, + Description: def.Description, + InputSchema: def.InputSchema, + Annotations: processToolAnnotations, + }, processToolMCPHandler(sess)) + } + return reg +} + +// historyToolCallArgs is get_conversation_history's tools/call arguments +// shape — see historyToolInputSchema. +type historyToolCallArgs struct { + Offset int `json:"offset"` + Limit int `json:"limit"` +} + +// historyToolHandler returns the mcpserver.ToolHandler for +// get_conversation_history, closing over sess. It re-reads sess.History() +// on every call (never cached): the session may keep progressing between +// a tools/call and the process's own lifetime, and this handler has no +// reason to serve a stale snapshot. +func historyToolHandler(sess *engine.Session) mcpserver.ToolHandler { + return func(_ context.Context, raw json.RawMessage) (mcp.CallToolResult, error) { + var args historyToolCallArgs + if len(raw) > 0 { + if err := json.Unmarshal(raw, &args); err != nil { + return mcp.CallToolResult{}, fmt.Errorf("%s: invalid arguments: %w", historyToolName, err) + } + } + page := flattenHistory(sess.History(), args.Offset, args.Limit) + return mcp.CallToolResult{ + Content: []mcp.Content{{Type: mcp.ContentTypeText, Text: historyResultText(page)}}, + }, nil + } +} + +// processToolMCPHandler returns the mcpserver.ToolHandler for the native +// `process` tool, closing over sess. It routes every call through +// sess.RunTool (engine/engine.go) — the SAME generic dispatch path a +// native-loop process-tool call goes through (hooks, events, panic +// recovery included) — never a hand-rolled second implementation of the +// process tool's own action switch. RunTool's error already carries the +// tool-level failure text (e.g. "no such process"), and returning it +// directly from this handler makes mcpserver.Registry's own dispatch turn +// it into CallToolResult.IsError automatically — see ToolHandler's own doc +// comment for that contract. +func processToolMCPHandler(sess *engine.Session) mcpserver.ToolHandler { + return func(ctx context.Context, raw json.RawMessage) (mcp.CallToolResult, error) { + parts, err := sess.RunTool(ctx, engine.ProcessToolName, raw) + if err != nil { + return mcp.CallToolResult{}, err + } + return mcp.CallToolResult{Content: partsToMCPContent(parts)}, nil + } +} + +// partsToMCPContent flattens a native tool's message.Parts result into MCP +// Content items. Every native session tool (process included) answers +// with a single *message.Text part carrying a JSON- or plain-text result +// (see engine/process.go's jsonResult), so Parts.Text()'s own +// newline-joining rule already produces exactly the one string an MCP +// tool_result needs — this wraps it as the transport's required Content +// item rather than reimplementing Parts' own text-extraction logic. +func partsToMCPContent(parts message.Parts) []mcp.Content { + return []mcp.Content{{Type: mcp.ContentTypeText, Text: parts.Text()}} +} + +// historyPage is flattenHistory's result: a rendered page of a session's +// message history plus enough bookkeeping for historyResultText to tell +// the caller where it is and whether to ask for more. +type historyPage struct { + Text string + Total int + Offset int // the actual, clamped starting offset this page begins at + Returned int + NextOffset int + HasMore bool +} + +// flattenHistory renders msgs[offset:offset+limit] (clamped to msgs' +// bounds) into readable "Role: text" lines — see writeFlattenedMessage for +// the per-role rendering. offset/limit index MESSAGES, oldest first, +// matching Session.History()'s own order; offset < 0 clamps to 0, and +// limit <= 0 or > historyMaxLimit falls back to historyDefaultLimit (an +// absent or malformed pagination arg gets a sane default, never an +// unbounded read or a zero-length response). +// +// A ToolResult's tool NAME (message.ToolResult carries only its CallID) is +// resolved by scanning every ToolCall in msgs BEFORE the page even when +// the matching call itself fell on an earlier page — a caller paging +// through a long history one chunk at a time must still see readable tool +// names on every page, not just the one that happens to include the +// original call. +func flattenHistory(msgs []message.Message, offset, limit int) historyPage { + if offset < 0 { + offset = 0 + } + if limit <= 0 || limit > historyMaxLimit { + limit = historyDefaultLimit + } + total := len(msgs) + if offset > total { + offset = total + } + end := offset + limit + if end > total { + end = total + } + + callNames := make(map[string]string) + for _, m := range msgs[:offset] { + recordToolCallNames(m, callNames) + } + + var b strings.Builder + for _, m := range msgs[offset:end] { + recordToolCallNames(m, callNames) + writeFlattenedMessage(&b, m, callNames) + } + + returned := end - offset + nextOffset := offset + returned + return historyPage{ + Text: b.String(), + Total: total, + Offset: offset, + Returned: returned, + NextOffset: nextOffset, + HasMore: nextOffset < total, + } +} + +func recordToolCallNames(m message.Message, callNames map[string]string) { + for _, p := range m.Parts { + if tc, ok := p.(*message.ToolCall); ok { + callNames[tc.CallID] = tc.Name + } + } +} + +// writeFlattenedMessage appends one message's own readable rendering to b: +// +// - RoleUser: "User: " (or "(no text)" for a part-less trigger +// message — see message.OriginEngine's own doc comment for the one +// shape that can be this bare). +// - RoleAssistant: one "Assistant: " line per non-empty Text part, +// one "Assistant: [thinking]" line per Reasoning part (the reasoning +// TEXT itself is deliberately omitted — it is verbose, provider- +// internal narration, not conversational content a catch-up read +// needs), and one "Assistant called tool NAME(args)" summary per +// ToolCall part. +// - RoleTool: one "Tool result (NAME, ok|error): " line per +// ToolResult part, NAME resolved via callNames (falling back to the +// bare CallID if no matching ToolCall was ever seen). +// +// Every inlined tool-call-argument or tool-result-content string is +// truncated to historyContentTruncateBytes — see that constant's own doc +// comment. +func writeFlattenedMessage(b *strings.Builder, m message.Message, callNames map[string]string) { + switch m.Role { + case message.RoleUser: + text := m.Parts.Text() + if text == "" { + text = "(no text)" + } + fmt.Fprintf(b, "User: %s\n", text) + + case message.RoleAssistant: + for _, p := range m.Parts { + switch part := p.(type) { + case *message.Text: + if part.Text != "" { + fmt.Fprintf(b, "Assistant: %s\n", part.Text) + } + case *message.Reasoning: + b.WriteString("Assistant: [thinking]\n") + case *message.ToolCall: + fmt.Fprintf(b, "Assistant called tool %s(%s)\n", part.Name, truncateForHistory(string(part.Arguments))) + } + } + + case message.RoleTool: + for _, p := range m.Parts { + tr, ok := p.(*message.ToolResult) + if !ok { + continue + } + name := callNames[tr.CallID] + if name == "" { + name = tr.CallID + } + status := "ok" + if tr.IsError { + status = "error" + } + fmt.Fprintf(b, "Tool result (%s, %s): %s\n", name, status, truncateForHistory(tr.Content.Text())) + } + } +} + +// truncateForHistory bounds s to historyContentTruncateBytes, appending a +// byte-count note when it cuts anything off — see that constant's own doc +// comment. +func truncateForHistory(s string) string { + if len(s) <= historyContentTruncateBytes { + return s + } + return fmt.Sprintf("%s... (truncated, %d bytes total)", s[:historyContentTruncateBytes], len(s)) +} + +// historyResultText wraps a historyPage into the tool_result text a model +// actually reads: a leading label making unmistakably clear this is PRIOR, +// already-happened context (not a new instruction to act on), the +// page's own rendered lines, and — when more of the history remains — a +// trailing hint naming the next_offset to continue with. +func historyResultText(page historyPage) string { + if page.Total == 0 { + return "No prior conversation history for this session." + } + var b strings.Builder + b.WriteString("The following is PRIOR conversation history for this session that already happened. It is context for you to read, not a new message to respond to.\n\n") + if page.Returned == 0 { + // Offset landed at or past the end of history (e.g. a clamped + // out-of-range offset — see flattenHistory) — there is no + // message range to report, so this must not print a "Showing + // messages X-Y" line at all: with Returned == 0, page.Offset+1 > + // page.Offset+page.Returned, which would otherwise read as the + // nonsensical "Showing messages 5-4 of 4". + fmt.Fprintf(&b, "(no messages in the requested range; %d total)\n", page.Total) + } else { + fmt.Fprintf(&b, "Showing messages %d-%d of %d total (oldest first).\n\n", page.Offset+1, page.Offset+page.Returned, page.Total) + b.WriteString(page.Text) + } + if page.HasMore { + fmt.Fprintf(&b, "\nMore history is available. Call %s again with offset=%d to continue.\n", historyToolName, page.NextOffset) + } + return b.String() +} + +// handleSessionMCP implements POST /session/{id}/mcp: the MCP server role's +// Streamable HTTP endpoint for sess's own harness-hosted tools +// (get_conversation_history, plus `process` when configured — see this +// file's package doc). A fresh mcpserver.Registry is built per request +// rather than cached: registration is cheap (at most two map entries) and +// this keeps the handler free of any registry lifecycle to manage across a +// session's possibly-long resident lifetime. +func (s *Server) handleSessionMCP(w http.ResponseWriter, r *http.Request) { + id, ok := s.sessionIDOrNotFound(w, r) + if !ok { + return + } + sess, ok := s.lookupSession(id) + if !ok { + writeErr(w, http.StatusNotFound, "no such session") + return + } + newSessionMCPRegistry(sess, s.opts.Version).ServeHTTP(w, r) +} diff --git a/server/mcp_history_test.go b/server/mcp_history_test.go new file mode 100644 index 00000000..401e9835 --- /dev/null +++ b/server/mcp_history_test.go @@ -0,0 +1,472 @@ +package server + +import ( + "bytes" + "encoding/json" + "fmt" + "net/http" + "strings" + "testing" + + "github.com/majorcontext/harness/mcp" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/process" +) + +// seedMessages builds a small, readable history: a user question, an +// assistant reply that calls a tool, the tool's own result, a reasoning +// block, and a final assistant text reply — enough shapes to exercise +// every branch of writeFlattenedMessage in one seed. +func seedMessages() []message.Message { + return []message.Message{ + {ID: "1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "what is in the repo root?"}}}, + {ID: "2", Role: message.RoleAssistant, Parts: message.Parts{ + &message.Reasoning{Text: "I should list the directory."}, + &message.ToolCall{CallID: "call_1", Name: "bash", Arguments: json.RawMessage(`{"command":"ls"}`)}, + }}, + {ID: "3", Role: message.RoleTool, Parts: message.Parts{ + &message.ToolResult{CallID: "call_1", Content: message.Parts{&message.Text{Text: "README.md\nmain.go"}}}, + }}, + {ID: "4", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "The repo root has README.md and main.go."}}}, + } +} + +// TestFlattenHistoryRendersRolesReadably proves every message shape +// (user text, assistant reasoning, assistant tool call, tool result, +// final assistant text) renders into a readable, role-labeled line. +func TestFlattenHistoryRendersRolesReadably(t *testing.T) { + page := flattenHistory(seedMessages(), 0, 0) + if page.Total != 4 || page.Returned != 4 || page.HasMore { + t.Fatalf("page = %+v, want Total=4 Returned=4 HasMore=false", page) + } + + wantSubstrings := []string{ + "User: what is in the repo root?", + "Assistant: [thinking]", + `Assistant called tool bash({"command":"ls"})`, + "Tool result (bash, ok): README.md\nmain.go", + "Assistant: The repo root has README.md and main.go.", + } + for _, want := range wantSubstrings { + if !strings.Contains(page.Text, want) { + t.Errorf("flattened text missing %q\ngot:\n%s", want, page.Text) + } + } +} + +// TestFlattenHistoryToolResultErrorStatus proves an IsError ToolResult is +// labeled "error", not "ok". +func TestFlattenHistoryToolResultErrorStatus(t *testing.T) { + msgs := []message.Message{ + {ID: "1", Role: message.RoleAssistant, Parts: message.Parts{&message.ToolCall{CallID: "c1", Name: "bash", Arguments: json.RawMessage(`{}`)}}}, + {ID: "2", Role: message.RoleTool, Parts: message.Parts{&message.ToolResult{CallID: "c1", Content: message.Parts{&message.Text{Text: "command not found"}}, IsError: true}}}, + } + page := flattenHistory(msgs, 0, 0) + if !strings.Contains(page.Text, "Tool result (bash, error): command not found") { + t.Errorf("flattened text = %q, want an (bash, error) tool result line", page.Text) + } +} + +// TestFlattenHistoryToolNameResolvedAcrossPageBoundary proves a +// ToolResult's tool name is still resolved correctly even when its +// matching ToolCall fell on an EARLIER page than the one being rendered — +// a caller paging through a long history one chunk at a time must still +// see readable names on every page. +func TestFlattenHistoryToolNameResolvedAcrossPageBoundary(t *testing.T) { + msgs := []message.Message{ + {ID: "1", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "q1"}}}, + {ID: "2", Role: message.RoleAssistant, Parts: message.Parts{&message.ToolCall{CallID: "call_1", Name: "grep", Arguments: json.RawMessage(`{}`)}}}, + {ID: "3", Role: message.RoleTool, Parts: message.Parts{&message.ToolResult{CallID: "call_1", Content: message.Parts{&message.Text{Text: "match"}}}}}, + } + // Page 2 starts AFTER the ToolCall message (offset 2), so only the + // ToolResult message itself is on this page. + page := flattenHistory(msgs, 2, 10) + if page.Returned != 1 { + t.Fatalf("page.Returned = %d, want 1", page.Returned) + } + if !strings.Contains(page.Text, "Tool result (grep, ok): match") { + t.Errorf("flattened text = %q, want the tool name \"grep\" resolved from an earlier page", page.Text) + } +} + +// TestFlattenHistoryPagination drives offset/limit across a 10-message +// history (5 user/assistant pairs) and checks each page's own bookkeeping. +func TestFlattenHistoryPagination(t *testing.T) { + var msgs []message.Message + for i := 0; i < 5; i++ { + msgs = append(msgs, + message.Message{ID: fmt.Sprintf("u%d", i), Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: fmt.Sprintf("question %d", i)}}}, + message.Message{ID: fmt.Sprintf("a%d", i), Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: fmt.Sprintf("answer %d", i)}}}, + ) + } + + p1 := flattenHistory(msgs, 0, 4) + if p1.Offset != 0 || p1.Returned != 4 || p1.NextOffset != 4 || !p1.HasMore { + t.Errorf("page 1 = %+v, want Offset=0 Returned=4 NextOffset=4 HasMore=true", p1) + } + p2 := flattenHistory(msgs, p1.NextOffset, 4) + if p2.Offset != 4 || p2.Returned != 4 || p2.NextOffset != 8 || !p2.HasMore { + t.Errorf("page 2 = %+v, want Offset=4 Returned=4 NextOffset=8 HasMore=true", p2) + } + p3 := flattenHistory(msgs, p2.NextOffset, 4) + if p3.Offset != 8 || p3.Returned != 2 || p3.NextOffset != 10 || p3.HasMore { + t.Errorf("page 3 = %+v, want Offset=8 Returned=2 NextOffset=10 HasMore=false", p3) + } + if p1.Total != 10 || p2.Total != 10 || p3.Total != 10 { + t.Errorf("Total = %d/%d/%d, want 10 on every page", p1.Total, p2.Total, p3.Total) + } +} + +// TestFlattenHistoryDefaultsAndClampsLimit proves a non-positive or +// oversized limit falls back to historyDefaultLimit, and a negative offset +// clamps to 0, rather than an unbounded or empty read. +func TestFlattenHistoryDefaultsAndClampsLimit(t *testing.T) { + msgs := seedMessages() + for _, limit := range []int{0, -1, historyMaxLimit + 1} { + page := flattenHistory(msgs, 0, limit) + if page.Returned != len(msgs) { + t.Errorf("limit=%d: Returned = %d, want %d (all of a short history)", limit, page.Returned, len(msgs)) + } + } + page := flattenHistory(msgs, -5, 0) + if page.Offset != 0 { + t.Errorf("negative offset clamped to %d, want 0", page.Offset) + } +} + +// TestFlattenHistoryTruncatesLargeToolResult proves an oversized tool +// result is bounded rather than inlined in full, with a byte-count note. +func TestFlattenHistoryTruncatesLargeToolResult(t *testing.T) { + big := strings.Repeat("x", historyContentTruncateBytes+500) + msgs := []message.Message{ + {ID: "1", Role: message.RoleAssistant, Parts: message.Parts{&message.ToolCall{CallID: "c1", Name: "cat", Arguments: json.RawMessage(`{}`)}}}, + {ID: "2", Role: message.RoleTool, Parts: message.Parts{&message.ToolResult{CallID: "c1", Content: message.Parts{&message.Text{Text: big}}}}}, + } + page := flattenHistory(msgs, 0, 0) + if strings.Contains(page.Text, big) { + t.Error("flattened text inlines the full oversized tool result, want it truncated") + } + if !strings.Contains(page.Text, "truncated") { + t.Errorf("flattened text = %q, want a truncation note", page.Text) + } +} + +// TestHistoryResultTextLabelsPriorContextAndHints proves the wrapped tool +// result text clearly labels itself as prior/already-happened context and +// names the next_offset to continue with when more history remains. +func TestHistoryResultTextLabelsPriorContextAndHints(t *testing.T) { + msgs := seedMessages() + page := flattenHistory(msgs, 0, 2) + text := historyResultText(page) + if !strings.Contains(strings.ToLower(text), "prior") { + t.Errorf("result text = %q, want it labeled as PRIOR context", text) + } + if !strings.Contains(text, "Showing messages 1-2 of 4") { + t.Errorf("result text = %q, want a \"Showing messages 1-2 of 4\" line", text) + } + if !strings.Contains(text, fmt.Sprintf("offset=%d", page.NextOffset)) { + t.Errorf("result text = %q, want a hint naming offset=%d", text, page.NextOffset) + } +} + +// TestHistoryResultTextEmptyHistory proves a session with no history at +// all gets an explicit "no history" message rather than an empty or +// confusingly-numbered "Showing messages 1-0 of 0" line. +func TestHistoryResultTextEmptyHistory(t *testing.T) { + page := flattenHistory(nil, 0, 0) + text := historyResultText(page) + if !strings.Contains(text, "No prior conversation history") { + t.Errorf("result text = %q, want an explicit no-history message", text) + } +} + +// TestHistoryResultTextClampedOffsetOmitsNonsensicalShowingLine proves an +// offset clamped to (or past) the end of history — flattenHistory's own +// clamp, e.g. an offset a prior call's next_offset hint no longer covers +// after the history shrank, or one a model simply got wrong — renders as +// an explicit "no messages" line, never a "Showing messages X-Y of N" +// line with X > Y (Returned == 0 makes page.Offset+1 > page.Offset+0). +func TestHistoryResultTextClampedOffsetOmitsNonsensicalShowingLine(t *testing.T) { + msgs := seedMessages() // 4 messages total + page := flattenHistory(msgs, 10, 5) + if page.Returned != 0 || page.Total != 4 { + t.Fatalf("page = %+v, want Returned=0 Total=4 (offset clamped past the end)", page) + } + text := historyResultText(page) + if strings.Contains(text, "Showing messages") { + t.Errorf("result text = %q, want no \"Showing messages\" line when Returned is 0", text) + } + if !strings.Contains(text, "no messages") { + t.Errorf("result text = %q, want an explicit \"no messages\" line", text) + } + if !strings.Contains(text, "4") { + t.Errorf("result text = %q, want the total (4) still mentioned somewhere", text) + } +} + +// --- HTTP wiring: POST /session/{id}/mcp --- + +// TestHandleSessionMCPFullLifecycle drives initialize, tools/list, and +// tools/call against a REAL session (built cold, on disk, the same way +// TestMessagesUnparameterizedIsUnchanged's coldMessages helper does) over +// the actual HTTP route, proving handleSessionMCP's session lookup and +// mcpserver.Registry wiring end to end. +func TestHandleSessionMCPFullLifecycle(t *testing.T) { + dir := t.TempDir() + sess := coldMessages(t, dir, 2) // 4 messages: ask 0/reply 0, ask 1/reply 1 + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + initResp, initData := h.do("POST", "/session/"+sess.ID+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "initialize", + "params": map[string]any{"protocolVersion": "2025-11-25"}, + }) + if initResp.StatusCode != 200 { + t.Fatalf("initialize status = %d: %s", initResp.StatusCode, initData) + } + var initMsg struct { + Result mcp.InitializeResult `json:"result"` + } + if err := json.Unmarshal(initData, &initMsg); err != nil { + t.Fatalf("decoding initialize response: %v (%s)", err, initData) + } + if initMsg.Result.Capabilities.Tools == nil { + t.Error("initialize response has no tools capability") + } + + listResp, listData := h.do("POST", "/session/"+sess.ID+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "2", "method": "tools/list", + }) + if listResp.StatusCode != 200 { + t.Fatalf("tools/list status = %d: %s", listResp.StatusCode, listData) + } + var listMsg struct { + Result mcp.ListToolsResult `json:"result"` + } + if err := json.Unmarshal(listData, &listMsg); err != nil { + t.Fatalf("decoding tools/list response: %v (%s)", err, listData) + } + if len(listMsg.Result.Tools) != 1 || listMsg.Result.Tools[0].Name != historyToolName { + t.Fatalf("tools/list Tools = %+v, want exactly [%s]", listMsg.Result.Tools, historyToolName) + } + + callResp, callData := h.do("POST", "/session/"+sess.ID+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "3", "method": "tools/call", + "params": map[string]any{"name": historyToolName}, + }) + if callResp.StatusCode != 200 { + t.Fatalf("tools/call status = %d: %s", callResp.StatusCode, callData) + } + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call returned an error: %+v", callMsg.Error) + } + if len(callMsg.Result.Content) != 1 { + t.Fatalf("tools/call Content = %+v, want exactly one text item", callMsg.Result.Content) + } + got := callMsg.Result.Content[0].Text + for _, want := range []string{"ask 0", "reply 0", "ask 1", "reply 1"} { + if !strings.Contains(got, want) { + t.Errorf("tools/call text missing %q from the session's real history:\n%s", want, got) + } + } +} + +// TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured proves a +// session with no Config.Processes (the ordinary newHarness session, no +// process manager wired at all) advertises ONLY get_conversation_history — +// the same "process tool absent when unconfigured" rule the native loop +// already follows (engine's TestProcessToolAbsentWhenNoProcessesConfigured) +// applies identically to this MCP surface. +func TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("") + + _, listData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/list", + }) + var listMsg struct { + Result mcp.ListToolsResult `json:"result"` + } + if err := json.Unmarshal(listData, &listMsg); err != nil { + t.Fatalf("decoding tools/list response: %v (%s)", err, listData) + } + if len(listMsg.Result.Tools) != 1 || listMsg.Result.Tools[0].Name != historyToolName { + t.Fatalf("tools/list Tools = %+v, want exactly [%s]", listMsg.Result.Tools, historyToolName) + } +} + +// TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations proves a +// session WITH Config.Processes configured advertises the native +// `process` tool alongside get_conversation_history, each carrying the +// annotation this file's package doc promises: readOnlyHint on history, +// destructiveHint on process — plus process's own Description/InputSchema +// passed through from the engine's real tool definition (ToolDef), not a +// hand-duplicated copy. +func TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations(t *testing.T) { + h, _ := newProcessHarness(t, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", "true"}}, + }) + id := h.createSession("") + + _, listData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/list", + }) + var listMsg struct { + Result mcp.ListToolsResult `json:"result"` + } + if err := json.Unmarshal(listData, &listMsg); err != nil { + t.Fatalf("decoding tools/list response: %v (%s)", err, listData) + } + if len(listMsg.Result.Tools) != 2 { + t.Fatalf("tools/list Tools = %+v, want exactly 2 (history + process)", listMsg.Result.Tools) + } + + var hist, proc *mcp.Tool + for i := range listMsg.Result.Tools { + switch listMsg.Result.Tools[i].Name { + case historyToolName: + hist = &listMsg.Result.Tools[i] + case "process": + proc = &listMsg.Result.Tools[i] + } + } + if hist == nil { + t.Fatal("tools/list missing get_conversation_history") + } + // json.Marshal compacts an embedded json.RawMessage (no insignificant + // whitespace survives the round trip through writeResult), so the + // wire form is "readOnlyHint":true, not "readOnlyHint": true. + if !strings.Contains(string(hist.Annotations), `"readOnlyHint":true`) { + t.Errorf("get_conversation_history Annotations = %s, want readOnlyHint true", hist.Annotations) + } + if proc == nil { + t.Fatal("tools/list missing process") + } + if !strings.Contains(string(proc.Annotations), `"destructiveHint":true`) { + t.Errorf("process Annotations = %s, want destructiveHint true", proc.Annotations) + } + if !strings.Contains(proc.Description, "long-lived") { + t.Errorf("process Description = %q, want it to describe managing long-lived box processes", proc.Description) + } + if len(proc.InputSchema) == 0 { + t.Error("process InputSchema is empty, want the engine's own process-tool schema passed through") + } +} + +// TestHandleSessionMCPProcessToolCallStartsRealProcess proves tools/call +// for the process tool routes all the way through +// engine.Session.RunTool -- a "start" call over MCP leaves the process +// RUNNING in the same Manager a native-loop call would have used, not +// merely returning a plausible-looking response. +func TestHandleSessionMCPProcessToolCallStartsRealProcess(t *testing.T) { + h, mgr := newProcessHarness(t, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", `echo "Ready in 5ms"; sleep 100`}, ReadyRegex: "Ready in .*ms"}, + }) + id := h.createSession("") + + _, callData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/call", + "params": map[string]any{"name": "process", "arguments": map[string]any{"action": "start", "name": "dev"}}, + }) + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call returned a protocol error: %+v", callMsg.Error) + } + if callMsg.Result.IsError { + t.Fatalf("tools/call result IsError = true: %+v", callMsg.Result.Content) + } + + st, err := mgr.Status("dev") + if err != nil { + t.Fatalf("mgr.Status: %v", err) + } + if st.State != process.StateReady { + t.Fatalf("mgr.Status(dev) = %+v, want ready — tools/call must have actually started the process via RunTool", st) + } +} + +// TestHandleSessionMCPProcessToolCallFailureIsToolError proves a process +// tool call that fails at the ACTION level (an unknown action here) comes +// back as a successful JSON-RPC response carrying CallToolResult.IsError — +// the same TOOL-level-vs-protocol-level distinction +// TestRegistryToolsCallHandlerErrorBecomesIsErrorResult already locks in +// for mcpserver generically — never a protocol-level RPCError, since +// "process" IS a known, registered tool. +func TestHandleSessionMCPProcessToolCallFailureIsToolError(t *testing.T) { + h, _ := newProcessHarness(t, map[string]process.Def{ + "dev": {Command: []string{"sh", "-c", "true"}}, + }) + id := h.createSession("") + + _, callData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/call", + "params": map[string]any{"name": "process", "arguments": map[string]any{"action": "not_a_real_action", "name": "dev"}}, + }) + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call returned a protocol-level error for a tool-level failure: %+v", callMsg.Error) + } + if !callMsg.Result.IsError { + t.Fatalf("tools/call Result.IsError = false, want true for an unknown action") + } + if len(callMsg.Result.Content) != 1 || !strings.Contains(callMsg.Result.Content[0].Text, "not_a_real_action") { + t.Errorf("tools/call Content = %+v, want it to name the unknown action", callMsg.Result.Content) + } +} + +// TestHandleSessionMCPUnknownSessionNotFound proves the route 404s for a +// session id that does not exist, mirroring every other {id}-keyed route. +func TestHandleSessionMCPUnknownSessionNotFound(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, data := h.do("POST", "/session/ses_0000000000000000/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "initialize", + }) + if resp.StatusCode != 404 { + t.Errorf("status = %d, want 404: %s", resp.StatusCode, data) + } +} + +// TestHandleSessionMCPRequiresAuth proves the route carries the same +// bearer-token gate as every other session route — unlike GET /health, it +// is never publicly reachable. +func TestHandleSessionMCPRequiresAuth(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("") + + body, err := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": "1", "method": "initialize"}) + if err != nil { + t.Fatal(err) + } + req, err := http.NewRequest("POST", h.ts.URL+"/session/"+id+"/mcp", bytes.NewReader(body)) + if err != nil { + t.Fatal(err) + } + // Deliberately no Authorization header — see s.auth/s.authorized in + // server.go, the same gate every other {id}-keyed route sits behind. + resp, err := h.ts.Client().Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusUnauthorized { + t.Errorf("status = %d, want 401 with no Authorization header", resp.StatusCode) + } +} diff --git a/server/process_handlers.go b/server/process_handlers.go index 192ec1b4..f733af44 100644 --- a/server/process_handlers.go +++ b/server/process_handlers.go @@ -4,6 +4,7 @@ import ( "context" "errors" "net/http" + "strconv" "github.com/majorcontext/harness/process" ) @@ -43,6 +44,49 @@ func (s *Server) handleProcessRestart(w http.ResponseWriter, r *http.Request) { }) } +// processLogsJSON is GET /process/{name}/logs' response shape — a +// console's processes panel wants both the trailing log content and the +// process's own current status in one round trip, rather than a second +// request to GET /process for the status half. +type processLogsJSON struct { + Content string `json:"content"` + Status process.Status `json:"status"` +} + +// handleProcessLogs answers GET /process/{name}/logs?tail=N: the last N +// lines of name's log file (process.Manager.Logs' own default, 50, when +// tail is absent or not a positive integer) plus its current status. Same +// small-handler pattern as handleProcessAction, but Logs' own three-value +// return (content, status, error) doesn't fit that helper's +// Status-only processAction shape, so this gets its own body. +func (s *Server) handleProcessLogs(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("name") + if name == "" { + writeErr(w, http.StatusBadRequest, "process name is required") + return + } + if s.opts.Processes == nil { + writeErr(w, http.StatusNotFound, "no such process") + return + } + tail := 0 + if v := r.URL.Query().Get("tail"); v != "" { + if n, err := strconv.Atoi(v); err == nil { + tail = n + } + } + content, st, err := s.opts.Processes.Logs(name, tail) + if err != nil { + if errors.Is(err, process.ErrUnknownProcess) { + writeErr(w, http.StatusNotFound, "no such process") + return + } + writeErr(w, http.StatusInternalServerError, err.Error()) + return + } + writeJSON(w, http.StatusOK, processLogsJSON{Content: content, Status: st}) +} + // handleProcessAction resolves {name} and, if Processes is configured and // the name is a declared process, runs fn (Start/Stop/Restart) and // answers its resulting Status. A nil Options.Processes, or a name naming diff --git a/server/process_handlers_test.go b/server/process_handlers_test.go index c6d532ab..3d974756 100644 --- a/server/process_handlers_test.go +++ b/server/process_handlers_test.go @@ -121,14 +121,62 @@ func TestHandleProcessUnknownName404(t *testing.T) { h, _ := newProcessHarness(t, map[string]process.Def{ "dev": {Command: []string{"sh", "-c", "true"}}, }) - for _, action := range []string{"start", "stop", "restart"} { - resp, body := h.do(http.MethodPost, "/process/nope/"+action, nil) + for _, action := range []string{"start", "stop", "restart", "logs"} { + method := http.MethodPost + if action == "logs" { + method = http.MethodGet + } + resp, body := h.do(method, "/process/nope/"+action, nil) if resp.StatusCode != http.StatusNotFound { t.Errorf("%s: status = %d, body = %s, want 404", action, resp.StatusCode, body) } } } +// TestHandleProcessLogs proves GET /process/{name}/logs returns the +// process's own log content alongside its current status — the processes +// panel's one-request source for both. +func TestHandleProcessLogs(t *testing.T) { + h, _ := newProcessHarness(t, map[string]process.Def{ + "dev": { + Command: []string{"sh", "-c", `echo "Ready in 5ms"; sleep 100`}, + ReadyRegex: "Ready in .*ms", + ReadyTimeout: 5 * time.Second, + }, + }) + resp, body := h.do(http.MethodPost, "/process/dev/start", nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("start status = %d, body = %s", resp.StatusCode, body) + } + + resp, body = h.do(http.MethodGet, "/process/dev/logs", nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("logs status = %d, body = %s", resp.StatusCode, body) + } + var got processLogsJSON + if err := json.Unmarshal(body, &got); err != nil { + t.Fatalf("unmarshal: %v (%s)", err, body) + } + if !strings.Contains(got.Content, "Ready in 5ms") { + t.Errorf("Content = %q, want the ready line", got.Content) + } + if got.Status.State != process.StateReady { + t.Errorf("Status = %+v, want ready", got.Status) + } +} + +// TestHandleProcessLogsNotConfigured404s proves the endpoint 404s (not a +// panic, not a bare empty body) when no Options.Processes is configured +// at all, the same "unconfigured looks like unknown" rule +// handleProcessAction already applies to start/stop/restart. +func TestHandleProcessLogsNotConfigured404(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, body := h.do(http.MethodGet, "/process/dev/logs", nil) + if resp.StatusCode != http.StatusNotFound { + t.Errorf("status = %d, body = %s, want 404", resp.StatusCode, body) + } +} + // TestHandleProcessListNeverLeaksEnvValues is the HTTP-layer counterpart // of process.TestDeclareAndUndeclare's "env names never expose values" // case: GET /process is the one endpoint an orchestrator (or a curious diff --git a/server/server.go b/server/server.go index 7d40debd..320ef1c9 100644 --- a/server/server.go +++ b/server/server.go @@ -937,6 +937,15 @@ func (s *Server) routes() { mux.HandleFunc("GET /session/{id}/message", s.auth(s.handleMessages)) mux.HandleFunc("GET /session/{id}/journal", s.auth(s.handleJournal)) mux.HandleFunc("GET /session/{id}/request", s.auth(s.handleRequest)) + // The MCP server role for sess's own harness-hosted tools (currently + // get_conversation_history — see mcp_history.go's package doc). A + // delegated Claude Code CLI turn is handed this endpoint's own URL in + // its --mcp-config (engine.ClaudeCodeConfig.HTTPBaseURL), so it carries + // the same s.auth bearer-token gate as every other route: an operator + // wiring HTTPAuthToken (or an Unauthenticated loopback bind) is what + // makes the delegated child able to reach it, exactly like any other + // authenticated route. + mux.HandleFunc("POST /session/{id}/mcp", s.auth(s.handleSessionMCP)) mux.HandleFunc("POST /session/{id}/prompt_async", s.auth(s.handlePrompt)) mux.HandleFunc("POST /session/{id}/enqueue", s.auth(s.handleEnqueue)) mux.HandleFunc("GET /session/{id}/queue", s.auth(s.handleQueueGet)) @@ -958,6 +967,9 @@ func (s *Server) routes() { mux.HandleFunc("POST /process/{name}/start", s.auth(s.handleProcessStart)) mux.HandleFunc("POST /process/{name}/stop", s.auth(s.handleProcessStop)) mux.HandleFunc("POST /process/{name}/restart", s.auth(s.handleProcessRestart)) + // GET /process/{name}/logs: the processes panel's log tail, behind the + // same auth as every other process route — see handleProcessLogs. + mux.HandleFunc("GET /process/{name}/logs", s.auth(s.handleProcessLogs)) // /debug/goroutines: an authed HTTP alternative to sending SIGQUIT (see // handleGoroutines's doc comment and cmd/harness/main.go's serveCmd, // which confirms SIGQUIT still produces Go's default all-goroutine dump diff --git a/server/server_test.go b/server/server_test.go index 087b10a9..b4ef0302 100644 --- a/server/server_test.go +++ b/server/server_test.go @@ -164,6 +164,13 @@ func newServer(t *testing.T, dir string, prov provider.Provider, maxResident int SessionDir: dir, OnEvent: func(ev engine.Event) { srv.Publish(ev) }, GoalTool: !opts.GoalEvaluator.IsZero(), + // Processes mirrors production's own mkCfg (cmd/harness/main.go): + // every session gets the SAME Options.Processes a mutate func + // may have set, so a session created live (handleCreate) or + // cold-loaded (LoadSession) both see the native `process` tool + // exactly like a real served box does — see + // TestHandleSessionMCPProcessTool. + Processes: opts.Processes, } } opts = Options{ From 129749b3af56cfaf43e3a74458b8dcff9938c4af Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 00:12:23 -0400 Subject: [PATCH 34/95] feat(server,engine): expose task and list-only model over the MCP shim (#223) The delegated claude-code lane's MCP shim exposed only history and process. It could not spawn a cross-family harness child session or enumerate the models configured on this box, so a delegated turn had no way to delegate further or pick a family for such a delegation. Generalize the shim into a tools server that routes tools/call through Session.RunTool, the same generic external-dispatch seam a future harness-hosted tool would use. Expose two more tools on top of history: task (non-blocking spawn plus pull-collect status/log, cross-family, readOnlyHint false) and a list-only model (list action enforced in a dedicated handler before dispatch, readOnlyHint true). Keep model's set and status actions off the shim entirely so a delegated caller can never re-point the parent session's own live model; task's own spawn(model:...) override is the supported way to pick a model, for a child session, never this one. This also folds in two history-directive fixes: a delegated turn's prior history is restored correctly, and the directive re-fires after a switch back from native. tools/list now advertises task (readOnlyHint false) and model (readOnlyHint true, action enum ["list"]) alongside history and process. This is a rebase of andybons/claude-code-history-mcp (HEAD 863847b) onto current main: the branch predated main's own #221 (append_system_prompt) and #222 (harness-hosted MCP shim) and had independently rebuilt the same shim. append_system_prompt wiring is untouched; cmd/harness/main.go, engine/claude_code_backend.go, and engine/engine.go needed a 3-way merge but resolved with zero net diff against main, since main's #222 had already converged on the same shim/task/model wiring in those three files. Verification: go build ./..., gofmt -l ., go vet ./... all clean; go test -race ./... is green except a pre-existing, unrelated TestAppendSystemPromptReachesModelOnServe failure that reproduces identically on a clean origin/main (a macOS /var vs /private/var symlink mismatch in t.TempDir(), not a rebase regression). TestHandleSessionMCPModelToolSetAndStatusRejectedOverShim was red-verified: removing modelListOnlyMCPHandler's action != "list" guard makes the model actually swap and the test fail, confirming the guard is load-bearing; restoring it makes the test pass again. --- engine/model_tool.go | 61 ++++- engine/model_tool_test.go | 42 +++ engine/task_tool.go | 8 + server/mcp_history.go | 230 +++++++++++++---- server/mcp_history_test.go | 510 +++++++++++++++++++++++++++++++++++-- 5 files changed, 782 insertions(+), 69 deletions(-) diff --git a/engine/model_tool.go b/engine/model_tool.go index 8f37bc83..7b82de56 100644 --- a/engine/model_tool.go +++ b/engine/model_tool.go @@ -6,11 +6,14 @@ // model is current (see engine.go). This tool is the model-facing surface over // that machinery. // -// Two actions only: status (read-only) reports the current model, the -// configured aliases, and the configured provider names; set(model) resolves a -// one-level alias, validates the target provider is configured, and calls -// SetModel. There is deliberately no clear action — a model always has a model; -// there is nothing to clear. +// Three actions: status (read-only) reports the current model, the +// configured aliases, and the configured provider names; list (read-only) is +// status's data minus the current model, for a caller that only wants the +// choices — e.g. a delegated caller picking a family for another tool's own +// model override (task's spawn action) rather than swapping THIS session's +// model; set(model) resolves a one-level alias, validates the target +// provider is configured, and calls SetModel. There is deliberately no clear +// action — a model always has a model; there is nothing to clear. // // Gated by Config.ModelTool: registered in newSession only when the host opts // in. Unlike GoalTool (opt-in, gated on a configured evaluator), the CLI/server @@ -34,6 +37,14 @@ import ( // modelToolName is the session tool's fixed name. const modelToolName = "model" +// ModelToolName exports modelToolName for a caller outside this package +// that needs to name the SAME tool RunTool/ToolDef dispatch by — +// server/mcp_history.go's harness-hosted MCP `model` tool entry, notably — +// without hand-duplicating the literal "model" and risking it silently +// drifting from this package's own internal name. Mirrors +// engine/process.go's identical ProcessToolName export. +const ModelToolName = modelToolName + // modelToolArgs is the model tool's input shape. type modelToolArgs struct { Action string `json:"action"` @@ -50,6 +61,20 @@ type modelToolResult struct { Providers []string `json:"providers,omitempty"` } +// modelListResult is the list action's return: the configured provider +// families and aliases a caller can spawn or set into, WITHOUT this +// session's current model — a delegated caller (e.g. a claude-code-lane +// agent reaching this tool through the harness-hosted MCP shim, +// server/mcp_history.go) asking "what models are available" is not asking +// about this particular session's own state, unlike status. Backed by the +// exact same configuredProviderNames()/ModelAliases data modelToolStatus +// reads (see modelToolList) — never a second, independent data source +// that could drift from it. +type modelListResult struct { + Providers []string `json:"providers"` + Aliases map[string]string `json:"aliases,omitempty"` +} + // modelTool builds the `model` session tool. See the package doc for the // action contract. func modelTool() Tool { @@ -65,11 +90,14 @@ func modelTool() Tool { "alias — it takes effect on the NEXT request in this session. set fails, and " + "changes nothing, if the target names an unconfigured provider (the error lists the " + "valid aliases and provider names). " + + "list() reports the configured provider families and aliases only, with no " + + "current-model or session state — useful to pick a family/model for another tool's " + + "own model override (e.g. task's spawn action) without swapping this session's own model. " + "There is no action to clear the model — a session always has a model.", InputSchema: json.RawMessage(`{ "type": "object", "properties": { - "action": {"type": "string", "enum": ["status", "set"], "description": "The operation to perform"}, + "action": {"type": "string", "enum": ["status", "set", "list"], "description": "The operation to perform"}, "model": {"type": "string", "description": "The target model: a \"provider/model\" ref or a configured alias (required for set)"} }, "required": ["action"] @@ -97,6 +125,9 @@ func runModelTool(s *Session, raw json.RawMessage) (message.Parts, error) { case "status": return jsonResult(s.modelToolStatus()) + case "list": + return jsonResult(s.modelToolList()) + case "set": if in.Model == "" { return nil, fmt.Errorf("model: set requires a non-empty model (%s)", s.modelChoicesHint()) @@ -128,7 +159,7 @@ func runModelTool(s *Session, raw json.RawMessage) (message.Parts, error) { return jsonResult(s.modelToolStatus()) default: - return nil, fmt.Errorf("model: unknown action %q (valid actions: status, set — there is no clear action)", in.Action) + return nil, fmt.Errorf("model: unknown action %q (valid actions: status, set, list — there is no clear action)", in.Action) } } @@ -159,6 +190,22 @@ func (s *Session) modelToolStatus() modelToolResult { return res } +// modelToolList builds the list action's result: the configured provider +// families and aliases only — the same underlying data modelToolStatus +// reads (configuredProviderNames/ModelAliases), just without the current +// model field a bare "what models are available" query has no use for. +func (s *Session) modelToolList() modelListResult { + res := modelListResult{Providers: s.configuredProviderNames()} + if len(s.cfg.ModelAliases) > 0 { + aliases := make(map[string]string, len(s.cfg.ModelAliases)) + for k, v := range s.cfg.ModelAliases { + aliases[k] = v + } + res.Aliases = aliases + } + return res +} + // configuredProviderNames returns the configured provider names, sorted. func (s *Session) configuredProviderNames() []string { if len(s.cfg.Providers) == 0 { diff --git a/engine/model_tool_test.go b/engine/model_tool_test.go index 4e8eefa8..a809ea5b 100644 --- a/engine/model_tool_test.go +++ b/engine/model_tool_test.go @@ -64,6 +64,48 @@ func newModelToolSession(t *testing.T) *Session { }) } +// runModelToolListAction runs the model tool's "list" action directly +// against s and decodes the result as modelListResult. t.Fatal on a tool +// error. +func runModelToolListAction(t *testing.T, s *Session, args string) modelListResult { + t.Helper() + tool, ok := s.tools[modelToolName] + if !ok { + t.Fatal("model tool absent") + } + parts, err := tool.Run(context.Background(), s, []byte(args)) + if err != nil { + t.Fatalf("model tool run(%s): %v", args, err) + } + text, ok := parts[0].(*message.Text) + if !ok { + t.Fatalf("model tool result is not text: %#v", parts[0]) + } + var res modelListResult + if err := json.Unmarshal([]byte(text.Text), &res); err != nil { + t.Fatalf("model tool result not valid JSON: %v (%s)", err, text.Text) + } + return res +} + +// TestModelToolList proves the list action reports the same configured +// provider families and aliases modelToolStatus reads — the data a +// delegated caller (e.g. a claude-code-lane agent over the MCP shim) needs +// to pick a family for task's spawn(model:...) override, without a second +// data source that could drift from status's own. +func TestModelToolList(t *testing.T) { + s := newModelToolSession(t) + + res := runModelToolListAction(t, s, `{"action":"list"}`) + want := []string{"other", "test"} + if strings.Join(res.Providers, ",") != strings.Join(want, ",") { + t.Fatalf("list providers = %v, want %v (sorted)", res.Providers, want) + } + if res.Aliases["fast"] != "test/m2" { + t.Fatalf("list aliases = %+v, want fast->test/m2", res.Aliases) + } +} + func TestModelToolStatus(t *testing.T) { s := newModelToolSession(t) diff --git a/engine/task_tool.go b/engine/task_tool.go index b857352e..c7115f2f 100644 --- a/engine/task_tool.go +++ b/engine/task_tool.go @@ -18,6 +18,14 @@ import ( // see installTaskToolLocked and Spawn in session_manager.go. const taskToolName = "task" +// TaskToolName exports taskToolName for a caller outside this package that +// needs to name the SAME tool RunTool/ToolDef dispatch by — +// server/mcp_history.go's harness-hosted MCP `task` tool entry, notably — +// without hand-duplicating the literal "task" and risking it silently +// drifting from this package's own internal name. Mirrors +// engine/process.go's identical ProcessToolName export. +const TaskToolName = taskToolName + // The task tool's five actions. "" (the JSON zero value, so an omitted // action field) is treated as taskActionSpawn — the original, pre-verbs // shape of this tool's arguments (agent/prompt/model, no action at all) diff --git a/server/mcp_history.go b/server/mcp_history.go index 8d8e529c..5694c934 100644 --- a/server/mcp_history.go +++ b/server/mcp_history.go @@ -1,5 +1,5 @@ // This file implements harness's own hosted MCP tools: get_conversation_history -// and, when configured, the native `process` tool. +// and, when configured, the native `process`, `task`, and `model` tools. // // # Why this exists // @@ -29,21 +29,52 @@ // // # Beyond history: a generalized harness-tools server // -// The same endpoint also advertises harness's native `process` tool -// (engine/process.go) when the session has one configured (Config.Processes -// non-nil) — the tool a box's `pnpm dev`-style long-lived processes are -// started, stopped, and inspected through. A delegated claude-code turn -// otherwise has NO way to reach it: it drives its own tool loop entirely -// inside the `claude` binary, never through this package's native -// runToolCall path, so without an MCP entry a delegated turn simply could -// not start/stop/check a managed process at all. tools/call routes through -// engine.Session.RunTool (see its own doc comment), the SAME generic -// external-dispatch seam a future harness-hosted tool would use — this file -// deliberately exposes ONLY these two tools, not the redundant file tools -// (read/write/edit/glob/grep/ls — a delegated `claude` process already has -// its own, native equivalents) or the loop-internal ones (session_info, -// goal, model, mcp, task, read_tool_result — meaningless outside the -// native agentic loop this MCP surface exists to route AROUND). +// The same endpoint also advertises three more of harness's native session +// tools when the session has each configured: `process` (engine/process.go, +// gated on Config.Processes non-nil) — the tool a box's `pnpm dev`-style +// long-lived processes are started, stopped, and inspected through; `task` +// (engine/task_tool.go, gated on Config.SessionManager) — spawn/cancel/ +// send/status/log against a child session, INCLUDING a model override +// naming a different provider family than the one driving this delegated +// turn (a claude-code-lane agent can spawn a `sol`/`codex`/any-configured +// child this way); and `model` (engine/model_tool.go, gated on +// Config.ModelTool) — but ONLY its list action (the configured provider +// families and aliases to pick a target from), never the engine's own +// status/set: this surface deliberately narrows `model` down from its full +// ToolDef (see modelToolShimInputSchema/modelListOnlyMCPHandler) because +// set re-points THIS session's own live model — a real hijack of whichever +// lane is driving this very delegated turn, not merely an unwanted read — +// and status leaks current-session state a delegated caller has no +// legitimate need for; list is the one action such a caller actually +// needs, to pick a family for task's own spawn(model:...) override +// instead. A delegated claude-code turn otherwise has NO way to reach any +// of the three: it drives its own tool loop entirely inside the `claude` +// binary, never through this package's native runToolCall path, so +// without an MCP entry a delegated turn simply could not manage a +// process, delegate to a subagent, or discover a model family to delegate +// to at all. +// tools/call routes through engine.Session.RunTool (see its own doc +// comment), the SAME generic external-dispatch seam a future harness-hosted +// tool would use — this file deliberately exposes ONLY these four tools, +// not the redundant file tools (read/write/edit/glob/grep/ls — a delegated +// `claude` process already has its own, native equivalents) or the +// remaining loop-internal ones (session_info, goal, mcp, read_tool_result — +// still meaningless outside the native agentic loop this MCP surface exists +// to route AROUND; task and model, by contrast, are genuinely useful to a +// delegated caller and are exposed above). +// +// task's spawn action is already non-blocking (SessionManager.Spawn returns +// the child's session id at once and runs its turn in a goroutine) and its +// status/log actions already pull the child's live status/result directly +// from its own settled node state (SessionManager.DescendantInfo/ +// DescendantTranscript) rather than from the native push-delivery path (the +// EngineContext notification a child's completion normally delivers to its +// parent's NEXT native-loop turn) — a delegated turn has no such next turn +// this MCP surface would ever see, so status/log's pull semantics are what +// make collecting a spawned child's result possible here at all. Neither +// property required any change to engine/task_tool.go itself; see +// engine.TaskToolName's own doc comment for the one export this file +// needed. package server @@ -111,27 +142,68 @@ var historyToolInputSchema = json.RawMessage(`{ // in its own tools/list cache. const historyToolDescription = "Read the PRIOR conversation history for this session: messages that already happened before this turn, either on a different model or in a part of this conversation you have not seen. Call this once, before responding, whenever you are continuing a conversation you have not already read. Supports offset/limit pagination for long histories." -// historyToolAnnotations and processToolAnnotations are each tool's -// mcp.Tool.Annotations object (the spec's ToolAnnotations hints, +// historyToolAnnotations, processToolAnnotations, taskToolAnnotations, and +// modelToolAnnotations are each tool's mcp.Tool.Annotations object (the +// spec's ToolAnnotations hints, // https://modelcontextprotocol.io/specification/2025-11-25/server/tools#annotations) // — 2026-era MCP client UIs surface these to a human (or gate a // destructive call behind confirmation), so an honest hint here is worth -// setting even though this server does not enforce either one itself. +// setting even though this server does not enforce any of them itself. +// // get_conversation_history is read-only by construction (flattenHistory -// never mutates sess); the `process` tool is the opposite — its -// start/stop/restart actions can kill a running process — so it gets -// destructiveHint instead, never readOnlyHint. +// never mutates sess). `process`'s start/stop/restart actions can kill a +// running process, so it gets destructiveHint instead of readOnlyHint. +// `task` bundles spawn/cancel/send (each mutates a session — but only ones +// the CALLER itself spawned, directly or transitively; see task_tool.go's +// own cancel/send doc comments) alongside read-only status/log, so it gets +// readOnlyHint false rather than destructiveHint: a task call can create or +// stop the caller's OWN subagents, never touch anything outside that +// caller-owned subtree, which is a materially smaller blast radius than +// `process`'s ability to kill a shared, box-wide long-lived process. +// `model` is readOnlyHint true — and that is honest ONLY because this +// file's shim restricts the exposed `model` surface to the list action +// (see modelToolShimInputSchema/modelListOnlyMCPHandler below); the +// engine's own `model` tool also has a mutating set action (SetModel: +// persistModel + EventModelChanged), which would make readOnlyHint a lie +// if this shim ever passed it through. var ( historyToolAnnotations = json.RawMessage(`{"readOnlyHint": true}`) processToolAnnotations = json.RawMessage(`{"destructiveHint": true}`) + taskToolAnnotations = json.RawMessage(`{"readOnlyHint": false}`) + modelToolAnnotations = json.RawMessage(`{"readOnlyHint": true}`) ) +// modelToolShimInputSchema and modelToolShimDescription are the `model` +// tool's SHIM-published tools/list surface — DELIBERATELY NOT the engine's +// own full ToolDef (status/set/list), unlike process/task just below. A +// delegated caller only ever needs to enumerate provider families/aliases +// to pick one for task's own spawn(model:...) override; it must never +// reach set (re-points THIS session's own main model — the parent's live +// delegation to whatever lane is driving this very turn, a real behavioral +// hijack, not merely an unwanted read) or status (leaks this session's own +// current-model state, which list's own result deliberately omits — see +// modelListResult, engine/model_tool.go). Restricting the PUBLISHED schema +// alone is not enough — a caller can send whatever action it wants +// regardless of what tools/list advertised — so modelListOnlyMCPHandler +// enforces the same restriction on every actual tools/call, before +// dispatch. +var modelToolShimInputSchema = json.RawMessage(`{ + "type": "object", + "properties": { + "action": {"type": "string", "enum": ["list"], "description": "The operation to perform; only \"list\" is exposed on this surface"} + }, + "required": ["action"] +}`) + +const modelToolShimDescription = "List the provider families and aliases configured on this box. Use this to pick a family for task's own spawn(model:...) override when delegating to a child session. This surface exposes ONLY the list action — inspecting or changing THIS session's own current model is not available here; task's model override is the way to select a model, for a CHILD session, not this one." + // newSessionMCPRegistry builds the per-request mcpserver.Registry for // sess's own /session/{id}/mcp endpoint (handleSessionMCP): always -// get_conversation_history, plus the native `process` tool whenever sess -// has one configured (Config.Processes non-nil — see -// engine.Session.ToolDef) — see this file's package doc for why only -// these two. version is reported as the MCP server's own implementation +// get_conversation_history, plus the native `process`, `task`, and `model` +// tools whenever sess has each one configured (Config.Processes non-nil, +// Config.SessionManager set, Config.ModelTool true, respectively — see +// engine.Session.ToolDef) — see this file's package doc for why exactly +// these four. version is reported as the MCP server's own implementation // version (Options.Version — the same harness build version every other // endpoint already reports, not a version of the tool's own wire shape). func newSessionMCPRegistry(sess *engine.Session, version string) *mcpserver.Registry { @@ -149,21 +221,46 @@ func newSessionMCPRegistry(sess *engine.Session, version string) *mcpserver.Regi Annotations: historyToolAnnotations, }, historyToolHandler(sess)) - // def comes from the engine's OWN process tool registration - // (engine/process.go's processTool) via ToolDef, not a second, - // hand-duplicated copy of its Description/InputSchema — the two would - // otherwise be free to silently drift apart. ok is false exactly when - // Config.Processes is nil (see ToolDef's own doc comment), the same - // condition that hides the native `process` tool from the model - // entirely — this MCP surface must not advertise a tool a delegated - // turn could call and get "unknown tool" back from RunTool. + // def for process/task below comes from the engine's OWN tool + // registration via ToolDef, never a second, hand-duplicated copy of + // its Description/InputSchema — the two would otherwise be free to + // silently drift apart. ok is false exactly when the tool's owning + // Config field is unset (see each ToolDef's own doc comment), the same + // condition that hides the native tool from the model entirely — this + // MCP surface must not advertise a tool a delegated turn could call + // and get "unknown tool" back from RunTool. if def, ok := sess.ToolDef(engine.ProcessToolName); ok { reg.RegisterTool(mcp.Tool{ Name: def.Name, Description: def.Description, InputSchema: def.InputSchema, Annotations: processToolAnnotations, - }, processToolMCPHandler(sess)) + }, runToolMCPHandler(sess, engine.ProcessToolName)) + } + if def, ok := sess.ToolDef(engine.TaskToolName); ok { + reg.RegisterTool(mcp.Tool{ + Name: def.Name, + Description: def.Description, + InputSchema: def.InputSchema, + Annotations: taskToolAnnotations, + }, runToolMCPHandler(sess, engine.TaskToolName)) + } + // model is the ONE exception to the "pass the engine's own ToolDef + // through verbatim" rule above: ok still gates on the real ToolDef (so + // this surface disappears exactly when Config.ModelTool is off, + // matching the native tool's own availability), but the Description/ + // InputSchema this file PUBLISHES are the hand-written, list-only + // modelToolShimDescription/modelToolShimInputSchema, never the + // engine's full status/set/list schema — see their own doc comment for + // why passing that through would misrepresent (and worse, invite) a + // mutating call this shim must never allow. + if _, ok := sess.ToolDef(engine.ModelToolName); ok { + reg.RegisterTool(mcp.Tool{ + Name: engine.ModelToolName, + Description: modelToolShimDescription, + InputSchema: modelToolShimInputSchema, + Annotations: modelToolAnnotations, + }, modelListOnlyMCPHandler(sess)) } return reg } @@ -195,19 +292,58 @@ func historyToolHandler(sess *engine.Session) mcpserver.ToolHandler { } } -// processToolMCPHandler returns the mcpserver.ToolHandler for the native -// `process` tool, closing over sess. It routes every call through -// sess.RunTool (engine/engine.go) — the SAME generic dispatch path a -// native-loop process-tool call goes through (hooks, events, panic -// recovery included) — never a hand-rolled second implementation of the -// process tool's own action switch. RunTool's error already carries the -// tool-level failure text (e.g. "no such process"), and returning it -// directly from this handler makes mcpserver.Registry's own dispatch turn -// it into CallToolResult.IsError automatically — see ToolHandler's own doc -// comment for that contract. -func processToolMCPHandler(sess *engine.Session) mcpserver.ToolHandler { +// runToolMCPHandler returns the mcpserver.ToolHandler for the native +// session tool registered under name (process, task, or model — see +// newSessionMCPRegistry), closing over sess. It routes every call through +// sess.RunTool — the SAME generic dispatch path a native-loop call to that +// tool goes through (hooks, events, panic recovery included) — never a +// hand-rolled second implementation of the tool's own action switch. +// RunTool's error already carries the tool-level failure text (e.g. "no +// such process", "unknown action"), and returning it directly from this +// handler makes mcpserver.Registry's own dispatch turn it into +// CallToolResult.IsError automatically — see ToolHandler's own doc comment +// for that contract. +func runToolMCPHandler(sess *engine.Session, name string) mcpserver.ToolHandler { return func(ctx context.Context, raw json.RawMessage) (mcp.CallToolResult, error) { - parts, err := sess.RunTool(ctx, engine.ProcessToolName, raw) + parts, err := sess.RunTool(ctx, name, raw) + if err != nil { + return mcp.CallToolResult{}, err + } + return mcp.CallToolResult{Content: partsToMCPContent(parts)}, nil + } +} + +// modelListOnlyMCPHandler returns the mcpserver.ToolHandler for the +// SHIM-restricted `model` tool, closing over sess: it enforces action == +// "list" itself, BEFORE ever dispatching through RunTool, rather than +// trusting modelToolShimInputSchema's published enum alone — a caller can +// send any action it likes over the wire regardless of what tools/list +// advertised, so the schema is documentation, not enforcement. Rejecting +// here, as an ordinary tool-level error (IsError, not a protocol failure — +// same ToolHandler contract every other handler in this file follows), +// is what actually closes off `set` (re-points THIS session's own live +// model — a real hijack of whichever lane is driving this very delegated +// turn) and `status` (leaks this session's own current-model state) — see +// modelToolShimInputSchema's own doc comment for the full reasoning. +// action == "list" still routes through sess.RunTool(engine.ModelToolName, +// ...) — the SAME generic dispatch path (hooks, events, panic recovery +// included) runToolMCPHandler's other callers use — so nothing about the +// engine's own `model` tool changes; only what THIS shim will forward to +// it is narrowed. +func modelListOnlyMCPHandler(sess *engine.Session) mcpserver.ToolHandler { + return func(ctx context.Context, raw json.RawMessage) (mcp.CallToolResult, error) { + var in struct { + Action string `json:"action"` + } + if len(raw) > 0 { + if err := json.Unmarshal(raw, &in); err != nil { + return mcp.CallToolResult{}, fmt.Errorf("model: invalid arguments: %w", err) + } + } + if in.Action != "list" { + return mcp.CallToolResult{}, fmt.Errorf("model: action %q is not available on this surface (only \"list\" is exposed here — use task's own model override to select a model for a child session)", in.Action) + } + parts, err := sess.RunTool(ctx, engine.ModelToolName, raw) if err != nil { return mcp.CallToolResult{}, err } diff --git a/server/mcp_history_test.go b/server/mcp_history_test.go index 401e9835..7593f419 100644 --- a/server/mcp_history_test.go +++ b/server/mcp_history_test.go @@ -5,12 +5,17 @@ import ( "encoding/json" "fmt" "net/http" + "net/http/httptest" + "sort" "strings" "testing" + "testing/synctest" + "github.com/majorcontext/harness/engine" "github.com/majorcontext/harness/mcp" "github.com/majorcontext/harness/message" "github.com/majorcontext/harness/process" + "github.com/majorcontext/harness/provider" ) // seedMessages builds a small, readable history: a user question, an @@ -279,10 +284,16 @@ func TestHandleSessionMCPFullLifecycle(t *testing.T) { // TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured proves a // session with no Config.Processes (the ordinary newHarness session, no -// process manager wired at all) advertises ONLY get_conversation_history — -// the same "process tool absent when unconfigured" rule the native loop -// already follows (engine's TestProcessToolAbsentWhenNoProcessesConfigured) -// applies identically to this MCP surface. +// process manager wired at all) advertises get_conversation_history and +// `task` but NEVER `process` — the same "process tool absent when +// unconfigured" rule the native loop already follows (engine's +// TestProcessToolAbsentWhenNoProcessesConfigured) applies identically to +// this MCP surface. `task` is present here (unlike `model`, gated purely on +// Config.ModelTool) because handleCreate's own h.createSession call path +// always runs SessionManager.AdoptRoot on a freshly created session (see +// server/handlers.go), which installs the native `task` tool unconditionally +// — see adoptRootLocked's own doc comment — independent of whatever +// Config.SessionManager the session was originally constructed with. func TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured(t *testing.T) { h := newHarness(t, &scriptedProvider{name: "test"}) id := h.createSession("") @@ -296,18 +307,26 @@ func TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured(t *testing.T if err := json.Unmarshal(listData, &listMsg); err != nil { t.Fatalf("decoding tools/list response: %v (%s)", err, listData) } - if len(listMsg.Result.Tools) != 1 || listMsg.Result.Tools[0].Name != historyToolName { - t.Fatalf("tools/list Tools = %+v, want exactly [%s]", listMsg.Result.Tools, historyToolName) + var names []string + for _, tl := range listMsg.Result.Tools { + names = append(names, tl.Name) + } + sort.Strings(names) + want := []string{historyToolName, "task"} + if strings.Join(names, ",") != strings.Join(want, ",") { + t.Fatalf("tools/list Tools = %v, want exactly %v (no process, no model)", names, want) } } // TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations proves a // session WITH Config.Processes configured advertises the native -// `process` tool alongside get_conversation_history, each carrying the -// annotation this file's package doc promises: readOnlyHint on history, -// destructiveHint on process — plus process's own Description/InputSchema -// passed through from the engine's real tool definition (ToolDef), not a -// hand-duplicated copy. +// `process` tool alongside get_conversation_history (and `task`, always +// present on a handleCreate-adopted session — see +// TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured's own doc +// comment), each carrying the annotation this file's package doc promises: +// readOnlyHint on history, destructiveHint on process — plus process's own +// Description/InputSchema passed through from the engine's real tool +// definition (ToolDef), not a hand-duplicated copy. func TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations(t *testing.T) { h, _ := newProcessHarness(t, map[string]process.Def{ "dev": {Command: []string{"sh", "-c", "true"}}, @@ -323,10 +342,14 @@ func TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations(t *testing. if err := json.Unmarshal(listData, &listMsg); err != nil { t.Fatalf("decoding tools/list response: %v (%s)", err, listData) } - if len(listMsg.Result.Tools) != 2 { - t.Fatalf("tools/list Tools = %+v, want exactly 2 (history + process)", listMsg.Result.Tools) - } - + // Per-name presence/absence, not a raw tool count (which drifts the + // moment any always-on tool — like `task`, unconditionally installed + // by handleCreate's own AdoptRoot call; see + // TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured's own + // doc comment — is added elsewhere): history and process are the + // tools THIS test cares about, task is expected-but-incidental here, + // and model must be absent (Config.ModelTool is not set by + // newProcessHarness). var hist, proc *mcp.Tool for i := range listMsg.Result.Tools { switch listMsg.Result.Tools[i].Name { @@ -334,6 +357,8 @@ func TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations(t *testing. hist = &listMsg.Result.Tools[i] case "process": proc = &listMsg.Result.Tools[i] + case "model": + t.Fatalf("tools/list Tools = %+v, want no model (Config.ModelTool not set)", listMsg.Result.Tools) } } if hist == nil { @@ -470,3 +495,458 @@ func TestHandleSessionMCPRequiresAuth(t *testing.T) { t.Errorf("status = %d, want 401 with no Authorization header", resp.StatusCode) } } + +// --- task/model tool exposure --- + +// newTaskModelHarness builds a harness whose sessions have BOTH the native +// `task` tool (Config.SessionManager wired to the server's own srv.sessMgr +// — the same wiring production's cmd/harness mkCfg uses, and the pattern +// journal_spawn_sync_test.go's +// TestChildJournaledAfterParentIdleEvictedAndReloaded already establishes +// for this package's tests) and the `model` tool (Config.ModelTool true) — +// the two tools this file's task+model MCP exposure tests need advertised +// together. The default newHarness/newServer helper (server_test.go) +// deliberately leaves both off (see that helper's own mkCfg), so a test +// that needs either builds its own Options here rather than mutating the +// shared default. +func newTaskModelHarness(t *testing.T, reg provider.Registry, defaultModel message.ModelRef) *harness { + t.Helper() + const token = "secret-run-token" + dir := t.TempDir() + var srv *Server + opts := Options{ + SessionDir: dir, + RunToken: token, + Version: "9.9.9", + NewSession: func(m message.ModelRef, workDir, parentSession string) (*engine.Session, error) { + if m.IsZero() { + m = defaultModel + } + return engine.NewSession(engine.Config{ + Providers: reg, + Model: m, + WorkDir: workDir, + ParentSession: parentSession, + SessionDir: dir, + OnEvent: func(ev engine.Event) { srv.Publish(ev) }, + SessionManager: srv.sessMgr, + ModelTool: true, + }), nil + }, + LoadSession: func(id string) (*engine.Session, error) { + return engine.LoadSession(engine.Config{ + Providers: reg, + SessionDir: dir, + OnEvent: func(ev engine.Event) { srv.Publish(ev) }, + SessionManager: srv.sessMgr, + ModelTool: true, + }, id) + }, + } + var err error + srv, err = New(opts) + if err != nil { + t.Fatal(err) + } + ts := httptest.NewServer(srv) + t.Cleanup(ts.Close) + return &harness{t: t, dir: dir, token: token, srv: srv, ts: ts} +} + +// TestHandleSessionMCPToolsListIncludesTaskAndModelToolsWithAnnotations +// proves a session with Config.SessionManager and Config.ModelTool both set +// advertises `task` and `model` alongside get_conversation_history, each +// carrying the annotation this file's package doc now promises: readOnlyHint +// false on `task` (it can mutate sessions the caller itself spawned — +// spawn/cancel/send — even though its status/log actions are read-only), +// readOnlyHint true on `model` — and, since that hint is only honest +// because this surface is list-only (see modelToolShimInputSchema), also +// proves the PUBLISHED schema itself advertises action enum ["list"] only, +// never "set" or "status". `task`'s Description/InputSchema are checked +// against the engine's own real ToolDef (never a hand-duplicated copy); +// `model`'s are checked against this file's own hand-written shim schema +// instead, since `model` is the one tool this surface deliberately does NOT +// pass the engine's full ToolDef through for. +func TestHandleSessionMCPToolsListIncludesTaskAndModelToolsWithAnnotations(t *testing.T) { + rootProv := &scriptedProvider{name: "root"} + childProv := &scriptedProvider{name: "child"} + h := newTaskModelHarness(t, provider.Registry{ + rootProv.Name(): rootProv, + childProv.Name(): childProv, + }, message.ModelRef{Provider: rootProv.Name(), Model: "m1"}) + id := h.createSession("") + + _, listData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/list", + }) + var listMsg struct { + Result mcp.ListToolsResult `json:"result"` + } + if err := json.Unmarshal(listData, &listMsg); err != nil { + t.Fatalf("decoding tools/list response: %v (%s)", err, listData) + } + + // Per-name presence, not a raw tool count — see + // TestHandleSessionMCPToolsListIncludesProcessToolWithAnnotations's + // identical reasoning. `process` is correctly absent here + // (Config.Processes unset by newTaskModelHarness), which the switch + // below would catch as an unhandled case if it ever regressed. + var hist, task, model *mcp.Tool + for i := range listMsg.Result.Tools { + switch listMsg.Result.Tools[i].Name { + case historyToolName: + hist = &listMsg.Result.Tools[i] + case "task": + task = &listMsg.Result.Tools[i] + case "model": + model = &listMsg.Result.Tools[i] + case "process": + t.Fatalf("tools/list Tools = %+v, want no process (Config.Processes not set)", listMsg.Result.Tools) + } + } + if hist == nil { + t.Fatal("tools/list missing get_conversation_history") + } + if task == nil { + t.Fatal("tools/list missing task") + } + if !strings.Contains(string(task.Annotations), `"readOnlyHint":false`) { + t.Errorf("task Annotations = %s, want readOnlyHint false", task.Annotations) + } + if len(task.InputSchema) == 0 || !strings.Contains(task.Description, "spawn") { + t.Errorf("task Description/InputSchema = %q/%s, want the engine's own task-tool schema passed through", task.Description, task.InputSchema) + } + if model == nil { + t.Fatal("tools/list missing model") + } + if !strings.Contains(string(model.Annotations), `"readOnlyHint":true`) { + t.Errorf("model Annotations = %s, want readOnlyHint true", model.Annotations) + } + // The load-bearing assertion: the PUBLISHED schema's action enum is + // list-only. json.Marshal compacts an embedded json.RawMessage (no + // insignificant whitespace survives the round trip through + // writeResult), so the wire form is exactly `"enum":["list"]`. + if !strings.Contains(string(model.InputSchema), `"enum":["list"]`) { + t.Errorf("model InputSchema = %s, want action enum [\"list\"] only", model.InputSchema) + } + if strings.Contains(string(model.InputSchema), `"set"`) || strings.Contains(string(model.InputSchema), `"status"`) { + t.Errorf("model InputSchema = %s, want it to never mention set or status", model.InputSchema) + } +} + +// TestHandleSessionMCPToolsListOmitsModelToolWhenDisabled proves a session +// built with Config.ModelTool false (the ordinary newHarness session) +// advertises history and `task` but never `model` — unlike `task` (always +// installed on a handleCreate-adopted session; see +// TestHandleSessionMCPToolsListOmitsProcessToolWhenNotConfigured's own doc +// comment), `model` has no such adopt-time install and stays governed +// purely by the Config.ModelTool flag the session was constructed with — +// the same "absent when unconfigured" rule already proven for `process`. +func TestHandleSessionMCPToolsListOmitsModelToolWhenDisabled(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("") + + _, listData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/list", + }) + var listMsg struct { + Result mcp.ListToolsResult `json:"result"` + } + if err := json.Unmarshal(listData, &listMsg); err != nil { + t.Fatalf("decoding tools/list response: %v (%s)", err, listData) + } + for _, tl := range listMsg.Result.Tools { + if tl.Name == "model" { + t.Fatalf("tools/list Tools = %+v, want no model (Config.ModelTool false)", listMsg.Result.Tools) + } + } +} + +// TestHandleSessionMCPModelToolListCallReturnsConfiguredFamilies proves +// tools/call for the `model` tool's list action routes through +// engine.Session.RunTool and returns the real configured provider families +// — the data a delegated caller (e.g. a claude-code-lane agent) needs to +// pick a family for task's own spawn(model:...) override. +func TestHandleSessionMCPModelToolListCallReturnsConfiguredFamilies(t *testing.T) { + rootProv := &scriptedProvider{name: "root"} + childProv := &scriptedProvider{name: "sol"} + h := newTaskModelHarness(t, provider.Registry{ + rootProv.Name(): rootProv, + childProv.Name(): childProv, + }, message.ModelRef{Provider: rootProv.Name(), Model: "m1"}) + id := h.createSession("") + + _, callData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/call", + "params": map[string]any{"name": "model", "arguments": map[string]any{"action": "list"}}, + }) + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call(model list) response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call(model list) returned a protocol error: %+v", callMsg.Error) + } + if callMsg.Result.IsError { + t.Fatalf("tools/call(model list) result IsError=true: %+v", callMsg.Result.Content) + } + got := callMsg.Result.Content[0].Text + for _, want := range []string{`"root"`, `"sol"`} { + if !strings.Contains(got, want) { + t.Errorf("model list result = %s, want it to list configured family %s", got, want) + } + } +} + +// TestHandleSessionMCPModelToolSetAndStatusRejectedOverShim is the +// regression guard for the hijack this file's `model` shim exists to +// close: without modelListOnlyMCPHandler's own action check, a delegated +// caller could send {"name":"model","arguments":{"action":"set", ...}} +// over this exact endpoint and re-point the PARENT session's own live +// model — the session actually driving this delegated turn — a real +// behavioral hijack (the next harness turn stops delegating to whichever +// lane issued the call), not merely an unwanted read. `status` is rejected +// too: it leaks this session's own current-model state, which `list` +// deliberately omits (see modelListResult, engine/model_tool.go). Both +// must come back as an ordinary CallToolResult.IsError tool failure — the +// same tool-level-vs-protocol-level distinction +// TestHandleSessionMCPProcessToolCallFailureIsToolError already locks in +// for `process` — never a protocol-level RPCError (model IS a known, +// registered tool) and never a 200 with the session's model actually +// changed. +// +// Red-verify: delete the `if in.Action != "list"` check in +// modelListOnlyMCPHandler (server/mcp_history.go) and this test fails — +// set's IsError assertion fails first (RunTool actually swaps the model +// and returns success), proving this test would have caught the hijack. +func TestHandleSessionMCPModelToolSetAndStatusRejectedOverShim(t *testing.T) { + rootProv := &scriptedProvider{name: "root"} + childProv := &scriptedProvider{name: "sol"} + h := newTaskModelHarness(t, provider.Registry{ + rootProv.Name(): rootProv, + childProv.Name(): childProv, + }, message.ModelRef{Provider: rootProv.Name(), Model: "m1"}) + id := h.createSession("") + + callModel := func(args map[string]any) mcp.CallToolResult { + t.Helper() + _, callData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/call", + "params": map[string]any{"name": "model", "arguments": args}, + }) + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call(model) response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call(model) returned a protocol-level error for a tool-level rejection: %+v", callMsg.Error) + } + return callMsg.Result + } + + setResult := callModel(map[string]any{"action": "set", "model": "sol/m1"}) + if !setResult.IsError { + t.Fatalf("tools/call(model set) IsError = false, want true — set must never be reachable over this shim: %+v", setResult.Content) + } + + statusResult := callModel(map[string]any{"action": "status"}) + if !statusResult.IsError { + t.Fatalf("tools/call(model status) IsError = false, want true — status must never be reachable over this shim: %+v", statusResult.Content) + } + + // The actual hijack check: the session's OWN model must be completely + // unaffected by the rejected set call above — read it back via the + // one action this shim DOES allow (list carries no current-model + // field, so this instead re-derives the session's live model via a + // direct engine-level check, the same session object the server + // itself is holding for id). + sess, ok := h.srv.sessMgr.Session(id) + if !ok { + t.Fatalf("session %s not tracked by sessMgr", id) + } + if got := sess.Model(); got != (message.ModelRef{Provider: rootProv.Name(), Model: "m1"}) { + t.Fatalf("session model = %v after a rejected set, want unchanged root/m1 — the shim's set rejection must be enforced BEFORE dispatch", got) + } +} + +// TestHandleSessionMCPTaskToolCallUnknownActionIsToolError proves an +// unknown `task` action over this MCP surface comes back as +// CallToolResult.IsError (a tool-level failure), not a protocol-level +// RPCError — the same distinction +// TestHandleSessionMCPProcessToolCallFailureIsToolError already locks in +// for `process`. +func TestHandleSessionMCPTaskToolCallUnknownActionIsToolError(t *testing.T) { + rootProv := &scriptedProvider{name: "root"} + h := newTaskModelHarness(t, provider.Registry{rootProv.Name(): rootProv}, message.ModelRef{Provider: rootProv.Name(), Model: "m1"}) + id := h.createSession("") + + _, callData := h.do("POST", "/session/"+id+"/mcp", map[string]any{ + "jsonrpc": "2.0", "id": "1", "method": "tools/call", + "params": map[string]any{"name": "task", "arguments": map[string]any{"action": "not_a_real_action"}}, + }) + var callMsg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(callData, &callMsg); err != nil { + t.Fatalf("decoding tools/call(task) response: %v (%s)", err, callData) + } + if callMsg.Error != nil { + t.Fatalf("tools/call(task) returned a protocol-level error for a tool-level failure: %+v", callMsg.Error) + } + if !callMsg.Result.IsError { + t.Fatalf("tools/call(task) Result.IsError = false, want true for an unknown action") + } + if len(callMsg.Result.Content) != 1 || !strings.Contains(callMsg.Result.Content[0].Text, "not_a_real_action") { + t.Errorf("tools/call(task) Content = %+v, want it to name the unknown action", callMsg.Result.Content) + } +} + +// TestHandleSessionMCPTaskToolSpawnIsNonBlockingAndStatusPullsResult is the +// end-to-end proof behind this file's task exposure: tools/call(task, +// spawn) over the MCP surface routes through engine.Session.RunTool into +// the REAL, already-non-blocking Session.Spawn (it launches the child's own +// turn in a goroutine and returns the child's session id at once — see +// SessionManager.Spawn's own doc comment) with a model override selecting a +// DIFFERENT configured family (child/m1, distinct from the root session's +// own root/m1) — and a later tools/call(task, status) pulls the child's +// result straight from its own settled node state +// (SessionManager.DescendantInfo), independent of the native push-delivery +// path this MCP surface deliberately routes around (see this file's package +// doc). +// +// childProv is a blockingProvider, deliberately never released until AFTER +// the spawn call and an immediate status check both complete: if +// runTaskSpawn (or RunTool's dispatch of it) ever became blocking — waiting +// on the child's own Prompt call before returning — this test would hang +// rather than merely race, the same "prove non-blocking by parking the +// dependency, not by guessing at scheduling order" technique +// server_test.go's other blockingProvider tests already use. The +// immediate-after-spawn status call is read BEFORE the child is released, +// so it deterministically observes the child still StatusRunning (set +// synchronously, under SessionManager's own lock, before Spawn ever +// returns) — never a race against how fast a real child would finish. +// synctest.Wait() then settles the child's turn (and any auto-resume +// notification it triggers on root, hence rootProv's own scripted "noted" +// turn) deterministically, with zero real wall-clock cost, mirroring +// journal_spawn_sync_test.go's identical use of Wait() for this exact +// purpose. +func TestHandleSessionMCPTaskToolSpawnIsNonBlockingAndStatusPullsResult(t *testing.T) { + dir := t.TempDir() + synctest.Test(t, func(t *testing.T) { + rootProv := &scriptedProvider{name: "root", turns: [][]provider.Event{ + asstTurn("noted"), // consumes the auto-resume notification turn. + }} + childProv := newBlockingProvider("child") + t.Cleanup(childProv.releaseAll) + reg := provider.Registry{rootProv.Name(): rootProv, childProv.Name(): childProv} + + var srv *Server + opts := Options{ + SessionDir: dir, + RunToken: "secret-run-token", + Version: "9.9.9", + NewSession: func(m message.ModelRef, workDir, parentSession string) (*engine.Session, error) { + return engine.NewSession(engine.Config{ + Providers: reg, + Model: m, + WorkDir: workDir, + ParentSession: parentSession, + SessionDir: dir, + OnEvent: func(ev engine.Event) { srv.Publish(ev) }, + SessionManager: srv.sessMgr, + ModelTool: true, + }), nil + }, + LoadSession: func(id string) (*engine.Session, error) { + return engine.LoadSession(engine.Config{ + Providers: reg, + SessionDir: dir, + OnEvent: func(ev engine.Event) { srv.Publish(ev) }, + SessionManager: srv.sessMgr, + ModelTool: true, + }, id) + }, + } + var err error + srv, err = New(opts) + if err != nil { + t.Fatal(err) + } + + rootID := createSessionDirect(t, srv, "root/m1") + + callMCP := func(body string) mcp.CallToolResult { + t.Helper() + rec := httptest.NewRecorder() + req := httptest.NewRequest("POST", "/session/"+rootID+"/mcp", strings.NewReader(body)) + req.SetPathValue("id", rootID) + srv.handleSessionMCP(rec, req) + if rec.Code != http.StatusOK { + t.Fatalf("tools/call status %d: %s", rec.Code, rec.Body) + } + var msg struct { + Result mcp.CallToolResult `json:"result"` + Error *mcp.RPCError `json:"error"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &msg); err != nil { + t.Fatalf("decoding tools/call response: %v (%s)", err, rec.Body) + } + if msg.Error != nil { + t.Fatalf("tools/call returned a protocol error: %+v", msg.Error) + } + return msg.Result + } + + spawnResult := callMCP(`{"jsonrpc":"2.0","id":"1","method":"tools/call","params":{"name":"task","arguments":{"action":"spawn","agent":"general-purpose","prompt":"find the answer","model":"child/m1"}}}`) + if spawnResult.IsError { + t.Fatalf("tools/call(spawn) IsError=true: %+v", spawnResult.Content) + } + var spawned struct { + SessionID string `json:"session_id"` + } + if len(spawnResult.Content) != 1 { + t.Fatalf("tools/call(spawn) Content = %+v, want exactly one text item", spawnResult.Content) + } + if err := json.Unmarshal([]byte(spawnResult.Content[0].Text), &spawned); err != nil { + t.Fatalf("decoding spawn result: %v (%s)", err, spawnResult.Content[0].Text) + } + if spawned.SessionID == "" { + t.Fatal("spawn result has no session_id") + } + + statusCall := `{"jsonrpc":"2.0","id":"2","method":"tools/call","params":{"name":"task","arguments":{"action":"status","session_id":"` + spawned.SessionID + `"}}}` + + // The child is still parked on blockingStream.Next (childProv not + // yet released) — this proves the spawn call above did not wait + // for it, and DescendantInfo's live status confirms the child is + // tracked as running, not merely "unknown" or "idle". + early := callMCP(statusCall) + if early.IsError { + t.Fatalf("tools/call(status) IsError=true: %+v", early.Content) + } + if !strings.Contains(early.Content[0].Text, `"status":"running"`) { + t.Fatalf("status immediately after spawn = %s, want status running (child still parked on its provider)", early.Content[0].Text) + } + + childProv.releaseAll() + synctest.Wait() + + final := callMCP(statusCall) + if final.IsError { + t.Fatalf("tools/call(status) IsError=true: %+v", final.Content) + } + if !strings.Contains(final.Content[0].Text, `"status":"done"`) { + t.Fatalf("status after settling = %s, want status done", final.Content[0].Text) + } + if !strings.Contains(final.Content[0].Text, "released") { + t.Fatalf("status after settling = %s, want the child's own result (\"released\") pulled from its settled node state", final.Content[0].Text) + } + }) +} From ae29764ef1dc939dfb82883ac3d33e23d8d79d9e Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 00:39:41 -0400 Subject: [PATCH 35/95] fix(server): race-close the transcript bootstrap resume point (#224) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * server: add a race-closed transcript bootstrap (?stream_from=1) A console that tail-loads a session's transcript and then opens the live SSE stream from "now" has a REPLAY window: a message journaled between the transcript read and the stream's own cursor can arrive on both, rendering it twice. boxes' read-path design names this the tail-load-versus-live-stream race and asks for a snapshot that also reports the exact durable seq it is synced through, so a client can resume GET /event strictly after it instead of guessing a cursor. Add Server.transcriptSyncedThrough (journal.go): syncMessages plus one extra locked read. sess.History()/sess.PersistErr() are read in the same unlocked window syncMessages already documents (session.mu released before server.mu, to avoid the lock-order cycle noted there), every unseen message in that snapshot is journaled under one s.mu hold exactly as syncMessages does, and the watermark is sampled in that SAME critical section right after the journaling loop, so nothing can be appended for the session between "the snapshot is durable" and "the watermark was read." It uses lookupSession, not liveSessionObject, so a cold on-disk session answers exactly like the existing unparameterized read does. handleMessages gets a third opt-in branch, `?stream_from=1`, alongside the untouched bare-array default and the before_seq/limit MessagePage branch; the answer is a new transcriptJSON/Transcript envelope ({messages, stream_from}). The watermark is deliberately NOT sessionSeqLocked (the plain highest seq journaled for the session, of any type): the unlocked read window above means a concurrent syncMessages call for the same session, with a fresher History() snapshot, can win the s.mu race and durably journal a message this call's own snapshot never saw. That message's seq would already count toward sessionSeqLocked's raw max even though it is absent from the returned history - exactly the gap a client resuming from that seq could never recover, since sse.go's replay is strict (`ev.Seq > from`). Confirmed by temporarily wiring the naive sessionSeqLocked into transcriptSyncedThrough and re-running TestTranscriptStreamFrom_ConcurrentJournalDuringSnapshot: it failed with the raced message absent from history yet seq <= stream_from. The fix, transcriptWatermarkLocked, restricts the max to journaled message records whose ID is present in the returned history. Every message in history was appended no later than this call's own History() snapshot, so any message excluded from it was necessarily appended strictly after - and every syncMessages-family loop journals a session's messages in history order under one lock, so a later-appended message can never receive a lower seq than an earlier one regardless of which call performs the journaling. That makes seq(excluded message) > seq(any message in history) always hold, so the watermark can never straddle a message the snapshot omits. Added a transcriptSyncRace test-only seam (server.go), mirroring sseRegisteredRace, firing between the unlocked reads and the s.mu hold. Added three tests (transcript_sync_test.go): the snapshot's messages are all journaled with seq <= stream_from and journaled IN THIS REQUEST for a session nothing had synced before; a message journaled after the call has seq > stream_from; and the concurrent race above never leaves the raced message in a state the resume cursor can't recover (present in history with seq <= stream_from, or absent with seq > stream_from - never anything else). Existing message/message-page tests (bare array, before_seq/limit paging) pass unchanged. Documented the new query parameter and Transcript schema in server/openapi.yaml per server/AGENTS.md. Verified: gofmt -l . clean; go vet ./... clean; go build ./... clean; go test -race ./... green across every package, including the new transcript_sync_test.go cases and the full server suite. * server: close a compaction-splice gap in the transcript watermark Adversarial review of the prior commit (the ?stream_from=1 race-closed transcript bootstrap) found a real gap in transcriptWatermarkLocked's "restrict the max to message IDs in history" rule: it assumed history only ever grows at the tail, so an excluded message is always chronologically (and therefore seq-wise) later than everything in a stale snapshot. engine/compact.go's Session.Compact breaks that assumption — it SPLICES a new summary message into an EARLIER array position, replacing the folded range, then journals the resulting history in array order. A compaction landing in transcriptSyncedThrough's unlocked read window can therefore journal its summary with a LOWER seq than an already-later message the stale snapshot already contains (that message sits after the summary's new position and gets journaled after it, in the same pass). The prior watermark would then count that later message's seq while the summary sits below it, excluded from history — and unlike an ordinary excluded message, which self-heals by arriving live, a summary in that state is permanently unrecoverable: its paired history.compacted reconciliation record never arrives either, since the client's SSE resume point already sits past it. Reverting only the fix and re-running the new regression test reproduced it directly: summary journaled at seq 2, reported stream_from 10. transcriptWatermarkLocked now caps the watermark below any compaction summary history excludes: for every evtHistoryCompacted record for the session whose CompactSummaryID is absent from history, the watermark is pulled down to strictly below that summary's own journaled seq, even when that lowers it below some in-history message's already- higher seq. The tradeoff is deliberate and asymmetric — a message dropping below the cap is merely redelivered live once more (the ordinary, self-correcting duplicate this endpoint exists to reduce), never permanently lost, whereas letting a summary slip above the watermark cannot be recovered at all. Added TestTranscriptStreamFrom_CompactionDuringSnapshotStaysRecoverable, which races a real POST /session/{id}/compact into the transcriptSyncRace gap against a session the harness has never synced, and red-verified it against the unfixed watermark before restoring the cap. Review also found: handleMessages silently let before_seq/limit win over a request naming stream_from alongside them, discarding the caller's second, incompatible intention instead of rejecting it — inconsistent with intParam's own "two intentions, pick one" rule for a repeated before_seq/limit value. Now 400s the combination; added TestTranscriptStreamFrom_RejectsCombinationWithPaging. Also added TestTranscriptStreamFrom_EmptyHistoryReportsZero (pins the empty- history-returns-0 behavior as deliberate, not an oversight — a higher fallback would reopen the exact race this endpoint exists to close, for a session's own first message) and TestTranscriptStreamFromUnknownSessionIsNotFound, and tightened the first concurrent-race test's assertion, which had a dead invariant arm (the seam runs synchronously, so the raced message is deterministically never in the snapshot — not a 50/50). Documented in server/openapi.yaml that a resuming client must pass GET /event's `session` filter alongside stream_from (a session with few messages relative to the instance's overall activity can report a stream_from far below the instance-wide journal's current position — correct for that session, but a footgun without the session filter), and fixed a doc comment naming a file that does not exist in this repo (the design note lives in meetneptune/boxes, not here). Verified: gofmt -l . clean; go vet ./... clean; go build ./... clean; go test -race ./... green across every package, server suite included (22.0s), with all seven transcript_sync_test.go cases passing and the existing message/message-page/compact suites unchanged. * server: never journal a synthetic orphan-repair message message.ResolveOrphanToolCalls' load-time repair (engine/store.go's LoadSession) folds a synthetic, is_error tool_result into a cold- loaded session's in-memory history for a tool_call with no matching result — purely to keep a rebuilt REQUEST protocol-valid. It exists only in that process's memory and is never itself persisted to the session's own log (message.IsSyntheticOrphanID's doc comment). The before_seq/limit page already treats this as a hard rule: durableOnly (handlers.go) drops these before paging, "a page must never give one a seq, whichever path produced the page" — the repair is re-derived fresh on every load, so nothing backs a durable "seen" mark for it across a restart. transcriptSyncedThrough's journaling loop (copied from syncMessages, which has this same latent gap already, just reachable only through a resident session) had no such guard: it would give one of these repairs a real seq the instant a cold session carrying an orphaned tool_call answered a stream_from request. Skip message.IsSyntheticOrphanID entries in the loop; the returned `messages` still includes the repair unchanged (this endpoint mirrors the unparameterized bare-array shape, which already includes it, unlike the stricter page). Added TestTranscriptStreamFrom_SyntheticOrphanRepairNeverJournaled, reusing engine/compact_test.go's NEP-5292 orphan fixture shape, and red-verified it against the unfixed loop. The fixture file is written AFTER the harness boots, not before: writing it first would let boot-time reconcile() — a separate loop with this identical characteristic, out of scope for this change — journal the repair before transcriptSyncedThrough ever ran, making the test exercise reconcile()'s behavior instead of the code it exists to cover. Verified: gofmt -l . clean; go vet ./... clean; go build ./... clean; go test -race ./... green across every package, server suite included (22.3s). * fix(server): cap transcript watermark on the summary event itself A compaction journals its summary and its history.compacted record in two separate Publish calls, each its own s.mu section (engine/compact.go emits the summary as an evtMessage first, then EventHistoryCompacted after releasing the engine lock). A bootstrap read (transcriptSyncedThrough) can acquire s.mu in the gap between the two: it sees the summary's evtMessage — so a stale-history message journaled in that same gap can push `highest` past the summary's seq — but not yet the evtHistoryCompacted record, so transcriptWatermarkLocked's pendingCeilings cap never engages. stream_from then lands above the summary's seq while the summary itself is absent from the returned history: the client never receives it on SSE resume (seq < from) but does receive the later history.compacted record that references it, a permanently dangling reference. Cap the watermark on the summary's own evtMessage record instead of waiting for its paired evtHistoryCompacted record. A summary excluded from history is identifiable on its own, from engine.IsCompactionSummaryID against the message ID already in hand — no second event required. Export isCompactionSummaryID (engine/compact.go) as engine.IsCompactionSummaryID, keeping the unexported predicate as the single source of truth. In transcriptWatermarkLocked's first pass, when a message is absent from history and its ID passes IsCompactionSummaryID, fold its seq into a new summaryCeiling alongside the existing pendingCeilings mechanism (still needed for a summary loaded from store whose evtHistoryCompacted record has no matching in-memory evtMessage); the final cap is the lower of whichever ceiling fires. The plain-append and no-compaction paths are unchanged: summaryCeiling stays -1 and the function returns `highest` exactly as before, paying only a cheap prefix check on messages already being skipped for being absent from history. Added TestTranscriptWatermarkLocked_CompactionSummarySandwich (server/transcript_sync_test.go), which builds the sandwich journal state directly via emitDurableLocked — an evtMessage summary absent from history, a later in-history message, and deliberately no evtHistoryCompacted record — since compaction's two Publish calls have no seam to stall between them yet. Red- verified: on the prior code it failed with "transcriptWatermarkLocked = 3, want 1 (summary cmpsum_test's own seq 2 minus one)"; it passes after this change. go build ./..., gofmt -l ., go vet ./..., and go test -race ./server/... ./engine/... are all clean; go test -race ./... has one pre-existing, unrelated failure (TestAppendSystemPromptReachesModelOnServe, a macOS tempdir symlink mismatch in e2e). --- engine/compact.go | 9 + server/handlers.go | 40 ++- server/journal.go | 219 ++++++++++++++ server/openapi.yaml | 77 ++++- server/server.go | 12 + server/transcript_sync_test.go | 521 +++++++++++++++++++++++++++++++++ 6 files changed, 869 insertions(+), 9 deletions(-) create mode 100644 server/transcript_sync_test.go diff --git a/engine/compact.go b/engine/compact.go index 3d873abd..45137e4e 100644 --- a/engine/compact.go +++ b/engine/compact.go @@ -103,6 +103,15 @@ func isCompactionSummaryID(id string) bool { return strings.HasPrefix(id, compactionSummaryIDTag+"_") } +// IsCompactionSummaryID is isCompactionSummaryID exported for callers +// outside this package (server/journal.go's transcriptWatermarkLocked caps +// the SSE resume watermark below a compaction summary's own evtMessage +// record — see that function's doc comment). Logic lives in +// isCompactionSummaryID; this is a thin wrapper, not a second copy. +func IsCompactionSummaryID(id string) bool { + return isCompactionSummaryID(id) +} + // compactionSystemPrompt is the dedicated system prompt for the tool-less // summarization call (see Session.Compact): concise, information-preserving, // never tool-call minutiae verbatim. diff --git a/server/handlers.go b/server/handlers.go index b5d31722..e814cb67 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -1201,10 +1201,31 @@ func (s *Server) handleMessages(w http.ResponseWriter, r *http.Request) { return } query := r.URL.Query() - if query.Has("before_seq") || query.Has("limit") { + paged := query.Has("before_seq") || query.Has("limit") + if paged && query.Has("stream_from") { + // Two different intentions named on one request: intParam (below) + // already rejects this shape for a repeated before_seq/limit value + // for the identical reason — answering one silently hides that the + // caller has a bug, rather than telling it. + writeErr(w, http.StatusBadRequest, "stream_from cannot be combined with before_seq or limit") + return + } + if paged { s.handleMessagePage(w, query, id) return } + if query.Has("stream_from") { + msgs, seq, ok := s.transcriptSyncedThrough(id) + if !ok { + writeErr(w, http.StatusNotFound, "no such session") + return + } + writeJSON(w, http.StatusOK, transcriptJSON{ + Messages: marshalMessages(msgs), + StreamFrom: seq, + }) + return + } sess, ok := s.lookupSession(id) if !ok { writeErr(w, http.StatusNotFound, "no such session") @@ -1214,6 +1235,23 @@ func (s *Server) handleMessages(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, marshalMessages(msgs)) } +// transcriptJSON is the ?stream_from=1 envelope: the session's whole message +// history PLUS the durable event-journal seq it is synced through (see +// transcriptSyncedThrough), so a client can open GET /event?from= +// immediately after this snapshot with no REPLAY window that can re-deliver +// or drop a message straddling the two reads — the "race-closed bootstrap" +// that closes the tail-load-versus-live-stream race behind the console's +// duplicate-render bug. +// +// A THIRD opt-in shape, alongside the bare-array default (no query +// parameter) and the before_seq/limit messagePageJSON page: a caller that +// never names stream_from keeps getting exactly what it always got, byte +// for byte. +type transcriptJSON struct { + Messages []json.RawMessage `json:"messages"` + StreamFrom int64 `json:"stream_from"` +} + // marshalMessages renders messages for the wire, one at a time, replacing // any that fails to marshal with a messagePlaceholder — see handleMessages' // own doc comment for the production incident that rule exists for. It diff --git a/server/journal.go b/server/journal.go index debcf0fb..147eb50a 100644 --- a/server/journal.go +++ b/server/journal.go @@ -818,6 +818,225 @@ func (s *Server) syncMessages(sessionID string) { } } +// transcriptSyncedThrough answers the "race-closed bootstrap" a client needs +// to tail-load a session's transcript and then resume GET /event strictly +// after it, with no REPLAY window that can re-deliver (or, symmetrically, +// permanently drop) a message straddling the two reads — the tail-load +// versus live-stream race the console's duplicate-render bug traces to (see +// the meetneptune/boxes repo's docs/console-read-path.md, and +// handleMessages' ?stream_from=1 branch, this function's only caller). +// +// It is syncMessages (above) PLUS one extra locked read: sess.History() and +// sess.PersistErr() are read in the exact same unlocked window, under the +// exact same lock-ordering invariant, that syncMessages documents — do not +// invert it. Every message in that snapshot not yet journaled is then +// journaled under one s.mu hold, exactly as syncMessages does, and the +// caller-visible watermark is sampled in that SAME critical section, +// immediately after the journaling loop: nothing can be appended to this +// session's journal between "the snapshot is fully durable" and "the +// watermark was read," because emitDurableLocked never runs without s.mu. +// +// It returns the SAME history slice it just journaled — never a second +// sess.History() call, and never syncMessages followed by a re-read — both +// of which would reopen an identical race one level down instead of closing +// it: this function's whole point is that the returned history and the +// returned seq describe the exact same instant. +// +// lookupSession, not liveSessionObject: a cold (on-disk, non-resident) +// session must answer this exactly like handleMessages' unparameterized read +// already does. This is a GET, not a journaling trigger tied to a live turn, +// so a caller must never be told "no such session" just because nothing in +// this process happens to be driving it right now. +func (s *Server) transcriptSyncedThrough(id string) (history []message.Message, seq int64, ok bool) { + sess, ok := s.lookupSession(id) + if !ok { + return nil, 0, false + } + history = sess.History() + persistErr := sess.PersistErr() + + if s.transcriptSyncRace != nil { + // Test-only seam: let a test force a concurrent Publish(EventMessage) + // call to land deterministically in the (now safe) gap between the + // unlocked reads above and the s.mu hold below. Always nil in + // production. + s.transcriptSyncRace() + } + + s.mu.Lock() + for i := range history { + m := history[i] + // A message.IsSyntheticOrphanID entry (message.ResolveOrphanToolCalls' + // load-time repair, folded into sess.History() for a cold-loaded + // session — see engine.LoadSession) exists only to keep a REQUEST + // protocol-valid; it is never itself persisted to the session's own + // log, so it has no durable identity to journal against. Giving it a + // seq would violate the same rule durableOnly (handlers.go) already + // enforces for the before_seq/limit page — "a page must never give + // one a seq, whichever path produced the page" — and, worse, is not + // even idempotent the way a real message's journaling is: nothing + // backs its "seen" mark across a restart, since it is re-derived + // fresh on every load rather than replayed from events.jsonl. + if message.IsSyntheticOrphanID(m.ID) { + continue + } + if s.isSeenLocked(id, m.ID) { + continue + } + s.markSeenLocked(id, m.ID) + s.emitDurableLocked(&Event{Type: evtMessage, SessionID: id, Message: &m}) + } + reportErr := s.checkPersistErrLocked(id, persistErr) + seq = s.transcriptWatermarkLocked(id, history) + s.mu.Unlock() + + if reportErr != nil { + s.reportError(reportErr) + } + return history, seq, true +} + +// transcriptWatermarkLocked returns the durable seq up to which sessionID's +// message transcript is FULLY represented by history: the highest seq among +// this session's journaled evtMessage records whose message ID appears in +// history — capped below any compaction summary history excludes (see the +// "compaction can reorder" section below). Returns 0 when history has +// nothing journaled yet (a brand-new or still cold session): the safe +// answer, not a gap — see the doc comment on transcriptSyncedThrough's only +// caller, handleMessages, for why 0 can never itself straddle a message. +// +// This is deliberately NOT sessionSeqLocked(sessionID) — the plain highest +// durable seq recorded for the session, of ANY event type. sess.History() +// and sess.PersistErr() in transcriptSyncedThrough above are read OUTSIDE +// s.mu (the same lock-ordering invariant syncMessages documents), so a +// concurrent syncMessages call for the SAME session — racing in that +// unlocked window with a FRESHER sess.History() snapshot that already +// includes a message this call's own (now stale) snapshot does not — can win +// the race for s.mu and durably journal that message before this call ever +// acquires it. sessionSeqLocked's raw, type-agnostic max would then already +// count that message's seq, even though the message is absent from the +// `history` this call is about to return: exactly the gap a client resuming +// GET /event?from= would never recover from, since a message with +// seq <= the reported watermark is never replayed (sse.go: `ev.Seq > from`). +// See TestTranscriptStreamFrom_ConcurrentJournalDuringSnapshot, which forces +// this exact interleaving via the transcriptSyncRace seam. +// +// Restricting the max to message IDs actually present in history closes +// that gap FOR A PLAIN APPEND: every message in history was appended to the +// session no later than this call's sess.History() snapshot, so any message +// NOT in history was necessarily appended strictly after it, and every +// syncMessages-family loop (this one included) journals a session's +// messages in history ARRAY order, under one s.mu hold — so as long as +// history only ever grows at the tail, a later-appended message can never +// be assigned a lower seq than an earlier one, whichever call performs the +// journaling. +// +// Compaction (engine/compact.go's Session.Compact) breaks that "only grows +// at the tail" assumption: it SPLICES a new summary message into an EARLIER +// array position, replacing the folded range, then journals the resulting +// history in array order — so the summary can receive a LOWER seq than a +// message that already sat later in this call's own (pre-compaction) stale +// snapshot, even though the summary was created (compacted) after that +// snapshot was taken. A summary excluded from history purely by this +// reordering must still end up with seq > the returned watermark, or its +// paired history.compacted reconciliation record (see publishHistoryCompacted) +// would sit at seq <= watermark while absent from history — permanently +// unrecoverable via SSE resume, unlike an ordinary excluded message, which +// self-heals by arriving live. So: for every evtHistoryCompacted record for +// this session whose CompactSummaryID is NOT in history, the watermark is +// capped to strictly below that summary's own journaled seq (looked up by +// ID) — even if that lowers it below some in-history message's already- +// higher seq. The tradeoff is deliberate and asymmetric: a message in +// history dropping below the cap is merely redelivered live once more (the +// ordinary duplicate this endpoint exists to reduce, still bounded and +// self-correcting by message ID), never lost — whereas letting the summary +// slip above the watermark is unrecoverable. See +// TestTranscriptStreamFrom_CompactionDuringSnapshotStaysRecoverable. +// +// # The two-emit sandwich window +// +// A compaction does not journal its summary and its reconciliation record +// atomically: engine/compact.go emits the summary as an ordinary evtMessage +// first, then EventHistoryCompacted separately, and server-side each goes +// through its own Publish call — its own separate s.mu critical section. +// A bootstrap read can acquire s.mu in the gap between the two: it observes +// the summary's evtMessage (so any stale-history message journaled after it +// in that same gap can raise `highest` past the summary's seq) but not yet +// the evtHistoryCompacted record, so pendingCeilings above is empty and the +// paragraph's cap never engages — stream_from would land above the summary +// while the summary is itself excluded from history, exactly the +// unrecoverable gap this function exists to prevent. Closing this requires +// no second event: a summary excluded from history is identifiable from its +// OWN evtMessage record via engine.IsCompactionSummaryID, independent of +// whether its evtHistoryCompacted record has arrived yet. summaryCeiling +// below captures that directly, in the same first pass, from the message's +// own seq — and is folded into the final cap alongside pendingCeilings, the +// lower of the two winning. The two mechanisms stay complementary rather +// than redundant: pendingCeilings still covers a summary loaded from store +// (its evtHistoryCompacted record present without a matching in-memory +// evtMessage), and summaryCeiling covers the newly-widened sandwich window +// above. See TestTranscriptWatermarkLocked_CompactionSummarySandwich. +// Caller holds s.mu. +func (s *Server) transcriptWatermarkLocked(sessionID string, history []message.Message) int64 { + inHistory := make(map[string]bool, len(history)) + for i := range history { + inHistory[history[i].ID] = true + } + + var highest int64 + // pendingCeilings collects, for every evtHistoryCompacted record whose + // summary is absent from history, that summary's OWN message ID — its + // seq is looked up in a second pass below, once highest is known, so the + // common (no concurrent compaction) case never pays for the lookup. + var pendingCeilings []string + // summaryCeiling caps the watermark directly from a compaction summary's + // own evtMessage record, without waiting for its evtHistoryCompacted + // record — see the "two-emit sandwich window" section above. -1 means no + // absent summary evtMessage was seen. + summaryCeiling := int64(-1) + for _, ev := range s.journal { + if ev.SessionID != sessionID { + continue + } + switch ev.Type { + case evtMessage: + if ev.Message == nil { + continue + } + if !inHistory[ev.Message.ID] { + if engine.IsCompactionSummaryID(ev.Message.ID) && (summaryCeiling == -1 || ev.Seq < summaryCeiling) { + summaryCeiling = ev.Seq + } + continue + } + if ev.Seq > highest { + highest = ev.Seq + } + case evtHistoryCompacted: + if !inHistory[ev.CompactSummaryID] { + pendingCeilings = append(pendingCeilings, ev.CompactSummaryID) + } + } + } + ceiling := summaryCeiling + if len(pendingCeilings) > 0 { + for _, ev := range s.journal { + if ev.Type != evtMessage || ev.SessionID != sessionID || ev.Message == nil { + continue + } + for _, id := range pendingCeilings { + if ev.Message.ID == id && (ceiling == -1 || ev.Seq < ceiling) { + ceiling = ev.Seq + } + } + } + } + if ceiling != -1 && ceiling-1 < highest { + return ceiling - 1 + } + return highest +} + // emitDurable assigns the next sequence number, journals the event, and fans // it out to connected clients. // diff --git a/server/openapi.yaml b/server/openapi.yaml index 1eb0d24a..7cd85c4f 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -737,6 +737,48 @@ components: type: boolean description: Whether at least one message older than `first_seq` exists. + Transcript: + type: object + description: > + The session's whole message history PLUS the durable event-journal + seq it is synced through — the response shape of GET + /session/{id}/message when the request names `stream_from`. + + A client uses this to bootstrap a session view without a race: it + renders `messages`, then opens GET /event?from=&session= + (or sends `stream_from` as the `Last-Event-ID`, alongside `session`) + to resume the live stream with no window in which a message can be + delivered twice (already in `messages`, then replayed live) or + dropped (journaled between the two reads, seen by neither). See + transcriptSyncedThrough's doc comment (server/journal.go) for why + this requires one extra locked read beyond the plain message sync. + + ALWAYS pass GET /event's `session` filter alongside `stream_from`. + `stream_from` is scoped to messages, so for a session with no + messages yet (or few relative to the instance's overall activity) + it can be far below the instance-wide event journal's current + position — correct for THIS session (nothing to duplicate or drop), + but an unfiltered GET /event?from= would then replay + every OTHER session's events with a higher seq too. + required: [messages, stream_from] + properties: + messages: + type: array + description: The whole history, oldest first. + items: + oneOf: + - $ref: "#/components/schemas/Message" + - $ref: "#/components/schemas/MessagePlaceholder" + stream_from: + type: integer + description: > + The durable event-journal seq `messages` is fully synced + through. Every message above is journaled with a seq no greater + than this value, and any message journaled after this call + returned has a seq strictly greater than it — pass it as + GET /event's `from` (or `Last-Event-ID`) to resume without a + gap or a duplicate. + Part: type: object required: [type] @@ -1980,14 +2022,19 @@ paths: /session/{id}/message: get: operationId: getMessages - summary: Canonical message history — whole, or one bounded page. + summary: Canonical message history — whole, one bounded page, or a race-closed bootstrap. description: > - Two response shapes, chosen by the request. WITHOUT `before_seq` and - `limit` the response is the bare array of the whole history it has - always been, unchanged for every existing caller. WITH either - parameter the response is a MessagePage envelope: one bounded page - of the session's durable message sequence, newest page by default, - read from the journal's tail rather than its whole length. + Three response shapes, chosen by the request. WITHOUT `before_seq`, + `limit`, or `stream_from` the response is the bare array of the + whole history it has always been, unchanged for every existing + caller. WITH `before_seq` or `limit` the response is a MessagePage + envelope: one bounded page of the session's durable message + sequence, newest page by default, read from the journal's tail + rather than its whole length. WITH `stream_from` the response is a + Transcript envelope: the whole history plus the durable + event-journal seq it is synced through, so a client can resume + GET /event strictly after it with no window that re-delivers or + drops a message. Either way, each message is marshaled independently: a single message that fails to marshal (e.g. a Reasoning part carrying an @@ -2022,6 +2069,18 @@ paths: above the maximum is REJECTED with 400, not clamped, so a generated client that enforces the schema and this server agree on what a request means. + - name: stream_from + in: query + required: false + schema: { type: integer } + description: > + Any value requests the Transcript envelope instead of the bare + array. The value itself is ignored — the response's own + `stream_from` is always freshly computed for this call, never + echoed back. Mutually exclusive with `before_seq`/`limit`: + naming `stream_from` alongside either is REJECTED with 400, the + same "two intentions, pick one" rule `before_seq`/`limit` + already enforce against each other. responses: "200": description: OK @@ -2031,7 +2090,8 @@ paths: oneOf: - type: array description: > - The whole history, for a request that names no page. + The whole history, for a request that names no page + and no stream_from. items: description: > A Message, or a MessagePlaceholder in place of any @@ -2040,6 +2100,7 @@ paths: - $ref: "#/components/schemas/Message" - $ref: "#/components/schemas/MessagePlaceholder" - $ref: "#/components/schemas/MessagePage" + - $ref: "#/components/schemas/Transcript" "400": { $ref: "#/components/responses/BadRequest" } "401": { $ref: "#/components/responses/Unauthorized" } "404": diff --git a/server/server.go b/server/server.go index 320ef1c9..5d0df7c3 100644 --- a/server/server.go +++ b/server/server.go @@ -619,6 +619,18 @@ type Server struct { // Always nil in production. sseRegisteredRace func() + // transcriptSyncRace is a test-only seam: when non-nil, + // transcriptSyncedThrough (journal.go) invokes it right after reading + // sess.History()/sess.PersistErr() but before acquiring s.mu — letting a + // test force a concurrent Publish(EventMessage) call to land + // deterministically in that exact gap and journal a message this call's + // own (now stale) history snapshot never saw, proving the returned + // (history, seq) pair still honors the gap-safety invariant instead of + // reporting a watermark past a message the snapshot omits (see + // TestTranscriptStreamFrom_ConcurrentJournalDuringSnapshot). Always nil + // in production. + transcriptSyncRace func() + // worktreeBase is the directory 'worktree'-isolation sessions create // their per-session git worktrees under (see worktree.go): / // worktrees when SessionDir is durable, otherwise a process-lifetime diff --git a/server/transcript_sync_test.go b/server/transcript_sync_test.go new file mode 100644 index 00000000..b0ae72aa --- /dev/null +++ b/server/transcript_sync_test.go @@ -0,0 +1,521 @@ +package server + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/majorcontext/harness/engine" + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// transcriptResponse mirrors transcriptJSON for a test that decodes it as a +// client would, without reaching into the server's own type for anything +// but the field names. +type transcriptResponse struct { + Messages []message.Message `json:"messages"` + StreamFrom int64 `json:"stream_from"` +} + +// getTranscript issues GET /session/{id}/message?stream_from=1 and decodes +// the transcriptJSON envelope. +func getTranscript(t *testing.T, h *harness, id string) (transcriptResponse, *responseMeta) { + t.Helper() + resp, data := h.do("GET", "/session/"+id+"/message?stream_from=1", nil) + meta := &responseMeta{status: resp.StatusCode, body: data} + if resp.StatusCode != 200 { + return transcriptResponse{}, meta + } + var got transcriptResponse + if err := json.Unmarshal(data, &got); err != nil { + t.Fatalf("decode transcript: %v (%s)", err, data) + } + return got, meta +} + +type responseMeta struct { + status int + body []byte +} + +// journalSeqByMessageID snapshots every evtMessage record's seq for +// sessionID, keyed by message ID, directly from the server's own durable +// journal. Caller must not hold s.mu. +func journalSeqByMessageID(s *Server, sessionID string) map[string]int64 { + s.mu.Lock() + defer s.mu.Unlock() + out := make(map[string]int64) + for _, ev := range s.journal { + if ev.Type == evtMessage && ev.SessionID == sessionID && ev.Message != nil { + out[ev.Message.ID] = ev.Seq + } + } + return out +} + +// TestTranscriptStreamFrom_ConsistentWithSnapshot is the endpoint's core +// promise: every message the snapshot returns is durably journaled with a +// seq no greater than the returned stream_from, and the journaling happens +// IN THIS REQUEST for a session nothing in this process had synced yet +// (mirrors the "spawned child never touched again" shape syncMessages' own +// doc comment describes) — proving transcriptSyncedThrough, not some +// earlier boot-time reconcile pass, is what produced these seqs. +func TestTranscriptStreamFrom_ConsistentWithSnapshot(t *testing.T) { + dir := t.TempDir() + // Build the server FIRST, against an empty dir, so its boot-time + // reconcile() finds nothing. The session below is written to the same + // dir only afterward — this process has never journaled a single byte + // of it before the GET below. + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + sess := coldMessages(t, dir, 3) // 6 messages + + h.srv.mu.Lock() + preSeq := h.srv.sessionSeqLocked(sess.ID) + h.srv.mu.Unlock() + if preSeq != 0 { + t.Fatalf("preSeq = %d, want 0 (nothing journaled for this session yet)", preSeq) + } + + got, meta := getTranscript(t, h, sess.ID) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + if len(got.Messages) != 6 { + t.Fatalf("got %d messages, want 6", len(got.Messages)) + } + + seqs := journalSeqByMessageID(h.srv, sess.ID) + for _, m := range got.Messages { + seq, ok := seqs[m.ID] + if !ok { + t.Errorf("message %s: not found in s.journal", m.ID) + continue + } + if seq > got.StreamFrom { + t.Errorf("message %s: journaled seq %d > stream_from %d", m.ID, seq, got.StreamFrom) + } + if seq <= preSeq { + t.Errorf("message %s: journaled seq %d <= pre-call watermark %d, want strictly greater (proves in-request journaling)", m.ID, seq, preSeq) + } + } + if got.StreamFrom <= preSeq { + t.Errorf("stream_from = %d, want strictly greater than pre-call watermark %d", got.StreamFrom, preSeq) + } +} + +// TestTranscriptStreamFrom_LaterMessageHasSeqAboveWatermark: a message +// journaled AFTER the snapshot call returns must have a seq strictly +// greater than stream_from — the property that makes GET +// /event?from= a safe, gap-free resume point. +func TestTranscriptStreamFrom_LaterMessageHasSeqAboveWatermark(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("first"), asstTurn("second")}} + h := newHarness(t, prov) + id := h.createSession("") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + got, meta := getTranscript(t, h, id) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + + sess := h.srv.residentSession(id) + if sess == nil { + t.Fatal("session not resident") + } + if _, err := sess.Prompt(context.Background(), "second"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + h.srv.mu.Lock() + newSeq := h.srv.sessionSeqLocked(id) + h.srv.mu.Unlock() + if newSeq <= got.StreamFrom { + t.Errorf("seq after later message = %d, want strictly greater than stream_from %d", newSeq, got.StreamFrom) + } +} + +// TestTranscriptStreamFrom_ConcurrentJournalDuringSnapshot forces a message +// to be fully journaled by a CONCURRENT Publish(EventMessage) call landing +// in the gap between transcriptSyncedThrough's unlocked sess.History() read +// and its s.mu.Lock() — via the transcriptSyncRace seam, the same pattern +// TestHandleEventDeliversEventPublishedBeforeHeadersFlush uses for +// handleEvent's own registration gap. +// +// The racing message is appended (and journaled, via a real second Prompt +// call) strictly AFTER this call's own sess.History() snapshot was taken, +// so it can never appear in the returned history — the gap-safety invariant +// requires it to land on the "NEITHER" side: absent from history AND its +// seq strictly greater than the returned stream_from. A naive watermark +// (the plain highest seq journaled anywhere for the session, unconditional +// on message identity) would instead already count the raced message's +// seq — this is the exact case transcriptWatermarkLocked's restriction to +// message IDs present in history exists to rule out. +func TestTranscriptStreamFrom_ConcurrentJournalDuringSnapshot(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("first"), asstTurn("raced")}} + h := newHarness(t, prov) + id := h.createSession("") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + var racedID string + h.srv.transcriptSyncRace = func() { + sess := h.srv.residentSession(id) + if sess == nil { + t.Error("transcriptSyncRace: session not resident") + return + } + asst, err := sess.Prompt(context.Background(), "trigger raced turn") + if err != nil { + t.Errorf("transcriptSyncRace: Prompt: %v", err) + return + } + racedID = asst.ID + } + t.Cleanup(func() { h.srv.transcriptSyncRace = nil }) + + got, meta := getTranscript(t, h, id) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + if racedID == "" { + t.Fatal("transcriptSyncRace never ran") + } + + inHistory := false + for _, m := range got.Messages { + if m.ID == racedID { + inHistory = true + } + } + // Deterministic, not a coin flip: the seam runs synchronously, in this + // same goroutine, strictly after transcriptSyncedThrough's own + // sess.History() call returned (that call already produced the + // `history` slice this request will return) and strictly before its + // s.mu.Lock() — so the raced message can never be in the snapshot this + // specific request answers with. + if inHistory { + t.Fatalf("raced message %s unexpectedly present in the snapshot — the seam did not run in the documented gap", racedID) + } + + seqs := journalSeqByMessageID(h.srv, id) + racedSeq, journaled := seqs[racedID] + if !journaled { + t.Fatalf("raced message %s was never journaled", racedID) + } + if racedSeq <= got.StreamFrom { + t.Errorf("gap-safety invariant violated: raced message absent from snapshot yet seq %d <= stream_from %d", racedSeq, got.StreamFrom) + } +} + +// TestTranscriptStreamFrom_CompactionDuringSnapshotStaysRecoverable is the +// regression test for the sharper version of the race above: +// engine/compact.go's Session.Compact does not merely APPEND to history, it +// SPLICES a new summary message into an EARLIER array position (replacing +// the folded range) and then journals the result in array order — so the +// summary can receive a LOWER seq than an already-later message this call's +// own stale snapshot already contains, even though the summary was created +// after that snapshot was taken. transcriptWatermarkLocked's plain +// "restrict to message IDs in history" rule alone is not enough here: it +// would let the summary's lower seq slip below the watermark while the +// summary sits outside history, and unlike an ordinary excluded message +// (which self-heals by arriving live later), a summary in that state is +// permanently unrecoverable — its paired history.compacted record would +// never redeliver either, since the client's SSE resume point already sits +// at or past it. +// +// This builds a session directly (bypassing the harness's own Publish +// wiring, like coldMessages) so the server has never synced a byte of it — +// exactly TestTranscriptStreamFrom_ConsistentWithSnapshot's setup — then +// races a REAL POST /session/{id}/compact into the transcriptSyncRace gap. +// Compaction runs against its own freshly-loaded *engine.Session (the cold +// path loads a new one via claimForPrompt), so it never touches this call's +// own already-captured `history`; the two only ever meet through the +// server's shared journal and s.mu. +func TestTranscriptStreamFrom_CompactionDuringSnapshotStaysRecoverable(t *testing.T) { + dir := t.TempDir() + // The harness's OWN provider only needs the compaction summarization + // call queued — the three turns below are written directly to disk by + // a throwaway session/provider pair, exactly like coldMessages, so the + // harness never sees them until the race below touches this session. + harnessProv := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactAsstTurn("SUMMARY", provider.Usage{InputTokens: 5}), + }} + h := newHarnessDir(t, dir, harnessProv) + + seedProv := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactAsstTurn("one", provider.Usage{InputTokens: 10}), + compactAsstTurn("two", provider.Usage{InputTokens: 20}), + compactAsstTurn("three", provider.Usage{InputTokens: 30}), + }} + seed := engine.NewSession(engine.Config{ + Providers: provider.Registry{seedProv.name: seedProv}, + Model: message.ModelRef{Provider: seedProv.name, Model: "m1"}, + SessionDir: dir, + WorkDir: dir, + }) + for i, text := range []string{"go1", "go2", "go3"} { + if _, err := seed.Prompt(context.Background(), text); err != nil { + t.Fatalf("seed Prompt %d: %v", i, err) + } + } + if err := seed.PersistErr(); err != nil { + t.Fatalf("seed PersistErr: %v", err) + } + + h.srv.transcriptSyncRace = func() { + resp, data := h.do("POST", "/session/"+seed.ID+"/compact", map[string]any{"keep_turns": 1}) + if resp.StatusCode != 200 { + t.Errorf("transcriptSyncRace: compact = %d: %s", resp.StatusCode, data) + } + } + t.Cleanup(func() { h.srv.transcriptSyncRace = nil }) + + got, meta := getTranscript(t, h, seed.ID) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + + // Find the summary: the one journaled evtHistoryCompacted record for + // this session names it. + h.srv.mu.Lock() + var summaryID string + for _, ev := range h.srv.journal { + if ev.Type == evtHistoryCompacted && ev.SessionID == seed.ID { + summaryID = ev.CompactSummaryID + } + } + h.srv.mu.Unlock() + if summaryID == "" { + t.Fatal("transcriptSyncRace never journaled a history.compacted record") + } + + inHistory := false + for _, m := range got.Messages { + if m.ID == summaryID { + inHistory = true + } + } + + seqs := journalSeqByMessageID(h.srv, seed.ID) + summarySeq, journaled := seqs[summaryID] + if !journaled { + t.Fatalf("summary %s was never journaled", summaryID) + } + + // The gap-safety invariant, exactly as the plain-message race test + // checks it: the summary must be in BOTH history and seq <= + // stream_from, or in NEITHER — never seq <= stream_from while absent. + switch { + case inHistory && summarySeq <= got.StreamFrom: + case !inHistory && summarySeq > got.StreamFrom: + default: + t.Errorf("gap-safety invariant violated for compaction summary: in history=%v, seq=%d, stream_from=%d", + inHistory, summarySeq, got.StreamFrom) + } +} + +// TestTranscriptWatermarkLocked_CompactionSummarySandwich is the regression +// test for the narrower race TestTranscriptStreamFrom_CompactionDuringSnapshotStaysRecoverable +// above cannot reach: that test's seam runs the whole POST /compact +// synchronously, so by the time the racing GET observes anything, BOTH the +// summary's evtMessage record and its paired evtHistoryCompacted record are +// already journaled — pendingCeilings always has something to cap on. In +// production the two are journaled in separate Publish calls (separate s.mu +// sections; see engine/compact.go's Compact and transcriptWatermarkLocked's +// own doc comment), so a bootstrap read can land in the gap between them: +// summary evtMessage present, evtHistoryCompacted absent. There is no +// existing seam that stalls a real compaction between its two Publish +// calls, so this constructs that exact journal state directly — via the +// same emitDurableLocked production code path syncMessages itself uses, not +// hand-rolled Event literals — and calls transcriptWatermarkLocked +// directly, the narrowest production-faithful entry point for a state that +// has no HTTP-level trigger yet. +func TestTranscriptWatermarkLocked_CompactionSummarySandwich(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + const sessionID = "sess_sandwich" + + // An ordinary message already in history, journaled before the summary. + first := message.Message{ID: "msg_first", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "first"}}} + // The compaction summary: a real cmpsum_-prefixed ID (engine.IsCompactionSummaryID + // gates on exactly this prefix — see compactionSummaryIDTag), journaled as + // a plain evtMessage. It is EXCLUDED from history below (compaction + // spliced it in, replacing the folded range) and — this is the sandwich — + // its evtHistoryCompacted record has NOT been journaled yet. + summary := message.Message{ID: "cmpsum_test", Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: engine.CompactionSummaryBanner + "summary"}}} + // A stale-history message journaled AFTER the summary, in the gap — the + // exact shape that pushes `highest` past the summary's own seq when + // nothing caps it. + stale := message.Message{ID: "msg_stale", Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "stale"}}} + + firstEv := &Event{Type: evtMessage, SessionID: sessionID, Message: &first} + summaryEv := &Event{Type: evtMessage, SessionID: sessionID, Message: &summary} + staleEv := &Event{Type: evtMessage, SessionID: sessionID, Message: &stale} + + h.srv.mu.Lock() + h.srv.emitDurableLocked(firstEv) + h.srv.emitDurableLocked(summaryEv) + h.srv.emitDurableLocked(staleEv) + // Deliberately no evtHistoryCompacted record: this is the gap between + // compaction's two separate emits, before the second one lands. + got := h.srv.transcriptWatermarkLocked(sessionID, []message.Message{first, stale}) + h.srv.mu.Unlock() + + if staleEv.Seq <= summaryEv.Seq { + t.Fatalf("test setup invariant broken: stale seq %d must be > summary seq %d", staleEv.Seq, summaryEv.Seq) + } + + want := summaryEv.Seq - 1 + if got != want { + t.Errorf("transcriptWatermarkLocked = %d, want %d (summary %s's own seq %d minus one) — "+ + "a summary absent from history must cap the watermark even with no evtHistoryCompacted "+ + "record yet, or its future history.compacted record becomes an unrecoverable dangling reference", + got, want, summary.ID, summaryEv.Seq) + } +} + +// TestTranscriptStreamFrom_EmptyHistoryReportsZero pins the deliberate +// choice for a session with nothing journaled yet: stream_from is 0, not +// some higher "current instant" value (e.g. the server's global seq +// counter). A higher fallback would reopen exactly the race +// transcriptWatermarkLocked exists to close, for the session's own FIRST +// message: if a message were mid-race-journaled by someone else between +// this call's unlocked reads and its lock, a global-counter fallback would +// already count it even though it is (trivially, history is empty) absent +// from `messages`. 0 has nothing to protect and nothing to straddle: every +// message that will ever exist for this session gets seq >= 1 > 0. +func TestTranscriptStreamFrom_EmptyHistoryReportsZero(t *testing.T) { + // Drive unrelated activity first so the process-wide seq counter is + // already well above 0 — proving 0 is a deliberate per-session answer, + // not just "nothing has happened in this process yet." + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("noise")}} + h := newHarness(t, prov) + other := h.createSession("") + resp, data := h.do("POST", "/session/"+other+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "go"}}, + }) + if resp.StatusCode != 202 { + t.Fatalf("prompt_async = %d: %s", resp.StatusCode, data) + } + h.waitIdle(other) + + id := h.createSession("") + got, meta := getTranscript(t, h, id) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + if len(got.Messages) != 0 { + t.Fatalf("got %d messages, want 0", len(got.Messages)) + } + if got.StreamFrom != 0 { + t.Errorf("stream_from = %d, want 0 for an empty transcript", got.StreamFrom) + } +} + +// TestTranscriptStreamFrom_RejectsCombinationWithPaging: stream_from names +// a different response envelope than before_seq/limit. Answering one +// silently (handleMessages used to let before_seq/limit win, discarding +// stream_from) hides that the caller named two incompatible intentions — +// intParam enforces the identical rule against a repeated before_seq or +// limit value for the same reason. +func TestTranscriptStreamFrom_RejectsCombinationWithPaging(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("") + + for _, query := range []string{"?stream_from=1&before_seq=5", "?stream_from=1&limit=5"} { + resp, data := h.do("GET", "/session/"+id+"/message"+query, nil) + if resp.StatusCode != 400 { + t.Errorf("GET message%s = %d, want 400: %s", query, resp.StatusCode, data) + } + } +} + +// TestTranscriptStreamFrom_SyntheticOrphanRepairNeverJournaled: a +// message.ResolveOrphanToolCalls repair (LoadSession's load-time patch for +// an assistant tool_call with no matching tool_result anywhere a provider's +// wire protocol requires one — see message.IsSyntheticOrphanID) exists only +// in this process's in-memory history; it is never itself persisted to the +// session's own log (see engine/store.go's LoadSession). It must still +// appear in the returned `messages` — this endpoint mirrors the +// unparameterized bare-array shape, which already includes it, unlike the +// before_seq/limit page (handleMessagePage's own doc comment: a page +// "never adds the load-time repair," reading verbatim from the durable log +// instead) — but it must never receive a durable seq: durableOnly +// (handlers.go) already enforces "a page must never give one a seq, +// whichever path produced the page" for the identical reason, and nothing +// backs its "seen" mark across a restart, since LoadSession re-derives it +// fresh on every load rather than replaying it from events.jsonl. +// +// The fixture file is written to disk AFTER the harness boots (like +// TestTranscriptStreamFrom_ConsistentWithSnapshot), not before: writing it +// first would let boot-time reconcile() — a separate, PRE-EXISTING loop +// with this exact same characteristic, unrelated to this change — journal +// the repair before transcriptSyncedThrough ever runs, which would pass or +// fail this test on reconcile()'s behavior instead of the code this test +// exists to cover. +func TestTranscriptStreamFrom_SyntheticOrphanRepairNeverJournaled(t *testing.T) { + dir := t.TempDir() + h := newHarnessDir(t, dir, &scriptedProvider{name: "test"}) + + id := "ses_5292000000000099" + // msg_2 is an assistant tool_call with no following tool-role result — + // the exact orphan shape message.ResolveOrphanToolCalls repairs, mirrors + // engine/compact_test.go's nep5292FixtureLines. + fixture := `{"type":"session","id":"` + id + `","created_at":"2025-01-02T03:04:05Z"} +{"type":"message","message":{"id":"msg_1","role":"user","parts":[{"type":"text","text":"task 1"}]}} +{"type":"message","message":{"id":"msg_2","role":"assistant","parts":[{"type":"tool_call","call_id":"A","name":"bash","arguments":{}}]}} +{"type":"message","message":{"id":"msg_3","role":"user","parts":[{"type":"text","text":"task 2"}]}} +{"type":"message","message":{"id":"msg_4","role":"assistant","parts":[{"type":"text","text":"done"}]}} +` + if err := os.WriteFile(filepath.Join(dir, id+".jsonl"), []byte(fixture), 0o644); err != nil { + t.Fatal(err) + } + + got, meta := getTranscript(t, h, id) + if meta.status != 200 { + t.Fatalf("GET stream_from = %d: %s", meta.status, meta.body) + } + if len(got.Messages) != 5 { + t.Fatalf("got %d messages, want 5 (4 raw + 1 synthetic repair)", len(got.Messages)) + } + var orphanID string + for _, m := range got.Messages { + if message.IsSyntheticOrphanID(m.ID) { + orphanID = m.ID + } + } + if orphanID == "" { + t.Fatal("no synthetic orphan-repair message in the returned messages — fixture did not trigger the repair") + } + + seqs := journalSeqByMessageID(h.srv, id) + if _, journaled := seqs[orphanID]; journaled { + t.Errorf("synthetic orphan-repair message %s was journaled with a durable seq — it has no durable identity to journal against", orphanID) + } +} + +// TestTranscriptStreamFromUnknownSessionIsNotFound: an id with no journal +// and no live session is a 404, exactly like the bare-array and +// before_seq/limit branches already report it. +func TestTranscriptStreamFromUnknownSessionIsNotFound(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, _ := h.do("GET", "/session/ses_0123456789abcdef/message?stream_from=1", nil) + if resp.StatusCode != 404 { + t.Fatalf("GET unknown session stream_from = %d, want 404", resp.StatusCode) + } +} From 62705f4b2add92caba38784f6f52401a7a4d7fed Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 08:21:12 -0400 Subject: [PATCH 36/95] feat(engine): disallow native subagent spawning in claude-code lane (#225) The claude-code lane delegates a turn to the real `claude` CLI, which carries its own built-in same-family subagent tools (Agent, and Workflow's own subagent fan-out). Harness separately exposes a cross-family, non-blocking "task" tool to that same CLI over its MCP shim (#223), so a delegated session already has two different ways to spawn a subagent: the CLI's native same-family path and harness's own cross-family path. Leaving both reachable lets the model bypass harness's task tool, which the rest of the system relies on for lifecycle, journaling, and cross-family routing of spawned work. runClaudeCodeTurn now always appends "--disallowedTools", "Agent,Workflow" to the claude argv, alongside the existing unconditional --forward-subagent-text flag. Per the CLI's own --disallowedTools flag, this is a single comma-separated value, matching the syntax already used for --allowedTools. Agent and Workflow are the CLI's only two built-in tools whose description names spawning a subagent (confirmed against the current tools-reference docs, which also renamed the older "Task" tool to "Agent"); TaskCreate/Get/ List/Update/Output/Stop and TodoWrite manage an existing task list or agent rather than spawning one, so they are left alone. This is an argv-only change: it does not touch the shared engine "task" tool's description (other callers consume it) and adds no box-runtime hook. Verified by a new TestClaudeCodeDisallowsNativeSpawnTools test, which failed before this change (argv carried no --disallowedTools flag) and passes after. go build ./..., gofmt -l ., go vet ./..., and go test -race ./engine/... all pass. --- engine/claude_code_backend.go | 8 ++++++++ engine/claude_code_backend_test.go | 22 ++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index bd82b05a..e356df9c 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -406,6 +406,14 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // which this driver always sets, so it is safe to pass // unconditionally. "--forward-subagent-text", + // All subagent spawning in the claude-code lane must go through + // harness's own cross-family "task" tool (server/mcp_history.go, + // #223), never the CLI's native same-family equivalent — the two + // tools below are the CLI's only built-in ways to spawn a + // same-family subagent (Agent: single subagent/teammate; Workflow: + // a script that fans out many subagents), so both are disallowed + // unconditionally rather than left for the model to choose between. + "--disallowedTools", "Agent,Workflow", } if model.Model != "" { args = append(args, "--model", model.Model) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 2e70c626..b12e1993 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -946,6 +946,28 @@ func TestClaudeCodeForwardSubagentTextAlwaysSet(t *testing.T) { } } +// TestClaudeCodeDisallowsNativeSpawnTools proves runClaudeCodeTurn always +// sends --disallowedTools naming every native Claude Code tool that spawns +// a same-family subagent (Agent, Workflow). All subagent spawning in the +// claude-code lane must go through harness's own cross-family "task" tool +// (server/mcp_history.go, #223), never the CLI's native same-family +// equivalent, so those tools are blocked at the argv level rather than +// relying on the model to prefer one path over the other. +func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + got, ok := argvValueAfter(invocations[0], "--disallowedTools") + if !ok { + t.Fatalf("argv has no --disallowedTools: %v", invocations[0]) + } + if want := "Agent,Workflow"; got != want { + t.Errorf("--disallowedTools = %q, want %q", got, want) + } +} + // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning From 21d18ea5692ad588d119d17c76c2516ecec8ab0b Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 10:14:18 -0400 Subject: [PATCH 37/95] Revert "feat(engine): disallow native subagent spawning in claude-code lane (#225)" (#226) This reverts commit 62705f4b2add92caba38784f6f52401a7a4d7fed. --- engine/claude_code_backend.go | 8 -------- engine/claude_code_backend_test.go | 22 ---------------------- 2 files changed, 30 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index e356df9c..bd82b05a 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -406,14 +406,6 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // which this driver always sets, so it is safe to pass // unconditionally. "--forward-subagent-text", - // All subagent spawning in the claude-code lane must go through - // harness's own cross-family "task" tool (server/mcp_history.go, - // #223), never the CLI's native same-family equivalent — the two - // tools below are the CLI's only built-in ways to spawn a - // same-family subagent (Agent: single subagent/teammate; Workflow: - // a script that fans out many subagents), so both are disallowed - // unconditionally rather than left for the model to choose between. - "--disallowedTools", "Agent,Workflow", } if model.Model != "" { args = append(args, "--model", model.Model) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index b12e1993..2e70c626 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -946,28 +946,6 @@ func TestClaudeCodeForwardSubagentTextAlwaysSet(t *testing.T) { } } -// TestClaudeCodeDisallowsNativeSpawnTools proves runClaudeCodeTurn always -// sends --disallowedTools naming every native Claude Code tool that spawns -// a same-family subagent (Agent, Workflow). All subagent spawning in the -// claude-code lane must go through harness's own cross-family "task" tool -// (server/mcp_history.go, #223), never the CLI's native same-family -// equivalent, so those tools are blocked at the argv level rather than -// relying on the model to prefer one path over the other. -func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { - s, logPath := claudeCodeTestSession(t, "normal") - if _, err := s.Prompt(context.Background(), "hi"); err != nil { - t.Fatalf("Prompt: %v", err) - } - invocations := readInvocations(t, logPath) - got, ok := argvValueAfter(invocations[0], "--disallowedTools") - if !ok { - t.Fatalf("argv has no --disallowedTools: %v", invocations[0]) - } - if want := "Agent,Workflow"; got != want { - t.Errorf("--disallowedTools = %q, want %q", got, want) - } -} - // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning From b6bd8a858c221cbf5c54fa95621bb3efbae28b4c Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 10:38:38 -0400 Subject: [PATCH 38/95] fix(engine): deliver and commit task notifications on claude-code lane (#227) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: a non-blocking task subagent settling on a claude-code delegated parent produced a live infinite loop. The engine kept firing an "A background task you started has finished..." resume turn, and the model kept answering "No action taken" forever. Root cause: runAgenticLoop's claudeCodeDelegated branch calls runClaudeCodeTurn and returns, entirely bypassing the native loop body's task-notification handling (checkoutTaskNotificationsSegment on the way in, commitTaskNotifications/requeueTaskNotifications on the way out). runClaudeCodeTurn built the CLI's stdin purely from lastUserMessageText, so the child's actual result never reached the model — only the bare trigger string did, hence the honest "No action taken." Because checkoutTaskNotificationsSegment was never called, the notification was never committed either, so hasPendingTaskNotifications stayed true and finalizeTurn kept re-triggering the resume every turn. Design: mirror the native path's two-phase checkout/commit/requeue discipline in the delegated lane instead of reinventing it. runClaudeCodeTurn now calls checkoutTaskNotificationsSegment and, when it returns a non-empty segment, splices the rendered "[tasks: ...]" block into the CLI turn's plain-text input on its own blank-line-separated block (there is no EngineContext wire concept for the stream-json stdin protocol, so it cannot ride a trust-tagged part the way a native request does). The block is deliberately not wrapped in RenderEngineContext's sentinel tags: that sentinel's meaning is taught only by the native base system prompt (ambientContextGuidance, cmd/harness/main.go), which a delegated turn never sends, so the tags would reach Claude Code's own system prompt as meaningless literal text. runAgenticLoop's delegated branch now calls commitTaskNotifications on success and requeueTaskNotifications on failure, exactly like the native branch's own calls, so the checkout this same turn made either clears (genuinely delivered) or returns to pending for a later attempt. Semantic change: a delegated turn that runs while a task notification is pending now (a) actually delivers the notification's content to the model, and (b) commits or requeues it depending on turn outcome, closing the resume loop. No change to the native path or to any turn with no pending notification. Verification: added engine/claude_code_backend_test.go coverage for both the delivery+commit and requeue-on-failure paths, driven through the fake claude CLI (engine/testdata/fakeclaude), which now records the exact bytes it received on stdin via FAKE_CLAUDE_STDIN_LOG. Red-verified against the pre-fix code: the delivery test failed with the CLI stdin containing only the bare trigger string, and the requeue test failed because checkout never ran at all. Both pass after the fix. go build ./..., gofmt -l ., go vet ./..., and go test -race ./engine/... are clean; go test -race ./... has one known pre-existing, unrelated failure (TestAppendSystemPromptReachesModelOnServe, a macOS tempdir symlink mismatch) reproduced identically on unmodified origin/main. --- engine/claude_code_backend.go | 35 ++++++++++ engine/claude_code_backend_test.go | 102 +++++++++++++++++++++++++++++ engine/engine.go | 15 +++++ engine/testdata/fakeclaude/main.go | 28 +++++--- 4 files changed, 171 insertions(+), 9 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index bd82b05a..ab811e30 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -363,6 +363,41 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro if text == "" { return nil, errors.New("engine: claude-code delegated turn found no pending user message to answer") } + // Deliver any pending task notifications (a settled child's Done/Failed + // result) into THIS turn's input, mirroring the native loop's own + // checkout step (engine.go's streamTurn, via withAmbientStatus) — see + // checkoutTaskNotificationsSegment's doc comment for the checkout/ + // commit/requeue two-phase handoff this call is one leg of; the other + // two legs (commit on success, requeue on failure) live one layer up, + // in runAgenticLoop's claudeCodeDelegated branch, exactly like the + // native path's own commit/requeue calls. + // + // There is no EngineContext wire concept for the stream-json CLI + // stdin protocol this file drives (unlike a native provider request, + // which carries the segment as its own trust-tagged part — see + // withAmbientStatus), so the rendered segment is spliced directly into + // the plain text the CLI receives instead, on its own blank-line- + // separated block. Deliberately NOT wrapped in RenderEngineContext's + // sentinel: that sentinel's meaning is taught + // by the NATIVE base system prompt's ambientContextGuidance + // (cmd/harness/main.go), which a delegated turn never sends — Claude + // Code drives this turn under its own, unrelated system prompt (see + // this file's package doc, "`--append-system-prompt` remains NOT + // auto-populated from s.cfg.System"), so the tag would reach the model + // as meaningless literal text rather than a recognized trust marker. + // The rendered segment is still self-delimited (renderTaskNotifications' + // own "[tasks:\n- ...\n]" shape) and still defended the same way a + // native request's block is: renderTaskNotifications already runs + // every free-text field (a child's own untrusted Result, or a + // provider's FailReason) through neutralizeNotificationText, which + // strips newlines so a child cannot manufacture a fake sibling "- ..." + // entry that reads as a second, forged notification — that protection + // is applied before this splice and does not depend on the sentinel. + // A no-op (text unchanged) when nothing is pending, matching + // withAmbientStatus's own no-op-on-empty-segment behavior. + if seg := s.checkoutTaskNotificationsSegment(); seg != "" { + text += "\n\n" + seg + } cfg := s.cfg.ClaudeCode binary := cfg.BinaryPath diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 2e70c626..bab8101f 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -289,6 +289,108 @@ func TestClaudeCodeErrorResultReturnsError(t *testing.T) { } } +// TestClaudeCodeDelegatedTurnDeliversAndCommitsTaskNotification is the +// regression test for the claude-code delegated lane's own bypass of the +// task-notification delivery/commit machinery — root-caused live as an +// infinite resume loop: a settled non-blocking child's notification never +// reached the model (runClaudeCodeTurn built its CLI turn purely from +// lastUserMessageText, never calling checkoutTaskNotificationsSegment the +// way the native loop body — engine.go's runAgenticLoop, via streamTurn — +// does on every call), so the model kept answering the bare trigger +// string with "No action taken," and the notification, never committed, +// stayed pending forever, so SessionManager.finalizeTurn's +// hasPendingTaskNotifications check kept re-firing triggerResumeLocked. +// +// Proves both halves of the fix, in one real Prompt call through the fake +// CLI: (a) the delivery half — the CLI's actual STDIN contains the +// checked-out notification's rendered content, not just the bare trigger +// string, so the model can act on the child's real result; (b) the commit +// half — a turn that then SUCCEEDS clears the pending set +// (hasPendingTaskNotifications false afterward, breaking the loop) and +// durably records delivery (a recTaskNotifyDelivered record on the +// session's own log, exactly like the native path's own commit). +func TestClaudeCodeDelegatedTurnDeliversAndCommitsTaskNotification(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + stdinLog := filepath.Join(t.TempDir(), "stdin.log") + t.Setenv("FAKE_CLAUDE_STDIN_LOG", stdinLog) + + s.enqueueTaskNotification(taskNotification{ + ChildID: "ses_child1", + Agent: "explore", + Status: StatusDone, + Result: "found the bug in foo.go", + }) + + if _, err := s.Prompt(context.Background(), taskResumeTriggerText); err != nil { + t.Fatalf("Prompt: %v", err) + } + + stdinBytes, err := os.ReadFile(stdinLog) + if err != nil { + t.Fatalf("reading captured CLI stdin: %v", err) + } + stdin := string(stdinBytes) + if !strings.Contains(stdin, "ses_child1") || !strings.Contains(stdin, "found the bug in foo.go") { + t.Fatalf("CLI stdin missing the checked-out notification's content (only the bare trigger reached the model): %s", stdin) + } + + if s.hasPendingTaskNotifications() { + t.Error("notification still pending after a successful delegated turn — commitTaskNotifications did not run, so the resume loop is not broken") + } + + data, err := os.ReadFile(filepath.Join(s.cfg.SessionDir, s.ID+".jsonl")) + if err != nil { + t.Fatal(err) + } + log := string(data) + if !strings.Contains(log, `"type":"task.notify_delivered"`) || !strings.Contains(log, `"child_id":"ses_child1"`) { + t.Fatalf("log missing task.notify_delivered record: %s", log) + } +} + +// TestClaudeCodeDelegatedTurnRequeuesTaskNotificationOnFailure is the +// companion failure-path proof: when the delegated turn itself errors (the +// CLI's own "error" mode here — an is_error result, see +// TestClaudeCodeErrorResultReturnsError), the notification checked out for +// that failed attempt must be REQUEUED, not lost — mirroring the native +// loop body's own requeueTaskNotifications call on its streamTurnWithRetry +// error path (engine.go's runAgenticLoop). +func TestClaudeCodeDelegatedTurnRequeuesTaskNotificationOnFailure(t *testing.T) { + s, _ := claudeCodeTestSession(t, "error") + stdinLog := filepath.Join(t.TempDir(), "stdin.log") + t.Setenv("FAKE_CLAUDE_STDIN_LOG", stdinLog) + + s.enqueueTaskNotification(taskNotification{ + ChildID: "ses_child1", + Status: StatusDone, + Result: "found the bug in foo.go", + }) + + if _, err := s.Prompt(context.Background(), taskResumeTriggerText); err == nil { + t.Fatal("Prompt returned no error for an is_error result") + } + + // The failed attempt must actually have CHECKED OUT the notification + // (folded it into the CLI input it sent) — otherwise "still pending" + // below would trivially hold even with no checkout/requeue wiring at + // all, proving nothing about the requeue path this test targets. + stdinBytes, err := os.ReadFile(stdinLog) + if err != nil { + t.Fatalf("reading captured CLI stdin: %v", err) + } + if !strings.Contains(string(stdinBytes), "ses_child1") { + t.Fatalf("failed attempt's own CLI stdin never carried the notification (checkout never ran, so requeue is not actually exercised): %s", stdinBytes) + } + + if !s.hasPendingTaskNotifications() { + t.Fatal("notification lost after a failed delegated turn — it was committed or dropped instead of requeued") + } + seg := s.checkoutTaskNotificationsSegment() + if !strings.Contains(seg, "ses_child1") { + t.Errorf("requeued notification missing from a later checkout: %q", seg) + } +} + // TestClaudeCodeAbortSignalsChild proves a canceled context makes // runClaudeCodeTurn return promptly (bounded well under fakeclaude's own // hour-long sleep) rather than hanging until the child exits on its own — diff --git a/engine/engine.go b/engine/engine.go index a1680820..716a0113 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -2536,9 +2536,24 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // (emitSessionError before returning) — a plugin host // watching for session errors must see a delegated turn's // failure exactly like a native one's. + // + // requeueTaskNotifications mirrors the native branch's own + // failure-path call (below, in the s.streamTurnWithRetry + // err != nil case): whatever runClaudeCodeTurn's own + // checkoutTaskNotificationsSegment call (claude_code_backend.go) + // checked out for this attempt never reached a request that + // survived, so return it to pending rather than lose it — + // see requeueTaskNotifications' doc comment. + s.requeueTaskNotifications() s.emitSessionError(err) return nil, err } + // This attempt succeeded and msg is about to be returned as the + // turn's real result: whatever was checked out for it really was + // delivered in the CLI input that produced msg. Commit BEFORE + // returning, mirroring the native branch's own commit-before-append + // ordering (below) — see commitTaskNotifications' doc comment. + s.commitTaskNotifications() return msg, nil } s.emitStatus("busy") diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 8713b983..f5de4663 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -35,6 +35,7 @@ import ( "bufio" "encoding/json" "fmt" + "io" "os" "os/exec" "strconv" @@ -87,16 +88,25 @@ func main() { } if mode != "fast_no_drain" { - // Drain (don't require) exactly one stdin line — the real driver - // writes one turn message and closes stdin; reading it keeps this - // stand-in honest about the protocol without validating its - // content. - go func() { - scanner := bufio.NewScanner(os.Stdin) - for scanner.Scan() { - // discard + // Read (and, when FAKE_CLAUDE_STDIN_LOG is set, record) the one + // turn-input line the real driver writes — the real driver writes + // its turn message and closes stdin BEFORE it ever reads this + // process's stdout (see runClaudeCodeTurn's own stdin write/close + // ordering, claude_code_backend.go), so a synchronous read-to-EOF + // here always completes promptly regardless of mode, with none of + // the goroutine-vs-process-exit race a background drain would + // carry for a test that actually wants the captured bytes. + // FAKE_CLAUDE_STDIN_LOG lets a test recover the EXACT bytes the + // driver sent — e.g. proving a checked-out task notification's + // rendered content actually reached the CLI's input, not just the + // bare trigger text (engine/claude_code_backend_test.go). + stdinBytes, _ := io.ReadAll(os.Stdin) + if logPath := os.Getenv("FAKE_CLAUDE_STDIN_LOG"); logPath != "" { + if f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644); err == nil { + _, _ = f.Write(stdinBytes) + f.Close() } - }() + } } sessionID := os.Getenv("FAKE_CLAUDE_SESSION_ID") From 4e632ec4d963dc09062d25a819069e28093eab41 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 10:57:11 -0400 Subject: [PATCH 39/95] Reapply "feat(engine): disallow native subagent spawning in claude-code lane (#225)" (#226) (#228) This reverts commit 21d18ea5692ad588d119d17c76c2516ecec8ab0b. --- engine/claude_code_backend.go | 8 ++++++++ engine/claude_code_backend_test.go | 22 ++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index ab811e30..84a9922f 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -441,6 +441,14 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // which this driver always sets, so it is safe to pass // unconditionally. "--forward-subagent-text", + // All subagent spawning in the claude-code lane must go through + // harness's own cross-family "task" tool (server/mcp_history.go, + // #223), never the CLI's native same-family equivalent — the two + // tools below are the CLI's only built-in ways to spawn a + // same-family subagent (Agent: single subagent/teammate; Workflow: + // a script that fans out many subagents), so both are disallowed + // unconditionally rather than left for the model to choose between. + "--disallowedTools", "Agent,Workflow", } if model.Model != "" { args = append(args, "--model", model.Model) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index bab8101f..f2ddd609 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1048,6 +1048,28 @@ func TestClaudeCodeForwardSubagentTextAlwaysSet(t *testing.T) { } } +// TestClaudeCodeDisallowsNativeSpawnTools proves runClaudeCodeTurn always +// sends --disallowedTools naming every native Claude Code tool that spawns +// a same-family subagent (Agent, Workflow). All subagent spawning in the +// claude-code lane must go through harness's own cross-family "task" tool +// (server/mcp_history.go, #223), never the CLI's native same-family +// equivalent, so those tools are blocked at the argv level rather than +// relying on the model to prefer one path over the other. +func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + got, ok := argvValueAfter(invocations[0], "--disallowedTools") + if !ok { + t.Fatalf("argv has no --disallowedTools: %v", invocations[0]) + } + if want := "Agent,Workflow"; got != want { + t.Errorf("--disallowedTools = %q, want %q", got, want) + } +} + // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning From c6cf78a9fc9db2e814c804f43910e5b44b7d1816 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 13:37:45 -0400 Subject: [PATCH 40/95] feat(engine,provider,server): add per-session Codex service tier (#229) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Boxes needs a per-session Codex speed-tier control (the boxes half of this feature): a dashboard picker that lets a caller choose standard/fast/ultrafast, the same way it already chooses a reasoning-effort level. Harness had no seam to carry that value: provider.Request had no field for it, no session state held it, and no route let a caller set it. This adds ServiceTier as a second per-session dimension, built by mirroring the existing Effort mechanism at every seam it uses: provider.Request.ServiceTier rides to provider/openai/transcode.go, which sets the Responses API "service_tier" field when non-empty and omits it otherwise (both the HTTP and WebSocket transports serialize the same apiRequest, so one transcode hook covers both). engine.Session gains serviceTier state, SetServiceTier/ServiceTier choke-point methods with the same lock-persist-emit discipline as SetEffort, and EventServiceTierChanged. A durable recServiceTier record persists changes and replays on LoadSession, wired through every place recEffort is folded: the header record, the tail replay switch, JournalRecord/projectJournalRecord (GET /session/{id}/journal), SessionIndex/indexFold (the fast listing path), and sessionSnapshot (capture and restore). The server adds POST /session/{id}/service-tier (mirroring handleSetThinking, including its session-resolves-before-body-validates order), a service_tier field on sessionJSON, and a durable "service_tier" SSE/journal event on Publish (mirroring the "effort" event) so a GET /session poller and a GET /event subscriber both see the current value the same way they see effort. The one deliberate divergence from Effort: harness performs NO validation of the tier value. Effort has ParseEffort against a fixed enum; ServiceTier is an opaque string forwarded verbatim. Boxes owns the model/plan gating table for which tiers exist, exactly as it already owns effort-level gating per model — adding a harness-side enum would just be a second, driftable copy of that table. Verified with the same test shapes the effort mechanism already has: SetServiceTier rides the next provider.Request and emits EventServiceTierChanged exactly once per real change (engine); transcodeRequestFamily includes "service_tier" when set and omits it when empty, both asserted on the marshaled wire JSON, not just the struct field (provider/openai); the durable record round-trips through LoadSession, including a Config.ServiceTier set at create time restored from the session header (engine); and the HTTP route round-trips through the session, journals exactly once, accepts an empty clear and an omitted-field clear, accepts an arbitrary value with no 400 (unlike effort's invalid-level case), and 404s on an unknown session (server). Red-verified the transcode test first: before the ServiceTier field existed, both new tests failed to compile against apiRequest, confirming they exercise the change and not a coincidentally-passing default. go build ./..., gofmt -l ., go vet ./..., and go test -race across provider/engine/server all pass. go test -race ./... passes except the pre-existing TestAppendSystemPromptReachesModelOnServe e2e failure, confirmed present and unrelated on unmodified origin/main (a macOS tempdir path-prefix issue, not a regression from this change). --- engine/engine.go | 80 ++++++++++++++-- engine/index.go | 10 +- engine/journal.go | 23 +++++ engine/service_tier_test.go | 117 +++++++++++++++++++++++ engine/session_info.go | 10 +- engine/snapshot.go | 7 +- engine/store.go | 34 ++++++- provider/openai/service_tier_test.go | 49 ++++++++++ provider/openai/transcode.go | 8 ++ provider/provider.go | 9 ++ server/handlers.go | 97 ++++++++++++++++--- server/journal.go | 48 +++++++--- server/openapi.yaml | 87 +++++++++++++++++ server/server.go | 1 + server/set_service_tier_test.go | 138 +++++++++++++++++++++++++++ 15 files changed, 677 insertions(+), 41 deletions(-) create mode 100644 engine/service_tier_test.go create mode 100644 provider/openai/service_tier_test.go create mode 100644 server/set_service_tier_test.go diff --git a/engine/engine.go b/engine/engine.go index 716a0113..a3bbaefa 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -136,6 +136,16 @@ type Event struct { // restores the effort on LoadSession. Effort message.Effort `json:"effort,omitempty"` + // ServiceTier is carried by EventServiceTierChanged only: the session's + // new speed-tier value after a SetServiceTier call that actually changed + // it (see SetServiceTier). It is the single event a service-tier swap + // emits, whatever the route, so the server journals each swap through + // one path (see server/journal.go's EventServiceTierChanged case). It is + // distinct from the durable recServiceTier resume record + // persistServiceTier writes (store.go), which restores the value on + // LoadSession. + ServiceTier string `json:"service_tier,omitempty"` + // Goal-loop fields (set on goal.* events; see goal.go and the state // machine documented atop goal.go). GoalCondition is carried by // goal.set and goal.updated (the new condition); GoalReason/GoalMet/GoalTurn by goal.eval; GoalReason/GoalTurn @@ -251,6 +261,13 @@ const ( // (see SetEffort). EventEffortChanged = "effort.changed" + // EventServiceTierChanged fires once per SetServiceTier call that + // actually changes the session's speed-tier value (never on a no-op set + // to the current value). It carries the new value in + // Event.ServiceTier and is the single observability event every + // service-tier-swap route funnels through (see SetServiceTier). + EventServiceTierChanged = "service_tier.changed" + // Goal-loop events (see goal.go). EventGoalSet = "goal.set" EventGoalUpdated = "goal.updated" @@ -294,9 +311,14 @@ type Config struct { Providers provider.Registry Model message.ModelRef // initial model; swap any time with SetModel Effort message.Effort // initial reasoning-effort level; swap with SetEffort (zero = provider default) - System []string // base system prompt segments - MaxTokens int // per-response cap; defaults to 8192 - WorkDir string // working directory for built-in tools + // ServiceTier is the initial Codex speed-tier value; swap with + // SetServiceTier (zero = provider default). An opaque, unvalidated + // string forwarded verbatim — harness does not gate which tiers a + // model or plan supports (see provider.Request.ServiceTier). + ServiceTier string + System []string // base system prompt segments + MaxTokens int // per-response cap; defaults to 8192 + WorkDir string // working directory for built-in tools // AppendSystemPrompt carries OPERATOR-supplied system prompt segments — // environment truth the agent cannot otherwise discover (a gateway URL @@ -966,12 +988,13 @@ type Session struct { cfg Config tools map[string]Tool - mu sync.Mutex - model message.ModelRef - effort message.Effort // reasoning-effort level; swap with SetEffort - history []message.Message - usage provider.Usage // cumulative, across every turn (see appendWithUsage) - createdAt time.Time + mu sync.Mutex + model message.ModelRef + effort message.Effort // reasoning-effort level; swap with SetEffort + serviceTier string // Codex speed-tier value; swap with SetServiceTier + history []message.Message + usage provider.Usage // cumulative, across every turn (see appendWithUsage) + createdAt time.Time // lastUsage/haveLastUsage carry the most recent model turn's own Usage // (input/output/cache tokens for that one request), distinct from the // cumulative usage field above — GET /session surfaces both (issue #62 @@ -1508,6 +1531,7 @@ func newSession(cfg Config) *Session { cfg: cfg, model: cfg.Model, effort: cfg.Effort, + serviceTier: cfg.ServiceTier, tools: make(map[string]Tool), createdAt: time.Now().UTC(), promptQueueNextID: 1, @@ -1716,6 +1740,37 @@ func (s *Session) Effort() message.Effort { return s.effort } +// SetServiceTier swaps the Codex speed-tier value for subsequent requests. A +// no-op set to the current value changes nothing and emits no event. The +// value rides every request the same way Effort does — the adapter forwards +// it to the provider's wire shape at transcode time, so there is no +// migration step, and harness itself never validates which tiers a model or +// plan supports (see provider.Request.ServiceTier). +// +// On a real change it persists the durable recServiceTier resume record and +// emits EventServiceTierChanged (carrying the new value), both while holding +// s.mu so event order matches log order — the same persist-and-emit-under- +// s.mu shape SetEffort uses. EventServiceTierChanged is the ONE event every +// service-tier-swap route funnels through, so the server journals every swap +// once via a single path. OnEvent must not call back into this Session. +func (s *Session) SetServiceTier(tier string) { + s.mu.Lock() + defer s.mu.Unlock() + if tier == s.serviceTier { + return + } + s.serviceTier = tier + s.persistServiceTier(tier) + s.emit(Event{Type: EventServiceTierChanged, ServiceTier: tier}) +} + +// ServiceTier returns the session's current Codex speed-tier value. +func (s *Session) ServiceTier() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.serviceTier +} + // CreatedAt returns when the session was created (or, for a loaded session, // when it was originally created per its log header). func (s *Session) CreatedAt() time.Time { @@ -2928,6 +2983,13 @@ func (s *Session) streamTurn(ctx context.Context, attempt int) (*message.Message // POST /thinking; the boxes picker does this by clamping the level to the // new model's supported set on switch. Effort: s.Effort(), + // ServiceTier is read straight from session state, mirroring Effort + // immediately above — read fresh every request (and every tool + // round) so a SetServiceTier swap takes effect on the NEXT request, + // with no chat.params routing (same v1 scope rationale as Effort: + // the boxes API owns per-model/per-plan tier gating, not a harness + // plugin). + ServiceTier: s.ServiceTier(), // SessionKey names this session for an adapter that forwards it as // a routing/cache-affinity hint (see provider.Request.SessionKey // doc comment; openaicompat sends it as the wire "user" field). diff --git a/engine/index.go b/engine/index.go index 9fc9107d..9434c26b 100644 --- a/engine/index.go +++ b/engine/index.go @@ -102,8 +102,9 @@ type SessionIndex struct { // predate the timestamp field. LastActivityAt time.Time `json:"last_activity_at"` - Model message.ModelRef `json:"model,omitzero"` - Effort message.Effort `json:"effort,omitempty"` + Model message.ModelRef `json:"model,omitzero"` + Effort message.Effort `json:"effort,omitempty"` + ServiceTier string `json:"service_tier,omitempty"` WorkDir string `json:"workdir,omitempty"` ParentSession string `json:"parent_session,omitempty"` @@ -243,6 +244,7 @@ type indexRecord struct { TaskDepth int `json:"task_depth,omitempty"` Model message.ModelRef `json:"model,omitzero"` Effort message.Effort `json:"effort,omitempty"` + ServiceTier string `json:"service_tier,omitempty"` Message *indexMessage `json:"message,omitempty"` Usage *provider.Usage `json:"usage,omitempty"` Goal *goalRecord `json:"goal,omitempty"` @@ -267,6 +269,7 @@ func indexRecordOf(rec record) indexRecord { TaskDepth: rec.TaskDepth, Model: rec.Model, Effort: rec.Effort, + ServiceTier: rec.ServiceTier, Usage: rec.Usage, Goal: rec.Goal, Prompt: rec.Prompt, @@ -354,6 +357,7 @@ func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { f.ix.TaskAgentType = rec.TaskAgentType f.ix.TaskDepth = rec.TaskDepth f.ix.Effort = rec.Effort + f.ix.ServiceTier = rec.ServiceTier case recMessage: if rec.Message == nil { if isLast { @@ -370,6 +374,8 @@ func (f *indexFold) applyIndexRecord(rec indexRecord, isLast bool) error { f.ix.Model = rec.Model case recEffort: f.ix.Effort = rec.Effort + case recServiceTier: + f.ix.ServiceTier = rec.ServiceTier case recGoalSet, recGoalUpdated, recGoalAchieved, recGoalCleared: f.ix.GoalActive, f.ix.GoalCondition = applyGoalRecord(f.ix.GoalActive, f.ix.GoalCondition, rec.Type, rec.Goal) case recPromptQueued: diff --git a/engine/journal.go b/engine/journal.go index c29a02ab..890a04b5 100644 --- a/engine/journal.go +++ b/engine/journal.go @@ -93,6 +93,17 @@ type JournalRecord struct { // value) and leaves it nil on every other record type. Effort *message.Effort `json:"effort,omitempty"` + // ServiceTier is a *string, not a bare string with omitempty, for the + // same reason Effort is a pointer above: a recServiceTier record's + // SetServiceTier clear writes ServiceTier == "" — an explicit, + // meaningful wire value ("service_tier":"") — which a bare string with + // omitempty would indistinguishably drop, reading identically to a + // record type that never carries a service-tier value at all. + // projectJournalRecord always sets a non-nil pointer on + // recSession/recServiceTier (even for an empty value) and leaves it nil + // on every other record type. + ServiceTier *string `json:"service_tier,omitempty"` + // Goal trace (Type is one of recGoalSet/Updated/Eval/Stalled/Achieved/ // Cleared/EvalFailed/Parked). GoalReason is sanitized: goal.stalled and // goal.eval_failed carry a raw provider/tool err.Error() here (see @@ -198,6 +209,7 @@ func projectJournalRecord(seq int, rec record) JournalRecord { out.TaskDepth = rec.TaskDepth out.Model = rec.Model out.Effort = effortPtr(rec.Effort) + out.ServiceTier = serviceTierPtr(rec.ServiceTier) case recMessage: if rec.Message != nil { out.MessageID = rec.Message.ID @@ -210,6 +222,8 @@ func projectJournalRecord(seq int, rec record) JournalRecord { out.Model = rec.Model case recEffort: out.Effort = effortPtr(rec.Effort) + case recServiceTier: + out.ServiceTier = serviceTierPtr(rec.ServiceTier) case recGoalSet, recGoalUpdated, recGoalEval, recGoalStalled, recGoalAchieved, recGoalCleared, recGoalEvalFailed, recGoalParked: if rec.Goal != nil { out.GoalCondition = rec.Goal.Condition @@ -270,3 +284,12 @@ func projectJournalRecord(seq int, rec record) JournalRecord { func effortPtr(e message.Effort) *message.Effort { return &e } + +// serviceTierPtr returns a non-nil pointer to a local copy of tier, always — +// even when tier is empty — so JournalRecord.ServiceTier's own doc comment +// holds: a cleared service tier renders as an explicit "service_tier":"" +// wire value, never an omitted key indistinguishable from "this record type +// never carries a service-tier value." Mirrors effortPtr. +func serviceTierPtr(tier string) *string { + return &tier +} diff --git a/engine/service_tier_test.go b/engine/service_tier_test.go new file mode 100644 index 00000000..f4ce68b3 --- /dev/null +++ b/engine/service_tier_test.go @@ -0,0 +1,117 @@ +package engine + +import ( + "context" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// TestSetServiceTierRidesRequest: SetServiceTier sets the value carried on +// the next provider.Request. It drives the real Prompt path and inspects the +// request via OnRequest, the same hook production observers use. Mirrors +// TestSetEffortRidesRequest (effort_test.go). +func TestSetServiceTierRidesRequest(t *testing.T) { + var got string + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "ok"}), + }} + cfg := Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + } + cfg.OnRequest = func(_ string, _ int, req *provider.Request) { got = req.ServiceTier } + s := NewSession(cfg) + + s.SetServiceTier("fast") + if _, err := s.Prompt(context.Background(), "go"); err != nil { + t.Fatal(err) + } + if got != "fast" { + t.Fatalf("request service tier = %q, want fast", got) + } +} + +// TestSetServiceTierNoopEmitsNothing: a set to the current value changes +// nothing and emits no event (the surplus-direction guard against a +// redundant emit). Mirrors TestSetEffortNoopEmitsNothing. +func TestSetServiceTierNoopEmitsNothing(t *testing.T) { + var changes int + prov := &scriptedProvider{name: "test"} + cfg := Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + ServiceTier: "standard", + } + cfg.OnEvent = func(ev Event) { + if ev.Type == EventServiceTierChanged { + changes++ + } + } + s := NewSession(cfg) + s.SetServiceTier("standard") // no-op: already standard + if changes != 0 { + t.Fatalf("EventServiceTierChanged fired %d times on no-op set, want 0", changes) + } + s.SetServiceTier("fast") // real change + if changes != 1 { + t.Fatalf("EventServiceTierChanged fired %d times, want 1", changes) + } +} + +// TestPersistServiceTierChangeReplay: a SetServiceTier change survives +// LoadSession — the production resume path, not a hand-built replay. Mirrors +// TestPersistEffortChangeReplay. +func TestPersistServiceTierChangeReplay(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "one"}), + asstTurn(provider.StopEndTurn, &message.Text{Text: "two"}), + }} + cfg := persistCfg(dir, prov) + s := NewSession(cfg) + + if _, err := s.Prompt(context.Background(), "first"); err != nil { + t.Fatal(err) + } + s.SetServiceTier("fast") + if _, err := s.Prompt(context.Background(), "second"); err != nil { + t.Fatal(err) + } + + loaded, err := LoadSession(cfg, s.ID) + if err != nil { + t.Fatal(err) + } + if loaded.ServiceTier() != "fast" { + t.Errorf("loaded service tier = %q, want fast", loaded.ServiceTier()) + } +} + +// TestServiceTierInitialValuePersists: a Config.ServiceTier set at create +// time survives a LoadSession round trip through the session header record. +// Mirrors TestEffortInitialValuePersists. +func TestServiceTierInitialValuePersists(t *testing.T) { + dir := t.TempDir() + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + asstTurn(provider.StopEndTurn, &message.Text{Text: "one"}), + }} + cfg := persistCfg(dir, prov) + cfg.ServiceTier = "ultrafast" + s := NewSession(cfg) + if _, err := s.Prompt(context.Background(), "first"); err != nil { + t.Fatal(err) + } + + // Reload with a cfg that does NOT set ServiceTier — the header must + // restore it. + reloadCfg := persistCfg(dir, prov) + loaded, err := LoadSession(reloadCfg, s.ID) + if err != nil { + t.Fatal(err) + } + if loaded.ServiceTier() != "ultrafast" { + t.Errorf("loaded service tier = %q, want ultrafast (from header)", loaded.ServiceTier()) + } +} diff --git a/engine/session_info.go b/engine/session_info.go index 6809d518..94ed9647 100644 --- a/engine/session_info.go +++ b/engine/session_info.go @@ -38,7 +38,14 @@ type sessionInfoResult struct { // server/handlers.go). Mid-session it changes only via Session.SetEffort, // the same choke point handleSetThinking calls; the create-time value // comes from Config.Effort and a resumed value from the recEffort record. - Effort message.Effort `json:"effort"` + Effort message.Effort `json:"effort"` + // ServiceTier is the session's current provider service tier (a Codex + // speed tier such as "fast"/"ultrafast"). Empty string means no tier has + // been sent, so the provider runs its own default — the same "" convention + // Effort uses. Set only via Session.SetServiceTier; the create-time value + // comes from Config.ServiceTier and a resumed value from the recServiceTier + // record. + ServiceTier string `json:"service_tier"` Usage provider.Usage `json:"usage"` System []string `json:"system"` Tools []string `json:"tools"` @@ -116,6 +123,7 @@ func (s *Session) sessionInfo(ctx context.Context) sessionInfoResult { SessionID: s.ID, Model: s.model.String(), Effort: s.effort, + ServiceTier: s.serviceTier, Usage: s.usage, System: system, Tools: tools, diff --git a/engine/snapshot.go b/engine/snapshot.go index c9aed2fa..1949fcc7 100644 --- a/engine/snapshot.go +++ b/engine/snapshot.go @@ -109,8 +109,9 @@ type sessionSnapshot struct { History []message.Message `json:"history"` - Model message.ModelRef `json:"model,omitzero"` - Effort message.Effort `json:"effort,omitempty"` + Model message.ModelRef `json:"model,omitzero"` + Effort message.Effort `json:"effort,omitempty"` + ServiceTier string `json:"service_tier,omitempty"` Usage provider.Usage `json:"usage,omitzero"` LastUsage provider.Usage `json:"last_usage,omitzero"` @@ -416,6 +417,7 @@ func (s *Session) captureSnapshotLocked() *sessionSnapshot { History: append([]message.Message(nil), s.history...), Model: s.model, Effort: s.effort, + ServiceTier: s.serviceTier, Usage: s.usage, LastUsage: s.lastUsage, HaveLastUsage: s.haveLastUsage, @@ -475,6 +477,7 @@ func (s *Session) restoreSnapshot(snap *sessionSnapshot) { s.model = snap.Model } s.effort = snap.Effort + s.serviceTier = snap.ServiceTier s.usage = snap.Usage s.lastUsage = snap.LastUsage s.haveLastUsage = snap.HaveLastUsage diff --git a/engine/store.go b/engine/store.go index 0d5021b1..5b4a2512 100644 --- a/engine/store.go +++ b/engine/store.go @@ -32,6 +32,7 @@ const ( recMessage = "message" recModel = "model" recEffort = "effort" + recServiceTier = "service_tier" recGoalSet = "goal.set" recGoalUpdated = "goal.updated" recGoalEval = "goal.eval" @@ -287,7 +288,14 @@ type record struct { // change). Omitted when EffortUnset, so a legacy log with no effort // restores as EffortUnset (provider default) — unchanged behavior. Effort message.Effort `json:"effort,omitempty"` - Goal *goalRecord `json:"goal,omitempty"` + // ServiceTier carries the Codex speed-tier value on the session header + // record (the value at create time) and on a recServiceTier record (a + // SetServiceTier change). Omitted when empty, so a legacy log with no + // service_tier restores as "" (provider default) — unchanged behavior. + // Mirrors Effort exactly, but opaque and unvalidated (see + // provider.Request.ServiceTier). + ServiceTier string `json:"service_tier,omitempty"` + Goal *goalRecord `json:"goal,omitempty"` // Prompt carries a prompt.queued/prompt.dequeued record's payload (see // promptRecord and queue.go). nil on every other record type. Prompt *promptRecord `json:"prompt,omitempty"` @@ -677,6 +685,22 @@ func (s *Session) persistEffort(e message.Effort) { } } +// persistServiceTier appends a service_tier record to the session log. It +// mirrors persistEffort exactly: a no-op until the log exists (lazy +// creation), caller holds s.mu. +func (s *Session) persistServiceTier(tier string) { + if s.cfg.SessionDir == "" || !s.logStarted { + return + } + if err := s.ensureLog(); err != nil { + s.lastPersistErr = err + return + } + if err := s.writeRecord(record{Type: recServiceTier, ServiceTier: tier}); err != nil { + s.lastPersistErr = err + } +} + // persistClaudeCodeSessionID appends a claude_code.session_id record to the // session log. It mirrors persistModel/persistEffort exactly: a no-op // until the log exists (lazy creation), caller holds s.mu. @@ -1085,7 +1109,7 @@ func (s *Session) ensureLog() error { // LoadSession already tolerates. var buf bytes.Buffer headerRecs := []record{ - {Type: recSession, ID: s.ID, CreatedAt: s.createdAt, WorkDir: s.cfg.WorkDir, ParentSession: s.cfg.ParentSession, TaskParentID: s.cfg.TaskParentID, TaskAgentType: s.cfg.TaskAgentType, TaskToolNames: taskToolNamesPtr(s.cfg.TaskToolNames), TaskDepth: s.cfg.TaskDepth, Effort: s.effort}, + {Type: recSession, ID: s.ID, CreatedAt: s.createdAt, WorkDir: s.cfg.WorkDir, ParentSession: s.cfg.ParentSession, TaskParentID: s.cfg.TaskParentID, TaskAgentType: s.cfg.TaskAgentType, TaskToolNames: taskToolNamesPtr(s.cfg.TaskToolNames), TaskDepth: s.cfg.TaskDepth, Effort: s.effort, ServiceTier: s.serviceTier}, {Type: recModel, Model: s.model}, } // A selection made before the log existed has no other durable @@ -1530,6 +1554,8 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.model = rec.Model case recEffort: s.effort = rec.Effort + case recServiceTier: + s.serviceTier = rec.ServiceTier case recClaudeCodeSessionID: s.claudeCodeCLISessionID = rec.ClaudeCodeSessionID case recClaudeCodeHistoryWatermark: @@ -1965,6 +1991,10 @@ func (s *Session) applySessionHeader(rec record) { // The effort at create time. Omitted (EffortUnset) on a legacy // header, which restores as the provider default — unchanged. s.effort = rec.Effort + // The service tier at create time. Omitted (empty) on a header + // predating this field, which restores as the provider default — + // unchanged. Mirrors the effort restore immediately above. + s.serviceTier = rec.ServiceTier } // toolResultHandleInTextPattern matches a canonical trh_N handle token diff --git a/provider/openai/service_tier_test.go b/provider/openai/service_tier_test.go new file mode 100644 index 00000000..385dead9 --- /dev/null +++ b/provider/openai/service_tier_test.go @@ -0,0 +1,49 @@ +package openai + +import ( + "encoding/json" + "strings" + "testing" + + "github.com/majorcontext/harness/message" +) + +// TestServiceTierSetsField: a non-empty Request.ServiceTier sets the +// Responses API top-level "service_tier" field to the same string. Mirrors +// TestSessionKeySetsPromptCacheKey (session_affinity_test.go) — the same +// pass-through shape as PromptCacheKey, forwarded verbatim with no +// validation of which tiers exist (boxes owns that gating table). +func TestServiceTierSetsField(t *testing.T) { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.ServiceTier = "fast" + out := mustTranscode(t, req) + if out.ServiceTier != "fast" { + t.Errorf("ServiceTier = %q, want %q", out.ServiceTier, "fast") + } + raw, err := json.Marshal(out) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(raw), `"service_tier":"fast"`) { + t.Errorf("wire missing service_tier field: %s", raw) + } +} + +// TestServiceTierEmptyOmitsField: an empty Request.ServiceTier (the zero +// value) omits the "service_tier" field entirely rather than sending an +// empty string. Mirrors TestSessionKeyEmptyOmitsPromptCacheKey. +func TestServiceTierEmptyOmitsField(t *testing.T) { + req := baseRequest(message.Message{Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: "hi"}}}) + req.ServiceTier = "" + out := mustTranscode(t, req) + if out.ServiceTier != "" { + t.Errorf("ServiceTier = %q, want empty", out.ServiceTier) + } + raw, err := json.Marshal(out) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(raw), `"service_tier":`) { + t.Errorf("wire must omit service_tier field: %s", raw) + } +} diff --git a/provider/openai/transcode.go b/provider/openai/transcode.go index 5a1e6c33..350f67a4 100644 --- a/provider/openai/transcode.go +++ b/provider/openai/transcode.go @@ -60,6 +60,11 @@ type apiRequest struct { // gateway; this one targets the Responses API directly, whose own // affinity hint is prompt_cache_key, not user. PromptCacheKey string `json:"prompt_cache_key,omitempty"` + // ServiceTier is the Responses API's speed-tier selector, set verbatim + // from Request.ServiceTier (see that field's own doc comment for the + // pass-through/no-validation contract this adapter follows). Empty sends + // no field, so the account's default tier applies. + ServiceTier string `json:"service_tier,omitempty"` } // apiReasoning is the OpenAI Responses reasoning control. Effort is one of @@ -186,6 +191,9 @@ func transcodeRequestFamily(req *provider.Request, family string, omitParams []s if req.SessionKey != "" { out.PromptCacheKey = req.SessionKey } + if req.ServiceTier != "" { + out.ServiceTier = req.ServiceTier + } effort, reasoningEnabled := reasoningEffort(req.Effort) // stripReasoning is DELIBERATELY asymmetric with reasoningEnabled and with // the anthropic adapter. reasoningEnabled is false for BOTH EffortUnset diff --git a/provider/provider.go b/provider/provider.go index 566bb7d1..9df3db8d 100644 --- a/provider/provider.go +++ b/provider/provider.go @@ -54,6 +54,15 @@ type Request struct { // rejects surfaces as a provider error, since the adapter cannot know // per-model support from the ref alone. Effort message.Effort + // ServiceTier is an opaque, per-session speed-tier hint (e.g. Codex's + // "standard"/"fast"/"ultrafast"), forwarded to the provider verbatim as + // the wire "service_tier" field. The zero value (empty string) sends no + // service_tier control. Like Effort, harness does NOT validate which + // tiers a model or plan supports — the caller (the boxes API) owns that + // gating table — so an adapter maps a non-empty value straight through, + // and a tier the target model or plan rejects surfaces as a provider + // error. + ServiceTier string // SessionKey is a stable, opaque identifier for the session this // request belongs to. An adapter MAY forward it to the provider as a // routing or cache-affinity hint. It is not a secret and is not diff --git a/server/handlers.go b/server/handlers.go index e814cb67..1082fa6c 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -33,7 +33,13 @@ type sessionJSON struct { // EffortUnset, the provider default). A dashboard reads it back here after // POST /session/{id}/thinking, the same way it reads Model. Effort message.Effort `json:"effort,omitempty"` - Status string `json:"status"` + // ServiceTier is the session's current Codex speed-tier value (empty = + // provider default). A dashboard reads it back here after POST + // /session/{id}/service-tier, the same way it reads Effort. Harness + // forwards this value verbatim and does not validate which tiers a + // model or plan supports (see provider.Request.ServiceTier). + ServiceTier string `json:"service_tier,omitempty"` + Status string `json:"status"` // State is the unambiguous composite: idle, busy, or goal-running. Kept // alongside Status (never replacing it) for backward compat. Precedence: // goal-running wins whenever a goal is active, REGARDLESS of the momentary @@ -3091,6 +3097,71 @@ func (s *Server) handleSetThinking(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, setThinkingResponseJSON{Effort: st.sess.Effort()}) } +// setServiceTierResponseJSON is the POST /session/{id}/service-tier response +// shape: the session's Codex speed-tier value after the swap (a same-value +// set is a durable no-op, echoing the current value — see +// engine.Session.SetServiceTier). +type setServiceTierResponseJSON struct { + ServiceTier string `json:"service_tier"` +} + +// handleSetServiceTier swaps a session's Codex speed-tier value, decoupled +// from prompting — a client/dashboard-driven swap that never claims the run +// slot (SetServiceTier is concurrency-safe and takes effect on the NEXT +// request). It mirrors handleSetThinking: an unknown session is 404. Unlike +// effort, there is no ParseEffort-equivalent validation at all — the value +// is an opaque string harness forwards verbatim, never checked against a +// known tier set, since a dashboard that must gate per model/plan (the +// boxes picker) holds its own mapping. An empty string is accepted and +// clears the value (provider default). On success SetServiceTier emits +// EventServiceTierChanged, which Publish journals as the durable +// "service_tier" record. +func (s *Server) handleSetServiceTier(w http.ResponseWriter, r *http.Request) { + id, ok := s.sessionIDOrNotFound(w, r) + if !ok { + return + } + if s.rejectManagedChildTurn(w, id) { + return + } + var body struct { + ServiceTier string `json:"service_tier"` + } + if err := decodeBody(r, &body); err != nil { + writeErr(w, http.StatusBadRequest, err.Error()) + return + } + + // Resolve the session FIRST, loading a cold one into residency with the + // same race handling handleSetThinking uses (two *engine.Session for one + // log must never both be mutated — SetServiceTier persists the durable + // recServiceTier record). + s.mu.Lock() + st := s.sessions[id] + s.mu.Unlock() + if st == nil { + sess, err := s.opts.LoadSession(id) + if err != nil { + writeErr(w, http.StatusNotFound, "no such session") + return + } + s.mu.Lock() + var evicted []*engine.Session + if ex := s.sessions[id]; ex != nil { + st = ex + } else { + st = &sessionState{sess: sess, lastUsed: time.Now()} + s.sessions[id] = st + evicted = s.evictResidentLocked() + } + s.mu.Unlock() + releaseEvicted(evicted) + } + + st.sess.SetServiceTier(body.ServiceTier) + writeJSON(w, http.StatusOK, setServiceTierResponseJSON{ServiceTier: st.sess.ServiceTier()}) +} + // evictResidentLocked unloads the longest-idle non-busy sessions from // s.sessions (this server's OWN residency bookkeeping) when the resident // count exceeds Options.MaxResident. Busy sessions are never evicted; @@ -3881,6 +3952,7 @@ func (s *Server) buildSession(lv liveSession) sessionJSON { CreatedAt: sess.CreatedAt(), Model: sess.Model(), Effort: sess.Effort(), + ServiceTier: sess.ServiceTier(), Status: status, State: compositeState(status == "busy", goal != nil && goal.Active, forcesIdlePause(goal)), Messages: len(sess.History()), @@ -3927,17 +3999,18 @@ func (s *Server) buildSessionFromIndex(ix engine.SessionIndex) sessionJSON { lastTurn := s.lastTurnJSONLocked(ix.ID) s.mu.Unlock() return sessionJSON{ - ID: ix.ID, - CreatedAt: ix.CreatedAt, - Model: ix.Model, - Effort: ix.Effort, - Status: "idle", - State: compositeState(false, goal != nil && goal.Active, forcesIdlePause(goal)), - Messages: ix.Messages, - Seq: seq, - Goal: goal, - WorkDir: ix.WorkDir, - LastTurn: lastTurn, + ID: ix.ID, + CreatedAt: ix.CreatedAt, + Model: ix.Model, + Effort: ix.Effort, + ServiceTier: ix.ServiceTier, + Status: "idle", + State: compositeState(false, goal != nil && goal.Active, forcesIdlePause(goal)), + Messages: ix.Messages, + Seq: seq, + Goal: goal, + WorkDir: ix.WorkDir, + LastTurn: lastTurn, Usage: usageJSON{ InputTokens: ix.Usage.InputTokens, OutputTokens: ix.Usage.OutputTokens, diff --git a/server/journal.go b/server/journal.go index 147eb50a..1b090e05 100644 --- a/server/journal.go +++ b/server/journal.go @@ -38,12 +38,20 @@ type Event struct { // So Publish's EventEffortChanged case ALWAYS sets it (even on a clear), // and a nil pointer is omitted on every other record. This mirrors the // "model" record, which never clears to empty. - Effort *message.Effort `json:"effort,omitempty"` - Text string `json:"text,omitempty"` - ToolCall *message.ToolCall `json:"tool_call,omitempty"` - Output message.Parts `json:"output,omitempty"` - IsError bool `json:"is_error,omitempty"` - Error string `json:"error,omitempty"` + Effort *message.Effort `json:"effort,omitempty"` + // ServiceTier is a *string, not a bare string with omitempty, mirroring + // Effort immediately above for the identical reason: a "service_tier" + // record must tell "cleared to the provider default" (an explicit + // "service_tier":"") apart from "this event type never carries a + // service-tier value" (key absent, every other record). Publish's + // EventServiceTierChanged case ALWAYS sets it (even on a clear), and a + // nil pointer is omitted on every other record type. + ServiceTier *string `json:"service_tier,omitempty"` + Text string `json:"text,omitempty"` + ToolCall *message.ToolCall `json:"tool_call,omitempty"` + Output message.Parts `json:"output,omitempty"` + IsError bool `json:"is_error,omitempty"` + Error string `json:"error,omitempty"` // request.meta fields: a durable, replayable record of the assembled model // request. SystemHash fingerprints the joined system segments; the full @@ -172,13 +180,17 @@ const ( evtMessage = "message" evtModel = "model" evtEffort = "effort" - evtRequestMeta = "request.meta" - evtGoalSet = "goal.set" - evtGoalUpdated = "goal.updated" - evtGoalEval = "goal.eval" - evtGoalStalled = "goal.stalled" - evtGoalAchieved = "goal.achieved" - evtGoalCleared = "goal.cleared" + // evtServiceTier mirrors evtEffort: the durable observability record + // journaled for every SetServiceTier swap (see the + // engine.EventServiceTierChanged case in Publish below). + evtServiceTier = "service_tier" + evtRequestMeta = "request.meta" + evtGoalSet = "goal.set" + evtGoalUpdated = "goal.updated" + evtGoalEval = "goal.eval" + evtGoalStalled = "goal.stalled" + evtGoalAchieved = "goal.achieved" + evtGoalCleared = "goal.cleared" // evtGoalEvalFailed mirrors engine.EventGoalEvalFailed (see // engine/goal.go's "Round 6" doc section): journaled once per // failed evaluator boundary — a provider error the retryable-class @@ -365,6 +377,16 @@ func (s *Server) Publish(ev engine.Event) { // carries the "effort" key — see the Event.Effort field comment. eff := ev.Effort s.emitDurable(Event{Type: evtEffort, SessionID: ev.SessionID, Effort: &eff}) + case engine.EventServiceTierChanged: + // Every service-tier swap funnels through SetServiceTier's single + // EventServiceTierChanged emit, so this ONE case journals the + // durable "service_tier" observability record — the same + // single-path shape the EventEffortChanged case above uses. + // ServiceTier is set explicitly (a pointer) even on a clear to "", + // so the record always carries the "service_tier" key — see the + // Event.ServiceTier field comment. + tier := ev.ServiceTier + s.emitDurable(Event{Type: evtServiceTier, SessionID: ev.SessionID, ServiceTier: &tier}) case engine.EventGoalSet, engine.EventGoalUpdated, engine.EventGoalEval, engine.EventGoalStalled, engine.EventGoalAchieved, engine.EventGoalCleared, engine.EventGoalEvalFailed, engine.EventGoalParked: s.publishGoal(ev) case engine.EventPromptQueued, engine.EventPromptDequeued: diff --git a/server/openapi.yaml b/server/openapi.yaml index 7cd85c4f..4624fefd 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -115,6 +115,13 @@ components: The session's current reasoning-effort level. Absent (or empty) means the provider default (EffortUnset). Set it with POST /session/{id}/thinking. + service_tier: + $ref: "#/components/schemas/ServiceTier" + description: > + The session's current Codex speed-tier value. Absent (or empty) + means the provider default. Set it with POST + /session/{id}/service-tier. Harness forwards this value verbatim + and does not validate which tiers a model or plan supports. status: $ref: "#/components/schemas/SessionStatus" description: > @@ -1035,6 +1042,7 @@ components: - message - model - effort + - service_tier - request.meta - goal.set - goal.updated @@ -1223,6 +1231,13 @@ components: on an "effort" event, even on a clear: an empty string means cleared to the provider default (EffortUnset), never a dropped field. Absent on every other event type. + service_tier: + $ref: "#/components/schemas/ServiceTier" + description: > + The new Codex speed-tier value on a "service_tier" event. Always + present on a "service_tier" event, even on a clear: an empty + string means cleared to the provider default, never a dropped + field. Absent on every other event type. text: type: string description: Delta text for text.delta / reasoning.delta. @@ -1444,6 +1459,7 @@ components: - message - model - effort + - service_tier - goal.set - goal.updated - goal.eval @@ -1502,6 +1518,8 @@ components: $ref: "#/components/schemas/ModelRef" effort: type: string + service_tier: + type: string goal_condition: type: string description: Only on goal.* records. @@ -1715,6 +1733,36 @@ components: $ref: "#/components/schemas/Effort" description: The session's reasoning-effort level after the swap. + ServiceTier: + type: string + description: > + An opaque, per-session Codex speed-tier value (e.g. "standard", + "fast", "ultrafast"). Harness does NOT validate this against a known + tier set — unlike Effort, it has no enum — because the caller (the + boxes API) owns per-model/per-plan tier gating; harness only stores + and forwards the value verbatim as the provider wire "service_tier" + field. An empty string means the provider default (no service_tier + control sent). + + SetServiceTierRequest: + type: object + properties: + service_tier: + $ref: "#/components/schemas/ServiceTier" + description: > + The new Codex speed-tier value. An empty string OR an omitted + field both clear it to the provider default. Any value is + accepted — harness performs no validation. The field is optional + precisely because absent and "" are the same clear operation. + + SetServiceTierResponse: + type: object + required: [service_tier] + properties: + service_tier: + $ref: "#/components/schemas/ServiceTier" + description: The session's Codex speed-tier value after the swap. + Request: type: object description: > @@ -2644,6 +2692,45 @@ paths: "401": { $ref: "#/components/responses/Unauthorized" } "404": { $ref: "#/components/responses/NotFound" } + /session/{id}/service-tier: + post: + operationId: setServiceTier + summary: Swap a session's Codex speed-tier value. + description: > + Swaps the session's Codex speed-tier value for subsequent requests, + decoupled from prompting — it never claims the run slot, so it + applies even while a turn is running and takes effect on the NEXT + request. The value rides every request as the provider wire + "service_tier" field (OpenAI Responses API). The swap is durable (a + recServiceTier record restores it on resume) and journals a single + "service_tier" event on /event. Unlike /session/{id}/thinking there + is NO validation at all: the value is an opaque string harness + forwards verbatim, never checked against a known tier set — a + dashboard that gates tiers per model/plan (the boxes picker) holds + its own mapping. An empty string clears the value to the provider + default. The current value is read back on GET /session + ("service_tier"). + parameters: + - $ref: "#/components/parameters/sessionID" + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/SetServiceTierRequest" } + responses: + "200": + description: The Codex speed-tier value after the swap. + content: + application/json: + schema: { $ref: "#/components/schemas/SetServiceTierResponse" } + "400": + description: The request body is malformed. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "401": { $ref: "#/components/responses/Unauthorized" } + "404": { $ref: "#/components/responses/NotFound" } + /session/{id}/abort: post: operationId: abortSession diff --git a/server/server.go b/server/server.go index 5d0df7c3..c9b5f7d1 100644 --- a/server/server.go +++ b/server/server.go @@ -967,6 +967,7 @@ func (s *Server) routes() { mux.HandleFunc("DELETE /session/{id}/goal", s.auth(s.handleGoalDelete)) mux.HandleFunc("POST /session/{id}/model", s.auth(s.handleSetModel)) mux.HandleFunc("POST /session/{id}/thinking", s.auth(s.handleSetThinking)) + mux.HandleFunc("POST /session/{id}/service-tier", s.auth(s.handleSetServiceTier)) mux.HandleFunc("POST /session/{id}/abort", s.auth(s.handleAbort)) // session.send (design doc, Stage 4): deliver a message to ANY session // this server's SessionManager tracks, root or child — see diff --git a/server/set_service_tier_test.go b/server/set_service_tier_test.go new file mode 100644 index 00000000..9bc274dc --- /dev/null +++ b/server/set_service_tier_test.go @@ -0,0 +1,138 @@ +package server + +import ( + "encoding/json" + "testing" +) + +// countServiceTierJournalRecords returns how many durable "service_tier" +// records the journal holds for id. Mirrors countEffortJournalRecords +// (set_thinking_test.go). +func countServiceTierJournalRecords(h *harness, id string) int { + h.t.Helper() + h.srv.mu.Lock() + defer h.srv.mu.Unlock() + n := 0 + for _, ev := range h.srv.journal { + if ev.SessionID == id && ev.Type == evtServiceTier { + n++ + } + } + return n +} + +// TestSetServiceTierEndpointChangesAndJournalsOnce drives the real POST +// /session/{id}/service-tier route: a swap returns 200 with the new value, +// changes the session service tier, surfaces it on GET /session, and +// journals EXACTLY ONE durable "service_tier" record (the surplus-direction +// guard against a double emit). Mirrors +// TestSetThinkingEndpointChangesAndJournalsOnce. +func TestSetServiceTierEndpointChangesAndJournalsOnce(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + + resp, data := h.do("POST", "/session/"+id+"/service-tier", map[string]string{"service_tier": "fast"}) + if resp.StatusCode != 200 { + t.Fatalf("set service tier status %d: %s", resp.StatusCode, data) + } + var got setServiceTierResponseJSON + if err := json.Unmarshal(data, &got); err != nil { + t.Fatal(err) + } + if got.ServiceTier != "fast" { + t.Fatalf("response service_tier = %q, want fast", got.ServiceTier) + } + + _, sdata := h.do("GET", "/session/"+id, nil) + var sess struct { + ServiceTier string `json:"service_tier"` + } + if err := json.Unmarshal(sdata, &sess); err != nil { + t.Fatal(err) + } + if sess.ServiceTier != "fast" { + t.Fatalf("GET /session service_tier = %q, want fast", sess.ServiceTier) + } + + if n := countServiceTierJournalRecords(h, id); n != 1 { + t.Fatalf("durable service_tier records = %d, want exactly 1 (no double emit)", n) + } +} + +// TestSetServiceTierEndpointClearsWithEmpty: an empty string clears the +// value, reported as absent on GET /session. Mirrors +// TestSetThinkingEndpointClearsWithEmpty. +func TestSetServiceTierEndpointClearsWithEmpty(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + + h.do("POST", "/session/"+id+"/service-tier", map[string]string{"service_tier": "fast"}) + resp, data := h.do("POST", "/session/"+id+"/service-tier", map[string]string{"service_tier": ""}) + if resp.StatusCode != 200 { + t.Fatalf("clear service tier status %d: %s", resp.StatusCode, data) + } + _, sdata := h.do("GET", "/session/"+id, nil) + var sess struct { + ServiceTier string `json:"service_tier"` + } + if err := json.Unmarshal(sdata, &sess); err != nil { + t.Fatal(err) + } + if sess.ServiceTier != "" { + t.Fatalf("GET /session service_tier = %q after clear, want empty", sess.ServiceTier) + } +} + +// TestSetServiceTierEndpointOmittedFieldClears: an omitted service_tier +// field (body {}) is treated the same as "" — a clear, 200. Mirrors +// TestSetThinkingEndpointOmittedFieldClears. +func TestSetServiceTierEndpointOmittedFieldClears(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + h.do("POST", "/session/"+id+"/service-tier", map[string]string{"service_tier": "fast"}) + + resp, data := h.doRaw("POST", "/session/"+id+"/service-tier", `{}`) + if resp.StatusCode != 200 { + t.Fatalf("omitted-service_tier status %d: %s, want 200 (clear)", resp.StatusCode, data) + } + _, sdata := h.do("GET", "/session/"+id, nil) + var sess struct { + ServiceTier string `json:"service_tier"` + } + if err := json.Unmarshal(sdata, &sess); err != nil { + t.Fatal(err) + } + if sess.ServiceTier != "" { + t.Fatalf("GET /session service_tier = %q after omitted-field clear, want empty", sess.ServiceTier) + } +} + +// TestSetServiceTierEndpointAcceptsArbitraryValue: harness does NOT validate +// which tiers exist — boxes owns that gating table, exactly as it gates +// effort levels — so any non-empty string round-trips verbatim, with no 400 +// for an "unknown" value. +func TestSetServiceTierEndpointAcceptsArbitraryValue(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + resp, data := h.do("POST", "/session/"+id+"/service-tier", map[string]string{"service_tier": "whatever-boxes-sends"}) + if resp.StatusCode != 200 { + t.Fatalf("status %d: %s, want 200 (harness does not validate tiers)", resp.StatusCode, data) + } + var got setServiceTierResponseJSON + if err := json.Unmarshal(data, &got); err != nil { + t.Fatal(err) + } + if got.ServiceTier != "whatever-boxes-sends" { + t.Fatalf("response service_tier = %q, want verbatim passthrough", got.ServiceTier) + } +} + +// TestSetServiceTierEndpointUnknownSession: an unknown session is 404. +// Mirrors TestSetThinkingEndpointUnknownSession. +func TestSetServiceTierEndpointUnknownSession(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + resp, data := h.do("POST", "/session/ses_01000000000000000000000000/service-tier", map[string]string{"service_tier": "fast"}) + if resp.StatusCode != 404 { + t.Fatalf("unknown-session status %d: %s, want 404", resp.StatusCode, data) + } +} From a01deac367a8d3fe5f6a0fb24f4836d0e8b005f0 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 13:52:42 -0400 Subject: [PATCH 41/95] feat(server): accept a client-supplied user-message id (#230) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The console pre-mints an id for its own optimistic render of a prompt, then has no honest way to reconcile that bubble with the real message once it arrives over SSE — only fragile text matching, which breaks on duplicate or edited text. OpenCode hit the same problem and its fix (a client-minted, time-sortable id) introduced a worse bug: sorting history by that id instead of append order lets client clock skew reorder a conversation. This server is reached only by a trusted, authenticated first-party client, so an id it supplies is used verbatim with exactly one fail-safe guard, never heavy validation: empty, or a reserved provenance prefix engine mints for a different synthetic message kind (cmpsum_ for a compaction summary, synthetic-orphan-tool-result- for a synthesized orphaned tool result), is silently ignored in favor of a fresh server mint. The prompt is never rejected for a bad id. Ordering is untouched: history order stays server-assigned append order, never a sort on this id. POST /session/{id}/prompt_async and POST /session/{id}/send both take an optional `id` field and echo the resolved id back as `message_id`, synchronously, before the turn actually runs — a queued prompt (behind a busy session) resolves its id once, at enqueue time, and both the response and the eventual message agree on that exact value, never a second, independently-minted one at drain time. engine.Session.EnqueuePrompt's queued-prompt record now carries a MessageID field alongside its queue-slot ID and Text; a record written before this field existed folds back with an empty MessageID on replay, and PromptWithOrigin's own mint site resolves that exactly like any other unset id, at dispatch time — no replay error, no rejected prompt. Verification: TestPromptAsyncUsesSuppliedMessageID red-verified by temporarily reverting PromptWithOrigin's two mint sites to their old unconditional newID("msg") call — the test failed with "transcript user message id = msg_01m1eqbxa..., want the supplied id console-optimistic-1", confirming it actually exercises the fix. New tests also cover the reserved-prefix and empty-id mint fallback, the busy-session queue path, backward compatibility with no supplied id, and session.send. go build, gofmt, go vet, and go test -race are clean across engine, server, and the full repo, except the pre-existing, unrelated TestAppendSystemPromptReachesModelOnServe macOS-tempdir e2e failure. --- engine/engine.go | 24 ++- engine/goal_retry_dedup_test.go | 8 +- engine/goal_retry_reuse_test.go | 2 +- engine/goal_toolcall_boundary_test.go | 2 +- engine/goal_update_test.go | 8 +- engine/id.go | 55 +++++- engine/index_test.go | 4 +- engine/max_tokens_continue_test.go | 2 +- engine/message_id_test.go | 59 ++++++ engine/queue.go | 41 +++- engine/queue_persist_order_test.go | 4 +- engine/queue_replay_model_test.go | 2 +- engine/queue_test.go | 18 +- engine/queue_toolcall_boundary_test.go | 4 +- engine/session_manager.go | 8 +- engine/session_replay_model_test.go | 2 +- engine/snapshot_test.go | 2 +- engine/store.go | 8 + engine/store_repair_test.go | 4 +- engine/task_external_turn_test.go | 2 +- server/handlers.go | 66 +++++-- server/openapi.yaml | 17 ++ server/prompt_message_id_test.go | 252 ++++++++++++++++++++++++ server/queue_clear_race_test.go | 2 +- server/queue_delete_nonresident_test.go | 4 +- server/queue_test.go | 12 +- server/session_tree.go | 46 +++-- server/session_tree_test.go | 2 +- 28 files changed, 576 insertions(+), 84 deletions(-) create mode 100644 engine/message_id_test.go create mode 100644 server/prompt_message_id_test.go diff --git a/engine/engine.go b/engine/engine.go index a3bbaefa..82cae110 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -2455,7 +2455,7 @@ func (s *Session) emitSessionError(err error) { // PromptWithOrigin for the overwhelming majority of callers that never need // to set one. func (s *Session) Prompt(ctx context.Context, text string) (*message.Message, error) { - return s.PromptWithOrigin(ctx, text, "") + return s.PromptWithOrigin(ctx, text, "", "") } // PromptEngineResume is Prompt's sibling for SessionManager.triggerResumeLocked's @@ -2466,7 +2466,7 @@ func (s *Session) Prompt(ctx context.Context, text string) (*message.Message, er // or programmatic turn driver (the goal loop's own directive text, notably) // still goes through plain Prompt, unchanged. func (s *Session) PromptEngineResume(ctx context.Context, text string) (*message.Message, error) { - return s.PromptWithOrigin(ctx, text, message.OriginEngine) + return s.PromptWithOrigin(ctx, text, message.OriginEngine, "") } // PromptWithOrigin is Prompt/PromptEngineResume's shared, exported body, @@ -2479,7 +2479,21 @@ func (s *Session) PromptEngineResume(ctx context.Context, text string) (*message // PromptEngineResume's own two-arm choice a second time on top of a choice // the caller already made. One parameterized entry point for the value // means a future third Origin value is handled in exactly one place. -func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string) (*message.Message, error) { +// +// id is the appended user message's own ID: a caller that already holds one +// (server.Server's prompt_async/session.send handlers, forwarding a +// client-supplied or already-resolved ID — see engine.ResolveMessageID) +// passes it through here; every other caller (Prompt, PromptEngineResume, +// and every purely-internal driver — the goal loop's own directive text, +// notably) passes "". Either way id is resolved exactly once, at the mint +// site below, via ResolveMessageID: used verbatim when +// usableClientMessageID accepts it, or replaced with a fresh server mint +// otherwise — id is an identity/reconciliation key only and this resolution +// never fails the prompt. The resolved value never drives ordering: history +// order is (and remains) append order, never a sort on this ID — see +// ResolveMessageID's own doc comment for why a client-minted, time-sortable +// ID must never be trusted for chronology. +func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string, id string) (*message.Message, error) { // A session delegated to the Claude Code CLI (ClaudeCodeProviderFamily // — see engine/claude_code_backend.go) dispatches here, FIRST, before // every check and assembly step below: ContextWindowErr, @@ -2495,7 +2509,7 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri // runAgenticLoop retry call, which never reaches this function at all. if s.claudeCodeDelegated() { s.append(message.Message{ - ID: newID("msg"), + ID: ResolveMessageID(id), Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: text}}, CreatedAt: time.Now().UTC(), @@ -2538,7 +2552,7 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri // compaction never blocks the real turn (see maybeAutoCompact). s.maybeAutoCompact(ctx) s.append(message.Message{ - ID: newID("msg"), + ID: ResolveMessageID(id), Role: message.RoleUser, Parts: message.Parts{&message.Text{Text: text}}, CreatedAt: time.Now().UTC(), diff --git a/engine/goal_retry_dedup_test.go b/engine/goal_retry_dedup_test.go index 22fed378..7a519b27 100644 --- a/engine/goal_retry_dedup_test.go +++ b/engine/goal_retry_dedup_test.go @@ -112,7 +112,7 @@ func (h *denyAndEnqueueHooks) ShellEnv(_ context.Context, _ *plugin.ShellEnvRequ func (h *denyAndEnqueueHooks) ToolExecuteBefore(_ context.Context, _ *plugin.ToolExecuteBeforeRequest) (json.RawMessage, string) { if !h.enqueued { h.enqueued = true - if _, err := h.s.EnqueuePrompt(h.text); err != nil { + if _, _, err := h.s.EnqueuePrompt(h.text, ""); err != nil { panic("denyAndEnqueueHooks: EnqueuePrompt failed: " + err.Error()) } } @@ -265,7 +265,7 @@ func TestPursueGoalDeterministicParkKeepsEmbeddedOperatorMessage(t *testing.T) { prov := &goalProvider{failWorker: workerErr} s := goalSession(t, prov, t.TempDir()) - if _, err := s.EnqueuePrompt("urgent operator directive"); err != nil { + if _, _, err := s.EnqueuePrompt("urgent operator directive", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } @@ -301,7 +301,7 @@ func TestPursueGoalRetryableBudgetExhaustedParkKeepsEmbeddedOperatorMessage(t *t } s := goalSession(t, prov, t.TempDir()) - if _, err := s.EnqueuePrompt("urgent operator directive"); err != nil { + if _, _, err := s.EnqueuePrompt("urgent operator directive", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } @@ -333,7 +333,7 @@ func TestPursueGoalStreamTruncatedParkKeepsEmbeddedOperatorMessage(t *testing.T) } s := goalSession(t, prov, t.TempDir()) - if _, err := s.EnqueuePrompt("urgent operator directive"); err != nil { + if _, _, err := s.EnqueuePrompt("urgent operator directive", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } diff --git a/engine/goal_retry_reuse_test.go b/engine/goal_retry_reuse_test.go index f47d2c75..00f2ee2c 100644 --- a/engine/goal_retry_reuse_test.go +++ b/engine/goal_retry_reuse_test.go @@ -217,7 +217,7 @@ func TestPursueGoalRetryReuseNeverLosesEmbeddedOperatorMessage(t *testing.T) { // Enqueued BEFORE PursueGoal starts, so turn 1's OWN turn-boundary // drain (before promptTurnWithRetry ever runs) bakes it into turn // 1's directive string — not a separately-appended message. - if _, err := s.EnqueuePrompt("urgent operator directive"); err != nil { + if _, _, err := s.EnqueuePrompt("urgent operator directive", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } diff --git a/engine/goal_toolcall_boundary_test.go b/engine/goal_toolcall_boundary_test.go index 53f36852..5cc12895 100644 --- a/engine/goal_toolcall_boundary_test.go +++ b/engine/goal_toolcall_boundary_test.go @@ -55,7 +55,7 @@ func TestGoalWorkerTurnInheritsMidTurnInjection(t *testing.T) { <-entered // turn 1's tool call is genuinely executing - if _, err := s.EnqueuePrompt("operator mid worker tool"); err != nil { + if _, _, err := s.EnqueuePrompt("operator mid worker tool", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } close(release) diff --git a/engine/goal_update_test.go b/engine/goal_update_test.go index ba325c2d..a62faa7d 100644 --- a/engine/goal_update_test.go +++ b/engine/goal_update_test.go @@ -828,10 +828,10 @@ func TestGoalInjectsQueuedPromptsAtBoundary(t *testing.T) { <-entered // turn 1's worker call is genuinely in flight - if _, err := s.EnqueuePrompt("first operator message"); err != nil { + if _, _, err := s.EnqueuePrompt("first operator message", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } - if _, err := s.EnqueuePrompt("second operator message"); err != nil { + if _, _, err := s.EnqueuePrompt("second operator message", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } @@ -920,7 +920,7 @@ func TestGoalInjectionSurvivesConditionUpdate(t *testing.T) { if err := s.UpdateGoal("the NEW condition"); err != nil { t.Fatalf("UpdateGoal = %v", err) } - if _, err := s.EnqueuePrompt("operator says hi"); err != nil { + if _, _, err := s.EnqueuePrompt("operator says hi", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } } @@ -983,7 +983,7 @@ func TestInjectedPromptsNotRedeliveredAfterStaleDiscard(t *testing.T) { if err := s.RegisterGoal("original condition"); err != nil { t.Fatal(err) } - if _, err := s.EnqueuePrompt("do not lose me"); err != nil { + if _, _, err := s.EnqueuePrompt("do not lose me", ""); err != nil { t.Fatal(err) } diff --git a/engine/id.go b/engine/id.go index cf2eaef9..136366de 100644 --- a/engine/id.go +++ b/engine/id.go @@ -1,6 +1,11 @@ package engine -import "github.com/majorcontext/harness/typeid" +import ( + "strings" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/typeid" +) // legacyHexLen is the length, in characters, of the random suffix in a // pre-TypeID session ID: "ses_" + 16 lowercase hex digits (8 bytes of @@ -40,6 +45,54 @@ func ValidSessionID(id string) bool { return err == nil && tid.Prefix() == "ses" } +// usableClientMessageID reports whether id is safe to use verbatim as a +// user message's ID: non-empty, and not one of the reserved provenance +// prefixes engine itself mints for a DIFFERENT kind of synthetic message — +// compactionSummaryIDTag ("cmpsum", compact.go) for a compaction summary, or +// message.SyntheticOrphanIDPrefix ("synthetic-orphan-tool-result-") for a +// synthesized orphaned tool result. Both prefixes are load-bearing markers +// elsewhere in this package (isCompactionSummaryID, +// message.IsSyntheticOrphanID) that assume only engine itself ever mints an +// ID with that shape; a client-supplied ID colliding with one would make a +// genuine user message indistinguishable from that synthetic kind. +// +// This is cheap insurance against an accidental collision, not attacker +// defense: PromptWithOrigin's caller (server.Server, this repo's only HTTP +// surface for it) is reached exclusively by a trusted, authenticated +// first-party client, so an id that fails this check is silently ignored +// in favor of a fresh server-minted one — see ResolveMessageID — never a +// rejected request. +func usableClientMessageID(id string) bool { + if id == "" { + return false + } + if strings.HasPrefix(id, compactionSummaryIDTag) { + return false + } + if strings.HasPrefix(id, message.SyntheticOrphanIDPrefix) { + return false + } + return true +} + +// ResolveMessageID returns id unchanged when usableClientMessageID accepts +// it, or mints a fresh "msg" TypeID otherwise. This is the exact rule +// PromptWithOrigin applies at each of its two mint sites (the claude-code- +// delegated and native branches) to the id a caller supplies for the +// appended user message; it is exported so a caller that must know a +// prompt's resolved message ID before the turn actually runs — a queued +// prompt's synchronous accept response, notably, dispatched into +// PromptWithOrigin only later, asynchronously, once its turn is drained — +// can compute the SAME value PromptWithOrigin will use, once, without +// duplicating the reserved-prefix rule or risking a second, different mint +// for the same logical prompt. +func ResolveMessageID(id string) string { + if usableClientMessageID(id) { + return id + } + return newID("msg") +} + // isLegacyHexID reports whether id is prefix + "_" + exactly legacyHexLen // lowercase hex digits. func isLegacyHexID(id, prefix string) bool { diff --git a/engine/index_test.go b/engine/index_test.go index 88609b6a..2d6806f7 100644 --- a/engine/index_test.go +++ b/engine/index_test.go @@ -211,10 +211,10 @@ func TestSessionIndexMatchesLoadSession(t *testing.T) { name: "prompt queue", turns: [][]provider.Event{}, drive: func(t *testing.T, s *Session) { - if _, err := s.EnqueuePrompt("first"); err != nil { + if _, _, err := s.EnqueuePrompt("first", ""); err != nil { t.Fatal(err) } - if _, err := s.EnqueuePrompt("second"); err != nil { + if _, _, err := s.EnqueuePrompt("second", ""); err != nil { t.Fatal(err) } if _, _, err := s.EnqueuePromptDurable("third", 1); err != nil { diff --git a/engine/max_tokens_continue_test.go b/engine/max_tokens_continue_test.go index c0d22ad8..e9d86e12 100644 --- a/engine/max_tokens_continue_test.go +++ b/engine/max_tokens_continue_test.go @@ -488,7 +488,7 @@ func TestMaxTokensContinuationDrainsQueuedPrompt(t *testing.T) { Model: message.ModelRef{Provider: "test", Model: "m1"}, MaxTokensContinuations: 3, }) - if _, err := s.EnqueuePrompt("steer now"); err != nil { + if _, _, err := s.EnqueuePrompt("steer now", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } diff --git a/engine/message_id_test.go b/engine/message_id_test.go new file mode 100644 index 00000000..337e0e72 --- /dev/null +++ b/engine/message_id_test.go @@ -0,0 +1,59 @@ +package engine + +import ( + "strings" + "testing" + + "github.com/majorcontext/harness/message" +) + +// TestUsableClientMessageID is the RED test for the reserved-prefix guard: +// a client-supplied user-message ID is usable verbatim unless it is empty +// or begins with one of engine's own reserved provenance prefixes for a +// DIFFERENT synthetic message kind (a compaction summary or a synthesized +// orphaned tool result). +func TestUsableClientMessageID(t *testing.T) { + cases := []struct { + id string + want bool + }{ + {"", false}, + {"cmpsum_01abc", false}, + {compactionSummaryIDTag, false}, + {message.SyntheticOrphanIDPrefix + "3-toolcall1", false}, + {"msg_client_supplied", true}, + {"anything-a-trusted-client-picks", true}, + {"cmp", true}, // shares no prefix boundary with "cmpsum" + } + for _, c := range cases { + if got := usableClientMessageID(c.id); got != c.want { + t.Errorf("usableClientMessageID(%q) = %v, want %v", c.id, got, c.want) + } + } +} + +// TestResolveMessageIDUsesSuppliedIDVerbatim is the RED test for +// ResolveMessageID's happy path: a usable client-supplied ID passes +// through unchanged, never replaced by a server mint. +func TestResolveMessageIDUsesSuppliedIDVerbatim(t *testing.T) { + const supplied = "console-optimistic-id-1" + got := ResolveMessageID(supplied) + if got != supplied { + t.Fatalf("ResolveMessageID(%q) = %q, want the supplied id unchanged", supplied, got) + } +} + +// TestResolveMessageIDMintsOnReservedOrEmpty covers both fail-safe cases: +// an empty id and a reserved-prefix id must NEVER be used verbatim — a +// fresh "msg" TypeID is minted instead, and the prompt is never rejected. +func TestResolveMessageIDMintsOnReservedOrEmpty(t *testing.T) { + for _, supplied := range []string{"", "cmpsum_hijack", message.SyntheticOrphanIDPrefix + "0-x"} { + got := ResolveMessageID(supplied) + if got == supplied { + t.Fatalf("ResolveMessageID(%q) = %q, want a freshly minted id, not the supplied value", supplied, got) + } + if !strings.HasPrefix(got, "msg_") { + t.Fatalf("ResolveMessageID(%q) = %q, want a msg_-prefixed minted id", supplied, got) + } + } +} diff --git a/engine/queue.go b/engine/queue.go index f301c74b..e25c5652 100644 --- a/engine/queue.go +++ b/engine/queue.go @@ -35,6 +35,16 @@ type QueuedPrompt struct { // via EnqueuePromptDurable (see store.go's promptRecord.Seq); 0 for a // plain EnqueuePrompt, which has no idempotency contract. Seq int64 + // MessageID is the ID the user message this prompt eventually becomes + // will carry — already resolved (see ResolveMessageID) by EnqueuePrompt + // at enqueue time, so it is stable and known before this prompt is ever + // dispatched: a caller reporting a synchronous "queued" response can + // promise the exact ID PromptWithOrigin will use later, at drain time, + // with no risk of a second, different mint for the same prompt. Empty + // on a record folded from an older session log written before this + // field existed — PromptWithOrigin's own mint site resolves that case + // exactly like any other unset id, at dispatch time. + MessageID string } // promptQueueFold replays prompt.queued/prompt.dequeued records into the @@ -96,7 +106,7 @@ type promptQueueFold struct { // let a malformed record's ID move the counter, which is exactly what the // guard rejects it for. func (f *promptQueueFold) queued(p promptRecord) { - q := QueuedPrompt{ID: p.ID, Text: p.Text, Seq: p.Seq} + q := QueuedPrompt{ID: p.ID, Text: p.Text, Seq: p.Seq, MessageID: p.MessageID} valid := q.ID > 0 for _, existing := range f.queue { if existing.ID == q.ID { @@ -164,23 +174,33 @@ var ErrEmptyPromptText = errors.New("engine: prompt text must not be empty or wh // or whitespace-only, matching RegisterGoal's non-empty-condition rule. The // stored/emitted text is trimmed, same as a goal condition. // +// messageID is the caller's own (possibly empty, possibly client-supplied) +// message ID for the prompt; EnqueuePrompt resolves it exactly once, via +// ResolveMessageID, and both persists and returns that SAME resolved value +// — never resolved a second time at dispatch, which would risk minting a +// DIFFERENT fresh ID than whatever this call's own return value already +// promised a caller reporting a synchronous "queued" response. Pass "" for +// a caller with no client ID of its own (a server-minted ID is what +// PromptWithOrigin would have chosen anyway). +// // The enqueued prompt does not touch s.history and is not visible to any // provider request started before it is actually delivered (see // DequeuePrompt/dequeueAllLocked) — see the package doc comment. -func (s *Session) EnqueuePrompt(text string) (int64, error) { +func (s *Session) EnqueuePrompt(text string, messageID string) (id int64, resolvedMessageID string, err error) { trimmed := strings.TrimSpace(text) if trimmed == "" { - return 0, ErrEmptyPromptText + return 0, "", ErrEmptyPromptText } + resolved := ResolveMessageID(messageID) s.mu.Lock() - p := s.enqueueMemoryOnlyLocked(trimmed) - s.persistPromptQueueLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text}) + p := s.enqueueMemoryOnlyLocked(trimmed, resolved) + s.persistPromptQueueLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text, MessageID: p.MessageID}) // Emit while still holding s.mu (see ClearGoal in goal.go): keeps event // order matching log order under a concurrent dequeue. OnEvent must not // call back into this Session — that would deadlock on s.mu, held here. s.emit(Event{Type: EventPromptQueued, QueueID: p.ID, QueueText: p.Text, QueueLen: len(s.promptQueue)}) s.mu.Unlock() - return p.ID, nil + return p.ID, p.MessageID, nil } // enqueueMemoryOnlyLocked is EnqueuePrompt's memory-only half: assigns @@ -209,12 +229,15 @@ func (s *Session) EnqueuePrompt(text string) (int64, error) { // // text is assumed already validated non-empty and trimmed — the one // other caller (SendToDescendant) applies the same validation -// EnqueuePrompt does above, on its own copy of the text. Caller holds +// EnqueuePrompt does above, on its own copy of the text. messageID is +// stored as given — already resolved by EnqueuePrompt's own caller, or "" +// for a caller (SendToDescendant) with no client message ID of its own, +// left for PromptWithOrigin to resolve at dispatch time. Caller holds // s.mu. -func (s *Session) enqueueMemoryOnlyLocked(text string) QueuedPrompt { +func (s *Session) enqueueMemoryOnlyLocked(text string, messageID string) QueuedPrompt { id := s.promptQueueNextID s.promptQueueNextID++ - p := QueuedPrompt{ID: id, Text: text} + p := QueuedPrompt{ID: id, Text: text, MessageID: messageID} s.promptQueue = append(s.promptQueue, p) return p } diff --git a/engine/queue_persist_order_test.go b/engine/queue_persist_order_test.go index a6e86826..3115b5aa 100644 --- a/engine/queue_persist_order_test.go +++ b/engine/queue_persist_order_test.go @@ -132,7 +132,7 @@ func TestEnqueuePromptDurableDrainsParkedRecordsFirst(t *testing.T) { // SendToDescendant's running-target branch, memory half: item A is in // the queue, its durable record parked, its flush not run yet. s.mu.Lock() - a := s.enqueueMemoryOnlyLocked("message A") + a := s.enqueueMemoryOnlyLocked("message A", "") s.queueRecordDeferredLocked(recPromptQueued, promptRecord{ID: a.ID, Text: a.Text}, Event{Type: EventPromptQueued, QueueID: a.ID, QueueText: a.Text, QueueLen: 1}) s.mu.Unlock() @@ -187,7 +187,7 @@ func TestDeferredQueuedRecordStillPrecedesItsDequeuedRecord(t *testing.T) { // SendToDescendant's running-target branch, memory half: append and // park the durable record under ONE s.mu hold, no disk write. s.mu.Lock() - p := s.enqueueMemoryOnlyLocked("message A") + p := s.enqueueMemoryOnlyLocked("message A", "") s.queueRecordDeferredLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text}, Event{Type: EventPromptQueued, QueueID: p.ID, QueueText: p.Text, QueueLen: 1}) s.mu.Unlock() diff --git a/engine/queue_replay_model_test.go b/engine/queue_replay_model_test.go index fcc2d7d3..07e69627 100644 --- a/engine/queue_replay_model_test.go +++ b/engine/queue_replay_model_test.go @@ -205,7 +205,7 @@ func (m *queueModel) DurableEnqueue(t *rapid.T) { // watermark movement), which shares the same folded queue and ID space. func (m *queueModel) PlainEnqueue(t *rapid.T) { text := fmt.Sprintf("plain-%d", rapid.IntRange(0, 1<<20).Draw(t, "textSeed")) - if _, err := m.s.EnqueuePrompt(text); err != nil { + if _, _, err := m.s.EnqueuePrompt(text, ""); err != nil { t.Fatalf("EnqueuePrompt: %v", err) } } diff --git a/engine/queue_test.go b/engine/queue_test.go index 8ef63ae9..a2d9a4b9 100644 --- a/engine/queue_test.go +++ b/engine/queue_test.go @@ -17,7 +17,7 @@ func TestEnqueuePromptPersistsAndEmits(t *testing.T) { var evs []Event s.cfg.OnEvent = func(ev Event) { evs = append(evs, ev) } - id, err := s.EnqueuePrompt("do the thing") + id, _, err := s.EnqueuePrompt("do the thing", "") if err != nil { t.Fatalf("EnqueuePrompt = %v", err) } @@ -61,7 +61,7 @@ func TestEnqueuePromptPersistsAndEmits(t *testing.T) { func TestEnqueueRejectsEmpty(t *testing.T) { s := NewSession(Config{}) - if _, err := s.EnqueuePrompt(" \n\t "); err == nil { + if _, _, err := s.EnqueuePrompt(" \n\t ", ""); err == nil { t.Fatal("EnqueuePrompt with whitespace-only text should error") } if len(s.QueuedPrompts()) != 0 { @@ -75,11 +75,11 @@ func TestDequeueFIFOAndJournalsReason(t *testing.T) { var evs []Event s.cfg.OnEvent = func(ev Event) { evs = append(evs, ev) } - id1, err := s.EnqueuePrompt("first") + id1, _, err := s.EnqueuePrompt("first", "") if err != nil { t.Fatal(err) } - id2, err := s.EnqueuePrompt("second") + id2, _, err := s.EnqueuePrompt("second", "") if err != nil { t.Fatal(err) } @@ -154,7 +154,7 @@ func TestQueuedPromptsAbsentFromHistory(t *testing.T) { System: []string{"base"}, }) - if _, err := s.EnqueuePrompt("queued text should never appear"); err != nil { + if _, _, err := s.EnqueuePrompt("queued text should never appear", ""); err != nil { t.Fatal(err) } if h := s.History(); len(h) != 0 { @@ -205,14 +205,14 @@ func TestLoadSessionRefoldsQueue(t *testing.T) { dir := t.TempDir() s := NewSession(Config{SessionDir: dir}) - if _, err := s.EnqueuePrompt("a"); err != nil { + if _, _, err := s.EnqueuePrompt("a", ""); err != nil { t.Fatal(err) } - id2, err := s.EnqueuePrompt("b") + id2, _, err := s.EnqueuePrompt("b", "") if err != nil { t.Fatal(err) } - id3, err := s.EnqueuePrompt("c") + id3, _, err := s.EnqueuePrompt("c", "") if err != nil { t.Fatal(err) } @@ -241,7 +241,7 @@ func TestLoadSessionRefoldsQueue(t *testing.T) { // The next-ID counter must continue past the highest folded ID, never // reissuing an ID already used on disk. - id4, err := loaded.EnqueuePrompt("d") + id4, _, err := loaded.EnqueuePrompt("d", "") if err != nil { t.Fatal(err) } diff --git a/engine/queue_toolcall_boundary_test.go b/engine/queue_toolcall_boundary_test.go index ff0727f8..559e95e2 100644 --- a/engine/queue_toolcall_boundary_test.go +++ b/engine/queue_toolcall_boundary_test.go @@ -76,7 +76,7 @@ func TestMidTurnInjectionAtToolBoundary(t *testing.T) { <-entered // the tool call is genuinely executing - if _, err := s.EnqueuePrompt("operator says hi mid-turn"); err != nil { + if _, _, err := s.EnqueuePrompt("operator says hi mid-turn", ""); err != nil { t.Fatalf("EnqueuePrompt = %v", err) } @@ -170,7 +170,7 @@ func TestNoToolCallTurnLeavesQueueForTail(t *testing.T) { System: []string{"base"}, }) - if _, err := s.EnqueuePrompt("still waiting"); err != nil { + if _, _, err := s.EnqueuePrompt("still waiting", ""); err != nil { t.Fatal(err) } diff --git a/engine/session_manager.go b/engine/session_manager.go index 29347408..c632d293 100644 --- a/engine/session_manager.go +++ b/engine/session_manager.go @@ -3689,8 +3689,12 @@ func (m *SessionManager) SendToDescendant(callerID, targetID, text string) (queu // then resurrected the delivered prompt. s := n.session s.mu.Lock() - p := s.enqueueMemoryOnlyLocked(text) - s.queueRecordDeferredLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text}, + // "": SendToDescendant carries no client message ID of its own (a + // task-tool descendant delivery, not an HTTP prompt/send caller) — + // PromptWithOrigin's own mint site resolves it at dispatch time, + // exactly like a pre-this-feature session log record would. + p := s.enqueueMemoryOnlyLocked(text, "") + s.queueRecordDeferredLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text, MessageID: p.MessageID}, Event{Type: EventPromptQueued, QueueID: p.ID, QueueText: p.Text, QueueLen: len(s.promptQueue)}) s.mu.Unlock() m.deferQueueRecordFlush(s) diff --git a/engine/session_replay_model_test.go b/engine/session_replay_model_test.go index eb906071..e74b31d1 100644 --- a/engine/session_replay_model_test.go +++ b/engine/session_replay_model_test.go @@ -263,7 +263,7 @@ func (m *sessionModel) DurableEnqueue(t *rapid.T) { // PlainEnqueue mirrors queueModel.PlainEnqueue. func (m *sessionModel) PlainEnqueue(t *rapid.T) { text := fmt.Sprintf("plain-%d", rapid.IntRange(0, 1<<20).Draw(t, "textSeed")) - if _, err := m.s.EnqueuePrompt(text); err != nil { + if _, _, err := m.s.EnqueuePrompt(text, ""); err != nil { t.Fatalf("EnqueuePrompt: %v", err) } } diff --git a/engine/snapshot_test.go b/engine/snapshot_test.go index 7faf541a..4c44ebac 100644 --- a/engine/snapshot_test.go +++ b/engine/snapshot_test.go @@ -225,7 +225,7 @@ func TestSnapshotCarriesEveryFoldedField(t *testing.T) { if err := s.RegisterGoal("ship it"); err != nil { t.Fatalf("RegisterGoal: %v", err) } - if _, err := s.EnqueuePrompt("queued one"); err != nil { + if _, _, err := s.EnqueuePrompt("queued one", ""); err != nil { t.Fatalf("EnqueuePrompt: %v", err) } if _, _, err := s.EnqueuePromptDurable("queued two", 7); err != nil { diff --git a/engine/store.go b/engine/store.go index 5b4a2512..ac10c630 100644 --- a/engine/store.go +++ b/engine/store.go @@ -484,6 +484,14 @@ type promptRecord struct { // the recPromptQueued replay case for why that heals torn fsync // failures. Seq int64 `json:"seq,omitempty"` + // MessageID is the resolved ID (see ResolveMessageID) the queued + // prompt's eventual user message will carry — set on prompt.queued, + // carried through to QueuedPrompt.MessageID by promptQueueFold.queued + // on replay. Omitted (empty) on a record written before this field + // existed; PromptWithOrigin's own mint site resolves that case exactly + // like any other unset id, at dispatch time — a backward-compatible + // fallback, not a replay error. + MessageID string `json:"message_id,omitempty"` } // taskSpawnRecord is a recTaskSpawned record's payload — see that diff --git a/engine/store_repair_test.go b/engine/store_repair_test.go index 58382360..187a6f8e 100644 --- a/engine/store_repair_test.go +++ b/engine/store_repair_test.go @@ -108,7 +108,7 @@ func TestPlainEnqueueRepairsTornTailAfterValidRecords(t *testing.T) { } // The first write after load is where the repair actually fires. - id2, err := s.EnqueuePrompt("second") + id2, _, err := s.EnqueuePrompt("second", "") if err != nil { t.Fatalf("EnqueuePrompt: %v", err) } @@ -120,7 +120,7 @@ func TestPlainEnqueueRepairsTornTailAfterValidRecords(t *testing.T) { if err != nil { t.Fatalf("first reload: %v", err) } - id3, err := reloaded.EnqueuePrompt("third") + id3, _, err := reloaded.EnqueuePrompt("third", "") if err != nil { t.Fatalf("EnqueuePrompt on reloaded session: %v", err) } diff --git a/engine/task_external_turn_test.go b/engine/task_external_turn_test.go index 2b285166..729fac34 100644 --- a/engine/task_external_turn_test.go +++ b/engine/task_external_turn_test.go @@ -47,7 +47,7 @@ func TestReportTurnEndDoesNotReDriveQueuedPrompt(t *testing.T) { t.Fatal("Session: child not found") } for _, text := range []string{"message A", "message B"} { - if _, err := child.EnqueuePrompt(text); err != nil { + if _, _, err := child.EnqueuePrompt(text, ""); err != nil { t.Fatalf("EnqueuePrompt %q: %v", text, err) } } diff --git a/server/handlers.go b/server/handlers.go index 1082fa6c..d231c96b 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -1609,6 +1609,13 @@ type promptAsyncResponse struct { Seq int64 `json:"seq"` Status string `json:"status"` Queued int `json:"queued,omitempty"` + // MessageID is the ID this request's own prompt was actually recorded + // under — the caller's own `id` echoed back verbatim, or a freshly + // minted one when `id` was empty or a reserved-prefix collision (see + // engine.ResolveMessageID) — so a caller that pre-minted an id for its + // own optimistic render can confirm which id to reconcile against, + // whether this prompt started immediately or is still queued. + MessageID string `json:"message_id"` } // handlePrompt is POST /session/{id}/prompt_async (see docs/plans/2026-07-19- @@ -1636,6 +1643,15 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { Text string `json:"text"` } `json:"parts"` Model message.ModelRef `json:"model"` + // ID is an OPTIONAL client-minted ID for the user message this + // prompt becomes — the console pre-mints one to render its own + // optimistic bubble, then reconciles it by ID once the real message + // arrives over SSE, rather than by fragile text-matching. This + // server is reached only by a trusted, authenticated first-party + // caller, so ID is used verbatim with exactly one fail-safe guard + // (see msgID/engine.ResolveMessageID below) — never validated for + // uniqueness and never a reason to reject the prompt. + ID string `json:"id"` } if err := decodeBody(r, &body); err != nil { writeErr(w, http.StatusBadRequest, err.Error()) @@ -1654,6 +1670,15 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { texts = append(texts, p.Text) } text := strings.Join(texts, "\n") + // Resolved ONCE, here, regardless of which branch below actually ends + // up delivering this prompt (immediate dispatch, or enqueued behind a + // busy/non-empty queue): every branch reports this SAME value as its + // response's message_id, and threads it through to whichever call + // (EnqueuePrompt, or runPrompt directly) actually appends the user + // message — so the caller's response is never a promise that a LATER, + // second, differently-minted id ends up in the transcript instead. See + // engine.ResolveMessageID's own doc comment. + msgID := engine.ResolveMessageID(body.ID) // Resolve the session and atomically claim its prompt slot (also does the // wg.Add under the admission gate). See claimForPrompt for the ordering that @@ -1665,7 +1690,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { writeErr(w, code, fmt.Sprintf("workdir busy: held by session %s", holder)) case code == http.StatusConflict: // Same-session busy: queue-on-busy (invariant 9), not a 409. - s.enqueueOrDispatch(w, id, text) + s.enqueueOrDispatch(w, id, text, msgID) case code == http.StatusServiceUnavailable: writeErr(w, code, "server shutting down") default: @@ -1686,7 +1711,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // text) into the run slot just claimed above. See // dispatchQueueHead and enqueueOrDispatch's identical shape for the // same-session-BUSY counterpart of this same rule. - ourID, err := st.sess.EnqueuePrompt(text) + ourID, _, err := st.sess.EnqueuePrompt(text, msgID) if err != nil { // handlePrompt already rejects an empty parts list and joins // non-empty text above, so this is not reachable in practice; @@ -1722,7 +1747,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // promptAsyncResponse's queued field doc for why depth 0 is // possible here. writeJSON(w, http.StatusAccepted, promptAsyncResponse{ - Seq: fromSeq, Status: "queued", Queued: len(st.sess.QueuedPrompts()), + Seq: fromSeq, Status: "queued", Queued: len(st.sess.QueuedPrompts()), MessageID: msgID, }) return } @@ -1730,7 +1755,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { if head.ID == ourID { status = "started" } - resp := promptAsyncResponse{Seq: fromSeq, Status: status} + resp := promptAsyncResponse{Seq: fromSeq, Status: status, MessageID: msgID} if status == "queued" { // remaining, not a fresh QueuedPrompts() re-read — see // dispatchQueueHead's own doc comment for the race that used @@ -1785,8 +1810,8 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { s.emitDurable(Event{Type: evtSessionStatus, SessionID: id, Status: "busy"}) - go s.runPrompt(ctx, id, st, text, "") - writeJSON(w, http.StatusAccepted, promptAsyncResponse{Seq: fromSeq, Status: "started"}) + go s.runPrompt(ctx, id, st, text, "", msgID) + writeJSON(w, http.StatusAccepted, promptAsyncResponse{Seq: fromSeq, Status: "started", MessageID: msgID}) } // enqueueOrDispatch implements handlePrompt's same-session-busy branch: @@ -1826,7 +1851,13 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // slot to carry a per-prompt model override through to a future drain. A // caller that needs a model swap to take effect should re-issue it once its // prompt is confirmed "started". -func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string) { +// +// msgID is handlePrompt's own already-resolved message id (see +// engine.ResolveMessageID) for text — resolved exactly once, before either +// of handlePrompt's two branches runs, so this function's own response +// promises the SAME id EnqueuePrompt persists and PromptWithOrigin later +// uses at dispatch. +func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string, msgID string) { sess := s.residentSession(id) if sess == nil { // Benign race window, identical to handleGoalBusy's (see its doc @@ -1838,7 +1869,7 @@ func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string writeErr(w, http.StatusConflict, "session is busy with another prompt") return } - ourID, err := sess.EnqueuePrompt(text) + ourID, _, err := sess.EnqueuePrompt(text, msgID) if err != nil { // handlePrompt already rejects an empty parts list and joins // non-empty text, so this is not reachable in practice; fail closed @@ -1856,7 +1887,7 @@ func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string // Lost the retry: still queued, whatever already occupies the slot // keeps running undisturbed. writeJSON(w, http.StatusAccepted, promptAsyncResponse{ - Seq: s.currentSeq(), Status: "queued", Queued: len(sess.QueuedPrompts()), + Seq: s.currentSeq(), Status: "queued", Queued: len(sess.QueuedPrompts()), MessageID: msgID, }) return } @@ -1876,7 +1907,7 @@ func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string // rather than a 500, which would misrepresent a benign, documented // race as a server bug. See TestQueueClearRaceDuringDispatchIsNotAnError. writeJSON(w, http.StatusAccepted, promptAsyncResponse{ - Seq: s.currentSeq(), Status: "queued", Queued: len(sess.QueuedPrompts()), + Seq: s.currentSeq(), Status: "queued", Queued: len(sess.QueuedPrompts()), MessageID: msgID, }) return } @@ -1885,7 +1916,7 @@ func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string if head.ID == ourID { status = "started" } - resp := promptAsyncResponse{Seq: s.currentSeq(), Status: status} + resp := promptAsyncResponse{Seq: s.currentSeq(), Status: status, MessageID: msgID} if status == "queued" { // remaining, not a fresh QueuedPrompts() re-read — see // dispatchQueueHead's own doc comment for the race that used to @@ -2225,7 +2256,7 @@ func (s *Server) dispatchQueueHead(id string, st *sessionState, ctx context.Cont // deliberately dispatches the QUEUE HEAD instead of its own trigger // text, exactly so a real queued message is never displaced by the // resume trigger — see runOrQueueText's own doc comment. - go s.runPrompt(ctx, id, st, head.Text, "") + go s.runPrompt(ctx, id, st, head.Text, "", head.MessageID) if s.dispatchQueueHeadRace != nil { // Test-only seam — see its own doc comment (server.go). s.dispatchQueueHeadRace() @@ -2250,7 +2281,14 @@ func (s *Server) dispatchQueueHead(id string, st *sessionState, ctx context.Cont // (dispatchQueueHead, whether or not the drain was itself provoked by a // resume trigger — see that function's own doc comment), or a session.send // delivery (sendTextToRoot). -func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, text string, origin string) { +// +// msgID is likewise forwarded to PromptWithOrigin verbatim: the caller's +// own already-resolved message id (handlePrompt's msgID, or a dequeued +// QueuedPrompt.MessageID) for an ordinary or queued prompt, or "" for +// runOrQueueText's synthetic resume trigger, which has no client message id +// of its own — PromptWithOrigin's own mint site resolves either case +// identically. +func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, text string, origin string, msgID string) { defer s.wg.Done() // ReportTurnStart/ReportTurnEnd bracket the ONE choke point every // ordinary (non-goal-loop) turn on a resident session funnels through @@ -2265,7 +2303,7 @@ func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, tex // hits, closing the "task tool broken after restart" gap a live // review caught. s.sessMgr.ReportTurnStart(st.sess) - msg, err := st.sess.PromptWithOrigin(ctx, text, origin) + msg, err := st.sess.PromptWithOrigin(ctx, text, origin, msgID) s.syncMessages(id) // catch any message not yet journaled switch { case err == nil: diff --git a/server/openapi.yaml b/server/openapi.yaml index 4624fefd..aedc07c9 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -888,11 +888,28 @@ components: prompt, before it could be dispatched. That is not an error — the prompt was durably accepted and journaled, it simply never ran — so this case is still a 202, never a 500. + message_id: + type: string + description: > + The id of the user message this prompt created — the caller's own + `id` when it supplied a usable one, otherwise the server-minted + `msg_...` id. Echoed so a client that pre-minted the id can confirm + it, and so a client that did not can learn it. The message also + arrives on /event carrying this same id. PromptRequest: type: object required: [parts] properties: + id: + type: string + description: > + Optional client-supplied id for the user message this prompt + creates, so a client can pre-mint the id and reconcile its + optimistic render by id. Used verbatim when usable; a reserved + prefix or an empty value is ignored and the server mints a fresh + id (reported back as `message_id`). Never drives ordering — the + server journal sequence does. parts: type: array minItems: 1 diff --git a/server/prompt_message_id_test.go b/server/prompt_message_id_test.go new file mode 100644 index 00000000..aaa18062 --- /dev/null +++ b/server/prompt_message_id_test.go @@ -0,0 +1,252 @@ +package server + +import ( + "encoding/json" + "net/http" + "strings" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// userMessages filters GET /session/{id}/message's transcript down to the +// RoleUser entries, in order — the oracle every test in this file checks a +// prompt's resolved id against. +func (h *harness) userMessages(id string) []message.Message { + h.t.Helper() + resp, data := h.do("GET", "/session/"+id+"/message", nil) + if resp.StatusCode != http.StatusOK { + h.t.Fatalf("GET message status %d: %s", resp.StatusCode, data) + } + var all []message.Message + if err := json.Unmarshal(data, &all); err != nil { + h.t.Fatalf("unmarshal transcript: %v (%s)", err, data) + } + var users []message.Message + for _, m := range all { + if m.Role == message.RoleUser { + users = append(users, m) + } + } + return users +} + +// TestPromptAsyncUsesSuppliedMessageID is the RED test for the feature's +// core promise: a caller-supplied `id` on POST /session/{id}/prompt_async +// is used VERBATIM as the resulting user message's own ID — both in the +// synchronous response's message_id field and in the durable transcript — +// never replaced by a server mint. +func TestPromptAsyncUsesSuppliedMessageID(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("done")}} + h := newHarness(t, prov) + id := h.createSession("test/m1") + + const supplied = "console-optimistic-1" + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "hello"}}, + "id": supplied, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + var pr promptAsyncResponse + if err := json.Unmarshal(data, &pr); err != nil { + t.Fatal(err) + } + if pr.MessageID != supplied { + t.Fatalf("response message_id = %q, want the supplied id %q", pr.MessageID, supplied) + } + + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1: %+v", len(users), users) + } + if users[0].ID != supplied { + t.Fatalf("transcript user message id = %q, want the supplied id %q", users[0].ID, supplied) + } +} + +// TestPromptAsyncMintsOnReservedOrEmptyMessageID is the fail-safe-guard +// test: an empty id, or one beginning with a reserved provenance prefix +// (cmpsum_ or message.SyntheticOrphanIDPrefix), must NEVER be used +// verbatim — the server mints a fresh msg_-prefixed id instead, the +// response and transcript agree on that same minted value, and the +// prompt still succeeds (never rejected). +func TestPromptAsyncMintsOnReservedOrEmptyMessageID(t *testing.T) { + cases := []struct { + name string + id string + explicit bool // whether to include "id" in the JSON body at all + }{ + {name: "empty_id_field_present", id: "", explicit: true}, + {name: "reserved_compaction_prefix", id: "cmpsum_hijack", explicit: true}, + {name: "reserved_synthetic_orphan_prefix", id: message.SyntheticOrphanIDPrefix + "0-x", explicit: true}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("done")}} + h := newHarness(t, prov) + id := h.createSession("test/m1") + + body := map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "hello"}}, + } + if c.explicit { + body["id"] = c.id + } + resp, data := h.do("POST", "/session/"+id+"/prompt_async", body) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + var pr promptAsyncResponse + if err := json.Unmarshal(data, &pr); err != nil { + t.Fatal(err) + } + if pr.MessageID == c.id { + t.Fatalf("response message_id = %q, want a freshly minted id, not the rejected supplied value %q", pr.MessageID, c.id) + } + if !strings.HasPrefix(pr.MessageID, "msg_") { + t.Fatalf("response message_id = %q, want a msg_-prefixed minted id", pr.MessageID) + } + + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1: %+v", len(users), users) + } + if users[0].ID != pr.MessageID { + t.Fatalf("transcript user message id = %q, want it to match the response's minted message_id %q", users[0].ID, pr.MessageID) + } + }) + } +} + +// TestPromptAsyncNoSuppliedIDMintsLikeBefore is the backward-compat test: a +// caller that never names `id` at all (every existing caller, before this +// feature) still gets a working prompt with a server-minted id — the +// existing, unmodified default behavior. +func TestPromptAsyncNoSuppliedIDMintsLikeBefore(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("done")}} + h := newHarness(t, prov) + id := h.createSession("test/m1") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "hello"}}, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + var pr promptAsyncResponse + if err := json.Unmarshal(data, &pr); err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(pr.MessageID, "msg_") { + t.Fatalf("response message_id = %q, want a msg_-prefixed minted id", pr.MessageID) + } + + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 || users[0].ID != pr.MessageID { + t.Fatalf("user messages = %+v, want exactly one whose id matches response message_id %q", users, pr.MessageID) + } +} + +// TestQueuedPromptDeliversSuppliedMessageID is the busy-queue RED test: a +// prompt submitted while the session is already busy is durably enqueued +// (not run immediately), yet its caller-supplied id still survives to the +// eventual user message once the occupying turn finishes and the queue +// drains — proving EnqueuePrompt's queued-prompt record carries the id +// through dispatchQueueHead, not just the immediate-dispatch fast path. +func TestQueuedPromptDeliversSuppliedMessageID(t *testing.T) { + prov := &queueProv{ + name: "test", + started: make(chan struct{}), + release: make(chan struct{}), + turns: [][]provider.Event{asstTurn("second done")}, + } + h := newHarness(t, prov) + id := h.createSession("test/m1") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "first"}}, + "id": "first-msg", + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("first prompt status %d: %s", resp.StatusCode, data) + } + <-prov.started + + const queuedID = "console-queued-2" + resp, data = h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]string{{"type": "text", "text": "second"}}, + "id": queuedID, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("second prompt status %d: %s", resp.StatusCode, data) + } + var qr promptAsyncResponse + if err := json.Unmarshal(data, &qr); err != nil { + t.Fatal(err) + } + if qr.Status != "queued" { + t.Fatalf("second prompt response = %+v, want status=queued", qr) + } + if qr.MessageID != queuedID { + t.Fatalf("queued response message_id = %q, want the supplied id %q, even though the prompt has not run yet", qr.MessageID, queuedID) + } + + close(prov.release) // let the first turn finish, which drains the queue + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 2 { + t.Fatalf("user messages = %d, want 2: %+v", len(users), users) + } + if users[0].ID != "first-msg" { + t.Fatalf("first user message id = %q, want %q", users[0].ID, "first-msg") + } + if users[1].ID != queuedID { + t.Fatalf("second (queued) user message id = %q, want the supplied id %q — a busy-session queue must deliver the client's id, not mint a new one at drain time", users[1].ID, queuedID) + } +} + +// TestSessionSendUsesSuppliedMessageID mirrors +// TestPromptAsyncUsesSuppliedMessageID for POST /session/{id}/send: the +// root-session send path shares runPrompt/PromptWithOrigin with +// prompt_async (see sendTextToRoot), so it gets the same client-id +// treatment — verified independently here rather than assumed. +func TestSessionSendUsesSuppliedMessageID(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{asstTurn("done")}} + h := newHarness(t, prov) + id := h.createSession("test/m1") + + const supplied = "console-send-1" + resp, data := h.do("POST", "/session/"+id+"/send", map[string]any{ + "text": "hello via send", + "id": supplied, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("session.send status %d: %s", resp.StatusCode, data) + } + var sr struct { + MessageID string `json:"message_id"` + } + if err := json.Unmarshal(data, &sr); err != nil { + t.Fatal(err) + } + if sr.MessageID != supplied { + t.Fatalf("response message_id = %q, want the supplied id %q", sr.MessageID, supplied) + } + + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 || users[0].ID != supplied { + t.Fatalf("user messages = %+v, want exactly one whose id is %q", users, supplied) + } +} diff --git a/server/queue_clear_race_test.go b/server/queue_clear_race_test.go index 64771ae9..0b8140cf 100644 --- a/server/queue_clear_race_test.go +++ b/server/queue_clear_race_test.go @@ -35,7 +35,7 @@ func TestQueueClearRaceDuringIdleDispatchIsNotAnError(t *testing.T) { if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("q1"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q1", ""); err != nil { t.Fatalf("EnqueuePrompt q1: %v", err) } diff --git a/server/queue_delete_nonresident_test.go b/server/queue_delete_nonresident_test.go index 531e2123..0ad19f48 100644 --- a/server/queue_delete_nonresident_test.go +++ b/server/queue_delete_nonresident_test.go @@ -41,10 +41,10 @@ func TestDeleteQueueColdSessionSurvivesResidencyRace(t *testing.T) { if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("q1"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q1", ""); err != nil { t.Fatalf("EnqueuePrompt q1: %v", err) } - if _, err := st.sess.EnqueuePrompt("q2"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q2", ""); err != nil { t.Fatalf("EnqueuePrompt q2: %v", err) } diff --git a/server/queue_test.go b/server/queue_test.go index 67dd8c74..d42c29d3 100644 --- a/server/queue_test.go +++ b/server/queue_test.go @@ -694,7 +694,7 @@ func TestQueueRestartRefoldNoAutoDispatch(t *testing.T) { if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("queued before restart"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("queued before restart", ""); err != nil { t.Fatalf("EnqueuePrompt: %v", err) } @@ -1022,10 +1022,10 @@ func TestIdlePromptWithQueueGoesFIFO(t *testing.T) { if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("q1"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q1", ""); err != nil { t.Fatalf("EnqueuePrompt q1: %v", err) } - if _, err := st.sess.EnqueuePrompt("q2"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q2", ""); err != nil { t.Fatalf("EnqueuePrompt q2: %v", err) } @@ -1152,10 +1152,10 @@ func TestIdlePromptWithQueueDispatchDoesNotRaceQueuedCountInResponse(t *testing. if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("q1"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q1", ""); err != nil { t.Fatalf("EnqueuePrompt q1: %v", err) } - if _, err := st.sess.EnqueuePrompt("q2"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q2", ""); err != nil { t.Fatalf("EnqueuePrompt q2: %v", err) } @@ -1233,7 +1233,7 @@ func TestQueuedArrivalDoesNotRetargetSessionModel(t *testing.T) { if st == nil { t.Fatal("session not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("q1"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("q1", ""); err != nil { t.Fatalf("EnqueuePrompt q1: %v", err) } diff --git a/server/session_tree.go b/server/session_tree.go index 5ed00714..ee51a6cb 100644 --- a/server/session_tree.go +++ b/server/session_tree.go @@ -258,7 +258,11 @@ func (s *Server) runOrQueueText(id, text string) engine.RunnerOutcome { // above dispatches the QUEUE HEAD instead — see this function's own doc // comment). Tagging it lets the console render it as a system notice // rather than a human-typed bubble — see message.Message.Origin. - go s.runPrompt(ctx, id, st, text, message.OriginEngine) + // "": text here is ALWAYS taskResumeTriggerText, a synthetic engine + // resume trigger with no client message id of its own — see the origin + // comment just above. PromptWithOrigin's own mint site resolves it like + // any other unset id. + go s.runPrompt(ctx, id, st, text, message.OriginEngine, "") return engine.RunnerHandled } @@ -289,7 +293,14 @@ func (s *Server) runOrQueueText(id, text string) engine.RunnerOutcome { // (404 unknown session, 503 draining, 409 with holder set for a // workdir-held conflict, 400 for the practically-unreachable empty-text // case handleSessionSend already guards against). -func (s *Server) sendTextToRoot(id, text string) (status string, queuedDepth int, errCode int, holder string) { +// +// msgID is handleSessionSend's own already-resolved message id (see +// engine.ResolveMessageID) for text, resolved exactly once before this +// call — every branch below threads that SAME value through to whichever +// of EnqueuePrompt or runPrompt actually delivers text, so the caller's +// own response always names the id that ends up in the transcript, never +// a second, independently-minted one. +func (s *Server) sendTextToRoot(id, text string, msgID string) (status string, queuedDepth int, errCode int, holder string) { st, ctx, _, code, holder := s.claimForPrompt(id) switch { case code == http.StatusNotFound: @@ -322,7 +333,7 @@ func (s *Server) sendTextToRoot(id, text string) (status string, queuedDepth int // this reason; mirror it. A live review caught this. return "", 0, http.StatusConflict, "" } - ourID, err := sess.EnqueuePrompt(text) + ourID, _, err := sess.EnqueuePrompt(text, msgID) if err != nil { return "", 0, http.StatusBadRequest, "" } @@ -343,7 +354,7 @@ func (s *Server) sendTextToRoot(id, text string) (status string, queuedDepth int return "queued", remaining, 0, "" default: // code == 0: claimed cleanly if len(st.sess.QueuedPrompts()) > 0 { - if _, err := st.sess.EnqueuePrompt(text); err != nil { + if _, _, err := st.sess.EnqueuePrompt(text, msgID); err != nil { s.releasePromptClaim(st) return "", 0, http.StatusBadRequest, "" } @@ -355,7 +366,7 @@ func (s *Server) sendTextToRoot(id, text string) (status string, queuedDepth int // (an MCP send_message_to_box call, or any other operator-authored // text), never the engine's own synthetic resume trigger — that one // goes exclusively through runOrQueueText above. - go s.runPrompt(ctx, id, st, text, "") + go s.runPrompt(ctx, id, st, text, "", msgID) return "started", 0, 0, "" } } @@ -396,7 +407,12 @@ func (s *Server) resumeSessionForTaskNotification(id, text string) engine.Runner // truthy status still sees success — but now ALSO reports "queued" with // a depth, honestly, exactly like prompt_async already does, rather than // claiming "sent" for a message that has not actually run yet. -func (s *Server) writeSendToRootResult(w http.ResponseWriter, id, status string, queuedDepth, errCode int, holder string) { +// +// messageID is handleSessionSend's own already-resolved message id (see +// engine.ResolveMessageID), reported back verbatim on every success shape +// — "sent" and "queued" alike — mirroring promptAsyncResponse's +// message_id field. +func (s *Server) writeSendToRootResult(w http.ResponseWriter, id, status string, queuedDepth, errCode int, holder, messageID string) { switch errCode { case 0: // fall through to the success response below @@ -426,7 +442,7 @@ func (s *Server) writeSendToRootResult(w http.ResponseWriter, id, status string, writeErr(w, http.StatusNotFound, "no such session") return } - resp := map[string]any{"session_id": id, "status": "sent"} + resp := map[string]any{"session_id": id, "status": "sent", "message_id": messageID} if status == "queued" { resp["status"] = "queued" resp["queued"] = queuedDepth @@ -441,6 +457,10 @@ func (s *Server) handleSessionSend(w http.ResponseWriter, r *http.Request) { } var body struct { Text string `json:"text"` + // ID mirrors prompt_async's optional client-minted message id (see + // handlePrompt's body.ID doc comment) — used verbatim with the same + // single fail-safe guard, never validated or rejected. + ID string `json:"id"` } if err := decodeBody(r, &body); err != nil { writeErr(w, http.StatusBadRequest, err.Error()) @@ -450,6 +470,10 @@ func (s *Server) handleSessionSend(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusBadRequest, "text is required") return } + // Resolved ONCE, exactly like handlePrompt's msgID — see its own doc + // comment for why every branch below must report and use this SAME + // value rather than resolving a second, possibly different, id later. + msgID := engine.ResolveMessageID(body.ID) sess, ok := s.sessMgr.Session(id) if !ok { // Not a tracked node yet — could be a root that exists on disk but @@ -463,8 +487,8 @@ func (s *Server) handleSessionSend(w http.ResponseWriter, r *http.Request) { // SessionManager registers it the instant Spawn creates it, so // "not a node" here only ever means "an as-yet-unadopted root" or // "genuinely unknown." - status, queuedDepth, errCode, holder := s.sendTextToRoot(id, body.Text) - s.writeSendToRootResult(w, id, status, queuedDepth, errCode, holder) + status, queuedDepth, errCode, holder := s.sendTextToRoot(id, body.Text, msgID) + s.writeSendToRootResult(w, id, status, queuedDepth, errCode, holder, msgID) return } // sess.TaskParentID() (durable), not the live tree's ParentID — a live @@ -487,8 +511,8 @@ func (s *Server) handleSessionSend(w http.ResponseWriter, r *http.Request) { // later reloaded: claimForPrompt's own cold-load path covers it, // unlike an earlier version of this handler that drove a stale // SessionManager-cached object in that case. - status, queuedDepth, errCode, holder := s.sendTextToRoot(id, body.Text) - s.writeSendToRootResult(w, id, status, queuedDepth, errCode, holder) + status, queuedDepth, errCode, holder := s.sendTextToRoot(id, body.Text, msgID) + s.writeSendToRootResult(w, id, status, queuedDepth, errCode, holder, msgID) return } // Child: SessionManager is its sole scheduler, always safe. Unlike a diff --git a/server/session_tree_test.go b/server/session_tree_test.go index c7a9aec4..89481186 100644 --- a/server/session_tree_test.go +++ b/server/session_tree_test.go @@ -896,7 +896,7 @@ func TestSessionSendToRootWithStrandedQueueIsNotLost(t *testing.T) { if st == nil { t.Fatal("root not resident right after creation") } - if _, err := st.sess.EnqueuePrompt("stranded head"); err != nil { + if _, _, err := st.sess.EnqueuePrompt("stranded head", ""); err != nil { t.Fatalf("EnqueuePrompt: %v", err) } From 7918b6da0d47f7d9224b1e48ad53f45c52b59dc2 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 16:46:43 -0400 Subject: [PATCH 42/95] fix(engine): keep claude-code stdin open for mid-turn queue delivery (#231) Observable problem: a prompt queued via POST /session/{id}/enqueue (or boxes' .../send) while a claude-code-lane session was busy sat in the durable queue, undelivered, for the ENTIRE remainder of that turn. Live- reproduced on session ses_01m1f4hbpee1nvwzam39b7fwm3 (box box_01m1f4g92bfb0a3e5863hqgbpw): a prompt queued at journal seq 701 (19:29:14Z) was still undelivered 6+ minutes later while the underlying `claude` turn kept running real tool calls. A second live repro on a fresh box confirmed delivery only happened once the whole turn ended, ~2.5 minutes after enqueue. A native-provider session on the SAME box delivered an equivalent mid-turn prompt in ~13 seconds, at the next tool-call boundary, via drainQueuedPromptsIntoHistory. Root cause: runAgenticLoop dispatches a claude-code-delegated turn to runClaudeCodeTurn, which drives the `claude` CLI as a subprocess over its --input-format/--output-format stream-json protocol. The old code wrote the turn's one input line and closed stdin immediately, before ever reading the child's stdout. A prompt queued after that close had no way to reach the already-running child; the ONLY delivery path was the server's ordinary end-of-turn tail dispatch (maybeDispatchQueued), which starts a brand new turn only after the current one fully ends. None of the native loop's own mid-turn drain points (the tool-call- boundary and max-tokens-continuation calls to drainQueuedPromptsIntoHistory, engine.go) apply to a claude-code turn, which runAgenticLoop dispatches to runClaudeCodeTurn before any of that native-loop machinery runs. Design: mirror the Claude Agent SDK's own streaming-input construct (@anthropic-ai/claude-agent-sdk, sdk.mjs) instead of inventing a bespoke protocol. Its ProcessTransport keeps a `claude` child's stdin open for the whole session and never closes it after one write; Query.streamInput pumps an app-supplied input source into it one message at a time and calls transport.endInput() (closes stdin) only once that source itself is exhausted -- as opposed to the SDK's plain single-string query() path, whose Query.readMessages instead closes stdin the moment the FIRST "result" event arrives, since a one-shot call has nothing further to send. This driver's own turn is a hybrid of those two SDK shapes for one child: it starts with exactly one message like the single-string path, but a prompt queued mid-turn is exactly the further input the SDK's streaming-input mode exists to carry into an ALREADY RUNNING child. Verified live (Step 1, before writing any code) against a real `claude` 2.1.251 binary, driven directly over its stream-json stdin/stdout protocol outside harness entirely: a second stdin line written while the first tool loop was still in flight was picked up and answered at the very next tool-call boundary, mid-turn, turn continuing -- not held until the final "result" event 10+ seconds later. That result is what makes stdin- injection the right mechanism at all, rather than an interrupt (which would end the turn instead of steering it, and which Claude Code's own docs describe as ending the current turn rather than injecting into it). Semantic change: runClaudeCodeTurn's stdin-writer is now a goroutine that stays alive for the child's whole lifetime instead of a single write-then-close. It sends the turn's own driving text first, then blocks on a per-turn wake channel (Session.claudeCodeQueueWake, engine.go) that Session.emit signals, non-blocking, whenever an EventPromptQueued fires -- the one choke point every enqueue path (EnqueuePrompt, EnqueuePromptDurable, the deferred-flush path) already shares. On each wake it drains the ENTIRE queue (DequeueAllPrompts, reason "injected"), renders it with the same operatorMessagesBlock the native loop's own mid-turn drain uses, appends it into session history (so the delivery is visible in the transcript, exactly like the native path), and writes it to the child's stdin as a further stream-json input line. Stdin is closed exactly once, by this same goroutine, right after consumeClaudeCodeStream returns (the child's own terminal "result" event) -- mirroring the SDK's single-string endInput()-on- result call, just reached from a longer-lived writer. A write failure on a mid-turn injection is best-effort, never fatal: the prompt is already durably dequeued and already in session history, so the next claude-code turn's existing claudeCodeHistoryDirectiveArgs mechanism detects the gap and feeds it forward via --resume -- a delay, never a loss. inputErr's existing contract (claudeCodeTurnResult: "no usable result at all, and the turn's own driving text never reached the child") is preserved unchanged, now sourced from the pump's first write specifically. Native-provider sessions are unaffected; nothing in the native loop or its own drain points changed. Verification: engine/testdata/fakeclaude/main.go's stand-in `claude` process needed a matching update -- it used to read stdin to EOF in one shot, which now blocks forever since the driver no longer closes stdin after the first line; it now reads stream-json input line by line, matching how the real CLI actually behaves. A new "queue_injection" mode blocks for a second stdin line after an initial marker message, so a driver that still closes stdin early sees an immediate EOF (a closed pipe never blocks) instead of a hang, and reports that mismatch as wrong content rather than a timeout. TestClaudeCodeQueueInjectedMidTurnViaOpenStdin red-verified against the pre-fix runClaudeCodeTurn (failed on wrong content, as expected) and green-verified against the fix; it asserts the queued text reaches the SAME running child as a genuine second stdin line, lands in session history, and the queue empties. go test -race ./engine/... and ./server/... pass; go build ./... and go vet ./... are clean; gofmt reports no diffs. Full go test -race ./... shows one pre-existing, unrelated e2e failure (TestAppendSystemPromptReachesModelOnServe, a system-prompt content mismatch on a native anthropic-lane test) reproduced identically on a clean checkout of this branch's own base commit, before any of this change. --- engine/claude_code_backend.go | 186 ++++++++++++++++++++++------- engine/claude_code_backend_test.go | 120 +++++++++++++++++++ engine/engine.go | 37 ++++++ engine/testdata/fakeclaude/main.go | 115 ++++++++++++++---- 4 files changed, 393 insertions(+), 65 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 84a9922f..c47dbc4e 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -542,47 +542,99 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro _, _ = io.Copy(&stderr, stderrPipe) }() - inputLine, err := json.Marshal(claudeCodeInputMessage{ - Type: "user", - Message: claudeCodeInputInnerMessage{ - Role: "user", - Content: text, - }, - }) - if err != nil { - _ = cmd.Process.Kill() - _ = cmd.Wait() - return nil, fmt.Errorf("engine: claude-code: encoding turn input: %w", err) - } - // inputErr captures a failure writing or closing stdin WITHOUT killing - // the child or returning early: a fast/trivial turn's child can - // legitimately finish its whole result and exit (closing its own end - // of the pipe) before this call finishes writing/closing its side, - // which turns an otherwise-harmless race into a broken-pipe/closed- - // pipe error right here. That is not a real failure — the child still - // has a complete, valid result waiting on stdout — so this call must - // keep going and read it: only if the turn ends with NO usable result - // at all does inputErr get promoted to the actual returned error, - // below. Deliberately not stdin.Close() after a failed Write: closing - // an already-broken pipe has nothing useful to report, and calling it - // anyway would risk overwriting a meaningful inputErr with a second, - // less informative one. - var inputErr error - if _, err := stdin.Write(append(inputLine, '\n')); err != nil { - inputErr = fmt.Errorf("engine: claude-code: writing turn input: %w", err) - } else if err := stdin.Close(); err != nil { - // Close stdin: this driver sends exactly one turn per `claude` - // child (continuity across harness turns is --resume, not a long- - // lived child — see this file's package doc), and an unclosed - // stdin would leave the CLI waiting indefinitely for a second - // message that is never coming, wedging cmd.Wait() below forever — - // but a Close failing is itself just as benign as a Write failing, - // for the exact same reason (the read end may already be gone - // because the child already finished), so it gets the same - // deferred treatment as the Write error above rather than an - // immediate kill-and-return. - inputErr = fmt.Errorf("engine: claude-code: closing stdin: %w", err) - } + // The stdin-writer pump: mirrors the Claude Agent SDK's own streaming- + // input construct rather than inventing a bespoke protocol — + // @anthropic-ai/claude-agent-sdk's Query.streamInput (sdk.mjs): its + // ProcessTransport keeps a `claude` child's stdin open for the whole + // session and never closes it after one write; streamInput pumps an + // app-supplied async-iterable of input messages into it one at a time + // (`for await (n of e) transport.write(JSON.stringify(n)+"\n")`) and + // calls `transport.endInput()` (closes stdin) only once THAT + // iterable is exhausted — as opposed to the SDK's plain single-string + // query() path, whose Query.readMessages instead calls endInput() the + // moment the FIRST "result" event arrives, because a one-shot call has + // nothing further to send. + // + // This driver's own turn is a hybrid of those two SDK shapes for one + // child: it starts with exactly one message (the turn's own driving + // text) like the single-string path, but a prompt queued mid-turn + // (EnqueuePrompt et al.) is exactly the further input the SDK's + // streaming-input mode exists to carry into an ALREADY RUNNING child + // rather than a fresh one. So the goroutine below plays BOTH SDK + // roles for this one child: it is the input source (draining + // s.promptQueue, woken by EventPromptQueued — see + // Session.claudeCodeQueueWake's own doc comment, engine.go) AND the + // thing that pumps each item to the child's stdin, closing stdin + // (mirrors endInput()) only once THIS driver's own "no more input" + // signal fires: stopPump, closed by the code below right after + // consumeClaudeCodeStream returns (i.e. once the child's OWN terminal + // "result" event arrives) — the exact moment the single-string SDK + // path's own endInput() call fires, just reached from a longer-lived + // writer instead of a one-off write. + // + // One writer, ever: this goroutine is the ONLY thing that touches + // stdin from the moment it starts until it returns (having already + // closed stdin itself on every exit path), so the rest of this + // function never writes to or closes stdin directly — it only closes + // stopPump and waits on pumpDone before doing anything else with the + // child (cmd.Wait(), below). + firstWriteErrCh := make(chan error, 1) + wake := make(chan struct{}, 1) + s.claudeCodeQueueWake.Store(&wake) + stopPump := make(chan struct{}) + pumpDone := make(chan struct{}) + go func() { + defer close(pumpDone) + defer s.claudeCodeQueueWake.Store(nil) + // The turn's own driving text (already carries any spliced + // task-notification segment, above) — sent through the SAME + // writer as every later mid-turn injection, never a separate + // one-off path. + firstWriteErrCh <- writeClaudeCodeInputMessage(stdin, text) + for { + select { + case <-wake: + queued := s.DequeueAllPrompts("injected") + if len(queued) == 0 { + // A concurrent DELETE /session/{id}/queue, or a wake + // coalesced behind one this same loop already + // drained: nothing left to send this time around. + continue + } + // Same rendering, and the same "append into real, + // durable history before anything else" shape, + // drainQueuedPromptsIntoHistory (engine.go) uses for the + // native loop's own mid-turn drain — a queued prompt's + // delivered text is identical whichever path answers it, + // and this append is what makes the delivery visible in + // the session transcript. + block := strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n") + s.append(message.Message{ + ID: newID("msg"), + Role: message.RoleUser, + Parts: message.Parts{&message.Text{Text: block}}, + CreatedAt: time.Now().UTC(), + }) + if err := writeClaudeCodeInputMessage(stdin, block); err != nil { + // Best-effort, exactly like the first write's own + // inputErr contract below: the prompt is already + // dequeued AND durably in s.history by the append + // just above, so a live write failure here only means + // THIS running child never saw it mid-turn — the next + // claude-code turn's claudeCodeHistoryDirectiveArgs + // detects the gap and feeds it via --resume, a delay + // never a loss (see engine/AGENTS.md: "Deliver each + // item once across tool and goal-turn drains"). The + // pipe is presumably gone either way, so stop pumping. + _ = stdin.Close() + return + } + case <-stopPump: + _ = stdin.Close() + return + } + } + }() // The signal-abort cascade: SIGINT first (Claude Code's own docs // describe this as ending the current turn gracefully, leaving it @@ -620,6 +672,25 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro }() finalMsg, started, turnErr := s.consumeClaudeCodeStream(stdout, model) + // No more input is coming for this child (mirrors the single-string + // SDK path's own endInput()-on-first-"result" call — see the pump + // goroutine's own doc comment above): signal it to stop and wait for + // it to actually exit BEFORE this goroutine touches the child again + // (cmd.Wait(), below) — see the pump's "one writer, ever" note for + // why this ordering matters. + close(stopPump) + <-pumpDone + // inputErr captures a failure on the FIRST write specifically — see + // its use in claudeCodeTurnResult below, whose contract is "no usable + // result at all, AND the turn's own driving text never reached the + // child" (a later, mid-turn queued-prompt write failure is always + // best-effort per the pump's own comment above, and by the time one + // could happen finalMsg is already non-nil, so claudeCodeTurnResult + // never consults inputErr for that case anyway). + inputErr := <-firstWriteErrCh + if inputErr != nil { + inputErr = fmt.Errorf("engine: claude-code: writing turn input: %w", inputErr) + } if started { // The CLI's own session actually came up for this turn — whether // or not turnErr is set (see claudeCodeTurnResult's own o.started @@ -1602,10 +1673,13 @@ func (s *Session) claudeCodeMCPConfigFile() (path string, cleanup func(), err er return name, func() { _ = os.Remove(name) }, nil } -// claudeCodeInputMessage is the stdin stream-json shape this driver writes -// — one line, one turn (see runClaudeCodeTurn's own doc comment on why a -// child is spawned fresh per harness turn rather than kept alive across -// several). +// claudeCodeInputMessage is the stdin stream-json shape this driver +// writes — one per line. A harness turn still spawns exactly one `claude` +// child (see runClaudeCodeTurn's own doc comment on why continuity across +// harness turns is --resume, not a long-lived child), but that one child +// can now receive SEVERAL of these lines across its lifetime: the turn's +// own driving text first, then zero or more prompts queued mid-turn — see +// runClaudeCodeTurn's stdin-writer pump. type claudeCodeInputMessage struct { Type string `json:"type"` Message claudeCodeInputInnerMessage `json:"message"` @@ -1616,6 +1690,28 @@ type claudeCodeInputInnerMessage struct { Content string `json:"content"` } +// writeClaudeCodeInputMessage marshals text as one stream-json user input +// line and writes it to w — mirrors the Claude Agent SDK's +// ProcessTransport.write (sdk.mjs: JSON.stringify(message) + "\n"). Used +// for both a turn's first, driving message and every later mid-turn +// queued-prompt injection, so the child's stdin only ever sees this one +// wire shape (see runClaudeCodeTurn's stdin-writer pump doc comment for +// the construct this mirrors). +func writeClaudeCodeInputMessage(w io.Writer, text string) error { + line, err := json.Marshal(claudeCodeInputMessage{ + Type: "user", + Message: claudeCodeInputInnerMessage{ + Role: "user", + Content: text, + }, + }) + if err != nil { + return fmt.Errorf("engine: claude-code: encoding turn input: %w", err) + } + _, err = w.Write(append(line, '\n')) + return err +} + // capBuffer is an io.Writer that retains at most cap bytes, silently // dropping anything beyond that — used to bound how much of the `claude` // child's stderr this file holds onto for a diagnostic message (see diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index f2ddd609..49255bcd 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1566,3 +1566,123 @@ func killLeakedFakeClaude(t *testing.T, pidFile string) { t.Logf("killLeakedFakeClaude: Kill(%d): %v (may have already exited)", pid, err) } } + +// TestClaudeCodeQueueInjectedMidTurnViaOpenStdin is the regression test for +// the live production bug reported as "queue doesn't seem to be working in +// opus subscription sessions" (box box_01m1f4g92bfb0a3e5863hqgbpw, session +// ses_01m1f4hbpee1nvwzam39b7fwm3): a prompt enqueued via POST +// .../sessions/{id}/send while a claude-code-lane turn was busy sat +// durably queued, undelivered, for the ENTIRE remainder of that turn — +// live-reproduced sitting queued 6+ minutes with the underlying `claude` +// turn still actively running — because runClaudeCodeTurn used to write +// its ONE input line and close stdin immediately, so a prompt queued after +// that close could never reach the already-running child; only the +// server's ordinary end-of-turn tail dispatch (a NEW turn) ever delivered +// it. A native-provider session on the same box, by contrast, delivers a +// mid-turn queued prompt within seconds via drainQueuedPromptsIntoHistory +// at the next tool-call boundary. +// +// This proves the fix directly against the mechanism: runClaudeCodeTurn's +// stdin-writer pump keeps the CLI child's stdin OPEN across the whole +// turn and writes a prompt queued via EnqueuePrompt to it as a SECOND +// stream-json input line, delivered to the SAME running child, before +// that child's own terminal "result" event — not after the process exits +// and a fresh one is dispatched. +// +// fakeclaude's "queue_injection" mode (testdata/fakeclaude/main.go) emits +// a "WAITING_FOR_QUEUE" marker message and then blocks reading a SECOND +// stdin line. This test waits for that marker via OnEvent — a +// deterministic, non-sleep synchronization point: by the time the driver +// has mapped that event, the child is already blocked in its second +// read — before calling EnqueuePrompt. If the fix were absent (a driver +// that still closes stdin right after its first write), fakeclaude's +// second read would see an immediate EOF (a closed pipe never blocks) and +// report "no second message received" instead of echoing the queued +// text — so an unfixed driver fails this test on WRONG CONTENT, not a +// timeout. +func TestClaudeCodeQueueInjectedMidTurnViaOpenStdin(t *testing.T) { + s, _ := claudeCodeTestSession(t, "queue_injection") + stdinLog := filepath.Join(t.TempDir(), "stdin.log") + t.Setenv("FAKE_CLAUDE_STDIN_LOG", stdinLog) + + waiting := make(chan struct{}) + var waitingOnce sync.Once + s.cfg.OnEvent = func(ev Event) { + if ev.Type == EventMessage && ev.Message != nil && ev.Message.Parts.Text() == "WAITING_FOR_QUEUE" { + waitingOnce.Do(func() { close(waiting) }) + } + } + + type outcome struct { + msg *message.Message + err error + } + done := make(chan outcome, 1) + go func() { + msg, err := s.Prompt(context.Background(), "start") + done <- outcome{msg, err} + }() + + select { + case res := <-done: + t.Fatalf("Prompt returned (%+v, %v) before fakeclaude ever emitted WAITING_FOR_QUEUE", res.msg, res.err) + case <-waiting: + case <-time.After(10 * time.Second): + t.Fatal("fakeclaude never emitted WAITING_FOR_QUEUE within 10s") + } + + // The turn is now mid-flight — Prompt has NOT returned, and + // fakeclaude is blocked on its own second stdin read. Enqueue while + // busy, exactly like the live bug's POST /session/{id}/send arriving + // while a claude-code turn is running. + if _, _, err := s.EnqueuePrompt("QUEUE-MARKER: please continue", ""); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + + var res outcome + select { + case res = <-done: + case <-time.After(10 * time.Second): + t.Fatal("Prompt did not return within 10s of EnqueuePrompt — the queued prompt was never " + + "delivered to the running child (fakeclaude stayed blocked on its second stdin read)") + } + if res.err != nil { + t.Fatalf("Prompt: %v", res.err) + } + if res.msg == nil || !strings.Contains(res.msg.Parts.Text(), "QUEUE-MARKER: please continue") { + t.Fatalf("Prompt's final message = %+v, want it to echo the mid-turn queued prompt's text "+ + "(proves fakeclaude's SAME process received line two)", res.msg) + } + + // The queue itself must be empty afterward: delivered, not stranded. + if q := s.QueuedPrompts(); len(q) != 0 { + t.Errorf("QueuedPrompts() after delivery = %+v, want empty", q) + } + + // The injected prompt must be visible in the session transcript — + // mirrors the native path's own drainQueuedPromptsIntoHistory, and is + // what lets the console render the delivery. + found := false + for _, m := range s.History() { + if m.Role == message.RoleUser && strings.Contains(m.Parts.Text(), "QUEUE-MARKER: please continue") { + found = true + } + } + if !found { + t.Error("session history has no user message carrying the queued prompt's text — mid-turn delivery did not append into the transcript") + } + + // The actual bytes reached the CLI's stdin as a genuine SECOND line, + // not just harness-side bookkeeping. + stdinBytes, err := os.ReadFile(stdinLog) + if err != nil { + t.Fatalf("reading captured CLI stdin: %v", err) + } + lines := strings.Split(strings.TrimRight(string(stdinBytes), "\n"), "\n") + if len(lines) != 2 { + t.Fatalf("CLI stdin carried %d lines, want 2 (initial turn text + the mid-turn injected prompt): %q", len(lines), string(stdinBytes)) + } + if !strings.Contains(lines[1], "QUEUE-MARKER: please continue") { + t.Errorf("CLI stdin's second line = %q, want it to carry the queued prompt's text", lines[1]) + } +} diff --git a/engine/engine.go b/engine/engine.go index 82cae110..293c2185 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -1412,6 +1412,28 @@ type Session struct { // order. Guarded by mu. deferredQueueRecords []deferredQueueRecord + // claudeCodeQueueWake, when non-nil, is a wake channel a currently + // running runClaudeCodeTurn (claude_code_backend.go) has registered + // for itself — its own stdin-writer pump's ONE way to learn that + // EnqueuePrompt/EnqueuePromptDurable/the deferred-flush path just + // added something to s.promptQueue, so it can drain and inject it + // into the live `claude` child's still-open stdin without polling + // (see emit's own EventPromptQueued case below, and + // runClaudeCodeTurn's doc comment for the Claude Agent SDK construct + // this pump mirrors). nil for every native-provider turn and for a + // claude-code turn that has not started its pump yet. Set at pump + // start and cleared at pump exit by runClaudeCodeTurn itself, via + // atomic.Pointer so a concurrent emit() (from ANY goroutine — see + // OnEvent's own "several goroutines at once" note above) can read it + // without taking s.mu, which emit() must not do (EnqueuePrompt calls + // it WHILE HOLDING s.mu). The pointed-to channel is buffered(1) and + // only ever sent to non-blocking (default: case) — a coalesced, + // dropped, or missed wake is harmless: the pump drains the ENTIRE + // queue on every wake it does see (DequeueAllPrompts), and anything + // still queued when the turn ends is picked up exactly as before this + // change, by the server's ordinary tail dispatch (maybeDispatchQueued). + claudeCodeQueueWake atomic.Pointer[chan struct{}] + // enqueueSeq is the durable-enqueue idempotency high-water mark (see // EnqueuePromptDurable in queue.go and promptRecord.Seq in store.go): // the largest caller-issued seq durably accepted. Monotonic; a seq at or @@ -2365,6 +2387,21 @@ func (s *Session) accumulateDiscardedTurnUsage(usage provider.Usage) { func (s *Session) emit(ev Event) { ev.SessionID = s.ID + if ev.Type == EventPromptQueued { + // Wake a running claude-code turn's stdin-writer pump, if one is + // registered — see claudeCodeQueueWake's own doc comment. This is + // the ONE choke point every enqueue path (EnqueuePrompt, + // EnqueuePromptDurable, and the deferred-flush path's own + // flushQueueRecordsLocked) already shares to emit this exact + // event, so hooking it here — instead of at each call site — + // covers all of them by construction. + if ch := s.claudeCodeQueueWake.Load(); ch != nil { + select { + case *ch <- struct{}{}: + default: + } + } + } if s.cfg.OnEvent != nil { s.cfg.OnEvent(ev) } diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index f5de4663..2c18a84f 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -25,20 +25,24 @@ // event with no overage in play at all — overageStatus "", isUsingOverage // false, overageResetsAt 0 — proving mapClaudeCodeRateLimit leaves // SubscriptionUsage.Overage nil rather than a hollow zero-value object), -// and "bg_leak" (reproduces a `claude --bg` turn whose detached child +// "bg_leak" (reproduces a `claude --bg` turn whose detached child // inherits this process's stdout AND stderr and outlives it — see its own // comment below and claude_code_backend.go's consumeClaudeCodeStream/ -// runClaudeCodeTurn doc comments for the wedge this proves fixed). +// runClaudeCodeTurn doc comments for the wedge this proves fixed), and +// "queue_injection" (blocks for a SECOND stdin line mid-turn, proving the +// driver's stdin-writer pump keeps stdin open and delivers a mid-turn +// queued prompt to THIS running child instead of a fresh one — see its own +// comment below). package main import ( "bufio" "encoding/json" "fmt" - "io" "os" "os/exec" "strconv" + "strings" "time" ) @@ -87,26 +91,46 @@ func main() { } } - if mode != "fast_no_drain" { - // Read (and, when FAKE_CLAUDE_STDIN_LOG is set, record) the one - // turn-input line the real driver writes — the real driver writes - // its turn message and closes stdin BEFORE it ever reads this - // process's stdout (see runClaudeCodeTurn's own stdin write/close - // ordering, claude_code_backend.go), so a synchronous read-to-EOF - // here always completes promptly regardless of mode, with none of - // the goroutine-vs-process-exit race a background drain would - // carry for a test that actually wants the captured bytes. - // FAKE_CLAUDE_STDIN_LOG lets a test recover the EXACT bytes the - // driver sent — e.g. proving a checked-out task notification's - // rendered content actually reached the CLI's input, not just the - // bare trigger text (engine/claude_code_backend_test.go). - stdinBytes, _ := io.ReadAll(os.Stdin) - if logPath := os.Getenv("FAKE_CLAUDE_STDIN_LOG"); logPath != "" { - if f, err := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644); err == nil { - _, _ = f.Write(stdinBytes) + // stdinR is kept OPEN and read line-by-line for the rest of main(), + // instead of the one-shot read-to-EOF this file used before the + // driver started keeping its own stdin open across a whole turn (see + // claude_code_backend.go's runClaudeCodeTurn doc comment on the + // stdin-writer pump it mirrors from the Claude Agent SDK's + // Query.streamInput). The real `claude` binary reads its input as a + // stream of newline-delimited JSON lines, not a single blob ending in + // EOF — an io.ReadAll here would now block forever, since the driver + // no longer closes stdin right after its first write. readStdinLine + // mirrors that real streaming-read shape: one line per call, blocking + // until it arrives (or stdin closes), which is also what lets the + // "queue_injection" mode below block for a SECOND line while the + // first turn is still open, exactly the mid-turn steering window this + // stand-in exists to prove. + stdinR := bufio.NewReader(os.Stdin) + readStdinLine := func() (line string, ok bool) { + b, err := stdinR.ReadString('\n') + if logPath := os.Getenv("FAKE_CLAUDE_STDIN_LOG"); logPath != "" && b != "" { + // Record (append) exactly the bytes read, same shape the old + // one-shot read-to-EOF left behind — a test recovers the + // EXACT bytes the driver sent on each line, e.g. proving a + // checked-out task notification's rendered content actually + // reached the CLI's input (engine/claude_code_backend_test.go). + if f, ferr := os.OpenFile(logPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644); ferr == nil { + _, _ = f.WriteString(b) f.Close() } } + return strings.TrimRight(b, "\n"), err == nil + } + + if mode != "fast_no_drain" { + // Read the ONE turn-input line every mode's own first message + // carries. Every mode below that does not itself read a further + // line (i.e. every mode except "queue_injection") never calls + // readStdinLine again, so its own stdin simply sits unread (but + // still open) until the driver eventually closes it — harmless, + // mirroring how the real CLI ignores stdin it has no more use for + // mid-turn. + readStdinLine() } sessionID := os.Getenv("FAKE_CLAUDE_SESSION_ID") @@ -168,6 +192,57 @@ func main() { // race. time.Sleep(time.Hour) return + case "queue_injection": + // Proves the driver keeps stdin OPEN across a turn and delivers a + // prompt queued mid-turn as a SECOND stream-json input line to + // THIS SAME running child, rather than closing stdin right after + // the first write (the pre-fix shape) or waiting until the whole + // turn ends. Emits a "WAITING_FOR_QUEUE" marker the test's OnEvent + // hook waits for (a deterministic, non-sleep synchronization + // point: the driver has finished mapping this event, so the child + // is now blocked in readStdinLine for line two) before the test + // enqueues a prompt. If a second line never arrives — a driver + // that still closes stdin after line one sees immediate EOF here, + // not a hang, since a closed pipe's read returns right away — + // this reports a distinguishable result instead of echoing + // anything, so the red run fails on content, not a timeout. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{{"type": "text", "text": "WAITING_FOR_QUEUE"}}, + }, + }) + line, ok := readStdinLine() + resultText := "no second message received" + if ok { + var second struct { + Message struct { + Content string `json:"content"` + } `json:"message"` + } + if err := json.Unmarshal([]byte(line), &second); err == nil { + resultText = "received queued: " + second.Message.Content + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{{"type": "text", "text": resultText}}, + }, + }) + } + } + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": resultText, + "usage": map[string]any{ + "input_tokens": 5, + "output_tokens": 5, + }, + }) + return case "crash": // Emitted "system"/"init" above, THEN exits nonzero without ever // emitting a "result" event — the non-deterministic mid-session From 9e0a1e803c49e1986ca849e07fb7c11d23dae776 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 17:37:22 -0400 Subject: [PATCH 43/95] fix(engine): stop the claude-code pump losing writes and wedging on stop (#232) Observable problems: an adversarial review of #231 (majorcontext/ harness#231, commit 7918b6d, the mid-turn claude-code queue-delivery fix) found two real defects in runClaudeCodeTurn's stdin-writer pump. (1) A mid-turn injection whose stdin write to the running `claude` child FAILS (its read end closes right as the injection lands -- a `claude --bg` turn, or any child racing its own exit against the wake) was silently and PERMANENTLY lost, contradicting the pump's own "a delay, never a loss" comment. The pump appends the injected block into session history BEFORE attempting the write (correct: honest, durable bookkeeping regardless of outcome), but the end-of-turn recordClaudeCodeHistoryWatermark(len(s.History())) call ran unconditionally whenever the child started, counting that now-durable-but-undelivered message as already incorporated. claudeCodeHistoryDirectiveArgs then saw priorCount == watermark on the NEXT claude-code turn, not priorCount > watermark, so it never re-fired the --append-system-prompt get_conversation_history pull that message's only remaining path to the model depended on. The transcript showed the prompt as delivered; the model never saw it. (2) close(stopPump); <-pumpDone ran before cmd.Wait(), but a goroutine blocked inside a live stdin.Write call can never reach its own select to observe stopPump closing. A `claude --bg` leaked grandchild holding stdin's read end open -- or simply a full pipe buffer at the exact turn-boundary instant -- left the pump wedged inside that Write with nothing but ctx cancellation ever able to rescue it: the same class of wedge the StdoutPipe/StderrPipe handling in this same function already exists to prevent, just one pipe over. Design: (1) track the session-history length immediately before each mid-turn injection's own append, and cap the recorded watermark there if that injection's write then fails -- the watermark must never claim the CLI incorporated a message whose write never actually landed. (2) stdin now has exactly ONE closer: the outer goroutine, called right after close(stopPump) and unconditionally BEFORE waiting on pumpDone, never the pump itself. Go's os.File.Close, for a pipe-backed file, safely interrupts another goroutine's concurrent Read/Write on the same *os.File (the runtime integrates pipe fds with its own poller for exactly this), so this Close call running concurrently with a stuck pump Write unblocks it with an ordinary write error immediately -- the pump's existing best-effort failure handling already tolerates that. Semantic change: a failed mid-turn injection write now leaves the watermark strictly below the failed message's own position, so a later claude-code turn's own directive check re-fires the history pull instead of silently treating the session as caught up -- a delivery delay, finally actually never a loss. A stop landing while the pump is blocked in Write now retires it immediately instead of only on ctx cancellation. Native-provider sessions are unaffected; nothing in the native loop changed. Verification: engine/testdata/fakeclaude/main.go gained two new modes. "queue_injection_broken_pipe" closes its own stdin read end right after a synchronization marker, so a mid-turn injection lands on an already-closed pipe and fails deterministically -- TestClaudeCodeMidTurnInjectionWriteFailureDoesNotStrandWatermark red-verified against the pre-fix code (the second claude-code turn's argv carried no --append-system-prompt re-pull directive at all) and is green after the fix. "queue_injection_blocked_write" spawns a detached grandchild that inherits stdin (mirroring the existing "bg_leak" mode's identical technique for stdout/stderr) and sleeps past any sane test timeout, so a many-megabyte mid-turn injection blocks inside the driver's own write(2) call with nothing left to ever drain it -- TestClaudeCodeStopRetiresPumpBlockedInStdinWrite red-verified as a 15s hard-timeout failure against the pre-fix code and is green after the fix. go test -race ./engine/... and ./server/... pass; go build ./..., go vet ./..., and gofmt -l are clean; no leaked fakeclaude processes survive either test. --- engine/claude_code_backend.go | 78 ++++++++++-- engine/claude_code_backend_test.go | 189 +++++++++++++++++++++++++++++ engine/testdata/fakeclaude/main.go | 119 +++++++++++++++++- 3 files changed, 376 insertions(+), 10 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index c47dbc4e..61c95a47 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -583,6 +583,23 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro s.claudeCodeQueueWake.Store(&wake) stopPump := make(chan struct{}) pumpDone := make(chan struct{}) + // injectionFailedAtLen, when >= 0, is the session-history length + // (len(s.History())) recorded IMMEDIATELY BEFORE the mid-turn + // injected message whose stdin write then failed — the watermark + // recorded below must never advance PAST this point, or a failed + // injection is silently and permanently lost: the message sits in + // s.history (honest bookkeeping — the append below always runs + // before the write is even attempted) but claudeCodeHistoryDirectiveArgs + // would see priorCount == watermark on the NEXT claude-code turn and + // never re-fire the get_conversation_history pull that message's + // only remaining path to the model depends on (an adversarial review + // finding on #231: "a delay, never a loss" was not actually true for + // this path). -1 means no injection ever failed to write this turn. + // Written only by the pump goroutine below; read only after + // <-pumpDone (this function's own tail) — the channel close + // establishes happens-before, so no lock is needed for this plain + // int despite the two different goroutines touching it. + injectionFailedAtLen := -1 go func() { defer close(pumpDone) defer s.claudeCodeQueueWake.Store(nil) @@ -608,6 +625,7 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // delivered text is identical whichever path answers it, // and this append is what makes the delivery visible in // the session transcript. + beforeAppendLen := len(s.History()) block := strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n") s.append(message.Message{ ID: newID("msg"), @@ -624,13 +642,20 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // claude-code turn's claudeCodeHistoryDirectiveArgs // detects the gap and feeds it via --resume, a delay // never a loss (see engine/AGENTS.md: "Deliver each - // item once across tool and goal-turn drains"). The - // pipe is presumably gone either way, so stop pumping. - _ = stdin.Close() + // item once across tool and goal-turn drains") — + // PROVIDED the watermark recorded below stays capped + // at beforeAppendLen, which is exactly what recording + // it here accomplishes. Does NOT close stdin itself — + // see the outer goroutine's own comment on why stdin + // has exactly one closer now. + injectionFailedAtLen = beforeAppendLen return } case <-stopPump: - _ = stdin.Close() + // Does NOT close stdin here — see the outer goroutine's + // own comment below for why stdin has exactly one + // closer, and why that closer runs BEFORE, not after, + // this select even has a chance to observe stopPump. return } } @@ -674,11 +699,31 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro finalMsg, started, turnErr := s.consumeClaudeCodeStream(stdout, model) // No more input is coming for this child (mirrors the single-string // SDK path's own endInput()-on-first-"result" call — see the pump - // goroutine's own doc comment above): signal it to stop and wait for - // it to actually exit BEFORE this goroutine touches the child again - // (cmd.Wait(), below) — see the pump's "one writer, ever" note for - // why this ordering matters. + // goroutine's own doc comment above): signal it to stop, THEN close + // stdin — in that order, but both from THIS goroutine, before ever + // waiting on pumpDone. + // + // stdin has exactly ONE closer now: this goroutine, here, never the + // pump itself (see its own two return paths above). That is what + // makes this safe against the wedge an adversarial review found on + // #231 (majorcontext/harness#231, commit 7918b6d): the pump can be + // BLOCKED inside stdin.Write when stopPump closes — a `claude --bg` + // leaked grandchild holding stdin's read end open, or simply a full + // pipe buffer at the exact turn-boundary instant — and a goroutine + // blocked in a syscall never reaches its own select to observe a + // closed channel. Go's os.File.Close, for a pipe-backed file like + // this one, safely interrupts any OTHER goroutine's concurrent + // Read/Write on the SAME *os.File (the runtime integrates pipe fds + // with its own poller for exactly this), so THIS Close call — + // running concurrently with a stuck pump Write — unblocks it with an + // error immediately, same as the pump's own ordinary write-failure + // path already tolerates. Without this, <-pumpDone below could block + // indefinitely (nothing but ctx cancellation would ever rescue it), + // which is the same class of wedge the StdoutPipe/StderrPipe + // handling elsewhere in this function exists to prevent — just one + // pipe over, on stdin instead of stdout/stderr. close(stopPump) + _ = stdin.Close() <-pumpDone // inputErr captures a failure on the FIRST write specifically — see // its use in claudeCodeTurnResult below, whose contract is "no usable @@ -703,7 +748,22 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // tell a genuinely stale resumed session (history grew via an // intervening native-provider turn) apart from one that is merely // mid-turn. - s.recordClaudeCodeHistoryWatermark(len(s.History())) + // + // Capped at injectionFailedAtLen when a mid-turn injection's own + // write failed (see the pump's own doc comment above): the CLI + // almost certainly did NOT incorporate that message or anything + // appended after it (its stdin write never landed), so recording + // the FULL current length here would tell + // claudeCodeHistoryDirectiveArgs the resumed session is caught up + // when it is not, permanently stranding the failed message — + // exactly the gap TestClaudeCodeMidTurnInjectionWriteFailureDoesNotStrandWatermark + // regresses. injectionFailedAtLen is always <= len(s.History()) + // here (nothing removes history), so the min is just the cap. + n := len(s.History()) + if injectionFailedAtLen >= 0 && injectionFailedAtLen < n { + n = injectionFailedAtLen + } + s.recordClaudeCodeHistoryWatermark(n) } // cmd.Wait() below does NOT reintroduce the EOF wait that diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 49255bcd..cf9ee08d 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1686,3 +1686,192 @@ func TestClaudeCodeQueueInjectedMidTurnViaOpenStdin(t *testing.T) { t.Errorf("CLI stdin's second line = %q, want it to carry the queued prompt's text", lines[1]) } } + +// TestClaudeCodeMidTurnInjectionWriteFailureDoesNotStrandWatermark is the +// regression test for an adversarial-review finding on #231 (PR +// majorcontext/harness#231, commit 7918b6d): a mid-turn queued prompt +// whose stdin write to the running `claude` child FAILS (the child's read +// end closes right as the injection lands — a `claude --bg` turn, or any +// child racing its own exit against the wake) was silently and +// PERMANENTLY lost, contradicting the pump's own "a delay, never a loss" +// doc comment (runClaudeCodeTurn, engine/claude_code_backend.go). +// +// Root cause: the pump appends the injected block into session history +// BEFORE attempting the write (so a failed write still leaves the block +// durably in s.history — correct, honest bookkeeping), but +// runClaudeCodeTurn's end-of-turn watermark recording +// (recordClaudeCodeHistoryWatermark(len(s.History()))) ran unconditionally +// whenever the child started, counting that now-durable-but-undelivered +// message as if the CLI's own resumed session already had it. The NEXT +// claude-code turn's claudeCodeHistoryDirectiveArgs then computes +// priorCount == watermark (not priorCount > watermark), so it never fires +// the --append-system-prompt get_conversation_history re-pull that would +// have been the injected prompt's last chance to actually reach the +// model — silently dropped, though the transcript shows it as delivered. +// +// This test proves the fix: after a mid-turn injection's write fails, the +// recorded watermark must be capped BELOW the failed message's own +// position, so a LATER claude-code turn's own directive check sees +// priorCount > watermark and re-fires the history pull — the lost +// prompt's one remaining path back to the model. +func TestClaudeCodeMidTurnInjectionWriteFailureDoesNotStrandWatermark(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "queue_injection_broken_pipe") + + ready := make(chan struct{}) + var readyOnce sync.Once + s.cfg.OnEvent = func(ev Event) { + if ev.Type == EventMessage && ev.Message != nil && ev.Message.Parts.Text() == "STDIN_CLOSED_READY" { + readyOnce.Do(func() { close(ready) }) + } + } + + type outcome struct { + msg *message.Message + err error + } + done := make(chan outcome, 1) + go func() { + msg, err := s.Prompt(context.Background(), "start") + done <- outcome{msg, err} + }() + + select { + case res := <-done: + t.Fatalf("Prompt returned (%+v, %v) before fakeclaude ever emitted STDIN_CLOSED_READY", res.msg, res.err) + case <-ready: + case <-time.After(10 * time.Second): + t.Fatal("fakeclaude never emitted STDIN_CLOSED_READY within 10s") + } + + // The turn is now mid-flight with fakeclaude's own stdin read end + // already closed. Enqueue now: the pump's injection write is + // guaranteed to land on the closed pipe and fail. + if _, _, err := s.EnqueuePrompt("LOST-IF-BUGGY: please handle this", ""); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + + var res outcome + select { + case res = <-done: + case <-time.After(10 * time.Second): + t.Fatal("first Prompt did not return within 10s") + } + if res.err != nil { + t.Fatalf("first Prompt: %v", res.err) + } + + // The queue itself is still empty (dequeued, not stranded there) and + // the prompt's text is still honestly present in history — this test + // is about the WATERMARK, not about re-queueing or hiding the attempt. + if q := s.QueuedPrompts(); len(q) != 0 { + t.Errorf("QueuedPrompts() after the failed injection = %+v, want empty (dequeued once, not requeued)", q) + } + found := false + for _, m := range s.History() { + if m.Role == message.RoleUser && strings.Contains(m.Parts.Text(), "LOST-IF-BUGGY: please handle this") { + found = true + } + } + if !found { + t.Error("session history has no user message carrying the failed injection's text") + } + + // The second claude-code turn must re-fire the history-directive + // re-pull: proof the watermark did not silently cover the lost + // message. Without the fix, this argv carries no + // --append-system-prompt at all (mirrors + // TestClaudeCodeHistoryDirectiveAbsentOnConsecutiveClaudeTurns' + // negative case) — the model's only remaining path to the queued + // prompt. + if _, err := s.Prompt(context.Background(), "continue"); err != nil { + t.Fatalf("second Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + if len(invocations) != 2 { + t.Fatalf("invocations = %d, want 2: %+v", len(invocations), invocations) + } + got, ok := argvValueAfter(invocations[1], "--append-system-prompt") + if !ok || got != claudeCodeHistoryDirective { + t.Fatalf("second invocation --append-system-prompt = %q, ok=%v, want the history directive %q -- "+ + "the failed mid-turn injection was silently stranded (watermark advanced past it)", got, ok, claudeCodeHistoryDirective) + } +} + +// TestClaudeCodeStopRetiresPumpBlockedInStdinWrite is the regression test +// for the second adversarial-review finding on #231 (PR +// majorcontext/harness#231, commit 7918b6d): a stop landing (the child's +// own terminal "result" event arrives, closing stopPump) while the +// stdin-writer pump is BLOCKED inside its own stdin.Write call must still +// retire the pump promptly — not wedge <-pumpDone (and so the whole +// runClaudeCodeTurn call) until ctx cancellation, the same `claude --bg` +// class of wedge the StdoutPipe/StderrPipe handling elsewhere in this file +// already exists to prevent, just one pipe over. +// +// fakeclaude's "queue_injection_blocked_write" mode never reads stdin +// again after its own first marker message, so the driver's own mid-turn +// injection write — many times larger than any real OS pipe buffer, so it +// cannot possibly complete in one buffered chunk — blocks inside the +// write(2) syscall with nothing on the other end ever draining it. The +// select below is this test's OWN hard-timeout guard: with the pre-fix +// code (stdin closed only from inside the pump's own select branches, +// never reachable while blocked in a live Write call), this test hangs +// until it times out; with the fix (the outer goroutine closes stdin +// itself, unconditionally, right after close(stopPump), before ever +// waiting on pumpDone), Go's os.File.Close interrupts the pump's blocked +// Write immediately and the turn completes normally. +func TestClaudeCodeStopRetiresPumpBlockedInStdinWrite(t *testing.T) { + s, _ := claudeCodeTestSession(t, "queue_injection_blocked_write") + pidFile := filepath.Join(t.TempDir(), "leaked.pid") + t.Setenv("FAKE_CLAUDE_LEAK_PID_FILE", pidFile) + + waiting := make(chan struct{}) + var waitingOnce sync.Once + s.cfg.OnEvent = func(ev Event) { + if ev.Type == EventMessage && ev.Message != nil && ev.Message.Parts.Text() == "WAITING_FOR_QUEUE" { + waitingOnce.Do(func() { close(waiting) }) + } + } + + type outcome struct { + msg *message.Message + err error + } + done := make(chan outcome, 1) + go func() { + msg, err := s.Prompt(context.Background(), "start") + done <- outcome{msg, err} + }() + + select { + case res := <-done: + t.Fatalf("Prompt returned (%+v, %v) before fakeclaude ever emitted WAITING_FOR_QUEUE", res.msg, res.err) + case <-waiting: + case <-time.After(10 * time.Second): + t.Fatal("fakeclaude never emitted WAITING_FOR_QUEUE within 10s") + } + + // Many times larger than any real pipe buffer (typically 16-64KiB) -- + // the driver's own write of this, plus its JSON/OPERATOR-MESSAGES + // framing overhead, cannot complete in one buffered chunk while + // fakeclaude never reads any of it. + huge := strings.Repeat("X", 8*1024*1024) + if _, _, err := s.EnqueuePrompt(huge, ""); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + + select { + case res := <-done: + if res.err != nil { + killLeakedFakeClaude(t, pidFile) + t.Fatalf("Prompt: %v", res.err) + } + case <-time.After(15 * time.Second): + // Cleaning up here too: if this branch ever fires, the leaked + // grandchild would otherwise outlive the test. + killLeakedFakeClaude(t, pidFile) + t.Fatal("Prompt did not return within 15s of the child's \"result\" event — " + + "the stdin-writer pump is wedged inside a blocked Write, never retired by the stop " + + "(the claude --bg-class stdin wedge this test guards against)") + } + killLeakedFakeClaude(t, pidFile) +} diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 2c18a84f..1e22b801 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -32,7 +32,14 @@ // "queue_injection" (blocks for a SECOND stdin line mid-turn, proving the // driver's stdin-writer pump keeps stdin open and delivers a mid-turn // queued prompt to THIS running child instead of a fresh one — see its own -// comment below). +// comment below), "queue_injection_broken_pipe" (closes its own stdin read +// end before the driver ever gets a chance to write a mid-turn injection, +// so that write fails — proves a failed injection's watermark accounting +// does not silently strand it, see its own comment below), and +// "queue_injection_blocked_write" (never reads stdin again after its own +// first marker, so a large mid-turn injection fills the pipe and blocks +// the driver's own Write — proves a stop landing mid-write still retires +// the pump promptly instead of wedging, see its own comment below). package main import ( @@ -243,6 +250,116 @@ func main() { }, }) return + case "queue_injection_broken_pipe": + // Proves the driver's own watermark bookkeeping never advances + // PAST a mid-turn injection whose stdin write actually failed -- + // the shape a real child's read end closing right as the + // injection lands produces (EPIPE on a `claude --bg` turn, or any + // child that exits between the wake firing and the write + // happening). Emits WAITING_FOR_QUEUE (the same synchronization + // marker "queue_injection" uses), then closes ITS OWN stdin read + // end and emits a SECOND marker, STDIN_CLOSED_READY, only after + // that close has actually completed -- the test waits for THIS + // marker (not WAITING_FOR_QUEUE) before enqueueing, so the + // driver's injection write is guaranteed to land on an + // already-closed pipe and fail, deterministically, never a race + // against whether the close beat the write. Never reads a second + // line at all -- there is nothing left open to read from -- and + // completes normally, exactly like a turn that finished its own + // work with no idea a queued prompt ever existed. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{{"type": "text", "text": "WAITING_FOR_QUEUE"}}, + }, + }) + _ = os.Stdin.Close() + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{{"type": "text", "text": "STDIN_CLOSED_READY"}}, + }, + }) + // Give the driver's stdin-writer pump time to actually wake, + // dequeue, and attempt (and fail) its write against the + // now-closed pipe before this process finishes the turn -- + // otherwise a fast finish could race ahead of the test's own + // EnqueuePrompt call and leave nothing for the pump to even + // attempt. This is cross-process timing against a real + // subprocess (see e2e/AGENTS.md's carve-out for exactly this + // class of wait), not a substitute for the Go test's own + // event-driven synchronization, which never sleeps. + time.Sleep(300 * time.Millisecond) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "STDIN_CLOSED_READY", + "usage": map[string]any{ + "input_tokens": 5, + "output_tokens": 5, + }, + }) + return + case "queue_injection_blocked_write": + // Proves a stop landing while the driver's stdin-writer pump is + // BLOCKED inside its own stdin.Write call still retires it + // promptly, instead of wedging <-pumpDone (and so the whole + // turn) until ctx cancellation -- the shape a `claude --bg` + // leaked grandchild holding stdin's read end open produces. This + // mirrors "bg_leak" below exactly, just on stdin instead of + // stdout/stderr: spawn a grandchild that inherits THIS process's + // stdin (a DUPLICATED copy of the same pipe read end) and sleeps + // well past any sane test timeout, then let THIS direct child + // exit normally and quickly right after its own "result" -- + // cmd.Wait() on the direct child returns promptly either way (the + // existing EOF-avoidance design this file's own doc comment + // covers), but the pipe's read end stays held open by the + // grandchild regardless of the direct child's own lifetime, so + // ONLY an explicit stdin.Close() on the driver's side can ever + // unblock a write still in flight against it. The test kills the + // grandchild by PID (FAKE_CLAUDE_LEAK_PID_FILE, same convention + // "bg_leak" uses) once it has proved the driver did not wedge on + // it. + // + // Emits WAITING_FOR_QUEUE, spawns the leaker, then gives the + // driver's own mid-turn injection write (the test enqueues + // something many times larger than any real pipe buffer) time to + // actually start and block inside the OS write(2) call before + // this process emits "result" and exits -- cross-process timing + // against a real subprocess, not a substitute for the Go test's + // own event-driven synchronization (see e2e/AGENTS.md's carve-out + // for this class of wait). + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{{"type": "text", "text": "WAITING_FOR_QUEUE"}}, + }, + }) + leaker := exec.Command(os.Args[0]) + leaker.Env = []string{"FAKE_CLAUDE_MODE=bg_leak_child"} + leaker.Stdin = os.Stdin + if err := leaker.Start(); err == nil { + if pidFile := os.Getenv("FAKE_CLAUDE_LEAK_PID_FILE"); pidFile != "" { + _ = os.WriteFile(pidFile, []byte(strconv.Itoa(leaker.Process.Pid)), 0o644) + } + _ = leaker.Process.Release() + } + time.Sleep(500 * time.Millisecond) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "done despite a blocked mid-turn write", + "usage": map[string]any{ + "input_tokens": 5, + "output_tokens": 5, + }, + }) + return case "crash": // Emitted "system"/"init" above, THEN exits nonzero without ever // emitting a "result" event — the non-deterministic mid-session From dd5afee2904c3ef4670afc284d19aa77585ea259 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 21:11:22 -0400 Subject: [PATCH 44/95] fix(engine): store a reasoning turn as one assistant message with both parts (#234) * fix(engine): store a reasoning turn as one assistant message with both parts The `claude` CLI streams each content block of one model turn as its own stream-json "assistant" envelope: a "thinking" block arrives as a complete envelope on its own, immediately followed by a separate envelope for the answer text. consumeClaudeCodeStream mapped each envelope to its own message.Message, so a reasoning turn persisted as two adjacent assistant messages instead of one. Downstream, a one-bubble-per-message console rendered a single turn as two "Agent" bubbles ("Thought for a few seconds", then the answer). message.Message.Parts already supports a mixed Reasoning+Text message; the bug was in emission, not the data model. consumeClaudeCodeStream now buffers a reasoning-only envelope (pendingReasoning) and reattaches it to the front of the very next envelope's own parts before appending, so the durable and replayed record is one message with Parts = [Reasoning, Text] (or [Reasoning, ToolCall], etc.), in emission order. A differently-parented envelope (a subagent frame) or the stream ending before a follow-up arrives flushes the buffered reasoning standalone instead of dropping it. This narrows the crash-durability grain for a reasoning envelope from "one content block" to "one turn segment": a crash in the brief window between two already-written stdout lines now loses the buffered reasoning instead of leaving it journaled as an orphaned message. This driver was never durable per-token (an "assistant" envelope is a whole content block, not a delta), and a turn dying mid-stream with an orphaned final message is an existing, named failure mode (server/journal.go's recordTurnEnd already documents it), not a new risk this change introduces. Every reader of session history already iterates message.Parts generically (compact.go's estimatePartsBytes, mcp_history.go's writeFlattenedMessage, the session index/replay fold), so merging reasoning and text into one message needed no reader changes: byte totals, transcript text, and token/usage accounting are unchanged in aggregate. No historical-session coalescing is included by design scope: only newly emitted turns get the merged shape. Verified: TestClaudeCodeThinkingBlockDecodesToReasoningPart red-verified against the pre-fix code (asserted 3 messages; now asserts one merged message with a Reasoning part followed by a Text part) and TestClaudeCodeReasoningMergeDoesNotOverMerge (new, via fakeclaude's new "thinking_interleaved" mode) proves an unrelated leading text message is not swept into the merge and a text-reasoning-text turn does not collapse into one message. go test -race ./engine/... ./server/... ./message/... ./provider/..., go build ./..., go vet ./..., and gofmt are clean. e2e's TestAppendSystemPromptReachesModelOnServe fails identically on unmodified origin/main - pre-existing, unrelated, untouched here. * fix(engine): don't flush buffered reasoning on content-free activity Review of #234 found the pre-switch flush guard in consumeClaudeCodeStream fired on EVERY non-"assistant" envelope, including "system" and "rate_limit_event". A rate_limit_event can arrive mid-turn between a "thinking" envelope and its "text" envelope (its own doc comment already says limits can shift mid-turn), which flushed the buffered reasoning standalone and re-split the turn into two bubbles again - defeating the fix exactly on subscription/usage sessions, where rate_limit_events are common. Only "user" (tool_result) and "result" (turn-terminal) envelopes genuinely end the turn segment a buffered thinking block started. Narrow the guard to those two types and let content-free activity ("system", "rate_limit_event", any other non-terminating type) pass through without disturbing the buffer. The review also flagged that the two flush branches - a differently-parented envelope interrupting a buffered thinking block, and the stream ending before a follow-up ever arrives - had no test coverage of their own, so the "reasoning is flushed standalone, never dropped" guarantee was unverified. Added three fakeclaude modes and their tests: - "thinking_ratelimit_text" / TestClaudeCodeReasoningMergeSurvivesRateLimitEvent: thinking, then a rate_limit_event, then the completing text - proves the narrowed guard still merges into one message. Red-verified against the pre-fix (over-eager) guard: History() came back with 3 messages instead of 2. - "thinking_then_crash" / TestClaudeCodeReasoningFlushesStandaloneOnCrash: a thinking block immediately followed by a nonzero exit with no "result" event - proves the post-loop flush. - "thinking_then_subagent" / TestClaudeCodeReasoningFlushesStandaloneAcrossSubagentBoundary: a top-level thinking block immediately followed by an assistant envelope on a different parent_tool_use_id - proves the different-parent flush. The latter two already passed against the current code (the flush branches were already correct; only their test coverage was missing), so each was red-verified against its own named mechanism instead: with flushPendingReasoning temporarily forced to a no-op, both failed (History() came back one message short in each case), confirming the tests actually exercise the flush path before restoring the real implementation. go test -race ./engine/... ./server/..., go build ./..., go vet ./..., and gofmt are clean. --- engine/claude_code_backend.go | 177 +++++++++++++++++++++-- engine/claude_code_backend_test.go | 223 +++++++++++++++++++++++++++-- engine/testdata/fakeclaude/main.go | 167 ++++++++++++++++++++- 3 files changed, 545 insertions(+), 22 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 61c95a47..8f85a693 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -55,19 +55,29 @@ // (claudeCodeCLISessionID) for --resume on this harness session's next // delegated turn. Any other subtype (e.g. "api_retry") is activity // only — observed, never fatal. -// - "assistant": one COMPLETE API-level assistant message (text, thinking, -// and/or tool_use content blocks together) — NOT a token-by-token -// delta; this driver does not pass --include-partial-messages, so -// there is nothing more granular to forward. Decoded into one -// message.Message (Reasoning, Text, and ToolCall parts, in order), -// appended via plain Session.append (no usage — see the usage-mapping -// note below) and emitted as EventMessage, with one EventReasoningDelta -// per non-empty thinking part, one EventTextDelta per non-empty text -// part (folding the whole block's text into the message in a single -// "delta", the closest honest match to the native -// EventTextDelta/EventReasoningDelta contract given the CLI hands over -// complete blocks), and one EventToolStart per tool_use part. The -// envelope's own parent_tool_use_id (null at top level, the spawning +// - "assistant": one COMPLETE content block (text, thinking, or a single +// tool_use), NOT a token-by-token delta; this driver does not pass +// --include-partial-messages, so there is nothing more granular to +// forward. A real `claude` binary streams each content block of one +// logical model turn as its OWN "assistant" envelope — most visibly, a +// "thinking" block arrives as a complete envelope on its own, +// immediately followed by a SEPARATE envelope for whatever the model +// emits next. consumeClaudeCodeStream's pendingReasoning buffers a +// reasoning-only envelope and reattaches it to the front of the very +// next envelope's own parts, so a reasoning turn still persists (and +// replays) as ONE message.Message with a Reasoning part followed by +// whatever completed that turn segment (Text and/or ToolCall parts), +// matching message.Message.Parts's own documented shape — rather than +// two adjacent assistant messages a one-bubble-per-message console +// would render as two separate turns. Appended via plain +// Session.append (no usage — see the usage-mapping note below) and +// emitted as EventMessage, with one EventReasoningDelta per non-empty +// thinking part, one EventTextDelta per non-empty text part (folding +// the whole block's text into the message in a single "delta", the +// closest honest match to the native EventTextDelta/EventReasoningDelta +// contract given the CLI hands over complete blocks), and one +// EventToolStart per tool_use part. The envelope's own +// parent_tool_use_id (null at top level, the spawning // tool_use id inside a subagent's own turn) rides onto the appended // message as Message.ParentToolUseID, unmodified. // - "user": Claude Code's own tool execution results, arriving in the @@ -1017,6 +1027,72 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // any realistic single stream-json line without an unbounded read. scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024) + // pendingReasoning buffers the Parts of an "assistant" envelope whose + // ONLY content is a "thinking"/"redacted_thinking" block — see the + // "assistant" case below and reasoningOnlyParts. A real `claude` + // binary streams every content block of one logical model-turn + // segment as its OWN top-level "assistant" envelope: a "thinking" + // block arrives as a complete envelope on its own, immediately + // followed by a SEPARATE envelope for whatever the model emits next + // (the answer text, a tool_use, or more thinking), even though + // Anthropic's own Messages API returns all of it as ONE assistant + // message with multiple content blocks. Appending each envelope as + // its own message.Message therefore persisted a reasoning turn as + // two adjacent assistant messages — one Reasoning-only, one + // Text-only — which a one-bubble-per-message console rendered as two + // separate "Agent" bubbles for a single turn ("Thought for a few + // seconds" then the answer). Buffering a reasoning-only envelope and + // reattaching it to the FRONT of the next envelope's own parts + // (below) restores the one-message-per-turn-segment shape this + // file's event-mapping doc above already assumed. + // + // Durability: this narrows the crash-durability grain for a + // reasoning-only envelope from "one content block" to "one turn + // segment" — a process crash in the brief window between reading this + // stdout line and the next one (both already written by the child; + // no external call or delay separates them) now loses the buffered + // reasoning instead of leaving it journaled as an orphaned message. + // This is not a new class of risk, only a narrower window on an + // already-accepted one: this driver's durability was never per-token + // (the "assistant" case above is NOT a token-by-token delta — nothing + // is durable before a whole content block has been read), and a turn + // that dies mid-stream with an orphaned final message is an existing, + // named failure mode the engine surfaces as such rather than + // preventing (see server/journal.go's recordTurnEnd doc comment, + // "final assistant message reasoning-only, no text, no tool call"). + var pendingReasoning message.Parts + var pendingReasoningParent string + + // flushPendingReasoning appends any buffered reasoning as a + // standalone assistant message. This is the uncommon path: it fires + // only when a differently-parented envelope interrupts a buffered + // thinking block (a subagent frame interleaving with the main + // thread's own reasoning — never observed live against a real + // binary, but never silently dropped either) or when the stream ends + // before a reasoning-only envelope is ever followed by another one + // (an aborted or crashed turn). The common case — thinking + // immediately followed by the rest of its own turn segment — never + // reaches here; it merges instead, in the "assistant" case below. + flushPendingReasoning := func() { + if len(pendingReasoning) == 0 { + return + } + msg := message.Message{ + ID: newID("msg"), + Role: message.RoleAssistant, + Parts: pendingReasoning, + Model: model, + Origin: message.OriginClaudeCode, + CreatedAt: time.Now().UTC(), + ParentToolUseID: pendingReasoningParent, + } + s.append(msg) + s.emit(Event{Type: EventMessage, Message: &msg}) + finalMsg = &msg + pendingReasoning = nil + pendingReasoningParent = "" + } + for scanner.Scan() { line := bytes.TrimSpace(scanner.Bytes()) if len(line) == 0 { @@ -1029,6 +1105,24 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // see this file's package doc on permissive decoding. continue } + if (env.Type == "user" || env.Type == "result") && len(pendingReasoning) > 0 { + // Only a "user" (tool_result) or "result" (turn-terminal) + // envelope genuinely ends the turn segment a buffered + // thinking block started — either means the model turn that + // owns this reasoning is over, so flush now rather than risk + // losing it if the turn ends abnormally right after. + // + // Deliberately NOT any non-"assistant" type: "system" and + // "rate_limit_event" are content-free activity that can + // legitimately land BETWEEN a "thinking" envelope and the + // text envelope that completes its own turn segment — see + // rate_limit_event's own doc comment ("a long-running turn + // can see its own limits shift mid-turn"). Flushing on those + // would re-split the very turn this buffer exists to keep + // merged, exactly on subscription/usage sessions where + // rate_limit_events are common. + flushPendingReasoning() + } switch env.Type { case "system": // ANY "system" event — not only subtype "init" — is proof the @@ -1049,6 +1143,42 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( if len(msg.Parts) == 0 { continue } + if len(pendingReasoning) > 0 { + if env.ParentToolUseID == pendingReasoningParent { + // The common case: this envelope is the rest of the + // turn segment the buffered thinking block started — + // reattach it to the front rather than flush it as + // its own message. A fresh backing array avoids + // aliasing either slice's own storage. + merged := make(message.Parts, 0, len(pendingReasoning)+len(msg.Parts)) + merged = append(merged, pendingReasoning...) + merged = append(merged, msg.Parts...) + msg.Parts = merged + } else { + // A different parent thread interrupted the buffered + // thinking block: flush it standalone rather than + // merge reasoning from one thread onto content from + // another. + flushPendingReasoning() + } + pendingReasoning = nil + pendingReasoningParent = "" + } + if reasoningOnlyParts(msg.Parts) { + // Buffer it — do not append or emit EventMessage yet — + // and wait for the envelope that completes this turn + // segment (see pendingReasoning's own doc comment). Still + // emit EventReasoningDelta immediately below so live + // streaming UX is unaffected by the buffering. + pendingReasoning = msg.Parts + pendingReasoningParent = env.ParentToolUseID + for _, p := range msg.Parts { + if r, ok := p.(*message.Reasoning); ok && r.Text != "" { + s.emit(Event{Type: EventReasoningDelta, Text: r.Text}) + } + } + continue + } s.append(msg) s.emit(Event{Type: EventMessage, Message: &msg}) for _, p := range msg.Parts { @@ -1149,9 +1279,30 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // Any other top-level "type" (this driver has none documented // beyond the four above) is inert activity, per the package doc. } + // The scanner loop ended without ever reaching "result" (a crashed or + // truncated stream) — flush any buffered thinking block now rather + // than silently drop it. See flushPendingReasoning's own doc comment. + flushPendingReasoning() return finalMsg, started, turnErr } +// reasoningOnlyParts reports whether parts is non-empty and every part is +// a *message.Reasoning — the shape claudeCodeAssistantMessage produces for +// an "assistant" envelope carrying nothing but "thinking"/ +// "redacted_thinking" blocks. See pendingReasoning's own doc comment in +// consumeClaudeCodeStream. +func reasoningOnlyParts(parts message.Parts) bool { + if len(parts) == 0 { + return false + } + for _, p := range parts { + if _, ok := p.(*message.Reasoning); !ok { + return false + } + } + return true +} + // claudeCodeEnvelope is the outer discriminator every line of `claude // --output-format stream-json` decodes into. Fields are read permissively // (encoding/json ignores JSON fields with no matching Go field, and every diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index cf9ee08d..3d291f92 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1073,7 +1073,19 @@ func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning -// part, is appended to history, and is emitted as EventReasoningDelta. +// part, is appended to history AS PART OF THE SAME MESSAGE as the text that +// follows it, and is emitted as EventReasoningDelta. +// +// The real `claude` binary streams the "thinking" block and the "text" +// block that completes the same turn segment as TWO separate stream-json +// "assistant" envelopes (see fakeclaude's "thinking" mode and +// consumeClaudeCodeStream's pendingReasoning doc comment). Before the fix +// this regresses, the engine appended one message.Message per envelope, +// so a single reasoning turn persisted as two adjacent assistant +// messages — one Reasoning-only, one Text-only — which a one-bubble- +// per-message console rendered as two separate "Agent" bubbles for one +// turn. The correct shape is ONE assistant message carrying both parts, +// in emission order. func TestClaudeCodeThinkingBlockDecodesToReasoningPart(t *testing.T) { s, _ := claudeCodeTestSession(t, "thinking") @@ -1089,27 +1101,222 @@ func TestClaudeCodeThinkingBlockDecodesToReasoningPart(t *testing.T) { } hist := s.History() - // user prompt, assistant(thinking), assistant(text) = 3 messages. - if len(hist) != 3 { - t.Fatalf("History() len = %d, want 3: %+v", len(hist), hist) + // user prompt, assistant(reasoning+text merged into ONE message) = 2 + // messages — NOT 3 (the pre-fix shape: user, assistant(thinking), + // assistant(text)). + if len(hist) != 2 { + t.Fatalf("History() len = %d, want 2: %+v", len(hist), hist) + } + asst := hist[1] + if asst.Role != message.RoleAssistant { + t.Fatalf("hist[1].Role = %q, want assistant", asst.Role) } - reasoning, ok := hist[1].Parts[0].(*message.Reasoning) - if hist[1].Role != message.RoleAssistant || !ok || reasoning.Text != "Let me reason about this." { - t.Fatalf("hist[1] = %+v, want an assistant Reasoning(%q)", hist[1], "Let me reason about this.") + if len(asst.Parts) != 2 { + t.Fatalf("hist[1].Parts = %+v, want exactly 2 parts (Reasoning then Text)", asst.Parts) + } + reasoning, ok := asst.Parts[0].(*message.Reasoning) + if !ok || reasoning.Text != "Let me reason about this." { + t.Fatalf("hist[1].Parts[0] = %+v, want a Reasoning(%q)", asst.Parts[0], "Let me reason about this.") } if len(reasoning.ProviderData) == 0 { t.Error("Reasoning.ProviderData is empty, want the thinking block's signature carried through") } + text, ok := asst.Parts[1].(*message.Text) + if !ok || text.Text != "Here is my answer." { + t.Fatalf("hist[1].Parts[1] = %+v, want a Text(%q)", asst.Parts[1], "Here is my answer.") + } - var sawReasoningDelta bool + var sawReasoningDelta, sawTextDelta bool for _, ev := range events { if ev.Type == EventReasoningDelta && ev.Text == "Let me reason about this." { sawReasoningDelta = true } + if ev.Type == EventTextDelta && ev.Text == "Here is my answer." { + sawTextDelta = true + } } if !sawReasoningDelta { t.Error("no EventReasoningDelta for the thinking block") } + if !sawTextDelta { + t.Error("no EventTextDelta for the text block") + } + + // Exactly one EventMessage for the whole merged turn (not one per + // envelope): proves the reasoning-only envelope was buffered, not + // flushed as its own message. + var messageEvents int + for _, ev := range events { + if ev.Type == EventMessage { + messageEvents++ + } + } + if messageEvents != 1 { + t.Errorf("EventMessage count = %d, want 1 (one merged message for the turn)", messageEvents) + } +} + +// TestClaudeCodeReasoningMergeDoesNotOverMerge drives fakeclaude's +// "thinking_interleaved" sequence (text, then thinking, then the text that +// completes THAT thinking block's own turn segment) and proves the +// pendingReasoning merge in consumeClaudeCodeStream attaches ONLY forward: +// the independent leading text stays its own message, and only the +// reasoning + the text immediately after it merge into one. A merge rule +// that instead swept every adjacent assistant envelope together (or +// merged backward) would either over-merge this into a single message or +// drop the leading text — this proves neither happens. +func TestClaudeCodeReasoningMergeDoesNotOverMerge(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking_interleaved") + + msg, err := s.Prompt(context.Background(), "think about it, twice") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if got := msg.Parts.Text(); got != "And here is the rest." { + t.Errorf("final message text = %q", got) + } + + hist := s.History() + // user prompt, assistant("First, a quick note."), + // assistant(reasoning+"And here is the rest." merged) = 3 messages. + if len(hist) != 3 { + t.Fatalf("History() len = %d, want 3: %+v", len(hist), hist) + } + if hist[1].Role != message.RoleAssistant || len(hist[1].Parts) != 1 || hist[1].Parts.Text() != "First, a quick note." { + t.Fatalf("hist[1] = %+v, want a standalone assistant Text(%q)", hist[1], "First, a quick note.") + } + asst := hist[2] + if asst.Role != message.RoleAssistant || len(asst.Parts) != 2 { + t.Fatalf("hist[2] = %+v, want an assistant message with exactly 2 parts", asst) + } + reasoning, ok := asst.Parts[0].(*message.Reasoning) + if !ok || reasoning.Text != "Now let me reason about the rest." { + t.Fatalf("hist[2].Parts[0] = %+v, want a Reasoning(%q)", asst.Parts[0], "Now let me reason about the rest.") + } + text, ok := asst.Parts[1].(*message.Text) + if !ok || text.Text != "And here is the rest." { + t.Fatalf("hist[2].Parts[1] = %+v, want a Text(%q)", asst.Parts[1], "And here is the rest.") + } +} + +// TestClaudeCodeReasoningMergeSurvivesRateLimitEvent drives fakeclaude's +// "thinking_ratelimit_text" sequence (thinking, then a rate_limit_event, +// then the text that completes the thinking block's own turn segment) and +// proves the pre-switch flush guard in consumeClaudeCodeStream does not +// treat a content-free "rate_limit_event" as ending the turn segment. A +// guard that flushes pendingReasoning on every non-"assistant" envelope +// re-splits the turn right here — exactly on subscription/usage sessions, +// where rate_limit_events are common (see rate_limit_event's own doc +// comment: "a long-running turn can see its own limits shift mid-turn"). +func TestClaudeCodeReasoningMergeSurvivesRateLimitEvent(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking_ratelimit_text") + + msg, err := s.Prompt(context.Background(), "think about it, with a rate limit event") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if got := msg.Parts.Text(); got != "Here is my answer after the rate-limit event." { + t.Errorf("final message text = %q", got) + } + + hist := s.History() + // user prompt, assistant(reasoning+text merged into ONE message) = 2 + // messages — NOT 3 (a rate_limit_event wrongly flushing the buffer + // between the thinking and text envelopes). + if len(hist) != 2 { + t.Fatalf("History() len = %d, want 2: %+v", len(hist), hist) + } + asst := hist[1] + if asst.Role != message.RoleAssistant || len(asst.Parts) != 2 { + t.Fatalf("hist[1] = %+v, want an assistant message with exactly 2 parts", asst) + } + reasoning, ok := asst.Parts[0].(*message.Reasoning) + if !ok || reasoning.Text != "Reasoning across a rate-limit event." { + t.Fatalf("hist[1].Parts[0] = %+v, want a Reasoning(%q)", asst.Parts[0], "Reasoning across a rate-limit event.") + } + text, ok := asst.Parts[1].(*message.Text) + if !ok || text.Text != "Here is my answer after the rate-limit event." { + t.Fatalf("hist[1].Parts[1] = %+v, want a Text(%q)", asst.Parts[1], "Here is my answer after the rate-limit event.") + } + + // The rate_limit_event itself must still be mapped onto + // SubscriptionUsage — this test does not just prove the merge + // survives it, but that the event was genuinely processed, not + // silently skipped. + usage := s.SubscriptionUsage() + if usage == nil || len(usage.Windows) == 0 { + t.Error("SubscriptionUsage() is empty, want the rate_limit_event mapped through") + } +} + +// TestClaudeCodeReasoningFlushesStandaloneOnCrash drives fakeclaude's +// "thinking_then_crash" sequence (a thinking block immediately followed by +// a nonzero exit with no "result" event at all) and proves +// flushPendingReasoning's post-loop flush: the buffered reasoning must +// survive as a standalone assistant message rather than being silently +// dropped when consumeClaudeCodeStream's scanner loop ends via EOF/crash +// before any envelope ever completes its turn segment. +func TestClaudeCodeReasoningFlushesStandaloneOnCrash(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking_then_crash") + + _, err := s.Prompt(context.Background(), "think about it, then crash") + if err == nil { + t.Fatal("Prompt returned no error for a child that crashed without a result event") + } + + hist := s.History() + // user prompt, assistant(reasoning only, flushed standalone) = 2 + // messages. NOT 1 (the reasoning silently dropped). + if len(hist) != 2 { + t.Fatalf("History() len = %d, want 2: %+v", len(hist), hist) + } + asst := hist[1] + if asst.Role != message.RoleAssistant || len(asst.Parts) != 1 { + t.Fatalf("hist[1] = %+v, want a standalone assistant message with exactly 1 part", asst) + } + reasoning, ok := asst.Parts[0].(*message.Reasoning) + if !ok || reasoning.Text != "Reasoning right before a crash." { + t.Fatalf("hist[1].Parts[0] = %+v, want a Reasoning(%q)", asst.Parts[0], "Reasoning right before a crash.") + } +} + +// TestClaudeCodeReasoningFlushesStandaloneAcrossSubagentBoundary drives +// fakeclaude's "thinking_then_subagent" sequence (a top-level thinking +// block, parent_tool_use_id "", immediately followed by an assistant +// envelope on a DIFFERENT parent_tool_use_id) and proves +// flushPendingReasoning's different-parent flush: the buffered reasoning +// must flush standalone rather than merge onto content from a different +// thread. +func TestClaudeCodeReasoningFlushesStandaloneAcrossSubagentBoundary(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking_then_subagent") + + msg, err := s.Prompt(context.Background(), "think, then spawn a subagent") + if err != nil { + t.Fatalf("Prompt: %v", err) + } + if got := msg.Parts.Text(); got != "Working inside the subagent." { + t.Errorf("final message text = %q", got) + } + + hist := s.History() + // user prompt, assistant(reasoning only, parent ""), + // assistant("Working inside the subagent.", parent "toolu_parent") = 3 + // messages — NOT a 2-message merge across the parent boundary. + if len(hist) != 3 { + t.Fatalf("History() len = %d, want 3: %+v", len(hist), hist) + } + reasoningMsg := hist[1] + if reasoningMsg.Role != message.RoleAssistant || len(reasoningMsg.Parts) != 1 || reasoningMsg.ParentToolUseID != "" { + t.Fatalf("hist[1] = %+v, want a standalone top-level assistant Reasoning message", reasoningMsg) + } + reasoning, ok := reasoningMsg.Parts[0].(*message.Reasoning) + if !ok || reasoning.Text != "Reasoning about which subagent to spawn." { + t.Fatalf("hist[1].Parts[0] = %+v, want a Reasoning(%q)", reasoningMsg.Parts[0], "Reasoning about which subagent to spawn.") + } + subagentMsg := hist[2] + if subagentMsg.Role != message.RoleAssistant || subagentMsg.ParentToolUseID != "toolu_parent" || subagentMsg.Parts.Text() != "Working inside the subagent." { + t.Fatalf("hist[2] = %+v, want an assistant Text on parent toolu_parent", subagentMsg) + } } // TestClaudeCodeParentToolUseIDCarriedOntoMessage proves the envelope's own diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 1e22b801..51bfeb23 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -10,7 +10,21 @@ // Beyond "normal"/"hang"/"error", a handful of narrower modes each cover // exactly one of the gaps claude_code_backend.go closes: "thinking" (a // thinking content block plus the result event's own ttft_ms/duration_ms -// timing fields), "subagent" (a null-then-set parent_tool_use_id pair), +// timing fields), "thinking_interleaved" (text, then thinking, then the +// text that completes THAT thinking block's own turn segment — proves the +// reasoning-buffer merge in consumeClaudeCodeStream attaches only forward, +// never sweeping in an unrelated, already-flushed text message), +// "thinking_ratelimit_text" (thinking, then a rate_limit_event, then the +// text that completes the thinking block's own turn segment — proves the +// pre-switch flush guard does not treat content-free activity as ending +// the turn segment, which would re-split it), "thinking_then_crash" (a +// thinking block immediately followed by a nonzero exit with no "result" +// event — proves the buffered reasoning survives the post-loop flush +// instead of being dropped), "thinking_then_subagent" (a top-level +// thinking block immediately followed by an assistant envelope on a +// DIFFERENT parent_tool_use_id — proves the buffered reasoning flushes +// standalone instead of merging across threads), "subagent" +// (a null-then-set parent_tool_use_id pair), // "rate_limit_error"/"deterministic_error" (a result event this file's own // claudeCodeRetryableClass must classify retryable/not-retryable, // respectively), "crash"/"crash_before_init" (a child that exits nonzero @@ -436,6 +450,157 @@ func main() { "duration_ms": 800, }) return + case "thinking_interleaved": + // Proves the reasoning-buffer merge (claude_code_backend.go's + // pendingReasoning) attaches ONLY forward, to the envelope that + // immediately follows a thinking block, never backward onto an + // unrelated text envelope that preceded it — i.e. a + // text-reasoning-text turn must not collapse into one giant + // message. Sequence: independent text, then a thinking block, + // then the text that completes ITS OWN turn segment. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "First, a quick note."}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Now let me reason about the rest.", "signature": "sig-def"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "And here is the rest."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "And here is the rest.", + "usage": map[string]any{ + "input_tokens": 30, + "output_tokens": 15, + }, + }) + return + case "thinking_ratelimit_text": + // Proves the pre-switch flush guard in consumeClaudeCodeStream + // does NOT flush a buffered thinking block on content-free + // activity: a "rate_limit_event" (the CLI's own subscription + // signal, which its own doc comment says can arrive mid-turn, + // shifting limits) lands BETWEEN the thinking envelope and the + // text envelope that completes its own turn segment. The result + // must still be ONE merged [Reasoning, Text] message, exactly + // like the "thinking" mode above — a guard that flushes on every + // non-"assistant" envelope re-splits the turn right here. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Reasoning across a rate-limit event.", "signature": "sig-rl"}, + }, + }, + }) + emit(map[string]any{ + "type": "rate_limit_event", + "rate_limit_info": map[string]any{ + "status": "allowed", + "resetsAt": 1788785267, + "rateLimitType": "five_hour", + "unifiedWindows": map[string]any{ + "five_hour": map[string]any{"utilization": 0.02, "resetsAt": 1788785267}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Here is my answer after the rate-limit event."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Here is my answer after the rate-limit event.", + "usage": map[string]any{ + "input_tokens": 22, + "output_tokens": 11, + }, + }) + return + case "thinking_then_crash": + // Proves flushPendingReasoning's post-loop flush: a thinking + // block arrives, then the child crashes (nonzero exit, no + // "result" event at all) — the buffered reasoning must still + // survive as a standalone assistant message rather than being + // silently dropped when consumeClaudeCodeStream's scanner loop + // ends via EOF. Mirrors the plain "crash" mode above, but with a + // thinking block emitted first. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Reasoning right before a crash.", "signature": "sig-crash"}, + }, + }, + }) + os.Exit(1) + case "thinking_then_subagent": + // Proves flushPendingReasoning's different-parent flush: a + // top-level (parent_tool_use_id "") thinking block is immediately + // followed by an assistant envelope belonging to a DIFFERENT + // thread (a subagent's own parent_tool_use_id) — the buffered + // reasoning must flush standalone rather than merge onto content + // from a different thread. + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Reasoning about which subagent to spawn.", "signature": "sig-subagent"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "parent_tool_use_id": "toolu_parent", + "message": map[string]any{ + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "Working inside the subagent."}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "Working inside the subagent.", + "usage": map[string]any{ + "input_tokens": 18, + "output_tokens": 9, + }, + }) + return case "subagent": emit(map[string]any{ "type": "assistant", From 5f8975a9d8b892ea669cdd6fc2608535fcbc39d9 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 21:37:42 -0400 Subject: [PATCH 45/95] feat(engine): surface claude-code session cost via subscription_usage (#235) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `claude` CLI's stream-json "result" event carries total_cost_usd, its own dollar-cost accounting for a delegated turn. This package decoded the field and then discarded it: provider.Usage has no cost concept (every other adapter derives cost from token counts), so there was nowhere for it to go. The boxes console's Claude usage readout consequently shows only a rate-limit percentage and goes silent the moment a subscription exhausts and turns start billing as pay-as-you-go overage, hiding the one number that actually matters at that point. total_cost_usd is not an overage-only signal: live-verified against a real `claude` 2.1.252 binary, it is present on every delegated turn's "result" event, plain-subscription turns included, reporting what that turn would have cost at metered API rates even though the user is not billed it. message.SubscriptionUsage.SessionCostUSD reflects that honestly: a *float64 that goes non-nil the moment a session completes its first "claude"-lane turn and only grows from there, not a field that lights up only during overage. A caller that wants to gate a dollar readout on real billing, not a hypothetical subscription-turn equivalent, reads Overage.InUse alongside it. Session.applyClaudeCodeUsage — the existing choke point that folds a "result" event's token usage into cumulative Usage/LastUsage — now also sums costUSD into a new claudeCodeSessionCostUSD accumulator, persisted in the same claude_code.usage journal record (a new pointer field, nil for a pre-existing record so replay does not mistake "no cost tracking yet" for "zero-cost turn") and folded back on LoadSession exactly like the token totals beside it. Session.SubscriptionUsage() merges the accumulator into its returned snapshot, synthesizing a minimal claude-provider snapshot for the edge case where cost has arrived but no rate_limit_event ever has. Verification: new tests in engine/claude_code_backend_test.go prove the total accumulates across turns, stays nil before any delegated turn completes, and survives a process reload via the journal; go test -race ./engine/... ./server/... ./message/..., go vet ./..., and gofmt all clean. server/openapi.yaml documents the new session_cost_usd field on SubscriptionUsage. --- engine/claude_code_backend.go | 41 ++++++++++++---- engine/claude_code_backend_test.go | 75 ++++++++++++++++++++++++++++++ engine/engine.go | 55 ++++++++++++++++++---- engine/store.go | 27 +++++++++-- message/subscription_usage.go | 24 ++++++++++ server/openapi.yaml | 14 ++++++ 6 files changed, 212 insertions(+), 24 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 8f85a693..9d059eb3 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -102,7 +102,11 @@ // for a known-transient shape (see claudeCodeRetryableClass), a plain // error otherwise. TotalCostUSD has no home in provider.Usage (no // adapter carries a cost field — every consumer derives cost from -// token counts) and is deliberately dropped, not persisted. +// token counts); applyClaudeCodeUsage instead sums it directly into +// the session's own message.SubscriptionUsage.SessionCostUSD, durable +// via the same recClaudeCodeUsage record as the token usage above — +// see that field's own doc comment for why this is reported every +// turn, not only during pay-as-you-go overage. // - "rate_limit_event": the CLI's own subscription rate-limit/quota // signal, typically the SECOND event of a turn (right after // "system"/"init"). Never appended as a message — it carries no @@ -326,7 +330,10 @@ func (s *Session) recordClaudeCodeHistoryWatermark(n int) { // applyClaudeCodeUsage folds a delegated turn's AGGREGATE usage (the // "result" event's own usage field, covering every internal API call // Claude Code made across the whole turn — not just the closing one) into -// Session.Usage()/LastUsage(), durably (recClaudeCodeUsage, store.go). +// Session.Usage()/LastUsage(), and costUSD (the same event's own +// total_cost_usd) into the session's cumulative +// message.SubscriptionUsage.SessionCostUSD (see that field's own doc +// comment), durably (recClaudeCodeUsage, store.go carries both). // // This is deliberately NOT routed through appendWithUsage: by the time a // "result" event arrives, every message this turn produced has already @@ -343,7 +350,12 @@ func (s *Session) recordClaudeCodeHistoryWatermark(n int) { // compaction never runs for a delegated session (see PromptWithOrigin's // dispatch), so there is no native trigger signal here to protect, and // GET /session should still report an accurate last-turn size. -func (s *Session) applyClaudeCodeUsage(usage provider.Usage) { +// +// costUSD is summed unconditionally, every turn, not gated on overage: +// see claudeCodeEnvelope.TotalCostUSD's own doc comment for why a plain +// subscription turn reports a real (if not actually billed) dollar +// figure too, live-verified against a real `claude` 2.1.252 binary. +func (s *Session) applyClaudeCodeUsage(usage provider.Usage, costUSD float64) { s.mu.Lock() defer s.mu.Unlock() s.usage.InputTokens += usage.InputTokens @@ -352,7 +364,9 @@ func (s *Session) applyClaudeCodeUsage(usage provider.Usage) { s.usage.CacheWriteTokens += usage.CacheWriteTokens s.lastUsage = usage s.haveLastUsage = true - s.persistClaudeCodeUsage(usage) + s.claudeCodeSessionCostUSD += costUSD + s.haveClaudeCodeCost = true + s.persistClaudeCodeUsage(usage, costUSD) } // runClaudeCodeTurn drives ONE turn through the `claude` CLI against s's @@ -1215,7 +1229,7 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( } case "result": usage := mapClaudeCodeUsage(env.Usage) - s.applyClaudeCodeUsage(usage) + s.applyClaudeCodeUsage(usage, env.TotalCostUSD) streamMillis := env.DurationMillis - env.TTFTMillis if streamMillis < 0 { // A CLI build that sends duration_ms but not ttft_ms (or @@ -1243,9 +1257,10 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( turnErr = provider.MarkRetryable(turnErr, class) } } - // TotalCostUSD is intentionally dropped here — see the - // package doc's "result" bullet: provider.Usage has no cost - // field for it to occupy. + // TotalCostUSD was already folded into the session's + // cumulative SessionCostUSD by applyClaudeCodeUsage above — + // see that call's own comment and message.SubscriptionUsage. + // SessionCostUSD's own doc comment. // // Return NOW rather than falling through to another // scanner.Scan(): "result" is the documented terminal event @@ -1317,8 +1332,14 @@ type claudeCodeEnvelope struct { Result string `json:"result,omitempty"` Usage *claudeCodeUsage `json:"usage,omitempty"` // TotalCostUSD is Claude Code's own dollar-cost accounting for the - // whole delegated turn — read but deliberately never mapped onto - // anything (see this file's package doc, "result" bullet). + // whole delegated turn — folded into the session's cumulative + // message.SubscriptionUsage.SessionCostUSD by applyClaudeCodeUsage + // (see this file's package doc, "result" bullet, and that field's own + // doc comment). Reported on every "result" event a real `claude` + // 2.1.252 binary sends, not only during pay-as-you-go overage; zero, + // not an error, for an older build that omits the field — this file's + // usual permissive-decoding philosophy, same as TTFTMillis/ + // DurationMillis below. TotalCostUSD float64 `json:"total_cost_usd,omitempty"` // ParentToolUseID is null (so absent, or explicit JSON null — either // decodes to "" for a plain string field, encoding/json's documented diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 3d291f92..e65146c0 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1457,6 +1457,81 @@ func TestClaudeCodeRateLimitEventWithNoOverageOmitsOverage(t *testing.T) { } } +// TestClaudeCodeSessionCostAccumulatesAcrossTurns proves a delegated +// session's message.SubscriptionUsage.SessionCostUSD sums the `claude` +// CLI's own per-turn total_cost_usd (fakeclaude's "normal" mode reports +// 0.0123 on every turn's "result" event) across successive turns, rather +// than reporting only the latest turn's figure — see +// Session.applyClaudeCodeUsage's own doc comment for why this is a +// cumulative session total, not a last-turn snapshot. +func TestClaudeCodeSessionCostAccumulatesAcrossTurns(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + + if _, err := s.Prompt(context.Background(), "first turn"); err != nil { + t.Fatalf("first Prompt: %v", err) + } + usage := s.SubscriptionUsage() + if usage == nil || usage.SessionCostUSD == nil { + t.Fatalf("SubscriptionUsage() after first turn = %+v, want a non-nil SessionCostUSD", usage) + } + if got, want := *usage.SessionCostUSD, 0.0123; got < want-1e-9 || got > want+1e-9 { + t.Errorf("SessionCostUSD after first turn = %v, want %v", got, want) + } + + if _, err := s.Prompt(context.Background(), "second turn"); err != nil { + t.Fatalf("second Prompt: %v", err) + } + usage = s.SubscriptionUsage() + if usage == nil || usage.SessionCostUSD == nil { + t.Fatalf("SubscriptionUsage() after second turn = %+v, want a non-nil SessionCostUSD", usage) + } + if got, want := *usage.SessionCostUSD, 0.0246; got < want-1e-9 || got > want+1e-9 { + t.Errorf("SessionCostUSD after second turn = %v, want %v (0.0123 summed twice)", got, want) + } +} + +// TestClaudeCodeSessionCostNilBeforeAnyTurn proves SessionCostUSD stays +// absent (nil, per message.SubscriptionUsage.SessionCostUSD's own doc +// comment) for a session that has not completed a "claude"-lane turn in +// this process yet — the counterpart to +// TestClaudeCodeSessionCostAccumulatesAcrossTurns, which proves the +// non-nil case. +func TestClaudeCodeSessionCostNilBeforeAnyTurn(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + + if got := s.SubscriptionUsage(); got != nil { + t.Fatalf("SubscriptionUsage() before any turn = %+v, want nil", got) + } +} + +// TestClaudeCodeSessionCostSurvivesReload proves SessionCostUSD is durable +// (recClaudeCodeUsage now carries the per-turn cost alongside token +// usage — see persistClaudeCodeUsage) — a process restart between two +// delegated turns must not silently reset the running dollar total to +// only the post-reload turn's own cost. +func TestClaudeCodeSessionCostSurvivesReload(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + + if _, err := s.Prompt(context.Background(), "first turn"); err != nil { + t.Fatalf("first Prompt: %v", err) + } + + reloaded, err := LoadSession(Config{ + SessionDir: s.cfg.SessionDir, + ClaudeCode: s.cfg.ClaudeCode, + }, s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + usage := reloaded.SubscriptionUsage() + if usage == nil || usage.SessionCostUSD == nil { + t.Fatalf("reloaded SubscriptionUsage() = %+v, want a non-nil SessionCostUSD", usage) + } + if got, want := *usage.SessionCostUSD, 0.0123; got < want-1e-9 || got > want+1e-9 { + t.Errorf("reloaded SessionCostUSD = %v, want %v", got, want) + } +} + // TestClaudeCodeRetryableClassification proves a "result" event this file // can actually name as transient provider weather (a rate-limit signal) is // wrapped provider.RetryableError, while a genuinely deterministic result diff --git a/engine/engine.go b/engine/engine.go index 293c2185..65a2c43a 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -1017,6 +1017,21 @@ type Session struct { // delegated/subscription turn regardless. subscriptionUsage *message.SubscriptionUsage + // claudeCodeSessionCostUSD/haveClaudeCodeCost carry this session's + // cumulative "claude"-lane delegated-turn dollar cost (see + // message.SubscriptionUsage.SessionCostUSD's own doc comment) — the + // running sum of every completed delegated turn's own + // claudeCodeEnvelope.TotalCostUSD, folded in by + // Session.applyClaudeCodeUsage and durable via the claude_code.usage + // journal record (see persistClaudeCodeUsage/store.go's + // recClaudeCodeUsage replay), unlike subscriptionUsage above which is + // process-local only. haveClaudeCodeCost distinguishes "no delegated + // turn has ever completed" (false, SessionCostUSD reports nil) from + // "a turn completed and its cost happened to be exactly zero" (true) — + // mirrors haveLastUsage's own role for lastUsage. + claudeCodeSessionCostUSD float64 + haveClaudeCodeCost bool + // turnUnsettled is SessionManager.recoverInterruptedTurnLocked's // restart-recovery signal, replacing an earlier, unreliable // heuristic (hasUnansweredTurn, since removed) that tried to infer @@ -2149,18 +2164,38 @@ func (s *Session) applySubscriptionUsage(u message.SubscriptionUsage) { func (s *Session) SubscriptionUsage() *message.SubscriptionUsage { s.mu.Lock() defer s.mu.Unlock() - if s.subscriptionUsage == nil { + if s.subscriptionUsage == nil && !s.haveClaudeCodeCost { return nil } - cp := *s.subscriptionUsage - cp.Windows = append([]message.SubscriptionUsageWindow(nil), s.subscriptionUsage.Windows...) - if s.subscriptionUsage.Overage != nil { - // Deep-copy Overage too, not just Windows: the struct copy above - // (cp := *s.subscriptionUsage) only copies the pointer value, so - // without this a caller mutating the returned snapshot's Overage - // would mutate this session's own stored one. - overage := *s.subscriptionUsage.Overage - cp.Overage = &overage + var cp message.SubscriptionUsage + if s.subscriptionUsage != nil { + cp = *s.subscriptionUsage + cp.Windows = append([]message.SubscriptionUsageWindow(nil), s.subscriptionUsage.Windows...) + if s.subscriptionUsage.Overage != nil { + // Deep-copy Overage too, not just Windows: the struct copy + // above (cp := *s.subscriptionUsage) only copies the pointer + // value, so without this a caller mutating the returned + // snapshot's Overage would mutate this session's own stored + // one. + overage := *s.subscriptionUsage.Overage + cp.Overage = &overage + } + } else { + // haveClaudeCodeCost is true but no rate_limit_event snapshot has + // ever arrived (a `claude` build that completed a delegated turn + // without ever sending one) — synthesize a minimal "claude"-lane + // snapshot so SessionCostUSD still has somewhere to live, rather + // than silently dropping a real cost figure because Windows/ + // Overage happen to be unknown. + cp = message.SubscriptionUsage{ + Provider: "claude", + Windows: []message.SubscriptionUsageWindow{}, + CapturedAt: s.cfg.Now().Unix(), + } + } + if s.haveClaudeCodeCost { + cost := s.claudeCodeSessionCostUSD + cp.SessionCostUSD = &cost } return &cp } diff --git a/engine/store.go b/engine/store.go index ac10c630..b2af3896 100644 --- a/engine/store.go +++ b/engine/store.go @@ -345,6 +345,14 @@ type record struct { // is never called with 0 in practice (a delegated turn always appends // at least the pending trigger message before this is recorded). ClaudeCodeHistoryWatermark int `json:"claude_code_history_watermark,omitempty"` + // ClaudeCodeCostUSD carries a recClaudeCodeUsage record's own + // per-turn total_cost_usd (see Session.applyClaudeCodeUsage and + // message.SubscriptionUsage.SessionCostUSD's own doc comment) — a + // pointer, not a plain float64, so a record written before this field + // existed decodes to nil (no cost accrual on replay) rather than + // being indistinguishable from an explicit zero-cost turn written by + // the current code, which always sets this non-nil (even to &0.0). + ClaudeCodeCostUSD *float64 `json:"claude_code_cost_usd,omitempty"` } // applyGoalRecord folds one goal.* record into the durable goal state a @@ -743,9 +751,11 @@ func (s *Session) persistClaudeCodeHistoryWatermark(n int) { } // persistClaudeCodeUsage appends a claude_code.usage record to the session -// log. It mirrors persistModel/persistEffort exactly: a no-op until the -// log exists (lazy creation), caller holds s.mu. -func (s *Session) persistClaudeCodeUsage(usage provider.Usage) { +// log, carrying both the turn's token usage and its own costUSD (see +// record.ClaudeCodeCostUSD's own doc comment). It mirrors persistModel/ +// persistEffort exactly: a no-op until the log exists (lazy creation), +// caller holds s.mu. +func (s *Session) persistClaudeCodeUsage(usage provider.Usage, costUSD float64) { if s.cfg.SessionDir == "" || !s.logStarted { return } @@ -753,7 +763,7 @@ func (s *Session) persistClaudeCodeUsage(usage provider.Usage) { s.lastPersistErr = err return } - if err := s.writeRecord(record{Type: recClaudeCodeUsage, Usage: &usage}); err != nil { + if err := s.writeRecord(record{Type: recClaudeCodeUsage, Usage: &usage, ClaudeCodeCostUSD: &costUSD}); err != nil { s.lastPersistErr = err } } @@ -1580,6 +1590,15 @@ func LoadSession(cfg Config, id string) (*Session, error) { s.lastUsage = *rec.Usage s.haveLastUsage = true } + // See record.ClaudeCodeCostUSD's own doc comment: nil means a + // record written before cost tracking existed, not a + // zero-cost turn, so this deliberately leaves + // haveClaudeCodeCost false in that case rather than folding a + // phantom zero into the running total. + if rec.ClaudeCodeCostUSD != nil { + s.claudeCodeSessionCostUSD += *rec.ClaudeCodeCostUSD + s.haveClaudeCodeCost = true + } case recMCPToolsSelected: // Union every record, in log order, into the restored selected // set (see mcp_lazy.go). Replay is defensive, like diff --git a/message/subscription_usage.go b/message/subscription_usage.go index 92335c3d..da3869de 100644 --- a/message/subscription_usage.go +++ b/message/subscription_usage.go @@ -38,6 +38,30 @@ type SubscriptionUsage struct { // CapturedAt is when this snapshot was captured — harness's own clock // (Config.Now), not a provider-reported time — Unix seconds. CapturedAt int64 `json:"captured_at"` + // SessionCostUSD is this session's cumulative dollar cost across every + // completed "claude"-lane delegated turn, summed turn over turn from + // the `claude` CLI's own per-turn total_cost_usd accounting (see + // engine/claude_code_backend.go's claudeCodeEnvelope.TotalCostUSD). + // Always nil for provider "codex" (its x-codex-* headers carry no + // cost figure). + // + // The `claude` CLI reports total_cost_usd on EVERY delegated turn's + // "result" event, live-verified against a real `claude` 2.1.252 + // binary — not only during pay-as-you-go overage. A plain- + // subscription turn still carries the dollar figure that turn would + // have cost at metered API rates, informational even though the user + // is not actually billed it. So this field goes non-nil the moment a + // session completes its FIRST "claude"-lane turn, whether or not + // Overage is ever set, and only grows from there. nil means "no + // delegated turn has completed in this process yet" — never "zero + // spend so far" — matching this type's own process-local capture + // contract (see this type's own doc comment). + // + // A caller that wants to gate a dollar readout on ACTUAL pay-as-you-go + // billing, not a hypothetical subscription-turn equivalent, must check + // Overage.InUse alongside this field — a non-nil SessionCostUSD alone + // is not proof the user was charged anything. + SessionCostUSD *float64 `json:"session_cost_usd,omitempty"` } // SubscriptionUsageWindow is one rate-limit window inside a diff --git a/server/openapi.yaml b/server/openapi.yaml index aedc07c9..43436b84 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -294,6 +294,20 @@ components: captured_at: type: integer description: When this snapshot was captured, Unix seconds. + session_cost_usd: + type: number + nullable: true + description: > + This session's cumulative dollar cost across every completed + "claude"-lane delegated turn, summed turn over turn from the + `claude` CLI's own per-turn total_cost_usd accounting. The CLI + reports total_cost_usd on EVERY delegated turn's result, not + only during pay-as-you-go overage, so this goes non-null the + moment a session completes its first "claude"-lane turn and + only grows from there — non-null does not by itself mean the + user was actually billed; check overage.in_use for that. + Always null for provider "codex" and for a session that has + not yet completed a "claude"-lane turn in this process. SubscriptionUsageWindow: type: object From 920117781fdff86b3a9c9bc04722eaa55f3df74b Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 21:58:50 -0400 Subject: [PATCH 46/95] feat(modelmeta): give codex a context-window entry like claude-code (#236) codex had no ContextWindow case, so a "codex"-provider ref (the form boxes mints for a ChatGPT Codex backend model, e.g. codex/gpt-5.6-sol) always missed. RequireContextWindow's fail-loud default then refused every codex session at create time, which forced boxes to disable that check globally with context_window_required: false rather than only for genuinely unknown models. Add a codex case that looks the model up in the same openaiContextWindows table the openai case already uses, since a codex ref and its openai/* counterpart name one underlying model over two transports. Unlike claudeCodeProvider, this case has no stand-in fallback: a codex model absent from the table still misses, so RequireContextWindow's refusal stays armed for a model this table genuinely does not know. Verified with go test -race ./modelmeta/... ./engine/..., go build ./..., go vet ./..., and gofmt -l . (all clean). Red-verified TestContextWindowCodex against pre-change code (missed: tokens=0, ok=false); TestContextWindowCodexUnknownModelStillMisses passed before the change too, since an unrecognized provider already misses trivially, but locks in that the new case must not gain a claude-code-style always-true fallback. --- modelmeta/modelmeta.go | 22 ++++++++++++++++++++++ modelmeta/modelmeta_test.go | 25 +++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/modelmeta/modelmeta.go b/modelmeta/modelmeta.go index ce894a34..52135505 100644 --- a/modelmeta/modelmeta.go +++ b/modelmeta/modelmeta.go @@ -205,6 +205,20 @@ func ContextWindow(ref message.ModelRef) (tokens int, ok bool) { model = stripBedrockVersionSuffix(suffix) } tokens, ok = openaiContextWindows[model] + case codexProvider: + // A ref routed through the ChatGPT Codex backend (see + // meetneptune/boxes internal/api/codex_models.go, which mints refs + // like "codex/gpt-5.6-sol") names the SAME underlying OpenAI model + // its "openai/*" counterpart does — openaiContextWindows already + // keys every codex model boxes uses (gpt-5.6-sol, gpt-5.6-terra, + // gpt-5.6-luna) — so this case looks the model up in that one + // table rather than duplicating it. Unlike claudeCodeProvider + // below, there is no stand-in fallback: a codex model absent from + // the table still misses, so engine.Config.RequireContextWindow's + // fail-loud refusal (see engine/context_window.go) stays armed for + // a genuinely unknown model instead of a boxes-side override + // disabling it globally. + tokens, ok = openaiContextWindows[model] case "amazon-bedrock": if suffix, isAnthropic := stripBedrockAnthropicPrefix(model); isAnthropic { tokens, ok = bedrockAnthropicContextWindows[stripBedrockVersionSuffix(suffix)] @@ -237,6 +251,14 @@ func ContextWindow(ref message.ModelRef) (tokens int, ok bool) { // engine's TestClaudeCodeProviderFamilyMatchesModelmeta. const claudeCodeProvider = "claude-code" +// codexProvider is the message.ModelRef.Provider value the boxes platform +// mints for a ChatGPT Codex backend model (see +// meetneptune/boxes internal/api/codex_models.go, e.g. "codex/gpt-5.6-sol") +// — distinct from provider/openai.CodexFamily, which names an "openai"-type +// provider's Client.Family for the same backend's wire format, not a +// message.ModelRef.Provider value this package switches on. +const codexProvider = "codex" + // claudeCodeContextWindow is the stand-in context-window figure reported // for claudeCodeProvider — see the ContextWindow case above for why its // exact value carries no real weight. diff --git a/modelmeta/modelmeta_test.go b/modelmeta/modelmeta_test.go index 8fa3212a..88fe9a27 100644 --- a/modelmeta/modelmeta_test.go +++ b/modelmeta/modelmeta_test.go @@ -23,6 +23,31 @@ func TestContextWindowOpenAI(t *testing.T) { } } +// TestContextWindowCodex proves a "codex"-provider ref (the boxes platform's +// form for a ChatGPT Codex backend model — see +// meetneptune/boxes internal/api/codex_models.go, which mints refs like +// "codex/gpt-5.6-sol") resolves from the SAME openaiContextWindows table the +// "openai" provider case already uses: gpt-5.6-sol is served over two +// different transports (openai/gpt-5.6-sol and codex/gpt-5.6-sol) but names +// one model, so it must report one context window. +func TestContextWindowCodex(t *testing.T) { + tokens, ok := ContextWindow(message.ModelRef{Provider: "codex", Model: "gpt-5.6-sol"}) + if !ok || tokens != 1_050_000 { + t.Fatalf("ContextWindow(codex/gpt-5.6-sol) = %d, %v; want 1050000, true", tokens, ok) + } +} + +// TestContextWindowCodexUnknownModelStillMisses proves the codex case does +// not fall back to a stand-in figure the way claudeCodeProvider does: a +// codex ref naming a model absent from openaiContextWindows must still miss, +// so engine.Config.RequireContextWindow's fail-loud refusal stays armed for +// a genuinely unknown model instead of silently reporting a guess. +func TestContextWindowCodexUnknownModelStillMisses(t *testing.T) { + if tokens, ok := ContextWindow(message.ModelRef{Provider: "codex", Model: "gpt-nonexistent"}); ok { + t.Errorf("ContextWindow(codex/gpt-nonexistent) = %d, true; want ok=false", tokens) + } +} + func TestContextWindowBedrockRegionPrefixes(t *testing.T) { cases := []string{ "anthropic.claude-opus-4-8", From 977fe4553d25328bbba4c9cb664ab1c3595bc050 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Tue, 1 Sep 2026 23:06:51 -0400 Subject: [PATCH 47/95] feat(server,engine): accept image and PDF attachments on prompt_async (#233) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(server,engine): accept image attachments on prompt_async The console composer has no way to send a picture. `prompt_async` rejects every non-text part with "v1 accepts text parts only", so an operator who wants to show a screenshot must describe it instead — even though the engine, every provider adapter, and imageclamp already handle `message.Blob` parts for images tools return. This lifts that limit on the interactive prompt route only, and threads attachments through every path a prompt can take, so an image can never be silently dropped between accept and delivery: - `decodePromptParts` (server/prompt_parts.go) validates each blob part before a run slot is claimed: media type in the same image set read_file returns as a blob, inline data (not a url), data that really decodes as the type it claims, and a 20MB per-attachment cap. A rejection appends no message and starts no turn. Validating here is the point: a user message is persisted into an append-only log before any provider sees it, so a blob nothing can render would 400 on every later turn from inside a durable transcript -- the wedge imageclamp exists to heal. - `Session.PromptWithOrigin` takes variadic blobs and builds its user message through `promptParts`: text first, then one Blob part per attachment. An attachment-only prompt is valid and carries no empty text part. - `QueuedPrompt.Blobs`/`promptRecord.Blobs` carry attachments through the durable queue, so a prompt sent while a turn is running -- or waiting across a restart -- still arrives with its picture. The bytes ride on prompt.queued only; prompt.dequeued names its entry by ID. - Both mid-turn drains deliver them: the tool-call-boundary append and the goal loop's turn-boundary injection. `operatorMessagesBlock` is text, so it announces each prompt's attachment count and the bytes ride beside it as Blob parts. - The claude-code lane sends the CLI's own content-block array on its stream-json stdin (text block plus base64 image blocks), keeping the bare-string content shape for every text-only turn. `POST /session/{id}/enqueue` stays text-only: its callers are machine relays whose upstream ack rides on the durable-accept contract, not the interactive composer. Verified: new server tests cover accept, image-only, each rejection, and a queued prompt keeping its image; new engine tests cover part shape, journal persistence, replay after restart, the operator-block marker, and the claude-code input line. Red-verified against "v1 accepts text parts only". The claude-code content-block shape was checked against the real CLI (`--input-format stream-json` with a base64 image block: the model answered from the image). Full `go test -race ./...` passes. * feat(server,engine): accept PDF attachments on prompt_async prompt_async accepted only images. A person with a PDF -- a spec, a resume, a report -- had no way to hand it to an agent, even though Anthropic and OpenAI both take documents natively and the claude-code CLI reads them from its stream-json stdin. The media-type set becomes a table of type -> verifier rather than a bare allowlist. Each entry now carries the check that proves the bytes really are what the caller claims, because the two types need different proof: an image is verified by decoding its header through the registered decoders, a PDF by its %PDF- magic. Keeping them in one table means adding a type is a row plus a verifier, and makes it impossible to add a type with no check at all. A type earns a row only when EVERY provider lane delivers it. A session switches models freely and an attachment stays in its history forever, so a type one lane drops would fail every later turn from inside a durable transcript. Images and application/pdf both clear that bar: anthropic and claude-code send image/document blocks, openai sends input_image/input_file. text/plain and docx do not -- they reach openai's transcodeBlob and openaicompat's blobURL as "unsupported blob media type" -- so they stay out. The size cap does more work for a PDF than for an image, and the comment now says so. imageclamp downscales an oversized image at transcode time, but nothing can rewrite a document: a PDF past a provider's own request ceiling (Anthropic's is 32MB) would fail on every turn with no repair path. The 20MB cap sits below that ceiling deliberately. claude_code_backend picks the stdin content block from the media type -- "document" for a PDF, "image" for an image -- instead of hardcoding "image". Verified end to end against the real claude-code CLI, which read a PDF's text back through stream-json stdin, and through the console: a PDF resume attached in the composer came back correctly summarized. A PNG mislabeled application/pdf is rejected by the %PDF- check rather than reaching a provider. * fix(engine,server): stop calling every prompt attachment an image Review of #233 found "image" wording left over from the commit that accepted images only, across queue.go, handlers.go, prompt_parts.go, openapi.yaml, and the queue doc. One of those was not prose. operatorMessagesBlock renders a marker into the OPERATOR MESSAGES block the model reads, and it said "[N image attachment(s) attached below]" for whatever the prompt carried -- so a PDF queued mid-turn was announced to the model as a picture. The marker is now "[N attachment(s) attached below]". Naming no media type is the right shape, not a shortcut. message/wire_normalize.go spells the same situation both ways: "image attachment(s)" on a path that only ever carries images, and a generic "[N attachment(s) omitted: ...]" where the type varies. This drain is the second kind -- a queued prompt carries whatever prompt_parts.go admits -- so it follows the generic form, and queue.go now records why. engine/prompt_attachments_test.go asserted the old marker literally and now asserts the new one, so the string is pinned in both directions rather than free to drift back. go test -race ./engine/ ./server/ clean. * fix(engine): deliver a queued prompt's attachments in the claude-code lane Review of #233 found the claude-code lane's mid-turn queue drain sending a queued prompt's TEXT while dropping its attachments: it appended a bare Text part to history and wrote stdin with no blobs. This was worse than losing the file quietly. operatorMessagesBlock had already rendered "[N attachment(s) attached below]" into the block the model reads, so the turn promised a file that never arrived -- the captured CLI stdin in the new test shows exactly that line with no image block after it. And because the durable history held no copy either, the next turn's --resume recovery had nothing to make good on, so engine/AGENTS.md's "a delay, never a loss" did not hold for the bytes. The drain now uses the same two helpers the native loop's own drainQueuedPromptsIntoHistory uses -- promptParts(block, queuedBlobs(queued)) for the history append, and the same blobs for the stdin write -- so both lanes deliver a queued attachment identically. The regression test asserts BOTH halves, because either alone would pass while the other still dropped the file: the child's captured stdin must carry an image content block with the PNG's own bytes, and the session history must hold exactly one Blob part with those bytes. Red-verified against the unfixed code: all three assertions fail. Also from the same review: verifyImageBytes reported a decoder format name where it had just compared two media types ("the data is jpeg" after "does not decode as image/png"), leaving the caller to infer that "jpeg" meant image/jpeg. It now reports "image/jpeg". That branch had no test -- the existing mislabel case feeds bytes that do not decode at all, which never reaches the comparison -- so a real JPEG claimed as image/png now covers it. go test -race ./engine/ ./server/ clean. * fix(server): bound the prompt body and require inline attachment data Third review pass on #233 found three things. An oversized request body could force unbounded allocation. Blob data arrives as base64 and encoding/json decodes it into a []byte during Unmarshal, so promptAttachmentMaxBytes -- checked per attachment inside decodePromptParts -- only ran AFTER the server had already paid for whatever the caller sent. Nothing capped how many attachments one body carried either, so N x 20MB was unbounded in aggregate. handlePrompt now wraps the body in http.MaxBytesReader at promptRequestMaxBytes (32 MiB, the same ceiling Anthropic applies to a whole request, which is the limit a prompt has to fit inside anyway) and answers 413 rather than decoding it. handleEnqueue needs no bound: its parts are text-only. The new test is the case the per-blob cap cannot catch on its own -- two attachments each UNDER the per-attachment limit that together exceed the request bound -- and asserts 413 plus an empty transcript. The OpenAPI request schema still let a prompt attachment be url-only, which decodePromptBlob rejects. BlobPart itself has to keep allowing either, because a blob in a TRANSCRIPT legitimately carries a url, so the tightening is a new PromptBlobPart (allOf BlobPart + required data) used only by PromptRequest.parts. Its description records why a url is refused: honoring one would make harness ask every provider, and imageclamp, to fetch a caller-supplied URL from inside the box. TestPromptAsyncRejectsUnusableBlob declared a pngBytes fixture it never used, kept alive by a `_ = pngBytes` no-op. Both are gone. go test -race ./engine/ ./server/ clean. * fix(server): make the 413 bound contractual and test what it claims Fourth review pass on #233. The oversize-body test did not demonstrate what its comment claimed. It built each attachment at promptRequestMaxBytes*2/3 (~21.3 MiB), which is LARGER than the 20 MiB per-attachment cap -- so the case it advertised, "each attachment individually legal, the request still too large", was never exercised. It passed because 413 fires first, which is exactly how a test can assert the wrong path and look right. Each attachment is now 2/3 of the PER-ATTACHMENT cap (~13.3 MiB), so two are ~26.7 MiB decoded and ~35.6 MiB base64 against the 32 MiB bound, and each is genuinely under the per-blob limit. A guard now pins that premise: if either constant later moves so the attachments become individually oversize, the request would still 413 and the test would still pass while silently covering the per-blob path instead, so it fails loudly rather than drifting. The OpenAPI description also promised that a bad attachment is "rejected with 400" without mentioning that an oversized body is refused with 413 before anything is decoded. A client could not tell the two apart from the contract: a 400 is a judgment about one attachment, a 413 says the request was never inspected. The description now says so, and prompt_async declares the 413 response it actually returns. Not addressed here, as it predates this change: the path declares no 400 response at all, though its description references 400 throughout. go test -race ./engine/ ./server/ clean. * docs(server): declare prompt_async's 400 response Fifth review pass on #233, and the finding I had raised myself when adding the 413: the path declared no 400 at all, though its request schema references 400 throughout and the handler returns it for every unusable attachment. A generated client could not represent the case, and the newly-declared 413 made the omission worse by implying 413 was the only rejection. The description spells out what a 400 covers -- unsupported media type, a url instead of inline data, bytes that are not the type claimed, one attachment past the per-attachment cap -- and says explicitly that each is a judgment about a specific part, in contrast to the 413's judgment about the request as a whole. Also puts the status codes back in ascending order; the 413 added in 2892d3c landed between 404 and 409. Contract only: no handler behavior changes. go test -race ./engine/ ./server/ clean. * fix(provider): omit an uncarryable attachment instead of wedging a session Sixth review pass on #233 found the real hole in this PR's own design rule, and it was mine. server/prompt_parts.go admits a media type only when every provider lane delivers it, "because a session switches models freely and the attachment stays in its history forever". I wrote that, then admitted application/pdf on the strength of anthropic, claude-code and openai -- without checking provider/openaicompat, the lane message/wire_normalize.go's own comment already names as the narrowest: its blobURL errors on anything that is not image/*, and says so explicitly about PDFs. Nothing repaired that downstream. NormalizeForWire only restructures tool calls and results; imageclamp downscales an oversized image but cannot rewrite a document. So the error reached transcodeUserMessage's `return nil, err` and failed the whole request -- and since a user attachment is DURABLE history, a session that attached a PDF under anthropic and then switched to any openaicompat provider would fail EVERY later turn, permanently, from inside its own transcript. Exactly the failure the rule exists to prevent. transcodeUserMessage now drops a non-image blob and appends "[N attachment(s) omitted: ]" -- the same wording and the same shape wire_normalize already uses when it drops a tool-result blob. The model is TOLD rather than left answering about a file it never received, and the session survives the provider switch with one degraded turn instead of a permanent wedge. blobURL still errors for any caller that reaches it directly; only the one shape a person can create is protected. TestTranscodeUserNonImageBlobErrors asserted the old behavior and is now TestTranscodeUserNonImageBlobOmitted, with a comment recording why the contract changed: erroring was safe while prompt_async took text only, because a non-image blob could then only come from a tool result. The two design comments that made the wrong claim -- prompt_parts.go's "every lane" list and wire_normalize.go's intersection -- now say what is actually true: PDF is the one entry that does not clear the bar on every lane, and openaicompat degrades rather than carrying it. go test -race ./... clean apart from the pre-existing tools/monitor/e2e TestRealEndToEnd failure, which fails identically on origin/main. * fix(engine): drop unusable blobs before the empty-prompt check Review of #233 (review 5084797022, which I missed on the first pass -- it never reached me and I reported "no findings since 4cca814" from the absence of a notification rather than from checking). EnqueuePrompt tested len(blobs) directly, so a caller passing a single nil satisfied "empty text is fine when a blob came with it". The queued prompt then persisted an unusable Blobs slice, and operatorMessagesBlock announced "[1 attachment(s) attached below]" to the model while promptParts skipped the nil on delivery -- the marker promising a file that never arrives, which is the same defect the claude-code mid-turn drain had (4cca814). usablePromptBlobs drops a nil entry and one carrying neither Data nor URL (every provider transcoder errors on that shape regardless of media type, per message/wire_normalize.go's intersection comment). It runs BEFORE the emptiness check, because the raw count is not a count of attachments, and its result is what gets persisted -- so len() answers "how many attachments will really be delivered" everywhere downstream. Red-verified: without it, three empty-text prompts carrying only unusable blobs enqueue, and the operator block counts them. Unrelated: TestAdoptRootRestoresLegacySettledChildAsUnknownFailureWhen LogCannotReconstruct is flaky on this branch AND on unmodified code (2 of 5 runs fail with my changes stashed) -- "mid's settled marker never landed durably". Not introduced here. * docs(server): state the attachment bar as deliver-or-degrade Review of #233 (approval recommended) caught the invariant contradicting its own exception: promptAttachmentTypes said a type belongs here only when EVERY lane DELIVERS it, then explained two paragraphs later that PDF is admitted because openaicompat drops it with a note. The bar was always deliver-or-degrade -- what it rules out is a lane that ERRORS, because an attachment lives in durable history and an error there fails every later turn of a session that merely switched into that lane. The sentence now says that, so the exception reads as the rule applying rather than as a hole in it. Comment only. --------- Co-authored-by: dev Co-authored-by: andybons --- docs/session-storage-and-queue.md | 32 +- engine/AGENTS.md | 3 +- engine/claude_code_backend.go | 139 +++++-- engine/claude_code_backend_test.go | 103 +++++ engine/engine.go | 58 ++- engine/goal.go | 20 +- engine/prompt_attachments_test.go | 318 ++++++++++++++++ engine/queue.go | 97 ++++- engine/store.go | 17 +- message/wire_normalize.go | 10 + provider/openaicompat/transcode.go | 35 ++ provider/openaicompat/transcode_test.go | 93 ++++- server/AGENTS.md | 13 +- server/handlers.go | 61 +-- server/openapi.yaml | 93 ++++- server/prompt_attachments_test.go | 480 ++++++++++++++++++++++++ server/prompt_parts.go | 250 ++++++++++++ 17 files changed, 1729 insertions(+), 93 deletions(-) create mode 100644 engine/prompt_attachments_test.go create mode 100644 server/prompt_attachments_test.go create mode 100644 server/prompt_parts.go diff --git a/docs/session-storage-and-queue.md b/docs/session-storage-and-queue.md index 094a498c..3985c31a 100644 --- a/docs/session-storage-and-queue.md +++ b/docs/session-storage-and-queue.md @@ -324,13 +324,25 @@ empty queue, and never touches a currently running turn — `POST /abort` is unrelated and does not touch the queue either way (it only cancels the in-flight turn's context). -Two v1 limits are deliberate, not gaps: **text-only** (queued prompts carry a -plain string — `QueuedPrompt{ID, Text}` — no attachment machinery, matching -the plain-prompt contract's `parts` being text-only already), and **a -per-request `model` override is silently dropped when the prompt is queued** -— there is no slot in `QueuedPrompt` to carry it through to a future drain, so -a caller that needs a model swap to take effect must re-issue the request once -it is confirmed `started`. +A queued prompt carries **text and file attachments** (images and PDFs, the +set every provider lane delivers): `QueuedPrompt{ID, Text, Blobs}`, persisted together on the `prompt.queued` record +(`promptRecord.Blobs`), so a file sent while a turn was running still +reaches the model when the queue drains — across a process restart included. +The bytes ride only on `prompt.queued`, never on `prompt.dequeued`, which +names its entry by ID. Every drain site delivers both halves: idle dispatch +and the tool-call boundary append the blobs as `Blob` parts of the user +message they build, and the goal loop's turn-boundary injection passes them +into its worker turn. The rendered `OPERATOR MESSAGES` block is text, so each +prompt that carries attachments announces them (`[N attachment(s) attached +below]`) — the marker names which numbered message an attachment belongs to. +It names no media type, because a queued prompt carries images and PDFs +alike. + +One v1 limit is still deliberate, not a gap: **a per-request `model` override +is silently dropped when the prompt is queued** — there is no slot in +`QueuedPrompt` to carry it through to a future drain, so a caller that needs a +model swap to take effect must re-issue the request once it is confirmed +`started`. `POST /session/{id}/enqueue` (docs/plans/2026-07-21-durable-enqueue.md) is `prompt_async`'s durable, idempotent sibling for a caller whose own upstream @@ -360,8 +372,10 @@ the pending queue (FIFO, `seq` present only on durable-enqueue entries), for an upstream recovering from its own crash to check what's already inside the durability domain instead of re-sending blind. `prompt_async` remains the right choice for an interactive client that has no upstream ack to protect — -it is not going away, and `POST /session/{id}/enqueue` adds no new limits -beyond what queued prompts already have (text-only, no model override). +it is not going away, and `POST /session/{id}/enqueue` adds one limit of its +own beyond what queued prompts have: **text parts only**. Its callers are +machine relays, not the interactive composer that uploads images, so the +durable-accept contract stays a string contract. The `fsync` in "write-ahead durability" above is itself mode-selectable: config's `session_sync` ("fsync", the default, or "volume") gates both this diff --git a/engine/AGENTS.md b/engine/AGENTS.md index 7467a5dd..3435b647 100644 --- a/engine/AGENTS.md +++ b/engine/AGENTS.md @@ -127,7 +127,8 @@ to a page. Bound scans to the indexed log size. - Let queued input beat goal auto-arm. - Deliver each item once across tool and goal-turn drains. - Do not auto-dispatch a restored queue at boot. -- Keep queued prompts text-only and model-override-free. +- Keep queued prompts model-override-free. +- Persist a queued prompt's attachments with it and deliver them at every drain. - Preserve durable sequence deduplication and its high-water mark. Keep the process manager box-scoped and shared. Runtime declarations are not diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 9d059eb3..784645bf 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -383,8 +383,8 @@ func (s *Session) applyClaudeCodeUsage(usage provider.Usage, costUSD float64) { // interruptedTurnError partial-append behavior (engine.go). func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, error) { history := s.History() - text := lastUserMessageText(history) - if text == "" { + text, blobs := lastUserMessageContent(history) + if text == "" && len(blobs) == 0 { return nil, errors.New("engine: claude-code delegated turn found no pending user message to answer") } // Deliver any pending task notifications (a settled child's Done/Failed @@ -631,7 +631,7 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // task-notification segment, above) — sent through the SAME // writer as every later mid-turn injection, never a separate // one-off path. - firstWriteErrCh <- writeClaudeCodeInputMessage(stdin, text) + firstWriteErrCh <- writeClaudeCodeInputMessage(stdin, text, blobs) for { select { case <-wake: @@ -651,13 +651,26 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // the session transcript. beforeAppendLen := len(s.History()) block := strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n") + // A queued prompt can carry attachments, so this drain + // delivers BOTH halves, exactly as the native loop's + // drainQueuedPromptsIntoHistory does with the same two + // helpers: promptParts puts the bytes in the durable + // history as Blob parts, and the stdin write below hands + // them to the running child as content blocks. Sending + // only the text would be worse than dropping the file + // quietly -- operatorMessagesBlock has already told the + // model "[N attachment(s) attached below]", so the turn + // would promise a file that never arrives, and the + // history would hold no copy for the next turn's + // --resume recovery to make good on. + injected := queuedBlobs(queued) s.append(message.Message{ ID: newID("msg"), Role: message.RoleUser, - Parts: message.Parts{&message.Text{Text: block}}, + Parts: promptParts(block, injected), CreatedAt: time.Now().UTC(), }) - if err := writeClaudeCodeInputMessage(stdin, block); err != nil { + if err := writeClaudeCodeInputMessage(stdin, block, injected); err != nil { // Best-effort, exactly like the first write's own // inputErr contract below: the prompt is already // dequeued AND durably in s.history by the append @@ -933,22 +946,31 @@ func claudeCodeTurnResult(o claudeCodeTurnOutcome) (*message.Message, error) { return o.finalMsg, nil } -// lastUserMessageText returns the Text of the LAST message in history if -// it is a RoleUser message, or "" otherwise. runClaudeCodeTurn's one +// lastUserMessageContent returns the text and the attachments (Blob parts, +// in order) of the LAST message in history if it is a RoleUser message, or +// "", nil otherwise. Both halves matter: Parts.Text() drops every non-text +// part, so reading text alone would silently strip an uploaded image on the +// way to the CLI. runClaudeCodeTurn's one // caller-contract requirement is that its caller has already appended // (or, for the goal-loop directive-reuse retry, left in place) exactly one // unanswered RoleUser message at the tail — this reads it back rather than // threading the text through an extra parameter, so both call sites (a // fresh append, and a retry that appends nothing new) share one path. -func lastUserMessageText(history []message.Message) string { +func lastUserMessageContent(history []message.Message) (string, []*message.Blob) { if len(history) == 0 { - return "" + return "", nil } last := history[len(history)-1] if last.Role != message.RoleUser { - return "" + return "", nil } - return last.Parts.Text() + var blobs []*message.Blob + for _, p := range last.Parts { + if b, ok := p.(*message.Blob); ok { + blobs = append(blobs, b) + } + } + return last.Parts.Text(), blobs } // claudeCodeHistoryDirective is the --append-system-prompt text @@ -969,7 +991,7 @@ const claudeCodeHistoryDirective = "You are continuing a conversation that happe // pair (flag plus value) whenever history holds conversation the CLI's own // resumed session (if any) has NOT already incorporated — priorCount > // watermark, where priorCount is len(history) minus the single pending -// trigger message runClaudeCodeTurn is about to answer (lastUserMessageText's +// trigger message runClaudeCodeTurn is about to answer (lastUserMessageContent's // own caller contract: history's last element is always that pending // message) and watermark is Session.claudeCodeHistoryWatermarkCount(), the // message count as of the end of whichever delegated turn last ran. nil @@ -1917,24 +1939,97 @@ type claudeCodeInputMessage struct { Message claudeCodeInputInnerMessage `json:"message"` } +// Content is a plain string for a text-only turn — the shape this driver +// has always written, kept byte-identical for the overwhelmingly common +// case — or the CLI's own content-block array when the turn's user message +// carries attachments. See claudeCodeInputContent. type claudeCodeInputInnerMessage struct { Role string `json:"role"` - Content string `json:"content"` + Content any `json:"content"` +} + +// claudeCodeInputBlock is one element of the content-block array form of +// claudeCodeInputInnerMessage.Content: the CLI accepts the Anthropic +// Messages API's own block shapes on its stream-json stdin, so an +// attachment rides as {"type":"image"|"document","source":{"type":"base64", +// ...}} exactly as it would in a native provider request. Text is set on a +// "text" block and Source on an attachment block; the other stays +// nil/empty and is omitted. +type claudeCodeInputBlock struct { + Type string `json:"type"` + Text string `json:"text,omitempty"` + Source *claudeCodeInputSource `json:"source,omitempty"` +} + +// claudeCodeInputSource is an image or document block's inline base64 +// payload. Data is []byte so encoding/json emits standard base64 — the same +// encoding message.Blob.Data uses on the wire. +type claudeCodeInputSource struct { + Type string `json:"type"` + MediaType string `json:"media_type"` + Data []byte `json:"data"` } -// writeClaudeCodeInputMessage marshals text as one stream-json user input -// line and writes it to w — mirrors the Claude Agent SDK's -// ProcessTransport.write (sdk.mjs: JSON.stringify(message) + "\n"). Used -// for both a turn's first, driving message and every later mid-turn -// queued-prompt injection, so the child's stdin only ever sees this one -// wire shape (see runClaudeCodeTurn's stdin-writer pump doc comment for -// the construct this mirrors). -func writeClaudeCodeInputMessage(w io.Writer, text string) error { +// claudeCodeInputContent renders one turn's input for the CLI's stdin: the +// bare string when the turn carries no attachments (unchanged wire shape), +// or a text block followed by one attachment block per blob when it does. +// +// The block TYPE follows the media type, matching the native anthropic +// adapter's own rule (provider/anthropic's transcodeBlob): an image/* blob +// is an "image" block, anything else — a PDF, today — is a "document" +// block. Both shapes are verified against the real CLI: it answered from a +// PNG's pixels and read a PDF's text back, each through this same +// stream-json stdin. +// +// A blob with no inline Data is SKIPPED rather than sent: the CLI's input +// protocol has no URL source, and a source object with an empty payload is +// worse than an honest omission — the model would be told a file exists and +// shown nothing. Every blob reaching a delegated turn arrives from a path +// that requires inline data (see the server's prompt validation), so this +// is a guard, not a routine case. +func claudeCodeInputContent(text string, blobs []*message.Blob) any { + var blocks []claudeCodeInputBlock + for _, b := range blobs { + if b == nil || len(b.Data) == 0 { + continue + } + blockType := "document" + if strings.HasPrefix(b.MediaType, "image/") { + blockType = "image" + } + blocks = append(blocks, claudeCodeInputBlock{ + Type: blockType, + Source: &claudeCodeInputSource{Type: "base64", MediaType: b.MediaType, Data: b.Data}, + }) + } + if len(blocks) == 0 { + return text + } + if text == "" { + return blocks + } + return append([]claudeCodeInputBlock{{Type: "text", Text: text}}, blocks...) +} + +// writeClaudeCodeInputMessage marshals one stream-json user input line and +// writes it to w — mirrors the Claude Agent SDK's ProcessTransport.write +// (sdk.mjs: JSON.stringify(message) + "\n"). Used for both a turn's first, +// driving message and every later mid-turn queued-prompt injection, so the +// child's stdin only ever sees this one wire shape (see runClaudeCodeTurn's +// stdin-writer pump doc comment for the construct this mirrors). +// +// blobs are the attachments to carry alongside text. BOTH call sites can +// have them: the turn's own driving message (lastUserMessageContent), and a +// mid-turn queued-prompt injection, since EnqueuePrompt takes blobs too and +// QueuedPrompt persists them. claudeCodeInputContent decides the content +// shape from them — a bare string when there are none, keeping the wire +// byte-identical to what this function sent before attachments existed. +func writeClaudeCodeInputMessage(w io.Writer, text string, blobs []*message.Blob) error { line, err := json.Marshal(claudeCodeInputMessage{ Type: "user", Message: claudeCodeInputInnerMessage{ Role: "user", - Content: text, + Content: claudeCodeInputContent(text, blobs), }, }) if err != nil { diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index e65146c0..6aa542a8 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -3,6 +3,7 @@ package engine import ( "bytes" "context" + "encoding/base64" "encoding/json" "errors" "fmt" @@ -2157,3 +2158,105 @@ func TestClaudeCodeStopRetiresPumpBlockedInStdinWrite(t *testing.T) { } killLeakedFakeClaude(t, pidFile) } + +// TestClaudeCodeQueueInjectedMidTurnCarriesAttachments is the regression for +// a mid-turn queued prompt LOSING its attachments in the claude-code lane. +// +// A prompt that arrives while a turn is running is queued with its blobs +// (QueuedPrompt.Blobs), and the native loop's own drain delivers them: +// drainQueuedPromptsIntoHistory (engine.go) builds its appended message with +// promptParts(block, queuedBlobs(queued)), so the bytes ride as Blob parts. +// This lane's drain appended a bare Text part and wrote stdin with no blobs, +// so an image or PDF sent mid-turn to a claude-code session was silently +// dropped on both halves at once — the running child never saw it, AND the +// durable history had no record of it for the next turn's --resume to +// recover. "A delay, never a loss" did not hold for the bytes. +// +// Both halves are asserted, because either alone would have passed while the +// other still dropped the file. +func TestClaudeCodeQueueInjectedMidTurnCarriesAttachments(t *testing.T) { + // Reuses fakeclaude's "queue_injection" mode — the one that blocks for a + // SECOND stdin line mid-turn — because that is exactly the delivery this + // regression is about; only what is ENQUEUED differs from the sibling + // test above. + s, _ := claudeCodeTestSession(t, "queue_injection") + stdinLog := filepath.Join(t.TempDir(), "stdin.log") + t.Setenv("FAKE_CLAUDE_STDIN_LOG", stdinLog) + + waiting := make(chan struct{}) + var waitingOnce sync.Once + s.cfg.OnEvent = func(ev Event) { + if ev.Type == EventMessage && ev.Message != nil && ev.Message.Parts.Text() == "WAITING_FOR_QUEUE" { + waitingOnce.Do(func() { close(waiting) }) + } + } + + done := make(chan error, 1) + go func() { + _, err := s.Prompt(context.Background(), "start") + done <- err + }() + + select { + case err := <-done: + t.Fatalf("Prompt returned (%v) before fakeclaude emitted WAITING_FOR_QUEUE", err) + case <-waiting: + case <-time.After(10 * time.Second): + t.Fatal("fakeclaude never emitted WAITING_FOR_QUEUE within 10s") + } + + // A 1x1 PNG, the same shape server/prompt_parts.go admits. + png := []byte("\x89PNG\r\n\x1a\nQUEUED-PNG-BYTES") + if _, _, err := s.EnqueuePrompt("look at this", "", &message.Blob{ + MediaType: "image/png", + Data: png, + }); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + + select { + case err := <-done: + if err != nil { + t.Fatalf("Prompt: %v", err) + } + case <-time.After(10 * time.Second): + t.Fatal("Prompt did not return within 10s of EnqueuePrompt") + } + + // Half one: the running child actually received the bytes. The stdin + // line must carry an image content block, not just the prompt's text. + stdinBytes, err := os.ReadFile(stdinLog) + if err != nil { + t.Fatalf("reading captured CLI stdin: %v", err) + } + lines := strings.Split(strings.TrimRight(string(stdinBytes), "\n"), "\n") + if len(lines) != 2 { + t.Fatalf("CLI stdin carried %d lines, want 2: %q", len(lines), string(stdinBytes)) + } + injected := lines[1] + if !strings.Contains(injected, `"type":"image"`) { + t.Errorf("CLI stdin's injected line carried no image block, so the queued attachment never "+ + "reached the running child: %q", injected) + } + if !strings.Contains(injected, base64.StdEncoding.EncodeToString(png)) { + t.Errorf("CLI stdin's injected line did not carry the queued PNG's own bytes: %q", injected) + } + + // Half two: the durable history records the attachment too, so the next + // turn's --resume recovery has something to recover. + var blobs int + for _, m := range s.History() { + if m.Role != message.RoleUser { + continue + } + for _, p := range m.Parts { + if b, ok := p.(*message.Blob); ok && bytes.Equal(b.Data, png) { + blobs++ + } + } + } + if blobs != 1 { + t.Errorf("history carried %d copies of the queued PNG, want exactly 1 (the mid-turn drain's "+ + "appended message must hold it as a Blob part, exactly once)", blobs) + } +} diff --git a/engine/engine.go b/engine/engine.go index 65a2c43a..817c9918 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -2521,6 +2521,41 @@ func (s *Session) emitSessionError(err error) { }}) } +// promptParts builds the canonical Parts of one prompt's user message: the +// typed text first, then one Blob part per attachment, in the caller's own +// order. It is the ONE place a prompt's text and attachments become a +// message, shared by PromptWithOrigin's two append sites (the native loop +// and the claude-code delegated lane), so the two can never disagree about +// where an attachment lands. +// +// Empty text with attachments is a real prompt — an uploaded screenshot +// with nothing typed beside it — and yields blob parts only, never an empty +// Text part in front of them: a leading empty text block is noise every +// provider transcoder would have to carry. Empty text with NO attachments +// keeps the exact single-empty-Text-part shape this function replaced, so +// nothing changes for a caller that prompts with "". +func promptParts(text string, blobs []*message.Blob) message.Parts { + if len(blobs) == 0 { + return message.Parts{&message.Text{Text: text}} + } + parts := make(message.Parts, 0, 1+len(blobs)) + if text != "" { + parts = append(parts, &message.Text{Text: text}) + } + for _, b := range blobs { + if b == nil { + continue + } + parts = append(parts, b) + } + if len(parts) == 0 { + // Every attachment was nil: fall back to the text-only shape rather + // than appending a part-less user message no provider can transcode. + return message.Parts{&message.Text{Text: text}} + } + return parts +} + // Prompt appends a user message and runs the agent loop — stream a turn, // execute any tool calls, feed results back — until the model ends its turn. // It returns the final assistant message. A thin, origin-less wrapper around @@ -2565,7 +2600,15 @@ func (s *Session) PromptEngineResume(ctx context.Context, text string) (*message // order is (and remains) append order, never a sort on this ID — see // ResolveMessageID's own doc comment for why a client-minted, time-sortable // ID must never be trusted for chronology. -func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string, id string) (*message.Message, error) { +// blobs are the prompt's attachments — an image a person uploaded beside +// their text, today. They ride as variadic trailing arguments so every one +// of this method's existing callers stays untouched and there is still +// exactly ONE parameterized entry point for appending a user message (the +// same reason origin is a parameter here rather than a third sibling +// method). Each blob becomes its own message.Blob part of the appended user +// message, after the text part — see promptParts. A prompt carrying at +// least one blob is valid with empty text. +func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin string, id string, blobs ...*message.Blob) (*message.Message, error) { // A session delegated to the Claude Code CLI (ClaudeCodeProviderFamily // — see engine/claude_code_backend.go) dispatches here, FIRST, before // every check and assembly step below: ContextWindowErr, @@ -2583,7 +2626,7 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri s.append(message.Message{ ID: ResolveMessageID(id), Role: message.RoleUser, - Parts: message.Parts{&message.Text{Text: text}}, + Parts: promptParts(text, blobs), CreatedAt: time.Now().UTC(), Origin: origin, }) @@ -2626,7 +2669,7 @@ func (s *Session) PromptWithOrigin(ctx context.Context, text string, origin stri s.append(message.Message{ ID: ResolveMessageID(id), Role: message.RoleUser, - Parts: message.Parts{&message.Text{Text: text}}, + Parts: promptParts(text, blobs), CreatedAt: time.Now().UTC(), Origin: origin, }) @@ -2897,10 +2940,17 @@ func (s *Session) runAgenticLoop(ctx context.Context) (*message.Message, error) // by reusing the identical mechanism rather than adding a second one. func (s *Session) drainQueuedPromptsIntoHistory() { if queued := s.DequeueAllPrompts("injected"); len(queued) > 0 { + // promptParts, not a bare Text part: a drained prompt can carry + // attachments (QueuedPrompt.Blobs), and operatorMessagesBlock + // renders TEXT only — it announces each prompt's attachment count + // and this append is what actually delivers the bytes, as Blob + // parts of the same injected user message. Without them an image + // queued while a turn was running would reach the model as a + // sentence about a picture it cannot see. s.append(message.Message{ ID: newID("msg"), Role: message.RoleUser, - Parts: message.Parts{&message.Text{Text: strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n")}}, + Parts: promptParts(strings.TrimSuffix(operatorMessagesBlock(queued, operatorContextTask), "\n"), queuedBlobs(queued)), CreatedAt: time.Now().UTC(), }) } diff --git a/engine/goal.go b/engine/goal.go index 150af962..edf8eb43 100644 --- a/engine/goal.go +++ b/engine/goal.go @@ -1220,7 +1220,12 @@ func (s *Session) PursueGoal(ctx context.Context, condition string, opts GoalOpt // the condition string itself stays clean. directive = operatorMessagesBlock(queued, operatorContextGoal) + directive } - if attempts, err := s.promptTurnWithRetry(ctx, directive, turn, snap.gen); err != nil { + // The drained batch's attachments ride to the worker turn beside + // that block: operatorMessagesBlock renders text only and announces + // each prompt's attachment count, so these bytes are what the count + // refers to. An image an operator sent mid-goal would otherwise be + // dropped at exactly this boundary — see queuedBlobs (queue.go). + if attempts, err := s.promptTurnWithRetry(ctx, directive, turn, snap.gen, queuedBlobs(queued)...); err != nil { if errors.Is(err, context.Canceled) { // Deliberate abort: leave the goal exactly as it is (a // drain must be resumable), no goal.stalled, no clear. @@ -1538,7 +1543,14 @@ func (s *Session) PursueGoal(ctx context.Context, condition string, opts GoalOpt // through to recordGoalStalled so a stall record for an attempt is never // journaled once an UpdateGoal has moved the goal past this turn's // generation — see recordGoalStalled and PursueGoal's stale-discard handling. -func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, turn int, gen uint64) (attempts int, err error) { +// blobs are the attachments PursueGoal's turn-boundary drain collected for +// this turn (nil on every turn with no operator mail). They ride with the +// directive on the attempts that actually APPEND one — attempt 1 and the +// fallback branch below — and are deliberately absent from the +// directive-reuse branch, which appends nothing at all: that branch re-runs +// the previous attempt's still-unanswered message, which already carries +// these very blob parts. +func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, turn int, gen uint64, blobs ...*message.Blob) (attempts int, err error) { var deterministicAttempt, retryableAttempt, truncatedAttempt, exhaustedAttempt int // anchorID identifies the message directiveReuseEligible and // dropUnansweredDirective both measure their tail from — the point @@ -1571,7 +1583,7 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur case attempts == 1: // Nothing to reuse yet: the ordinary path appends the directive // as history's first mention of it this turn. - _, perr = s.Prompt(ctx, directive) + _, perr = s.PromptWithOrigin(ctx, directive, "", "", blobs...) case s.directiveReuseEligible(anchorID): // The tail after anchorID is exactly the previous attempt's own // unanswered directive (see directiveReuseEligible) — reuse it @@ -1601,7 +1613,7 @@ func (s *Session) promptTurnWithRetry(ctx context.Context, directive string, tur // happens from here on, never the residue this fallback is // leaving behind for good. anchorID = s.lastMessageID() - _, perr = s.Prompt(ctx, directive) + _, perr = s.PromptWithOrigin(ctx, directive, "", "", blobs...) } if perr == nil { return attempts, nil diff --git a/engine/prompt_attachments_test.go b/engine/prompt_attachments_test.go new file mode 100644 index 00000000..10f1618c --- /dev/null +++ b/engine/prompt_attachments_test.go @@ -0,0 +1,318 @@ +package engine + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/majorcontext/harness/message" +) + +// tinyPNGBytes is a 1x1 PNG — real, decodable bytes, small enough to inline +// in a hand-authored journal line below. +var tinyPNGBytes = mustDecodeBase64("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==") + +func mustDecodeBase64(s string) []byte { + b, err := base64.StdEncoding.DecodeString(s) + if err != nil { + panic(err) + } + return b +} + +func testBlob() *message.Blob { + return &message.Blob{MediaType: "image/png", Data: tinyPNGBytes} +} + +// TestPromptPartsPlacesAttachmentsAfterText pins the shape every prompt path +// shares: the typed text first, then one Blob part per attachment, in the +// caller's order. +func TestPromptPartsPlacesAttachmentsAfterText(t *testing.T) { + a, b := testBlob(), &message.Blob{MediaType: "image/gif", Data: []byte("GIF89a")} + parts := promptParts("look at these", []*message.Blob{a, b}) + if len(parts) != 3 { + t.Fatalf("parts = %d, want text + 2 blobs: %+v", len(parts), parts) + } + text, ok := parts[0].(*message.Text) + if !ok || text.Text != "look at these" { + t.Fatalf("parts[0] = %+v, want the text part first", parts[0]) + } + if parts[1] != message.Part(a) || parts[2] != message.Part(b) { + t.Fatalf("attachments = %+v, want them in caller order after the text", parts[1:]) + } +} + +// TestPromptPartsOmitsEmptyTextWhenAttachedProves an image-only prompt +// carries no empty Text part: a leading empty text block is noise every +// transcoder would have to carry. +func TestPromptPartsOmitsEmptyTextWhenAttached(t *testing.T) { + parts := promptParts("", []*message.Blob{testBlob()}) + if len(parts) != 1 { + t.Fatalf("parts = %+v, want the blob alone", parts) + } + if _, ok := parts[0].(*message.Blob); !ok { + t.Fatalf("parts[0] = %+v, want the blob", parts[0]) + } +} + +// TestPromptPartsKeepsTextOnlyShape proves the no-attachment path is +// unchanged — one Text part, even for empty text, exactly as every caller +// predating attachments produced. +func TestPromptPartsKeepsTextOnlyShape(t *testing.T) { + for _, text := range []string{"hello", ""} { + parts := promptParts(text, nil) + if len(parts) != 1 { + t.Fatalf("parts for %q = %+v, want exactly one text part", text, parts) + } + got, ok := parts[0].(*message.Text) + if !ok || got.Text != text { + t.Fatalf("parts[0] for %q = %+v, want a Text part", text, parts[0]) + } + } +} + +// TestEnqueuePromptPersistsAttachments proves an enqueued prompt's +// attachments reach the journal on its prompt.queued record — the write half +// of surviving a restart. +func TestEnqueuePromptPersistsAttachments(t *testing.T) { + dir := t.TempDir() + s := NewSession(Config{SessionDir: dir, Model: message.ModelRef{Provider: "test", Model: "m1"}}) + if _, _, err := s.EnqueuePrompt("with a picture", "", testBlob()); err != nil { + t.Fatal(err) + } + + data, err := os.ReadFile(filepath.Join(dir, s.ID+".jsonl")) + if err != nil { + t.Fatal(err) + } + var queued *promptRecord + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + var rec record + if err := json.Unmarshal([]byte(line), &rec); err != nil { + t.Fatalf("unmarshaling session log line %q: %v", line, err) + } + if rec.Type == recPromptQueued { + queued = rec.Prompt + } + } + if queued == nil { + t.Fatalf("no prompt.queued record was written; log:\n%s", data) + } + if len(queued.Blobs) != 1 || !bytes.Equal(queued.Blobs[0].Data, tinyPNGBytes) { + t.Fatalf("record blobs = %+v, want the enqueued image", queued.Blobs) + } +} + +// TestEnqueuePromptAllowsAttachmentOnlyPrompt proves empty text is valid +// when an attachment carries the message — and still rejected when nothing +// does. +func TestEnqueuePromptAllowsAttachmentOnlyPrompt(t *testing.T) { + s := NewSession(Config{SessionDir: t.TempDir(), Model: message.ModelRef{Provider: "test", Model: "m1"}}) + if _, _, err := s.EnqueuePrompt(" ", "", testBlob()); err != nil { + t.Fatalf("attachment-only enqueue: %v, want it accepted", err) + } + if _, _, err := s.EnqueuePrompt(" ", ""); err != ErrEmptyPromptText { + t.Fatalf("empty enqueue error = %v, want ErrEmptyPromptText", err) + } +} + +// TestLoadSessionRestoresQueuedAttachments is the replay half: a queued +// prompt's image survives a process restart, so a box that was rescheduled +// while a prompt waited still answers the picture that was sent. +func TestLoadSessionRestoresQueuedAttachments(t *testing.T) { + dir := t.TempDir() + const id = "ses_0000000000000042" + blobJSON, err := json.Marshal([]*message.Blob{testBlob()}) + if err != nil { + t.Fatal(err) + } + writeSessionLog(t, dir, id, + `{"type":"session","id":"`+id+`","created_at":"2026-09-01T00:00:00Z"}`, + `{"type":"model","model":"test/m1"}`, + `{"type":"prompt.queued","prompt":{"id":1,"text":"what is this?","blobs":`+string(blobJSON)+`}}`, + ) + s, err := LoadSession(Config{SessionDir: dir}, id) + if err != nil { + t.Fatal(err) + } + q := s.QueuedPrompts() + if len(q) != 1 { + t.Fatalf("queue len = %d, want 1", len(q)) + } + if len(q[0].Blobs) != 1 || !bytes.Equal(q[0].Blobs[0].Data, tinyPNGBytes) { + t.Fatalf("restored blobs = %+v, want the queued image", q[0].Blobs) + } +} + +// TestOperatorMessagesBlockAnnouncesAttachments proves the rendered operator +// block tells the model which numbered message an attached image belongs to. +// The block is text; the bytes ride as separate Blob parts (queuedBlobs), so +// without this marker the model would find an unexplained picture at the end. +func TestOperatorMessagesBlockAnnouncesAttachments(t *testing.T) { + prompts := []QueuedPrompt{ + {ID: 1, Text: "plain"}, + {ID: 2, Text: "with a shot", Blobs: []*message.Blob{testBlob()}}, + } + block := operatorMessagesBlock(prompts, operatorContextTask) + if !bytes.Contains([]byte(block), []byte("2. with a shot\n [1 attachment(s) attached below]")) { + t.Fatalf("block = %q, want the attachment marker under its own message", block) + } + if bytes.Contains([]byte(block), []byte("1. plain\n [")) { + t.Fatalf("block = %q, want no marker on the text-only message", block) + } + if got := queuedBlobs(prompts); len(got) != 1 { + t.Fatalf("queuedBlobs = %d, want the one attachment", len(got)) + } +} + +// TestClaudeCodeInputContentCarriesImages proves the delegated lane's stdin +// line: a text-only turn keeps its historical bare-string content, and a turn +// with an attachment sends the CLI's own content-block array with a base64 +// image source (verified against the real CLI's stream-json input). +func TestClaudeCodeInputContentCarriesImages(t *testing.T) { + if got := claudeCodeInputContent("just text", nil); got != any("just text") { + t.Fatalf("text-only content = %#v, want the bare string", got) + } + + line, err := json.Marshal(claudeCodeInputMessage{ + Type: "user", + Message: claudeCodeInputInnerMessage{Role: "user", Content: claudeCodeInputContent("what is this?", []*message.Blob{testBlob()})}, + }) + if err != nil { + t.Fatal(err) + } + var got struct { + Message struct { + Content []struct { + Type string `json:"type"` + Text string `json:"text"` + Source struct { + Type string `json:"type"` + MediaType string `json:"media_type"` + Data []byte `json:"data"` + } `json:"source"` + } `json:"content"` + } `json:"message"` + } + if err := json.Unmarshal(line, &got); err != nil { + t.Fatalf("unmarshal input line: %v (%s)", err, line) + } + blocks := got.Message.Content + if len(blocks) != 2 { + t.Fatalf("content blocks = %d, want text + image: %s", len(blocks), line) + } + if blocks[0].Type != "text" || blocks[0].Text != "what is this?" { + t.Errorf("first block = %+v, want the text block", blocks[0]) + } + if blocks[1].Type != "image" || blocks[1].Source.Type != "base64" || blocks[1].Source.MediaType != "image/png" { + t.Errorf("second block = %+v, want a base64 image source", blocks[1]) + } + if !bytes.Equal(blocks[1].Source.Data, tinyPNGBytes) { + t.Errorf("image data = %d bytes, want the attachment's own bytes", len(blocks[1].Source.Data)) + } +} + +// TestClaudeCodeInputContentSendsDocumentBlockForNonImage proves the block +// TYPE follows the media type, matching the native anthropic adapter: a PDF +// is a "document" block, not an "image" one. Verified against the real CLI, +// which read a PDF's text back through this same stream-json shape. +func TestClaudeCodeInputContentSendsDocumentBlockForNonImage(t *testing.T) { + pdf := &message.Blob{MediaType: "application/pdf", Data: []byte("%PDF-1.4 fake body")} + content := claudeCodeInputContent("read this", []*message.Blob{pdf, testBlob()}) + blocks, ok := content.([]claudeCodeInputBlock) + if !ok { + t.Fatalf("content = %#v, want a content-block array", content) + } + if len(blocks) != 3 { + t.Fatalf("blocks = %d, want text + document + image", len(blocks)) + } + if blocks[1].Type != "document" || blocks[1].Source.MediaType != "application/pdf" { + t.Errorf("second block = %+v, want a pdf document block", blocks[1]) + } + if blocks[2].Type != "image" || blocks[2].Source.MediaType != "image/png" { + t.Errorf("third block = %+v, want an image block", blocks[2]) + } +} + +// TestClaudeCodeInputContentSkipsPayloadlessBlob proves a blob with no +// inline data is omitted rather than sent as an empty source: the CLI's +// input protocol has no URL image source, and announcing an image the model +// cannot see is worse than sending only the text. +func TestClaudeCodeInputContentSkipsPayloadlessBlob(t *testing.T) { + got := claudeCodeInputContent("see this", []*message.Blob{{MediaType: "image/png", URL: "https://example.com/x.png"}}) + if got != any("see this") { + t.Fatalf("content = %#v, want the bare text with the payloadless blob dropped", got) + } +} + +// TestLastUserMessageContentReturnsAttachments proves the delegated turn +// reads back BOTH halves of the pending user message. Parts.Text() drops +// non-text parts, so reading text alone silently stripped the upload. +func TestLastUserMessageContentReturnsAttachments(t *testing.T) { + history := []message.Message{ + {Role: message.RoleAssistant, Parts: message.Parts{&message.Text{Text: "earlier"}}}, + {Role: message.RoleUser, Parts: promptParts("what is this?", []*message.Blob{testBlob()})}, + } + text, blobs := lastUserMessageContent(history) + if text != "what is this?" { + t.Errorf("text = %q, want the pending prompt's text", text) + } + if len(blobs) != 1 || !bytes.Equal(blobs[0].Data, tinyPNGBytes) { + t.Fatalf("blobs = %+v, want the pending prompt's attachment", blobs) + } + + if text, blobs := lastUserMessageContent(history[:1]); text != "" || blobs != nil { + t.Errorf("assistant tail = (%q, %+v), want empty", text, blobs) + } +} + +// TestEnqueuePromptDropsUnusableBlobs: a blob that cannot be delivered must +// not make an empty prompt valid, and must not be counted in the operator +// block's attachment marker. +// +// EnqueuePrompt used to check len(blobs) directly, so a caller passing a +// single nil satisfied "empty text is fine when a blob came with it". The +// queued prompt then persisted an unusable Blobs slice, and +// operatorMessagesBlock announced "[1 attachment(s) attached below]" to the +// model while promptParts skipped the nil on delivery — the marker +// promising a file that never arrives, the same defect the claude-code +// mid-turn drain had. +func TestEnqueuePromptDropsUnusableBlobs(t *testing.T) { + s := NewSession(Config{SessionDir: t.TempDir()}) + + // Empty text plus only unusable blobs is an EMPTY prompt. + for _, blobs := range [][]*message.Blob{ + {nil}, + {{MediaType: "image/png"}}, // neither Data nor URL + {nil, {MediaType: "application/pdf"}}, // several, all unusable + } { + if _, _, err := s.EnqueuePrompt("", "", blobs...); !errors.Is(err, ErrEmptyPromptText) { + t.Errorf("EnqueuePrompt(%v) error = %v, want ErrEmptyPromptText", blobs, err) + } + } + if q := s.QueuedPrompts(); len(q) != 0 { + t.Fatalf("queue = %+v, want nothing enqueued", q) + } + + // A real blob beside an unusable one enqueues, carrying only the real + // one — so the marker counts what will actually be delivered. + real := &message.Blob{MediaType: "image/png", Data: []byte("\x89PNG\r\n\x1a\n")} + if _, _, err := s.EnqueuePrompt("look", "", nil, real); err != nil { + t.Fatalf("EnqueuePrompt: %v", err) + } + q := s.QueuedPrompts() + if len(q) != 1 { + t.Fatalf("queue = %d prompts, want 1", len(q)) + } + if len(q[0].Blobs) != 1 || q[0].Blobs[0] != real { + t.Errorf("queued blobs = %+v, want only the deliverable one", q[0].Blobs) + } + if block := operatorMessagesBlock(q, operatorContextTask); !strings.Contains(block, "[1 attachment(s) attached below]") { + t.Errorf("operator block = %q, want it to count ONE attachment", block) + } +} diff --git a/engine/queue.go b/engine/queue.go index e25c5652..c0ead187 100644 --- a/engine/queue.go +++ b/engine/queue.go @@ -21,6 +21,8 @@ import ( "errors" "fmt" "strings" + + "github.com/majorcontext/harness/message" ) // QueuedPrompt is one pending prompt in a session's durable FIFO queue (see @@ -45,6 +47,19 @@ type QueuedPrompt struct { // field existed — PromptWithOrigin's own mint site resolves that case // exactly like any other unset id, at dispatch time. MessageID string + // Blobs are the prompt's attachments — an uploaded image or PDF, the + // set server/prompt_parts.go admits — kept beside Text rather than + // folded into it because a Blob is binary content a provider + // transcodes as its own wire block — see message.Blob. They are + // persisted with the queued prompt (promptRecord.Blobs) and delivered + // as Blob parts of the user message this prompt becomes, so a prompt + // that waited behind a running turn, or behind a process restart, + // still arrives with its files. + // + // Nil for every text-only prompt, which is still the overwhelming + // majority: a queued prompt was text-only by contract until image + // input landed (see docs/session-storage-and-queue.md). + Blobs []*message.Blob } // promptQueueFold replays prompt.queued/prompt.dequeued records into the @@ -106,7 +121,7 @@ type promptQueueFold struct { // let a malformed record's ID move the counter, which is exactly what the // guard rejects it for. func (f *promptQueueFold) queued(p promptRecord) { - q := QueuedPrompt{ID: p.ID, Text: p.Text, Seq: p.Seq, MessageID: p.MessageID} + q := QueuedPrompt{ID: p.ID, Text: p.Text, Seq: p.Seq, MessageID: p.MessageID, Blobs: p.Blobs} valid := q.ID > 0 for _, existing := range f.queue { if existing.ID == q.ID { @@ -186,15 +201,51 @@ var ErrEmptyPromptText = errors.New("engine: prompt text must not be empty or wh // The enqueued prompt does not touch s.history and is not visible to any // provider request started before it is actually delivered (see // DequeuePrompt/dequeueAllLocked) — see the package doc comment. -func (s *Session) EnqueuePrompt(text string, messageID string) (id int64, resolvedMessageID string, err error) { +// blobs are the prompt's attachments, carried through the queue with it (see +// QueuedPrompt.Blobs). They are variadic so every existing text-only caller +// and its call sites stay untouched — one entry point still owns the enqueue +// rule, rather than a second near-identical method growing beside it. A +// prompt carrying at least one blob is valid with EMPTY text: an uploaded +// screenshot with nothing typed beside it is a real prompt, not an empty one. +// usablePromptBlobs drops the blobs a queued prompt cannot actually deliver: +// a nil entry, and one carrying neither Data nor URL (every provider +// transcoder errors on that shape regardless of media type — see +// message/wire_normalize.go's intersection comment). +// +// It runs BEFORE the empty-prompt check, because the raw count is not a +// count of attachments. A caller passing a single nil would otherwise +// satisfy "empty text is fine when a blob came with it", persist a prompt +// with an unusable Blobs slice, and then have operatorMessagesBlock +// announce "[1 attachment(s) attached below]" to the model while +// promptParts silently skipped the nil — the marker promising a file that +// never arrives, which is the same defect the claude-code drain had. +// +// Returns nil for an all-unusable input, so len() answers "how many +// attachments will really be delivered". +func usablePromptBlobs(blobs []*message.Blob) []*message.Blob { + usable := make([]*message.Blob, 0, len(blobs)) + for _, b := range blobs { + if b == nil || (len(b.Data) == 0 && b.URL == "") { + continue + } + usable = append(usable, b) + } + if len(usable) == 0 { + return nil + } + return usable +} + +func (s *Session) EnqueuePrompt(text string, messageID string, blobs ...*message.Blob) (id int64, resolvedMessageID string, err error) { trimmed := strings.TrimSpace(text) - if trimmed == "" { + usable := usablePromptBlobs(blobs) + if trimmed == "" && len(usable) == 0 { return 0, "", ErrEmptyPromptText } resolved := ResolveMessageID(messageID) s.mu.Lock() - p := s.enqueueMemoryOnlyLocked(trimmed, resolved) - s.persistPromptQueueLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text, MessageID: p.MessageID}) + p := s.enqueueMemoryOnlyLocked(trimmed, resolved, usable...) + s.persistPromptQueueLocked(recPromptQueued, promptRecord{ID: p.ID, Text: p.Text, MessageID: p.MessageID, Blobs: p.Blobs}) // Emit while still holding s.mu (see ClearGoal in goal.go): keeps event // order matching log order under a concurrent dequeue. OnEvent must not // call back into this Session — that would deadlock on s.mu, held here. @@ -234,10 +285,10 @@ func (s *Session) EnqueuePrompt(text string, messageID string) (id int64, resolv // for a caller (SendToDescendant) with no client message ID of its own, // left for PromptWithOrigin to resolve at dispatch time. Caller holds // s.mu. -func (s *Session) enqueueMemoryOnlyLocked(text string, messageID string) QueuedPrompt { +func (s *Session) enqueueMemoryOnlyLocked(text string, messageID string, blobs ...*message.Blob) QueuedPrompt { id := s.promptQueueNextID s.promptQueueNextID++ - p := QueuedPrompt{ID: id, Text: text, MessageID: messageID} + p := QueuedPrompt{ID: id, Text: text, MessageID: messageID, Blobs: blobs} s.promptQueue = append(s.promptQueue, p) return p } @@ -616,12 +667,44 @@ const ( // need to) that its enclosing Prompt call is being driven by PursueGoal; // only goal.go's OWN turn-boundary drain, which is actually building a // goal directive, uses the goal wording. +// A prompt carrying attachments renders its own count marker +// ("[N attachment(s) attached below]"): this block is TEXT, so the bytes +// themselves ride as separate Blob parts of whatever message the drain site +// builds around it (see queuedBlobs and drainQueuedPromptsIntoHistory). The +// marker is what ties the numbered entry to the files that follow it, so the +// model can tell which operator message an attachment belongs to instead of +// finding an unexplained blob at the end. +// +// The marker names no MEDIA TYPE, deliberately. message/wire_normalize.go +// spells the same situation two ways -- "[N image attachment(s) attached +// below]" on a path that only ever carries images, and a generic +// "[N attachment(s) omitted: ...]" where the type varies -- and this drain +// is the second kind: a queued prompt carries whatever +// server/prompt_parts.go admits, images and PDFs alike. Saying "image" +// here would tell the model a queued PDF is a picture. func operatorMessagesBlock(prompts []QueuedPrompt, ctx operatorContext) string { var b strings.Builder fmt.Fprintf(&b, "OPERATOR MESSAGES (address these, then continue the %s):\n", ctx) for i, p := range prompts { fmt.Fprintf(&b, "%d. %s\n", i+1, p.Text) + if len(p.Blobs) > 0 { + fmt.Fprintf(&b, " [%d attachment(s) attached below]\n", len(p.Blobs)) + } } b.WriteString("\n") return b.String() } + +// queuedBlobs collects every drained prompt's attachments, in FIFO prompt +// order, for a drain site that renders the batch's TEXT through +// operatorMessagesBlock above and must deliver the bytes alongside it. Nil +// when no drained prompt carried an attachment — the common case, and the +// one that keeps an injected operator message byte-identical to what it was +// before attachments existed. +func queuedBlobs(prompts []QueuedPrompt) []*message.Blob { + var blobs []*message.Blob + for _, p := range prompts { + blobs = append(blobs, p.Blobs...) + } + return blobs +} diff --git a/engine/store.go b/engine/store.go index b2af3896..a76a23c7 100644 --- a/engine/store.go +++ b/engine/store.go @@ -472,9 +472,8 @@ type goalRecord struct { } // promptRecord carries the durable payload of a prompt.queued/ -// prompt.dequeued record (see queue.go). String-only, mirroring goalRecord — -// v1's prompt contract is text parts only (see AGENTS.md), so no attachment -// machinery is needed here. ID is the queue-assigned, session-monotonic +// prompt.dequeued record (see queue.go). ID is the queue-assigned, +// session-monotonic // prompt ID. Text is the queued prompt, carried on BOTH record types (not // just prompt.queued) so a prompt.dequeued record is self-describing without // cross-referencing the matching prompt.queued one earlier in the log. @@ -500,6 +499,18 @@ type promptRecord struct { // like any other unset id, at dispatch time — a backward-compatible // fallback, not a replay error. MessageID string `json:"message_id,omitempty"` + // Blobs are the queued prompt's attachments (see QueuedPrompt.Blobs), + // written on prompt.queued ONLY — never on prompt.dequeued, unlike Text + // above. A dequeued record exists to name which queue entry left the + // queue, and it is matched by ID (promptQueueFold.dequeued), so copying + // an image's bytes into it would double every attachment in the journal + // for no reader. + // + // Omitted (nil) on every text-only prompt and on every record written + // before prompt attachments existed, which folds back to a QueuedPrompt + // with no attachments — the pre-feature behavior exactly, not a replay + // error. + Blobs []*message.Blob `json:"blobs,omitempty"` } // taskSpawnRecord is a recTaskSpawned record's payload — see that diff --git a/message/wire_normalize.go b/message/wire_normalize.go index 8b09ac42..60645ee4 100644 --- a/message/wire_normalize.go +++ b/message/wire_normalize.go @@ -553,10 +553,12 @@ func demoteWireInvalidToolResults(messages []Message) []Message { // ANY MediaType — image/* becomes an "image" block, anything else a // "document" block — as long as Data or URL is present; a blob with // neither errors "blob has neither data nor url". +// // - provider/openai/transcode.go's transcodeBlob (~line 298) accepts // image/* (Data or URL) or application/pdf (Data only — a PDF by URL // errors "is not supported"); anything else errors "unsupported blob // media type". +// // - provider/openaicompat/transcode.go's blobURL (~line 343) accepts // ONLY image/* (Data or URL); anything else — including // application/pdf, which openai alone tolerates — errors "unsupported @@ -564,6 +566,14 @@ func demoteWireInvalidToolResults(messages []Message) []Message { // intersection below: no PDF (openaicompat has no wire form for one at // all), and never a data-less, URL-less blob (every transcoder errors // on that regardless of media type). +// +// transcodeUserMessage does not let that error escape for a USER +// message: it drops the un-carryable blob and appends the same +// "[N attachment(s) omitted: ...]" note used below, because a user +// attachment is durable history and erroring would fail every later +// turn of a session that merely switched providers. The intersection +// here is still what a blob must satisfy to ride as REAL bytes on +// every lane. func buildSafeBlob(b *Blob) bool { return strings.HasPrefix(b.MediaType, "image/") && (len(b.Data) > 0 || b.URL != "") } diff --git a/provider/openaicompat/transcode.go b/provider/openaicompat/transcode.go index ba368585..0c849608 100644 --- a/provider/openaicompat/transcode.go +++ b/provider/openaicompat/transcode.go @@ -278,6 +278,10 @@ func transcodeUserMessage(m *message.Message) ([]apiMessage, error) { var texts []string var parts []apiContentPart hasBlob := false + // omittedBlobTypes collects the media types dropped below, so the model + // is told a file was withheld rather than left to answer about bytes it + // never received. + var omittedBlobTypes []string for _, p := range m.Parts { switch v := p.(type) { case *message.Text: @@ -301,6 +305,26 @@ func transcodeUserMessage(m *message.Message) ([]apiMessage, error) { texts = append(texts, t) parts = append(parts, apiContentPart{Type: "text", Text: t}) case *message.Blob: + // A blob this wire has no form for is OMITTED with a note, not + // an error. This lane is the narrowest of the three (see + // message/wire_normalize.go's intersection comment: no PDF at + // all), and an attachment lives in a session's DURABLE history + // — so erroring here would not fail one request, it would fail + // every turn from now on, permanently, for a session that + // merely switched to this provider after attaching a file the + // previous one accepted. There is no repair path: imageclamp + // downscales an oversized image but cannot rewrite a document. + // + // The note is the same shape wire_normalize already uses when + // it drops a tool-result blob, and it matters that the model + // SEES it: silently sending nothing would leave the model + // answering about a file it was never given, with no way to + // know. Dropping the bytes while saying so is the honest + // degradation. + if !strings.HasPrefix(v.MediaType, "image/") { + omittedBlobTypes = append(omittedBlobTypes, v.MediaType) + continue + } hasBlob = true url, err := blobURL(v) if err != nil { @@ -312,6 +336,17 @@ func transcodeUserMessage(m *message.Message) ([]apiMessage, error) { } } + if len(omittedBlobTypes) > 0 { + // Same wording wire_normalize uses for a dropped tool-result blob, + // so one vocabulary covers every omission the model can see. + note := fmt.Sprintf("[%d attachment(s) omitted: %s]", + len(omittedBlobTypes), strings.Join(omittedBlobTypes, ", ")) + texts = append(texts, note) + if hasBlob { + parts = append(parts, apiContentPart{Type: "text", Text: note}) + } + } + var content json.RawMessage var err error if hasBlob { diff --git a/provider/openaicompat/transcode_test.go b/provider/openaicompat/transcode_test.go index b0deaeb4..5dc71223 100644 --- a/provider/openaicompat/transcode_test.go +++ b/provider/openaicompat/transcode_test.go @@ -173,19 +173,38 @@ func TestTranscodeUserImage(t *testing.T) { } } -func TestTranscodeUserNonImageBlobErrors(t *testing.T) { +// TestTranscodeUserNonImageBlobOmitted: a non-image blob in a USER message +// is dropped with a note rather than failing the request. +// +// This test asserted the opposite until 2026-09-02 — that the request +// errors. That was safe while prompt_async took text only, because a +// non-image blob could then only come from a tool result. Once a person can +// ATTACH one, the same error becomes a permanent wedge: the attachment is +// in durable history, so a session that attached a PDF under anthropic and +// then switched to a provider on this lane would fail every later turn with +// no repair path. The error still exists in blobURL for any caller that +// reaches it directly; transcodeUserMessage just no longer lets it escape +// for the one shape a person can create. +func TestTranscodeUserNonImageBlobOmitted(t *testing.T) { req := baseRequest( message.Message{Role: message.RoleUser, Parts: message.Parts{ &message.Text{Text: "what is this"}, &message.Blob{MediaType: "application/pdf", Data: []byte{1, 2, 3}}, }}, ) - _, err := transcodeRequest(req, testFamily) - if err == nil { - t.Fatal("expected error for non-image blob, got nil") + got, err := transcodeRequest(req, testFamily) + if err != nil { + t.Fatalf("transcodeRequest: %v, want the PDF omitted rather than an error", err) + } + body, mErr := json.Marshal(got) + if mErr != nil { + t.Fatal(mErr) } - if !strings.Contains(err.Error(), "application/pdf") { - t.Errorf("error = %q, want it to name the media type application/pdf", err.Error()) + if !strings.Contains(string(body), "attachment(s) omitted: application/pdf") { + t.Errorf("request = %s, want it to name the omitted attachment", body) + } + if !strings.Contains(string(body), "what is this") { + t.Errorf("request = %s, want the user's text preserved", body) } } @@ -1009,3 +1028,65 @@ func TestTranscodeAssistantEngineContextRendered(t *testing.T) { t.Errorf("assistant Text content dropped:\n%s", got) } } + +// TestUserPDFOmittedWithNoteInsteadOfError is the durability regression for +// the narrowest lane. This transcoder has no wire form for a non-image blob +// (message/wire_normalize.go's intersection comment says so explicitly), and +// blobURL errors on one. Returning that error from here would not fail a +// single request: an attachment lives in the session's DURABLE history, so a +// session that attached a PDF under anthropic and then switched to a +// provider on this lane would fail EVERY later turn, forever, with no repair +// path — imageclamp downscales an oversized image but cannot rewrite a +// document. +// +// So the blob is dropped and the model is TOLD, which is the honest +// degradation: sending nothing silently would leave it answering about a +// file it never received. +func TestUserPDFOmittedWithNoteInsteadOfError(t *testing.T) { + msgs, err := transcodeUserMessage(&message.Message{ + Role: message.RoleUser, + Parts: message.Parts{ + &message.Text{Text: "what does this say?"}, + &message.Blob{MediaType: "application/pdf", Data: []byte("%PDF-1.4 body")}, + }, + }) + if err != nil { + t.Fatalf("transcodeUserMessage: %v, want a successful transcode with the PDF omitted", err) + } + if len(msgs) != 1 { + t.Fatalf("messages = %d, want 1", len(msgs)) + } + body := string(msgs[0].Content) + if !strings.Contains(body, "what does this say?") { + t.Errorf("content = %s, want it to keep the user's text", body) + } + if !strings.Contains(body, "attachment(s) omitted: application/pdf") { + t.Errorf("content = %s, want it to name the omitted attachment", body) + } + if strings.Contains(body, "JVBERi") || strings.Contains(body, "%PDF-") { + t.Errorf("content = %s, want the PDF's bytes NOT on the wire", body) + } +} + +// TestUserImageStillCarriedAlongsideOmittedPDF: the omission is per blob, +// not per message. An image in the same message must still ride as a real +// image part — dropping it too would turn a narrow gap into a wide one. +func TestUserImageStillCarriedAlongsideOmittedPDF(t *testing.T) { + msgs, err := transcodeUserMessage(&message.Message{ + Role: message.RoleUser, + Parts: message.Parts{ + &message.Blob{MediaType: "image/png", Data: []byte("\x89PNG\r\n\x1a\n")}, + &message.Blob{MediaType: "application/pdf", Data: []byte("%PDF-1.4 body")}, + }, + }) + if err != nil { + t.Fatalf("transcodeUserMessage: %v", err) + } + body := string(msgs[0].Content) + if !strings.Contains(body, "image_url") || !strings.Contains(body, "data:image/png;base64,") { + t.Errorf("content = %s, want the PNG carried as an image part", body) + } + if !strings.Contains(body, "attachment(s) omitted: application/pdf") { + t.Errorf("content = %s, want the PDF named as omitted", body) + } +} diff --git a/server/AGENTS.md b/server/AGENTS.md index bb2bdc7e..53eb700e 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -49,9 +49,20 @@ Another session that owns the workdir still conflicts. natural drain trigger. - Dispatch queued input before goal auto-arm. - Keep abort independent from queue clear. -- Keep `POST /enqueue` write-ahead and idempotent. +- Keep `POST /enqueue` write-ahead and idempotent. Keep it text-only. - Keep `GET /queue` as the reconciliation surface. +## Prompt attachments + +`prompt_async` takes file blob parts beside its text parts: images and PDFs +today. Validate every attachment before the run slot is claimed: allowed +media type, inline data, bytes that really are the claimed type, and the +size cap. Reject in the handler. Never persist an attachment a provider +cannot render. + +Add a media type only when EVERY provider adapter transcodes it. Each +accepted type owns a verifier in `promptAttachmentTypes`. + ## Goal and turn state Map a worker park to a distinct terminal outcome and keep the goal active. diff --git a/server/handlers.go b/server/handlers.go index d231c96b..e2fbb935 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -1638,11 +1638,8 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { return } var body struct { - Parts []struct { - Type string `json:"type"` - Text string `json:"text"` - } `json:"parts"` - Model message.ModelRef `json:"model"` + Parts []promptPartInput `json:"parts"` + Model message.ModelRef `json:"model"` // ID is an OPTIONAL client-minted ID for the user message this // prompt becomes — the console pre-mints one to render its own // optimistic bubble, then reconciles it by ID once the real message @@ -1653,7 +1650,21 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // uniqueness and never a reason to reject the prompt. ID string `json:"id"` } + // Bound the body BEFORE decoding it: blob data arrives as base64 and + // encoding/json allocates the decoded []byte during Unmarshal, so the + // per-attachment size check in decodePromptParts runs only after this + // server has already paid for whatever the caller sent -- and nothing + // bounds how many attachments one body carries. See + // promptRequestMaxBytes. handleEnqueue needs no such bound: its parts + // are text-only. + r.Body = http.MaxBytesReader(w, r.Body, promptRequestMaxBytes) if err := decodeBody(r, &body); err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + writeErr(w, http.StatusRequestEntityTooLarge, fmt.Sprintf( + "request body exceeds the %d-byte limit", promptRequestMaxBytes)) + return + } writeErr(w, http.StatusBadRequest, err.Error()) return } @@ -1661,15 +1672,17 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusBadRequest, "parts must be non-empty") return } - var texts []string - for _, p := range body.Parts { - if p.Type != "text" { - writeErr(w, http.StatusBadRequest, "v1 accepts text parts only") - return - } - texts = append(texts, p.Text) + // Text parts and attachment blob parts (images and PDFs), decoded and + // fully validated before + // any run slot is claimed or anything is enqueued — see + // decodePromptParts (prompt_parts.go) for why an attachment harness + // cannot deliver must be refused HERE rather than persisted first. + parts, code, err := decodePromptParts(body.Parts) + if err != nil { + writeErr(w, code, err.Error()) + return } - text := strings.Join(texts, "\n") + text, blobs := parts.Text, parts.Blobs // Resolved ONCE, here, regardless of which branch below actually ends // up delivering this prompt (immediate dispatch, or enqueued behind a // busy/non-empty queue): every branch reports this SAME value as its @@ -1690,7 +1703,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { writeErr(w, code, fmt.Sprintf("workdir busy: held by session %s", holder)) case code == http.StatusConflict: // Same-session busy: queue-on-busy (invariant 9), not a 409. - s.enqueueOrDispatch(w, id, text, msgID) + s.enqueueOrDispatch(w, id, text, msgID, blobs...) case code == http.StatusServiceUnavailable: writeErr(w, code, "server shutting down") default: @@ -1711,7 +1724,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // text) into the run slot just claimed above. See // dispatchQueueHead and enqueueOrDispatch's identical shape for the // same-session-BUSY counterpart of this same rule. - ourID, _, err := st.sess.EnqueuePrompt(text, msgID) + ourID, _, err := st.sess.EnqueuePrompt(text, msgID, blobs...) if err != nil { // handlePrompt already rejects an empty parts list and joins // non-empty text above, so this is not reachable in practice; @@ -1810,7 +1823,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { s.emitDurable(Event{Type: evtSessionStatus, SessionID: id, Status: "busy"}) - go s.runPrompt(ctx, id, st, text, "", msgID) + go s.runPrompt(ctx, id, st, text, "", msgID, blobs...) writeJSON(w, http.StatusAccepted, promptAsyncResponse{Seq: fromSeq, Status: "started", MessageID: msgID}) } @@ -1846,9 +1859,9 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // queue position, only "is my own prompt running or not, right now". // // A model override on a request whose prompt gets queued (either branch) is -// silently NOT applied: QueuedPrompt carries only ID and Text (see the plan's -// "text-only" locked decision — no attachment machinery), so there is no -// slot to carry a per-prompt model override through to a future drain. A +// silently NOT applied: QueuedPrompt carries the prompt's own text, +// attachments, and ids — not a model ref — so there is no slot to carry a +// per-prompt model override through to a future drain. A // caller that needs a model swap to take effect should re-issue it once its // prompt is confirmed "started". // @@ -1857,7 +1870,7 @@ func (s *Server) handlePrompt(w http.ResponseWriter, r *http.Request) { // of handlePrompt's two branches runs, so this function's own response // promises the SAME id EnqueuePrompt persists and PromptWithOrigin later // uses at dispatch. -func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string, msgID string) { +func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string, msgID string, blobs ...*message.Blob) { sess := s.residentSession(id) if sess == nil { // Benign race window, identical to handleGoalBusy's (see its doc @@ -1869,7 +1882,7 @@ func (s *Server) enqueueOrDispatch(w http.ResponseWriter, id string, text string writeErr(w, http.StatusConflict, "session is busy with another prompt") return } - ourID, _, err := sess.EnqueuePrompt(text, msgID) + ourID, _, err := sess.EnqueuePrompt(text, msgID, blobs...) if err != nil { // handlePrompt already rejects an empty parts list and joins // non-empty text, so this is not reachable in practice; fail closed @@ -2256,7 +2269,7 @@ func (s *Server) dispatchQueueHead(id string, st *sessionState, ctx context.Cont // deliberately dispatches the QUEUE HEAD instead of its own trigger // text, exactly so a real queued message is never displaced by the // resume trigger — see runOrQueueText's own doc comment. - go s.runPrompt(ctx, id, st, head.Text, "", head.MessageID) + go s.runPrompt(ctx, id, st, head.Text, "", head.MessageID, head.Blobs...) if s.dispatchQueueHeadRace != nil { // Test-only seam — see its own doc comment (server.go). s.dispatchQueueHeadRace() @@ -2288,7 +2301,7 @@ func (s *Server) dispatchQueueHead(id string, st *sessionState, ctx context.Cont // runOrQueueText's synthetic resume trigger, which has no client message id // of its own — PromptWithOrigin's own mint site resolves either case // identically. -func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, text string, origin string, msgID string) { +func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, text string, origin string, msgID string, blobs ...*message.Blob) { defer s.wg.Done() // ReportTurnStart/ReportTurnEnd bracket the ONE choke point every // ordinary (non-goal-loop) turn on a resident session funnels through @@ -2303,7 +2316,7 @@ func (s *Server) runPrompt(ctx context.Context, id string, st *sessionState, tex // hits, closing the "task tool broken after restart" gap a live // review caught. s.sessMgr.ReportTurnStart(st.sess) - msg, err := st.sess.PromptWithOrigin(ctx, text, origin, msgID) + msg, err := st.sess.PromptWithOrigin(ctx, text, origin, msgID, blobs...) s.syncMessages(id) // catch any message not yet journaled switch { case err == nil: diff --git a/server/openapi.yaml b/server/openapi.yaml index 43436b84..8c818168 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -839,6 +839,19 @@ components: data: { type: string, contentEncoding: base64 } url: { type: string } + PromptBlobPart: + description: > + A BlobPart sent as a PROMPT attachment, which must carry inline + base64 `data`. BlobPart itself allows `data` OR `url` because a + blob in a TRANSCRIPT legitimately carries either, but prompt_async + rejects a url-only attachment (decodePromptBlob): honoring one would + make harness ask every provider — and imageclamp, which has to decode + bytes to clamp them — to fetch a caller-supplied URL from inside the + box, which is an egress decision this route does not get to make. + allOf: + - $ref: "#/components/schemas/BlobPart" + - required: [data] + ToolCallPart: type: object required: [type, call_id, name, arguments] @@ -928,8 +941,37 @@ components: type: array minItems: 1 items: - $ref: "#/components/schemas/TextPart" - description: v1 accepts text parts only. + oneOf: + - $ref: "#/components/schemas/TextPart" + - $ref: "#/components/schemas/PromptBlobPart" + description: > + Text parts and file attachments. Text parts are joined by + newlines into the prompt's text; each blob part becomes a Blob + part of the user message, after the text, in the order given. + + + An attachment must be one of image/png, image/jpeg, image/gif, + image/webp, or application/pdf — every media type each provider + lane can actually deliver (an image block, or a document block + for a PDF). It must carry inline base64 `data` rather than a + `url`, must really be the type it claims (an image is decoded, a + PDF must carry its %PDF- header), and must not exceed 20971520 + bytes. Anything else is rejected with 400 before the prompt is + accepted, because a blob no provider can render would otherwise + be persisted into the durable transcript and re-sent on every + later turn. A prompt carrying at least one attachment is valid + with no text part at all. + + + Those 400s are per-attachment judgments, made after the body is + decoded. The WHOLE body is separately bounded at 33554432 bytes + and answered with 413 before any of it is decoded, since blob + `data` is base64 and decoding it allocates — so a client can tell + the two apart: a 400 means this attachment is unusable, a 413 + means the request was too large to inspect at all. Base64 costs + about 4/3, so the body bound is reached by roughly 24 MiB of + attachment bytes however they are divided up, which is why + several individually-legal attachments can still exceed it. model: $ref: "#/components/schemas/ModelRef" description: > @@ -938,11 +980,11 @@ components: as the CLI -model flag) — but ONLY when this prompt starts running immediately. When the session is busy and this request's prompt is durably queued instead (see prompt_async's 202 - `status: queued`), the override is silently dropped: `QueuedPrompt` - carries text only (docs/plans/2026-07-19-prompt-queue.md's - text-only v1 limit), so there is no slot to carry a per-prompt - override through to a future drain. Re-issue the request once it - is confirmed `started` if the override must take effect. + `status: queued`), the override is silently dropped: a queued + prompt carries its own text, attachments, and ids — not a model + ref — so there is no slot to carry a per-prompt override through + to a future drain. Re-issue the request once it is confirmed + `started` if the override must take effect. EnqueueRequest: type: object @@ -953,7 +995,12 @@ components: minItems: 1 items: $ref: "#/components/schemas/TextPart" - description: v1 accepts text parts only, same as PromptRequest. + description: > + Text parts only. Unlike PromptRequest, this write-ahead route + takes no attachments at all: its callers are machine relays + (an inbox poller, a coordinator) whose upstream ack rides on the + durable-accept contract, not the interactive composer a person + attaches a file to. seq: type: integer format: int64 @@ -964,10 +1011,10 @@ components: (the watermark simply jumps to it), but an out-of-order FRESH seq is indistinguishable from a duplicate and is silently dropped — see EnqueueResponse's `duplicate` status. No `model` - field: a durably-enqueued prompt is subject to the exact same - text-only, no-override limit as a queued PromptRequest (see - PromptRequest's `model` description) — there is simply no slot - to offer one here. + field: a durably-enqueued prompt is subject to the same + no-override limit as a queued PromptRequest (see PromptRequest's + `model` description) — there is simply no slot to offer one + here. EnqueueResponse: type: object @@ -2317,6 +2364,18 @@ paths: content: application/json: schema: { $ref: "#/components/schemas/PromptAsyncResponse" } + "400": + description: > + The body is malformed, `parts` is empty, or an attachment is + unusable — an unsupported media type, a url instead of inline + `data`, bytes that are not the type they claim, or one + attachment past 20971520 bytes. Each is a judgment about a + specific part, made after the body is decoded; contrast the 413 + below, which is about the request as a whole. The error message + names which attachment and why. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } "401": { $ref: "#/components/responses/Unauthorized" } "404": { $ref: "#/components/responses/NotFound" } "409": @@ -2329,6 +2388,16 @@ paths: content: application/json: schema: { $ref: "#/components/schemas/Error" } + "413": + description: > + The request body exceeds 33554432 bytes and was refused before + being decoded — blob `data` is base64 and decoding it allocates, + so an oversized body is stopped at the read rather than after + the cost is paid. Distinct from the per-attachment 400s in + `parts`: this one says nothing about any individual attachment. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } /session/{id}/enqueue: post: diff --git a/server/prompt_attachments_test.go b/server/prompt_attachments_test.go new file mode 100644 index 00000000..22b6642b --- /dev/null +++ b/server/prompt_attachments_test.go @@ -0,0 +1,480 @@ +package server + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json" + "fmt" + "image" + "image/color" + "image/jpeg" + "image/png" + "net/http" + "sync" + "testing" + + "github.com/majorcontext/harness/message" + "github.com/majorcontext/harness/provider" +) + +// testPNG returns the bytes of a small, structurally valid PNG — the +// stand-in for a console upload in every test below. Real bytes, not a +// fabricated prefix: the handler validates that a blob's data actually +// decodes as the image type it claims, so a fixture that only carried PNG +// magic bytes would be rejected for the wrong reason and prove nothing. +func testPNG(t *testing.T) []byte { + t.Helper() + img := image.NewRGBA(image.Rect(0, 0, 8, 8)) + for y := range 8 { + for x := range 8 { + img.Set(x, y, color.RGBA{R: 255, A: 255}) + } + } + var buf bytes.Buffer + if err := png.Encode(&buf, img); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +// testJPEG returns a small, valid JPEG — used to prove a REAL image of the +// wrong type is rejected, which is a different branch of verifyImageBytes +// than bytes that do not decode at all. +func testJPEG(t *testing.T) []byte { + t.Helper() + img := image.NewRGBA(image.Rect(0, 0, 8, 8)) + for y := range 8 { + for x := range 8 { + img.Set(x, y, color.RGBA{B: 255, A: 255}) + } + } + var buf bytes.Buffer + if err := jpeg.Encode(&buf, img, nil); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +// testPDF returns a small, structurally valid PDF carrying one line of text +// — the document counterpart of testPNG. Real bytes for the same reason: the +// handler proves an attachment is the type it claims. +func testPDF(t *testing.T) []byte { + t.Helper() + content := []byte("BT /F1 24 Tf 72 700 Td (attachment test) Tj ET") + objs := [][]byte{ + []byte("<< /Type /Catalog /Pages 2 0 R >>"), + []byte("<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + []byte("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"), + []byte("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>"), + fmt.Appendf(nil, "<< /Length %d >>\nstream\n%s\nendstream", len(content), content), + } + var out bytes.Buffer + out.WriteString("%PDF-1.4\n") + offsets := make([]int, 0, len(objs)) + for i, o := range objs { + offsets = append(offsets, out.Len()) + fmt.Fprintf(&out, "%d 0 obj\n%s\nendobj\n", i+1, o) + } + xref := out.Len() + fmt.Fprintf(&out, "xref\n0 %d\n0000000000 65535 f \n", len(objs)+1) + for _, off := range offsets { + fmt.Fprintf(&out, "%010d 00000 n \n", off) + } + fmt.Fprintf(&out, "trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n", len(objs)+1, xref) + return out.Bytes() +} + +// capturingProvider is scriptedProvider plus a copy of every request it was +// asked to stream, so a test can assert what the model actually RECEIVED — +// the only oracle that proves an uploaded image reached the provider rather +// than merely landing in the transcript. +type capturingProvider struct { + scripted *scriptedProvider + mu sync.Mutex + requests []*provider.Request +} + +func newCapturingProvider(turns ...[]provider.Event) *capturingProvider { + return &capturingProvider{scripted: &scriptedProvider{name: "test", turns: turns}} +} + +func (p *capturingProvider) Name() string { return p.scripted.Name() } + +func (p *capturingProvider) Stream(ctx context.Context, req *provider.Request) (provider.Stream, error) { + p.mu.Lock() + p.requests = append(p.requests, req) + p.mu.Unlock() + return p.scripted.Stream(ctx, req) +} + +// lastUserParts returns the parts of the last RoleUser message in the last +// request this provider was asked to stream. +func (p *capturingProvider) lastUserParts(t *testing.T) message.Parts { + t.Helper() + p.mu.Lock() + defer p.mu.Unlock() + if len(p.requests) == 0 { + t.Fatal("provider was never called") + } + msgs := p.requests[len(p.requests)-1].Messages + for i := len(msgs) - 1; i >= 0; i-- { + if msgs[i].Role == message.RoleUser { + return msgs[i].Parts + } + } + t.Fatal("no user message in the provider request") + return nil +} + +// blobParts filters ps down to its Blob parts, in order. +func blobParts(ps message.Parts) []*message.Blob { + var blobs []*message.Blob + for _, p := range ps { + if b, ok := p.(*message.Blob); ok { + blobs = append(blobs, b) + } + } + return blobs +} + +// attachmentPart builds one prompt_async blob part body for data. +func attachmentPart(mediaType string, data []byte) map[string]any { + return map[string]any{ + "type": "blob", + "media_type": mediaType, + "data": base64.StdEncoding.EncodeToString(data), + } +} + +// TestPromptAsyncAcceptsImageBlobPart is the RED test for the feature: a +// prompt_async body carrying a text part AND an image blob part is accepted, +// the blob survives into the durable transcript as a Blob part of the user +// message, and the provider request for that turn carries it too. Before +// this change the handler rejected every non-text part with 400 "v1 accepts +// text parts only". +func TestPromptAsyncAcceptsImageBlobPart(t *testing.T) { + prov := newCapturingProvider(asstTurn("red")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{ + map[string]any{"type": "text", "text": "what color is this?"}, + attachmentPart("image/png", pngBytes), + }, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1: %+v", len(users), users) + } + if got := users[0].Parts.Text(); got != "what color is this?" { + t.Errorf("transcript user text = %q, want the prompt text", got) + } + blobs := blobParts(users[0].Parts) + if len(blobs) != 1 { + t.Fatalf("transcript user blobs = %d, want 1 (parts %+v)", len(blobs), users[0].Parts) + } + if blobs[0].MediaType != "image/png" { + t.Errorf("blob media type = %q, want image/png", blobs[0].MediaType) + } + if !bytes.Equal(blobs[0].Data, pngBytes) { + t.Errorf("blob data = %d bytes, want the %d uploaded bytes", len(blobs[0].Data), len(pngBytes)) + } + + sent := blobParts(prov.lastUserParts(t)) + if len(sent) != 1 || !bytes.Equal(sent[0].Data, pngBytes) { + t.Fatalf("provider request carried %d blob parts, want the uploaded image", len(sent)) + } +} + +// TestPromptAsyncAcceptsImageOnlyPrompt proves an attachment-only prompt (no +// text part at all) is accepted: a person can send a screenshot with nothing +// to say about it. The old handler joined text parts and would have run a +// turn on an empty string. +func TestPromptAsyncAcceptsImageOnlyPrompt(t *testing.T) { + prov := newCapturingProvider(asstTurn("ok")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{attachmentPart("image/png", pngBytes)}, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1", len(users)) + } + if len(blobParts(users[0].Parts)) != 1 { + t.Fatalf("user parts = %+v, want exactly the image blob", users[0].Parts) + } + if got := users[0].Parts.Text(); got != "" { + t.Errorf("user text = %q, want empty for an image-only prompt", got) + } +} + +// TestPromptAsyncAcceptsPDFAttachment proves the contract is attachments, +// not images: a PDF is accepted, kept whole in the durable transcript, and +// handed to the provider as its own blob part. The claude-code lane sends it +// as a document block (claudeCodeInputContent); the native adapters already +// transcode a non-image blob as a document/input_file. +func TestPromptAsyncAcceptsPDFAttachment(t *testing.T) { + prov := newCapturingProvider(asstTurn("read it")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + pdfBytes := testPDF(t) + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{ + map[string]any{"type": "text", "text": "what does this say?"}, + attachmentPart("application/pdf", pdfBytes), + }, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("prompt_async status %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1", len(users)) + } + blobs := blobParts(users[0].Parts) + if len(blobs) != 1 || blobs[0].MediaType != "application/pdf" { + t.Fatalf("transcript blobs = %+v, want the pdf", blobs) + } + if !bytes.Equal(blobs[0].Data, pdfBytes) { + t.Errorf("blob data = %d bytes, want the %d uploaded bytes", len(blobs[0].Data), len(pdfBytes)) + } + sent := blobParts(prov.lastUserParts(t)) + if len(sent) != 1 || !bytes.Equal(sent[0].Data, pdfBytes) { + t.Fatalf("provider request carried %d blob parts, want the pdf", len(sent)) + } +} + +// TestPromptAsyncRejectsUnusableBlob proves each rejection the handler owns, +// and proves it rejects BEFORE anything runs: no user message is appended +// and the provider is never called. A blob the provider would reject later +// must fail here, synchronously, where the caller can still fix it — the +// wedge imageclamp exists to heal (an oversized image persisted into a +// durable transcript) starts exactly one accepted-but-unusable blob ago. +func TestPromptAsyncRejectsUnusableBlob(t *testing.T) { + cases := []struct { + name string + part map[string]any + want string + }{ + { + name: "unsupported media type", + part: attachmentPart("application/zip", []byte("PK\x03\x04not a zip")), + want: "unsupported blob media type", + }, + { + name: "document that is not a pdf", + part: attachmentPart("application/pdf", []byte("just text, no header")), + want: "%PDF- header", + }, + { + name: "data does not decode as the claimed type", + part: attachmentPart("image/png", []byte("this is plain text, not a PNG")), + want: "does not decode", + }, + { + // A REAL image, of the wrong type: this reaches + // verifyImageBytes' format comparison rather than its decode + // failure above, and the message must name what the data + // actually is in the SAME units the caller claimed it in + // ("image/jpeg", not the bare decoder format "jpeg") — the + // comparison is between two media types, so a caller should + // not have to guess that "jpeg" meant image/jpeg. + name: "a real image of a different type than claimed", + part: attachmentPart("image/png", testJPEG(t)), + want: "the data is image/jpeg", + }, + { + name: "no data and no url", + part: map[string]any{"type": "blob", "media_type": "image/png"}, + want: "neither data nor url", + }, + { + name: "url blob", + part: map[string]any{"type": "blob", "media_type": "image/png", "url": "https://example.com/x.png"}, + want: "inline data", + }, + { + name: "unknown part type", + part: map[string]any{"type": "reasoning", "text": "no"}, + want: "text and blob parts only", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + prov := newCapturingProvider(asstTurn("never")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{ + map[string]any{"type": "text", "text": "look"}, + tc.part, + }, + }) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("prompt_async status %d, want 400: %s", resp.StatusCode, data) + } + var errBody struct { + Error string `json:"error"` + } + if err := json.Unmarshal(data, &errBody); err != nil { + t.Fatal(err) + } + if !bytes.Contains([]byte(errBody.Error), []byte(tc.want)) { + t.Errorf("error = %q, want it to mention %q", errBody.Error, tc.want) + } + if users := h.userMessages(id); len(users) != 0 { + t.Errorf("user messages = %d, want none: a rejected prompt must append nothing", len(users)) + } + prov.mu.Lock() + calls := len(prov.requests) + prov.mu.Unlock() + if calls != 0 { + t.Errorf("provider calls = %d, want 0", calls) + } + }) + } +} + +// TestPromptAsyncOversizeBlobRejected proves the per-blob byte cap: a blob +// past promptAttachmentMaxBytes is refused with a message naming the limit, so a +// caller learns to downscale instead of silently poisoning the session. +func TestPromptAsyncOversizeBlobRejected(t *testing.T) { + prov := newCapturingProvider(asstTurn("never")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + + // A valid PNG header followed by enough bytes to pass the cap. The size + // check must run BEFORE the decode, so the padding never has to be a + // real image. + oversize := append(testPNG(t), bytes.Repeat([]byte("x"), promptAttachmentMaxBytes)...) + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{attachmentPart("image/png", oversize)}, + }) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("prompt_async status %d, want 400: %s", resp.StatusCode, data) + } + if !bytes.Contains(data, []byte("exceeds")) { + t.Errorf("error = %s, want it to name the size limit", data) + } +} + +// TestPromptAsyncOversizeBodyRejectedBeforeDecode proves the WHOLE-body +// bound, which is a different guarantee from the per-blob cap above. +// +// promptAttachmentMaxBytes is checked per attachment, but only after +// encoding/json has already base64-decoded that attachment into a []byte, +// and nothing caps how many attachments one body may carry -- so without +// this bound a caller could make the server allocate an unbounded amount +// before the first size check ran, then reject what it had already paid +// for. http.MaxBytesReader stops the read at promptRequestMaxBytes, so the +// body below is never fully decoded and the answer is 413, not 400. +func TestPromptAsyncOversizeBodyRejectedBeforeDecode(t *testing.T) { + prov := newCapturingProvider(asstTurn("never")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + + // Two attachments, each genuinely UNDER the per-attachment cap, that + // together exceed the request bound once base64 encoding is paid for: + // exactly the case the per-blob check alone cannot catch. Sized at 2/3 + // of the per-attachment cap, so two are ~26.7 MiB decoded and ~35.6 MiB + // on the wire, against a 32 MiB bound. + each := append(testPNG(t), bytes.Repeat([]byte("x"), promptAttachmentMaxBytes*2/3)...) + // Pin the premise rather than trusting the arithmetic above: if a future + // change to either constant made these attachments individually oversize, + // the request would still 413 and this test would still pass while + // silently testing the per-blob path instead of the body bound. + if len(each) >= promptAttachmentMaxBytes { + t.Fatalf("each attachment is %d bytes, which is not under the %d-byte per-attachment cap — "+ + "this test would no longer prove the body bound catches what the per-blob check cannot", + len(each), promptAttachmentMaxBytes) + } + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{ + attachmentPart("image/png", each), + attachmentPart("image/png", each), + }, + }) + if resp.StatusCode != http.StatusRequestEntityTooLarge { + t.Fatalf("prompt_async status %d, want 413: %s", resp.StatusCode, data) + } + if !bytes.Contains(data, []byte("limit")) { + t.Errorf("error = %s, want it to name the request limit", data) + } + if users := h.userMessages(id); len(users) != 0 { + t.Errorf("user messages = %d, want none", len(users)) + } +} + +// TestQueuedPromptKeepsItsImage is the durability half of the feature: an +// image sent while the session is BUSY is queued, and the queued prompt +// still carries its blob when the queue drains at the turn boundary. A queue +// that dropped attachments would lose the upload silently — the exact +// failure the text-only v1 queue contract used to guarantee. +func TestQueuedPromptKeepsItsImage(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + // First prompt claims the run slot and parks inside the provider. + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{map[string]any{"type": "text", "text": "first"}}, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("first prompt status %d: %s", resp.StatusCode, data) + } + <-prov.started + + resp, data = h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []any{ + map[string]any{"type": "text", "text": "and this screenshot"}, + attachmentPart("image/png", pngBytes), + }, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("queued prompt status %d: %s", resp.StatusCode, data) + } + var pr promptAsyncResponse + if err := json.Unmarshal(data, &pr); err != nil { + t.Fatal(err) + } + if pr.Status != "queued" { + t.Fatalf("status = %q, want queued", pr.Status) + } + + prov.releaseAll() + h.waitIdle(id) + + users := h.userMessages(id) + var withBlob int + for _, m := range users { + for _, b := range blobParts(m.Parts) { + if bytes.Equal(b.Data, pngBytes) { + withBlob++ + } + } + } + if withBlob != 1 { + t.Fatalf("user messages carrying the queued image = %d, want 1: %+v", withBlob, users) + } +} diff --git a/server/prompt_parts.go b/server/prompt_parts.go new file mode 100644 index 00000000..bc17c717 --- /dev/null +++ b/server/prompt_parts.go @@ -0,0 +1,250 @@ +package server + +import ( + "bytes" + "errors" + "fmt" + "image" + _ "image/gif" // register GIF decoder for image.DecodeConfig + _ "image/jpeg" // register JPEG decoder for image.DecodeConfig + _ "image/png" // register PNG decoder for image.DecodeConfig + "net/http" + "sort" + "strings" + + _ "golang.org/x/image/webp" // register WebP decoder for image.DecodeConfig + + "github.com/majorcontext/harness/message" +) + +// Prompt attachments: the parsing and validation half of "a person can send +// a file" -- an image or a PDF, the set promptAttachmentTypes admits below -- +// shared by every handler that accepts a prompt body's `parts` array. +// +// The wire shape is message.Blob's own JSON, verbatim +// ({"type":"blob","media_type":...,"data":}), so a caller building a +// prompt uses the same vocabulary the transcript hands back to it — no +// second, prompt-only attachment format to keep in sync. What this file +// adds is admission control: harness accepts a blob only if the model can +// actually be shown it, and it says so synchronously, while the caller can +// still fix the upload. +// +// Why validation belongs HERE and not at transcode time: a user message is +// appended to an append-only durable log before any provider ever sees it +// (engine.Session.PromptWithOrigin). A blob no provider can accept would +// therefore be persisted first and rejected on every turn afterwards — the +// exact wedge imageclamp exists to heal (see imageclamp's package doc: +// three Neptune boxes, oversized screenshots, a 400 that survived respawn). +// imageclamp handles the sizes it can repair by downscaling; this gate +// handles the ones nothing can repair — a type no provider decodes, bytes +// that are not the image they claim to be, an attachment with no payload. + +// promptAttachmentTypes is the set of attachment media types a prompt may +// carry, each paired with the check that proves the bytes really are that +// type. A media type belongs here only when EVERY provider lane harness can +// dispatch to either delivers it, or degrades visibly -- drops it and tells +// the model so -- rather than failing the request. That is the real bar, +// and it is set by durability rather than by capability: a session switches +// models freely and the attachment stays in its history forever, so a lane +// that ERRORS on a type would fail every later turn of a session that +// merely switched into it, with no repair path. +// +// - Images (the same set engine's read_file returns as a blob, see +// readFileImageMediaTypes): anthropic and claude-code send an image +// block, openai an input_image, openaicompat a data URL. imageclamp can +// also decode and downscale them, so an oversized one heals rather than +// wedging a session. +// - application/pdf: anthropic and claude-code send a document block, +// openai an input_file. Verified against the real claude-code CLI, which +// read a PDF's text back through its stream-json stdin. +// +// PDF is the entry that relies on the degrade half of that bar rather than +// the deliver half. provider/openaicompat has +// no wire form for a document at all (message/wire_normalize.go's +// intersection comment), so it OMITS a non-image blob and tells the model +// "[N attachment(s) omitted: application/pdf]" instead of erroring. That +// keeps the durability rule this list exists to serve: a session that +// attached a PDF under anthropic and later switched to an openaicompat +// provider degrades for that turn rather than failing every turn forever +// from inside its own transcript. A caller that wants a PDF actually READ +// should keep the session on a lane that carries one; the boxes console +// gates its attach control per model for exactly this reason. +// +// Everything else stays out for a concrete reason, not caution: a +// text/plain or docx blob reaches openai's transcodeBlob as "unsupported +// blob media type", so accepting one here would be a promise that lane +// cannot keep even in degraded form. Widening this set is a row plus a +// verifier — and a check of what EVERY provider adapter does with the new +// type, including whether it can degrade instead of erroring. +var promptAttachmentTypes = map[string]func(mediaType string, data []byte) error{ + "image/png": verifyImageBytes, + "image/jpeg": verifyImageBytes, + "image/gif": verifyImageBytes, + "image/webp": verifyImageBytes, + "application/pdf": verifyPDFBytes, +} + +// promptAttachmentMaxBytes bounds ONE decoded attachment. For an image it +// matches engine's read_file ceiling (readFileMaxImageBytes): one limit for +// "a picture harness will hold in a session", however it arrived. +// +// For a PDF the cap does more work, and is the only protection there is: +// imageclamp decodes and downscales an oversized IMAGE at transcode time, +// but it cannot rewrite a document, so a PDF past a provider's own request +// ceiling (Anthropic's is 32MB) would fail every turn from inside a durable +// transcript with nothing able to repair it. This ceiling sits below that. +const promptAttachmentMaxBytes = 20 * 1024 * 1024 + +// promptRequestMaxBytes bounds the WHOLE prompt request body, before any of +// it is decoded. +// +// promptAttachmentMaxBytes above is checked per attachment, but only AFTER +// encoding/json has already base64-decoded that attachment into a []byte -- +// and nothing bounds how many attachments one body may carry. So without a +// bound here, a single request could make this server allocate an unbounded +// amount before the first size check ever ran, and the check would then +// reject what it had already paid for. +// +// 32 MiB is the same ceiling Anthropic applies to a whole request, which is +// the real limit a prompt has to fit inside anyway. Base64 costs about 4/3, +// so this admits one attachment at the full 20 MiB per-attachment cap +// (~26.7 MiB encoded) plus its text, or several smaller ones -- while a body +// that could never be delivered to a provider is refused here, cheaply, +// instead of after the allocation. +const promptRequestMaxBytes = 32 * 1024 * 1024 + +// verifyImageBytes proves data decodes as the image type it claims. It +// reads the header only (dimensions, not pixels), so a 20MB image costs a +// header parse rather than a full decode. +func verifyImageBytes(mediaType string, data []byte) error { + cfg, format, err := image.DecodeConfig(bytes.NewReader(data)) + if err != nil { + return fmt.Errorf("does not decode as %s: %v", mediaType, err) + } + if decoded := "image/" + format; decoded != mediaType { + // Report the DECODED MEDIA TYPE, not image.DecodeConfig's bare + // format name ("jpeg"): the comparison just above is between two + // media types, so naming one of them in the other's units leaves + // the caller to guess whether "jpeg" meant image/jpeg. Both sides + // of a mismatch now read in the same vocabulary the caller sent. + return fmt.Errorf("does not decode as %s: the data is %s", mediaType, decoded) + } + if cfg.Width <= 0 || cfg.Height <= 0 { + return fmt.Errorf("does not decode as %s: it reports a %dx%d size", mediaType, cfg.Width, cfg.Height) + } + return nil +} + +// verifyPDFBytes proves data is a PDF by its header. A PDF file begins with +// %PDF- followed by its version (ISO 32000-1 §7.5.2); this is a +// mislabeling check, not a validity check — a structurally broken PDF is +// the provider's business, while a JPEG labeled application/pdf is this +// server's. +func verifyPDFBytes(mediaType string, data []byte) error { + if !bytes.HasPrefix(data, []byte("%PDF-")) { + return fmt.Errorf("does not begin with a %%PDF- header, so it is not a %s", mediaType) + } + return nil +} + +// promptParts is one decoded prompt body: the caller's text (every text +// part joined by newlines, exactly as the text-only contract always did) +// and its attachments in wire order. +type promptParts struct { + Text string + Blobs []*message.Blob +} + +// promptPartsInput is the JSON shape of one element of a prompt body's +// `parts` array — a text part or a blob part. Fields not valid for the +// part's own type are simply absent; decodePromptParts rejects a part whose +// type is neither. +type promptPartInput struct { + Type string `json:"type"` + Text string `json:"text"` + MediaType string `json:"media_type"` + Data []byte `json:"data"` + URL string `json:"url"` +} + +// errEmptyPromptParts reports a `parts` array that carried no deliverable +// content at all — no text, no attachment. Callers map it to the same 400 +// an empty array already produced. +var errEmptyPromptParts = errors.New("parts must carry text or at least one attachment") + +// decodePromptParts validates a prompt body's parts and folds them into the +// text-plus-attachments pair every prompt path downstream takes. It returns +// an HTTP status and error for a caller to write verbatim; a nil error +// means every part was usable. +// +// Order is preserved for attachments and irrelevant for text (joined, as +// before). Validation is total and happens BEFORE the caller claims a run +// slot or enqueues anything, so a rejected prompt leaves no trace: no +// message appended, no queue entry, no turn started. +func decodePromptParts(parts []promptPartInput) (promptParts, int, error) { + var out promptParts + var texts []string + for _, p := range parts { + switch p.Type { + case "text": + texts = append(texts, p.Text) + case "blob": + blob, err := decodePromptBlob(p) + if err != nil { + return promptParts{}, http.StatusBadRequest, err + } + out.Blobs = append(out.Blobs, blob) + default: + return promptParts{}, http.StatusBadRequest, fmt.Errorf("unsupported part type %q: text and blob parts only", p.Type) + } + } + out.Text = strings.Join(texts, "\n") + if strings.TrimSpace(out.Text) == "" && len(out.Blobs) == 0 { + return promptParts{}, http.StatusBadRequest, errEmptyPromptParts + } + return out, 0, nil +} + +// decodePromptBlob validates one blob part and returns the message.Blob it +// becomes. Every rejection names what is wrong with the attachment itself, +// never how to fix the request format, because the caller here is a UI +// forwarding a file a person chose. +func decodePromptBlob(p promptPartInput) (*message.Blob, error) { + verify, ok := promptAttachmentTypes[p.MediaType] + if !ok { + return nil, fmt.Errorf("unsupported blob media type %q: prompt attachments must be one of %s", p.MediaType, promptAttachmentTypeList()) + } + if len(p.Data) == 0 { + if p.URL != "" { + // A URL blob is a valid message.Blob and some providers accept + // one, but harness would then be asking every provider (and + // imageclamp, which must decode bytes to clamp them) to fetch + // caller-supplied URLs from inside the box. That is an + // egress decision this route does not get to make on its own. + return nil, errors.New("a prompt attachment must carry inline data, not a url") + } + return nil, errors.New("blob has neither data nor url") + } + if len(p.Data) > promptAttachmentMaxBytes { + return nil, fmt.Errorf("attachment is %d bytes, which exceeds the %d-byte limit for one prompt attachment", len(p.Data), promptAttachmentMaxBytes) + } + // Prove the bytes are the type they claim BEFORE the prompt is accepted: + // a mislabeled or truncated file passes a media-type string check and + // then fails at the provider, on every later turn, from inside a durable + // transcript. Each type brings its own verifier (promptAttachmentTypes). + if err := verify(p.MediaType, p.Data); err != nil { + return nil, fmt.Errorf("attachment %v", err) + } + return &message.Blob{MediaType: p.MediaType, Data: p.Data}, nil +} + +// promptAttachmentTypeList renders the accepted media types for an error +// message, sorted so the same request always names them in the same order. +func promptAttachmentTypeList() string { + types := make([]string, 0, len(promptAttachmentTypes)) + for mediaType := range promptAttachmentTypes { + types = append(types, mediaType) + } + sort.Strings(types) + return strings.Join(types, ", ") +} From f0c257e14ad9cbb0f4c9d32d2b66ec6b6120c870 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 12:45:28 -0400 Subject: [PATCH 48/95] fix(engine): ask the claude-code CLI for summarized thinking (#237) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(engine): ask the claude-code CLI for summarized thinking Every thinking block this driver journals is empty. Opus 4.7 silently flipped the API default of thinking.display from "summarized" to "omitted", and every model since has kept it. Under "omitted" the model still thinks and still bills for it, but the API returns the thinking block with an empty `thinking` field and only its provider signature. The whole chain then degrades quietly. claudeCodeAssistantMessage stores message.Reasoning{Text: ""}; consumeClaudeCodeStream's `r.Text != ""` guard emits no EventReasoningDelta, so the live stream never announces the block at all; and since #234 merged the block into its turn's own message, a reply segment reaches consumers as [reasoning(empty), text] on the wire while the row that streamed it holds only [text]. A consumer cannot render that part, and cannot align it either. The boxes console rendered a whole turn TWICE off exactly that asymmetry (meetneptune/boxes#599): its orphan-reconciliation compared streamed parts against canonical parts and failed on kind at part 0, so the streamed row survived alongside the durable message. --thinking-display is the CLI's own override for the default, and it is the only channel that carries it: the showThinkingSummaries setting does not reach the request (verified against a live session -- it still returns an empty thinking field), and nothing else in the CLI surfaces the API parameter. Passed unconditionally, next to --forward-subagent-text, for the same reason that one is: it is a property of what this driver needs from the CLI, not a per-session choice. The flag is real but NOT in `claude --help` (`claude --thinking-display bogus` answers "Allowed choices are summarized, omitted"), so this is a pinned-CLI dependency. A CLI that drops the flag fails the spawn outright instead of silently returning empty thinking again -- the loud failure is the one worth having, and the box image pins its own CLI version, so an upgrade is a deliberate, testable step. Summaries only: the raw chain of thought is never exposed by any model under any setting, so "summarized" is the maximum available. Thinking is billed identically under every display setting. Verified end to end against a live Opus subscription session, not just in argv: the same prompt returned 383 characters of summarized thinking with the flag and 0 characters without it, through --output-format stream-json. TestClaudeCodeThinkingDisplayAlwaysSummarized red-verifies against the unflagged argv ("argv has no --thinking-display: [--input-format stream-json ... --model sonnet]"). go test -race ./engine/ for the claude-code argv and reasoning tests is green; gofmt clean. * fix(engine): keep ExtraArgs from overriding --thinking-display Copilot review of #237 found the forced flag was not actually forced. cfg.ExtraArgs are appended AFTER every engine-owned flag, and the CLI keeps the LAST value of a repeated option, so a config carrying --thinking-display omitted would win silently and restore exactly the signature-only thinking blocks this PR exists to prevent. runClaudeCodeTurn now rejects the option in ExtraArgs, in both the separate-value and `=` forms, with the same shape and error style as the existing append-system-prompt conflict check. Both --thinking-display values are rejected, not just "omitted": the engine owns the option, and a config that happens to agree today is still a second owner of one wire contract tomorrow. The review's second finding was a doc overclaim of mine. The package doc said a thinking part is non-empty BECAUSE of the flag, while the code still correctly guards on `r.Text != ""` — an empty summary is a possible answer even with the display requested. The doc now says the flag makes a non-empty summary the ordinary case rather than an impossible one, and the flag's own comment says the same: a request, not a guarantee. Verified: TestClaudeCodeExtraArgsCannotOverrideThinkingDisplay red-verifies against the pre-fix code for both wire forms ("Prompt error = , want a --thinking-display conflict") and passes after. go test -race ./engine/ for the claude-code argv, append-prompt, and reasoning tests is green; gofmt clean. --------- Co-authored-by: andybons --- engine/claude_code_backend.go | 57 +++++++++++++++++++++++++++++- engine/claude_code_backend_test.go | 48 +++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 1 deletion(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 784645bf..01dba267 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -72,7 +72,11 @@ // would render as two separate turns. Appended via plain // Session.append (no usage — see the usage-mapping note below) and // emitted as EventMessage, with one EventReasoningDelta per non-empty -// thinking part, one EventTextDelta per non-empty text part (folding +// thinking part (runClaudeCodeTurn asks for a summary with +// --thinking-display summarized, which is what makes a non-empty one +// the ordinary case rather than the impossible one — the emission stays +// conditional, since the API can still answer with an empty summary), +// one EventTextDelta per non-empty text part (folding // the whole block's text into the message in a single "delta", the // closest honest match to the native EventTextDelta/EventReasoningDelta // contract given the CLI hands over complete blocks), and one @@ -438,6 +442,17 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro } } } + // --thinking-display is engine-owned, and ExtraArgs are appended AFTER + // every engine flag: the CLI keeps the LAST value of a repeated option, + // so an override here would win silently and restore the empty-thinking + // blocks the flag exists to prevent (see its own comment below). Reject + // it up front rather than let a config quietly defeat the driver's own + // wire contract — the same shape as the append-prompt conflict above. + for _, arg := range cfg.ExtraArgs { + if claudeCodeThinkingDisplayArg(arg) { + return nil, fmt.Errorf("engine: claude-code: Config.ClaudeCode.ExtraArgs contains %q, which is engine-owned; runClaudeCodeTurn always sends --thinking-display summarized, and an ExtraArgs value would override it and return empty thinking blocks", arg) + } + } // The --mcp-config file (if s.cfg.MCP has any servers to describe) is // written before the args slice below so its path can be included, and @@ -465,6 +480,38 @@ func (s *Session) runClaudeCodeTurn(ctx context.Context) (*message.Message, erro // which this driver always sets, so it is safe to pass // unconditionally. "--forward-subagent-text", + // Opus 4.7 silently flipped the API default of thinking.display from + // "summarized" to "omitted", and every model after it kept that + // default. Under "omitted" the API still THINKS and still bills for + // it, but returns the thinking block with an empty `thinking` field + // and only its provider signature: claudeCodeAssistantMessage stores + // message.Reasoning{Text: ""}, consumeClaudeCodeStream's + // `if r.Text != ""` guard emits no EventReasoningDelta, and every + // consumer sees a signature-only part it can neither render nor + // align against a streamed row. The boxes console rendered a turn + // TWICE off that asymmetry (meetneptune/boxes#599). + // + // --thinking-display is the CLI's own override for that default and + // the ONLY channel that carries it: the `showThinkingSummaries` + // setting does not reach the request (verified — it returns an empty + // thinking field), and the API parameter is not otherwise reachable + // through the CLI. Verified end to end against a live Opus + // subscription session: 383 characters of summarized thinking where + // the same prompt without the flag returned 0. + // + // The flag is real but NOT listed in `claude --help` (`claude + // --thinking-display bogus` answers "Allowed choices are summarized, + // omitted"), so treat it as a pinned-CLI dependency: a CLI that drops + // it fails the spawn outright rather than silently degrading, which + // is the loud failure this driver wants — the box image pins its own + // CLI version, so an upgrade is a deliberate, testable step. + // + // Summaries only, and a REQUEST rather than a guarantee: the raw + // chain of thought is never exposed by any model under any setting, + // "summarized" is the maximum available, and an empty summary is + // still a possible answer — every reader downstream stays guarded + // for it (consumeClaudeCodeStream's own `r.Text != ""`). + "--thinking-display", "summarized", // All subagent spawning in the claude-code lane must go through // harness's own cross-family "task" tool (server/mcp_history.go, // #223), never the CLI's native same-family equivalent — the two @@ -1748,6 +1795,14 @@ func claudeCodeAppendPromptArg(arg string) bool { strings.HasPrefix(arg, "--append-system-prompt-file=") } +// claudeCodeThinkingDisplayArg reports whether an ExtraArgs entry sets the +// engine-owned --thinking-display option, in either the separate-value or +// the `=` form — see runClaudeCodeTurn's rejection of it. +func claudeCodeThinkingDisplayArg(arg string) bool { + return arg == "--thinking-display" || + strings.HasPrefix(arg, "--thinking-display=") +} + // claudeCodeRetryableClass classifies a "result" event's own reported // failure — subtype plus the human-readable result text — as provider- // weather retryable, mirroring how a native adapter classifies an HTTP diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 6aa542a8..64f5f587 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1049,6 +1049,54 @@ func TestClaudeCodeForwardSubagentTextAlwaysSet(t *testing.T) { } } +// TestClaudeCodeThinkingDisplayAlwaysSummarized proves runClaudeCodeTurn +// always sends --thinking-display summarized. Opus 4.7 and later default +// thinking.display to "omitted", under which the API returns a thinking +// block whose `thinking` field is empty and whose signature is the only +// content: claudeCodeAssistantMessage then stores an empty +// message.Reasoning, consumeClaudeCodeStream emits no EventReasoningDelta +// for it (its `r.Text != ""` guard), and a consumer gets a durable part it +// cannot render and cannot align against the row that streamed the turn. +// The flag is the only channel that overrides that default — the +// showThinkingSummaries setting does not reach the request. +func TestClaudeCodeThinkingDisplayAlwaysSummarized(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + if _, err := s.Prompt(context.Background(), "hi"); err != nil { + t.Fatalf("Prompt: %v", err) + } + invocations := readInvocations(t, logPath) + got, ok := argvValueAfter(invocations[0], "--thinking-display") + if !ok { + t.Fatalf("argv has no --thinking-display: %v", invocations[0]) + } + if want := "summarized"; got != want { + t.Errorf("--thinking-display = %q, want %q", got, want) + } +} + +// TestClaudeCodeExtraArgsCannotOverrideThinkingDisplay proves a config +// cannot quietly defeat the engine-owned --thinking-display. ExtraArgs are +// appended AFTER every engine flag and the CLI keeps the LAST value of a +// repeated option, so an ExtraArgs entry would win silently and restore the +// signature-only thinking blocks the flag exists to prevent. Both wire +// forms are rejected, matching the append-prompt conflict check. +func TestClaudeCodeExtraArgsCannotOverrideThinkingDisplay(t *testing.T) { + for _, args := range [][]string{ + {"--thinking-display", "omitted"}, + {"--thinking-display=omitted"}, + {"--thinking-display", "summarized"}, + } { + t.Run(strings.Join(args, " "), func(t *testing.T) { + s, _ := claudeCodeTestSession(t, "normal") + s.cfg.ClaudeCode.ExtraArgs = args + _, err := s.Prompt(context.Background(), "hi") + if err == nil || !strings.Contains(err.Error(), "--thinking-display") { + t.Fatalf("Prompt error = %v, want a --thinking-display conflict", err) + } + }) + } +} + // TestClaudeCodeDisallowsNativeSpawnTools proves runClaudeCodeTurn always // sends --disallowedTools naming every native Claude Code tool that spawns // a same-family subagent (Agent, Workflow). All subagent spawning in the From 4e9e9c1a79156c62659f6a3369150c9cb3e9f825 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 13:16:51 -0400 Subject: [PATCH 49/95] feat(server,engine): accept blob attachments on the durable enqueue path (#238) boxes' pending-delivery drain forwards every message to a box's harness through POST /session/{id}/enqueue now, live or queued, running or freshly woken. handleEnqueue rejected any part whose type was not "text", so an attachment-bearing message against a box that was not already mid-turn surfaced to the end user as a 502 -- a real production incident, not a theoretical gap. prompt_async solved the identical problem for its own route: decodePromptParts (server/prompt_parts.go) validates a blob part against the same media-type, inline-data, decodability, and size checks a screenshot or PDF must pass, and EnqueuePrompt/QueuedPrompt/promptRecord already carry a plain-queued prompt's blobs through a restart and out through every drain site. handleEnqueue and EnqueuePromptDurable were the two pieces that predated that work and were never widened to match, because durable-enqueue (docs/plans/2026-07-21- durable-enqueue.md) was built and scoped text-only before attachments existed anywhere in harness. This widens both to the same shape, reusing rather than reimplementing: handleEnqueue now decodes its body with promptPartInput/decodePromptParts (the identical validator and constants prompt_async uses) behind the same http.MaxBytesReader(promptRequestMaxBytes) guard, and EnqueuePromptDurable(text, seq, blobs ...*message.Blob) filters blobs through usablePromptBlobs and persists them on its own prompt.queued record (promptRecord.Blobs, already a field -- only the durable write site needed to populate it). A blob rides on its prompt's own seq: EnqueuePromptDurable's idempotency contract (seq at or below the watermark is a clean duplicate no-op) is unchanged, and a retried enqueue that resends the same seq and the same attachment is the caller's existing responsibility, same as for text. dispatchQueueHead already forwarded a QueuedPrompt's Blobs to runPrompt (built for the plain-queue path), so once EnqueuePromptDurable sets Blobs on the in-memory queue entry, delivery needed no further change -- idle dispatch, the tool-call-boundary append, and the goal-turn-boundary injection all already carry a queued prompt's attachments generically. A rejected attachment 400s before any run slot is claimed or seq is consumed, mirroring prompt_async: the caller retries the same seq with a fixed attachment. A text-only enqueue body decodes through the same path and is byte-identical to before -- decodePromptParts's empty-parts rule (no text and no blobs) subsumes the old handler's separate "text must be non-empty" check. Verified: new server tests cover a text+blob body dispatched immediately, a blob-only body, a blob surviving the busy-queue-and-drain path, a duplicate seq with an attachment staying a no-op (no double delivery), an unsupported media type 400ing without consuming seq, the per-attachment size cap, and the whole-body size cap -- each red-verified against the unwidened handler before implementation. New engine tests cover EnqueuePromptDurable persisting blobs onto its own record, allowing an attachment-only durable enqueue, and dropping unusable blobs the same way the plain queue does. Every pre-existing enqueue/prompt_async/queue test in server and engine still passes unchanged, confirming the text-only path is untouched. go build ./..., go vet ./..., gofmt -l ., and go test -race ./... are clean except TestAppendSystemPromptReachesModelOnServe, which fails identically on unmodified origin/main (confirmed via git stash) and is unrelated to this change. --- docs/plans/2026-07-21-durable-enqueue.md | 20 ++ engine/prompt_attachments_test.go | 83 +++++++ engine/queue.go | 22 +- server/AGENTS.md | 13 +- server/enqueue_test.go | 291 +++++++++++++++++++++++ server/handlers.go | 79 +++--- server/openapi.yaml | 46 +++- 7 files changed, 506 insertions(+), 48 deletions(-) diff --git a/docs/plans/2026-07-21-durable-enqueue.md b/docs/plans/2026-07-21-durable-enqueue.md index e679dffe..7597fc6e 100644 --- a/docs/plans/2026-07-21-durable-enqueue.md +++ b/docs/plans/2026-07-21-durable-enqueue.md @@ -1026,3 +1026,23 @@ rewritten to match. resident-or-transient-load helper every other read endpoint uses, instead of the sketch's inline `s.residentSession` + raw `s.opts.LoadSession` pair. +- **`POST /enqueue` is no longer text-only.** This plan (and `handlePrompt`'s + own comment on the branch that added prompt attachments) scoped `enqueue` + to text parts because it was built before attachments existed anywhere in + harness — `EnqueuePromptDurable` had no `blobs` parameter and + `handleEnqueue` rejected any part whose type was not `text`. Once + `prompt_async` gained image/PDF attachments (the change that added + `QueuedPrompt.Blobs`/`promptRecord.Blobs` and threaded them through the + plain `EnqueuePrompt` queue, delivered by the same drain machinery this + plan's queue already uses), the remaining gap was narrow: `enqueue`'s own + HTTP body decode and `EnqueuePromptDurable`'s own signature. `enqueue` now + accepts a `blob` part exactly like `prompt_async` does — same + `decodePromptParts` validator, same per-attachment and whole-body caps, + reused verbatim rather than reimplemented — and `EnqueuePromptDurable` + carries a prompt's blobs on its own `promptRecord`, on the SAME `seq` as + its text, so an attachment durably queued behind a busy box (or across a + restart) survives and replays exactly like a plain-queued one already did. + This closed a real production 502: a caller (boxes' pending-delivery + drain) forwarding an attachment-bearing message through `enqueue` for any + box that was not mid-turn-live got rejected at the wire, with no + degrade-and-retry path at this layer. diff --git a/engine/prompt_attachments_test.go b/engine/prompt_attachments_test.go index 10f1618c..4901ad3c 100644 --- a/engine/prompt_attachments_test.go +++ b/engine/prompt_attachments_test.go @@ -316,3 +316,86 @@ func TestEnqueuePromptDropsUnusableBlobs(t *testing.T) { t.Errorf("operator block = %q, want it to count ONE attachment", block) } } + +// TestEnqueuePromptDurablePersistsAttachments is EnqueuePromptDurable's +// counterpart to TestEnqueuePromptPersistsAttachments: the durable, +// caller-seq-idempotent primitive POST /session/{id}/enqueue calls +// (docs/plans/2026-07-21-durable-enqueue.md) must carry a blob onto its own +// prompt.queued record exactly like the plain queue already does — the +// write half of a box's enqueued screenshot surviving a restart. +func TestEnqueuePromptDurablePersistsAttachments(t *testing.T) { + dir := t.TempDir() + s := NewSession(Config{SessionDir: dir, Model: message.ModelRef{Provider: "test", Model: "m1"}}) + if _, dup, err := s.EnqueuePromptDurable("with a picture", 1, testBlob()); err != nil || dup { + t.Fatalf("EnqueuePromptDurable: dup=%v err=%v", dup, err) + } + + data, err := os.ReadFile(filepath.Join(dir, s.ID+".jsonl")) + if err != nil { + t.Fatal(err) + } + var queued *promptRecord + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + var rec record + if err := json.Unmarshal([]byte(line), &rec); err != nil { + t.Fatalf("unmarshaling session log line %q: %v", line, err) + } + if rec.Type == recPromptQueued { + queued = rec.Prompt + } + } + if queued == nil { + t.Fatalf("no prompt.queued record was written; log:\n%s", data) + } + if queued.Seq != 1 { + t.Errorf("record seq = %d, want 1 (the caller's idempotency seq must ride with the blob)", queued.Seq) + } + if len(queued.Blobs) != 1 || !bytes.Equal(queued.Blobs[0].Data, tinyPNGBytes) { + t.Fatalf("record blobs = %+v, want the enqueued image", queued.Blobs) + } + + q := s.QueuedPrompts() + if len(q) != 1 || len(q[0].Blobs) != 1 || !bytes.Equal(q[0].Blobs[0].Data, tinyPNGBytes) { + t.Fatalf("in-memory queue = %+v, want the same attachment", q) + } +} + +// TestEnqueuePromptDurableAllowsAttachmentOnlyPrompt mirrors +// TestEnqueuePromptAllowsAttachmentOnlyPrompt for the durable path: an +// uploaded screenshot with nothing typed beside it is a real prompt, not an +// empty one, whether it arrives via the best-effort queue or the durable one. +func TestEnqueuePromptDurableAllowsAttachmentOnlyPrompt(t *testing.T) { + s := NewSession(Config{SessionDir: t.TempDir(), Model: message.ModelRef{Provider: "test", Model: "m1"}}) + if _, dup, err := s.EnqueuePromptDurable(" ", 1, testBlob()); err != nil || dup { + t.Fatalf("attachment-only durable enqueue: dup=%v err=%v, want it accepted", dup, err) + } + if _, _, err := s.EnqueuePromptDurable(" ", 2); err != ErrEmptyPromptText { + t.Fatalf("empty durable enqueue error = %v, want ErrEmptyPromptText", err) + } +} + +// TestEnqueuePromptDurableDropsUnusableBlobs mirrors +// TestEnqueuePromptDropsUnusableBlobs for the durable path: a blob nothing +// can deliver (nil, or carrying neither Data nor URL) must not make an +// otherwise-empty prompt valid, and must not survive into the persisted +// record — the same wedge usablePromptBlobs exists to prevent for the plain +// queue. +func TestEnqueuePromptDurableDropsUnusableBlobs(t *testing.T) { + s := NewSession(Config{SessionDir: t.TempDir()}) + + if _, _, err := s.EnqueuePromptDurable("", 1, nil, &message.Blob{MediaType: "image/png"}); !errors.Is(err, ErrEmptyPromptText) { + t.Fatalf("EnqueuePromptDurable with only unusable blobs: err = %v, want ErrEmptyPromptText", err) + } + if q := s.QueuedPrompts(); len(q) != 0 { + t.Fatalf("queue = %+v, want nothing enqueued", q) + } + + real := &message.Blob{MediaType: "image/png", Data: []byte("\x89PNG\r\n\x1a\n")} + if _, dup, err := s.EnqueuePromptDurable("look", 1, nil, real); err != nil || dup { + t.Fatalf("EnqueuePromptDurable: dup=%v err=%v", dup, err) + } + q := s.QueuedPrompts() + if len(q) != 1 || len(q[0].Blobs) != 1 || q[0].Blobs[0] != real { + t.Fatalf("queued blobs = %+v, want only the deliverable one", q) + } +} diff --git a/engine/queue.go b/engine/queue.go index c0ead187..e2ed73b0 100644 --- a/engine/queue.go +++ b/engine/queue.go @@ -422,9 +422,23 @@ func (s *Session) flushQueueRecordsLocked() { // dequeue record and the turn's completion loses the delivery — it is // never redelivered — while the watermark correctly continues to report // the message as accepted: lose-once-on-crash, not deliver-twice. -func (s *Session) EnqueuePromptDurable(text string, seq int64) (id int64, duplicate bool, err error) { +// +// blobs are the prompt's attachments, carried through exactly like +// EnqueuePrompt's own blobs parameter (see its doc comment): variadic so +// every existing text-only caller and call site stays untouched, filtered +// through usablePromptBlobs BEFORE the emptiness check so a caller passing +// only unusable blobs (nil, or one with neither Data nor URL) cannot +// satisfy "empty text is fine when a blob came with it" and durably persist +// a prompt.queued record promising an attachment that can never be +// delivered. They ride on the SAME seq as the prompt's text — there is no +// separate idempotency key for an attachment, so a retry that resends the +// identical seq is required to resend the identical blobs too (the caller +// reconstructs the same request on retry; this method has no way to detect +// a seq reused with DIFFERENT content, same as it already has none for text). +func (s *Session) EnqueuePromptDurable(text string, seq int64, blobs ...*message.Blob) (id int64, duplicate bool, err error) { trimmed := strings.TrimSpace(text) - if trimmed == "" { + usable := usablePromptBlobs(blobs) + if trimmed == "" && len(usable) == 0 { return 0, false, ErrEmptyPromptText } if seq < 1 { @@ -462,7 +476,7 @@ func (s *Session) EnqueuePromptDurable(text string, seq int64) (id int64, duplic // the record it is about to write. s.flushQueueRecordsLocked() const op = "enqueue_durable" - rec := record{Type: recPromptQueued, Prompt: &promptRecord{ID: id, Text: trimmed, Seq: seq}} + rec := record{Type: recPromptQueued, Prompt: &promptRecord{ID: id, Text: trimmed, Seq: seq, Blobs: usable}} if err := s.timedStorePhase(op, "write_record", func() error { return s.writeRecord(rec) }); err != nil { @@ -487,7 +501,7 @@ func (s *Session) EnqueuePromptDurable(text string, seq int64) (id int64, duplic return 0, false, err } } - s.promptQueue = append(s.promptQueue, QueuedPrompt{ID: id, Text: trimmed, Seq: seq}) + s.promptQueue = append(s.promptQueue, QueuedPrompt{ID: id, Text: trimmed, Seq: seq, Blobs: usable}) s.enqueueSeq = seq // Emit while still holding s.mu (see EnqueuePrompt above): keeps event // order matching log order under a concurrent dequeue. OnEvent must not diff --git a/server/AGENTS.md b/server/AGENTS.md index 53eb700e..d5bcf611 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -49,16 +49,17 @@ Another session that owns the workdir still conflicts. natural drain trigger. - Dispatch queued input before goal auto-arm. - Keep abort independent from queue clear. -- Keep `POST /enqueue` write-ahead and idempotent. Keep it text-only. +- Keep `POST /enqueue` write-ahead and idempotent. - Keep `GET /queue` as the reconciliation surface. ## Prompt attachments -`prompt_async` takes file blob parts beside its text parts: images and PDFs -today. Validate every attachment before the run slot is claimed: allowed -media type, inline data, bytes that really are the claimed type, and the -size cap. Reject in the handler. Never persist an attachment a provider -cannot render. +`prompt_async` and `POST /enqueue` both take file blob parts beside their +text parts: images and PDFs today. Validate every attachment before the run +slot is claimed: allowed media type, inline data, bytes that really are the +claimed type, and the size cap. Reject in the handler. Never persist an +attachment a provider cannot render. Share `decodePromptParts` and its caps +between both endpoints — do not fork a second validator. Add a media type only when EVERY provider adapter transcodes it. Each accepted type owns a verifier in `promptAttachmentTypes`. diff --git a/server/enqueue_test.go b/server/enqueue_test.go index 0295182a..3ad03dd7 100644 --- a/server/enqueue_test.go +++ b/server/enqueue_test.go @@ -1,6 +1,7 @@ package server import ( + "bytes" "encoding/json" "net/http" "net/http/httptest" @@ -21,6 +22,19 @@ func (h *harness) enqueue(id, text string, seq int64) (*http.Response, []byte) { }) } +// enqueueParts is enqueue's counterpart for a body whose parts are not all +// plain text — a blob attachment mixed in, or a blob-only body — built the +// same way prompt_attachments_test.go's prompt_async bodies are (a raw +// []any of part maps, so attachmentPart's own map shape drops in +// unchanged). +func (h *harness) enqueueParts(id string, parts []any, seq int64) (*http.Response, []byte) { + h.t.Helper() + return h.do("POST", "/session/"+id+"/enqueue", map[string]any{ + "parts": parts, + "seq": seq, + }) +} + // waitIdle blocks (via GET /session/{id}/wait?until=idle) until the session's // composite state reads idle, returning the final wait snapshot. func (h *harness) waitIdle(id string) waitJSON { @@ -493,3 +507,280 @@ func TestEnqueueWorkdirBusyRejected(t *testing.T) { t.Errorf("409 error = %q, want it to name holder session %s", e.Error, idA) } } + +// TestEnqueueAcceptsImageBlobPart is the RED test for the feature this file +// exists to add: POST /session/{id}/enqueue accepts a `blob` part beside its +// text part, exactly like prompt_async's own +// TestPromptAsyncAcceptsImageBlobPart — same wire shape, same validation +// (decodePromptParts, shared verbatim), reused here for the durable, +// idempotent route. Before this change every enqueue body with a non-text +// part 400ed "v1 accepts text parts only"; that rejection is what boxes' +// pending-delivery drain hit for any attachment-bearing message against a +// box whose harness had not woken (or was already running) — the production +// 502 this branch fixes. +func TestEnqueueAcceptsImageBlobPart(t *testing.T) { + prov := newCapturingProvider(asstTurn("red")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + resp, data := h.enqueueParts(id, []any{ + map[string]any{"type": "text", "text": "what color is this?"}, + attachmentPart("image/png", pngBytes), + }, 1) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("enqueue status %d: %s", resp.StatusCode, data) + } + var er enqueueResponse + if err := json.Unmarshal(data, &er); err != nil { + t.Fatal(err) + } + if er.Status != "started" || er.Watermark != 1 { + t.Fatalf("enqueue response = %+v, want status=started watermark=1", er) + } + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1: %+v", len(users), users) + } + if got := users[0].Parts.Text(); got != "what color is this?" { + t.Errorf("transcript user text = %q, want the prompt text", got) + } + blobs := blobParts(users[0].Parts) + if len(blobs) != 1 || blobs[0].MediaType != "image/png" || !bytes.Equal(blobs[0].Data, pngBytes) { + t.Fatalf("transcript user blobs = %+v, want the uploaded image", blobs) + } + + sent := blobParts(prov.lastUserParts(t)) + if len(sent) != 1 || !bytes.Equal(sent[0].Data, pngBytes) { + t.Fatalf("provider request carried %d blob parts, want the uploaded image", len(sent)) + } +} + +// TestEnqueueAcceptsBlobOnlyPrompt proves an attachment-only enqueue body (no +// text part) is accepted — an uploaded screenshot with nothing typed beside +// it is a real prompt, not an empty one, on the durable route exactly as it +// already is on prompt_async. +func TestEnqueueAcceptsBlobOnlyPrompt(t *testing.T) { + prov := newCapturingProvider(asstTurn("ok")) + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + resp, data := h.enqueueParts(id, []any{attachmentPart("image/png", pngBytes)}, 1) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("enqueue status %d: %s", resp.StatusCode, data) + } + h.waitIdle(id) + + users := h.userMessages(id) + if len(users) != 1 { + t.Fatalf("user messages = %d, want 1", len(users)) + } + if len(blobParts(users[0].Parts)) != 1 { + t.Fatalf("user parts = %+v, want exactly the image blob", users[0].Parts) + } + if got := users[0].Parts.Text(); got != "" { + t.Errorf("user text = %q, want empty for an attachment-only enqueue", got) + } +} + +// TestQueuedEnqueuePromptKeepsItsImage is the durability half of the +// feature, mirroring prompt_async's own TestQueuedPromptKeepsItsImage but +// through the durable/idempotent route: an image enqueued while the session +// is BUSY is durably queued (fsynced before the 202), and the attachment +// still rides along when the queue drains at the turn boundary — proving the +// blob survives the same busy-branch path boxes' pending-delivery drain +// exercises when a box is already running. +func TestQueuedEnqueuePromptKeepsItsImage(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + // First prompt claims the run slot and parks inside the provider. + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]any{{"type": "text", "text": "first"}}, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("first prompt status %d: %s", resp.StatusCode, data) + } + <-prov.started + + resp, data = h.enqueueParts(id, []any{ + map[string]any{"type": "text", "text": "and this screenshot"}, + attachmentPart("image/png", pngBytes), + }, 1) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("queued enqueue status %d: %s", resp.StatusCode, data) + } + var er enqueueResponse + if err := json.Unmarshal(data, &er); err != nil { + t.Fatal(err) + } + if er.Status != "queued" || er.Watermark != 1 { + t.Fatalf("enqueue response = %+v, want status=queued watermark=1", er) + } + + prov.releaseAll() + h.waitIdle(id) + + users := h.userMessages(id) + var withBlob int + for _, m := range users { + for _, b := range blobParts(m.Parts) { + if bytes.Equal(b.Data, pngBytes) { + withBlob++ + } + } + } + if withBlob != 1 { + t.Fatalf("user messages carrying the queued image = %d, want 1: %+v", withBlob, users) + } +} + +// TestEnqueueDuplicateSeqWithBlobIsNoOp proves the durability contract's seq +// idempotency extends to a blob-bearing prompt: a retried enqueue carrying +// the SAME seq (and the same attachment) must be a clean 200 duplicate +// no-op — not a second queue entry and not a second delivered attachment — +// exactly like a retried text-only enqueue already is +// (TestEnqueueBusyQueuesAndDeduplicates). The blob rides on its prompt's +// seq, not on a seq of its own. +func TestEnqueueDuplicateSeqWithBlobIsNoOp(t *testing.T) { + prov := newBlockingProvider("test") + h := newHarness(t, prov) + id := h.createSession("test/m1") + pngBytes := testPNG(t) + + resp, data := h.do("POST", "/session/"+id+"/prompt_async", map[string]any{ + "parts": []map[string]any{{"type": "text", "text": "occupant"}}, + }) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("occupant prompt status %d: %s", resp.StatusCode, data) + } + <-prov.started + + body := []any{ + map[string]any{"type": "text", "text": "with a picture"}, + attachmentPart("image/png", pngBytes), + } + resp, data = h.enqueueParts(id, body, 1) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("first enqueue status %d: %s", resp.StatusCode, data) + } + + // Same seq, same attachment: a clean duplicate no-op. + resp, data = h.enqueueParts(id, body, 1) + if resp.StatusCode != http.StatusOK { + t.Fatalf("duplicate enqueue status %d: %s", resp.StatusCode, data) + } + var dup enqueueResponse + if err := json.Unmarshal(data, &dup); err != nil { + t.Fatal(err) + } + if dup.Status != "duplicate" || dup.Watermark != 1 { + t.Fatalf("duplicate enqueue response = %+v, want status=duplicate watermark=1", dup) + } + + sess := h.getSessionJSON(id) + if sess.Queued != 1 { + t.Fatalf("queued depth = %d, want 1 (duplicate must not add a second entry)", sess.Queued) + } + + prov.releaseAll() + h.waitIdle(id) + + users := h.userMessages(id) + var withBlob int + for _, m := range users { + for _, b := range blobParts(m.Parts) { + if bytes.Equal(b.Data, pngBytes) { + withBlob++ + } + } + } + if withBlob != 1 { + t.Fatalf("user messages carrying the image = %d, want exactly 1 (a duplicate seq must not double-deliver)", withBlob) + } +} + +// TestEnqueueRejectsUnusableBlobPart proves the same validation caps +// prompt_async enforces (decodePromptParts, shared verbatim) apply to +// enqueue: an unsupported media type 400s, and — because rejection happens +// BEFORE any run slot is claimed or anything is durably accepted — the seq +// is NOT consumed, so a caller can retry the same seq with a fixed +// attachment. +func TestEnqueueRejectsUnusableBlobPart(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + + resp, data := h.enqueueParts(id, []any{attachmentPart("text/plain", []byte("not an image"))}, 1) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status %d, want 400: %s", resp.StatusCode, data) + } + + // The rejected attempt must not have consumed seq=1: GET /queue reports + // watermark 0 (nothing durably accepted yet). + resp, data = h.do("GET", "/session/"+id+"/queue", nil) + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET queue status %d: %s", resp.StatusCode, data) + } + var q queueGetResponse + if err := json.Unmarshal(data, &q); err != nil { + t.Fatal(err) + } + if q.Watermark != 0 { + t.Fatalf("watermark = %d, want 0 (a rejected attachment must not advance the durable watermark)", q.Watermark) + } + + // The same seq now succeeds with a usable attachment. + resp, data = h.enqueueParts(id, []any{attachmentPart("image/png", testPNG(t))}, 1) + if resp.StatusCode != http.StatusAccepted { + t.Fatalf("retry with fixed attachment status %d, want 202: %s", resp.StatusCode, data) + } +} + +// TestEnqueueOversizeBlobRejected proves the single-attachment cap +// (promptAttachmentMaxBytes, the SAME constant prompt_async enforces) is +// reused rather than reimplemented for enqueue. +func TestEnqueueOversizeBlobRejected(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + + oversized := bytes.Repeat([]byte("x"), promptAttachmentMaxBytes+1) + resp, data := h.enqueueParts(id, []any{attachmentPart("image/png", oversized)}, 1) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status %d, want 400: %s", resp.StatusCode, data) + } + if !bytes.Contains(data, []byte("limit")) { + t.Errorf("error = %s, want it to name the per-attachment limit", data) + } +} + +// TestEnqueueOversizeBodyRejectedBeforeDecode proves enqueue's body is +// bounded by the SAME promptRequestMaxBytes MaxBytesReader guard +// prompt_async uses (TestPromptAsyncOversizeBodyRejectedBeforeDecode) — two +// individually-legal attachments whose base64 encoding together exceeds the +// whole-body cap are rejected with 413 before decode even runs, not 400 from +// the per-attachment check. +func TestEnqueueOversizeBodyRejectedBeforeDecode(t *testing.T) { + h := newHarness(t, &scriptedProvider{name: "test"}) + id := h.createSession("test/m1") + + each := append(testPNG(t), bytes.Repeat([]byte("x"), promptAttachmentMaxBytes*2/3)...) + if len(each) >= promptAttachmentMaxBytes { + t.Fatalf("each attachment is %d bytes, which is not under the %d-byte per-attachment cap", + len(each), promptAttachmentMaxBytes) + } + resp, data := h.enqueueParts(id, []any{ + attachmentPart("image/png", each), + attachmentPart("image/png", each), + }, 1) + if resp.StatusCode != http.StatusRequestEntityTooLarge { + t.Fatalf("status %d, want 413: %s", resp.StatusCode, data) + } + if !bytes.Contains(data, []byte("limit")) { + t.Errorf("error = %s, want it to name the request limit", data) + } +} diff --git a/server/handlers.go b/server/handlers.go index e2fbb935..1d0d6838 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -1961,9 +1961,21 @@ type enqueueResponse struct { // session's watermark is a 200 duplicate no-op, so upstream retries are // always safe. Delivery is unchanged queue machinery: idle sessions // dispatch the queue head immediately, busy sessions drain at turn/tool -// boundaries. No model override (queued prompts carry text only — see -// enqueueOrDispatch's doc comment); the workdir-busy 409, draining 503, and -// unknown-session 404 mirror handlePrompt. +// boundaries. No model override (a durably-enqueued prompt is subject to +// the same no-override limit as a queued PromptRequest — see +// enqueueOrDispatch's doc comment: there is no slot in QueuedPrompt to +// carry one through to a future drain); the workdir-busy 409, draining 503, +// and unknown-session 404 mirror handlePrompt. +// +// Text parts and attachment blob parts (images and PDFs), decoded and +// validated by the SAME gate handlePrompt uses (decodePromptParts, +// prompt_parts.go) — reused verbatim rather than duplicated, so `enqueue` +// admits exactly what `prompt_async` admits, at the same per-attachment and +// whole-body size caps. A blob rides through the durable queue on its +// prompt's own seq (engine.Session.EnqueuePromptDurable's blobs parameter) +// and survives a restart with it, replaying through the same drain +// (dispatchQueueHead, tool-call-boundary append, goal-turn-boundary +// injection) that already carries a plain-queued prompt's attachments. func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { id, ok := s.sessionIDOrNotFound(w, r) if !ok { @@ -1973,13 +1985,23 @@ func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { return } var body struct { - Parts []struct { - Type string `json:"type"` - Text string `json:"text"` - } `json:"parts"` - Seq int64 `json:"seq"` - } + Parts []promptPartInput `json:"parts"` + Seq int64 `json:"seq"` + } + // Bound the body BEFORE decoding it, for the same reason handlePrompt + // does (see promptRequestMaxBytes's doc comment): blob data arrives as + // base64 and encoding/json allocates the decoded []byte during + // Unmarshal, so the per-attachment check in decodePromptParts runs only + // after this server has already paid for whatever the caller sent, and + // nothing else bounds how many attachments one body carries. + r.Body = http.MaxBytesReader(w, r.Body, promptRequestMaxBytes) if err := decodeBody(r, &body); err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + writeErr(w, http.StatusRequestEntityTooLarge, fmt.Sprintf( + "request body exceeds the %d-byte limit", promptRequestMaxBytes)) + return + } writeErr(w, http.StatusBadRequest, err.Error()) return } @@ -1991,26 +2013,21 @@ func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { writeErr(w, http.StatusBadRequest, "seq must be >= 1") return } - var texts []string - for _, p := range body.Parts { - if p.Type != "text" { - writeErr(w, http.StatusBadRequest, "v1 accepts text parts only") - return - } - texts = append(texts, p.Text) - } - text := strings.Join(texts, "\n") - // EnqueuePromptDurable rejects empty/whitespace-only text too, but by - // then we'd have already taken (or failed to take) the run-slot claim, - // and from the handler's side that engine error is indistinguishable - // from a genuine persist failure — both fall through to the 500 - // "enqueue not durable" mapping below, which tells the caller to retry - // with the same seq. An input that can never succeed must 400 instead, - // and before any claim is taken. - if strings.TrimSpace(text) == "" { - writeErr(w, http.StatusBadRequest, "text must be non-empty") + // Validation is total and happens BEFORE any run-slot claim or durable + // accept, exactly like handlePrompt: a rejected attachment must not + // consume the caller's seq, so a retry with the SAME seq and a fixed + // attachment succeeds — see decodePromptParts's own doc comment. This + // also folds the old handler's separate "text must be non-empty" check: + // decodePromptParts already rejects empty text with no attachments + // (errEmptyPromptParts), and accepts empty text when a usable + // attachment carries the message (an uploaded screenshot with nothing + // typed beside it). + parts, code, err := decodePromptParts(body.Parts) + if err != nil { + writeErr(w, code, err.Error()) return } + text, blobs := parts.Text, parts.Blobs st, ctx, _, code, holder := s.claimForPrompt(id) if code != 0 { @@ -2018,7 +2035,7 @@ func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { case code == http.StatusConflict && holder != "": writeErr(w, code, fmt.Sprintf("workdir busy: held by session %s", holder)) case code == http.StatusConflict: - s.enqueueDurableBusy(w, id, text, body.Seq) + s.enqueueDurableBusy(w, id, text, body.Seq, blobs...) case code == http.StatusServiceUnavailable: writeErr(w, code, "server shutting down") default: @@ -2030,7 +2047,7 @@ func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { // Idle: we hold the run slot. Durable-first, then dispatch the queue // HEAD — not necessarily this request's prompt (global FIFO, same rule // as handlePrompt's idle-with-queue branch). - ourID, dup, err := st.sess.EnqueuePromptDurable(text, body.Seq) + ourID, dup, err := st.sess.EnqueuePromptDurable(text, body.Seq, blobs...) if dup { s.releasePromptClaim(st) // Stranded-head liveness fix: THIS request's prompt was a no-op, @@ -2091,7 +2108,7 @@ func (s *Server) handleEnqueue(w http.ResponseWriter, r *http.Request) { // failure — never a silent 2xx), then ONE claim retry to close the // freed-slot race. See enqueueOrDispatch's doc comment for the race // analysis; only the enqueue call and response shape differ. -func (s *Server) enqueueDurableBusy(w http.ResponseWriter, id string, text string, seq int64) { +func (s *Server) enqueueDurableBusy(w http.ResponseWriter, id string, text string, seq int64, blobs ...*message.Blob) { sess := s.residentSession(id) if sess == nil { // Same benign race window as enqueueOrDispatch: busy occupant @@ -2100,7 +2117,7 @@ func (s *Server) enqueueDurableBusy(w http.ResponseWriter, id string, text strin writeErr(w, http.StatusConflict, "session is busy with another prompt") return } - ourID, dup, err := sess.EnqueuePromptDurable(text, seq) + ourID, dup, err := sess.EnqueuePromptDurable(text, seq, blobs...) if dup { writeJSON(w, http.StatusOK, enqueueResponse{Status: "duplicate", Watermark: sess.EnqueueSeq()}) return diff --git a/server/openapi.yaml b/server/openapi.yaml index 8c818168..4c31a292 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -994,13 +994,28 @@ components: type: array minItems: 1 items: - $ref: "#/components/schemas/TextPart" + oneOf: + - $ref: "#/components/schemas/TextPart" + - $ref: "#/components/schemas/PromptBlobPart" description: > - Text parts only. Unlike PromptRequest, this write-ahead route - takes no attachments at all: its callers are machine relays - (an inbox poller, a coordinator) whose upstream ack rides on the - durable-accept contract, not the interactive composer a person - attaches a file to. + Text parts and file attachments — the same shape PromptRequest + accepts, validated by the SAME gate (an attachment must be one + of image/png, image/jpeg, image/gif, image/webp, or + application/pdf; must carry inline base64 `data` rather than a + `url`; must really be the type it claims; must not exceed + 20971520 bytes — see PromptRequest's `parts` description for the + full rule and the 33554432-byte whole-body bound). A rejected + attachment 400s before any run slot is claimed or anything is + durably accepted, so the caller's `seq` is not consumed and the + SAME seq is safe to retry with a fixed attachment. + + + A blob rides through the durable queue on its prompt's own + `seq` — there is no separate idempotency key for an + attachment — and survives a process restart with it, replaying + through the same drain machinery that already carries a + plain-queued prompt's attachments (idle dispatch, tool-call- + boundary append, goal-turn-boundary injection). seq: type: integer format: int64 @@ -2472,7 +2487,13 @@ paths: schema: { $ref: "#/components/schemas/EnqueueResponse" } "400": description: > - `parts` is empty, a part's type is not `text`, or `seq` is < 1. + `parts` is empty, a part's type is neither `text` nor `blob`, + `seq` is < 1, or a blob attachment fails PromptRequest's own + admission checks (unsupported media type, no inline `data`, + oversize, or bytes that do not decode as the claimed type) — + see `EnqueueRequest`'s `parts` description. A 400 here means the + caller's `seq` was NOT consumed: retry the same seq once the + attachment is fixed. content: application/json: schema: { $ref: "#/components/schemas/Error" } @@ -2494,6 +2515,17 @@ paths: content: application/json: schema: { $ref: "#/components/schemas/Error" } + "413": + description: > + The request body exceeds 33554432 bytes and was refused before + being decoded — the same `promptRequestMaxBytes` guard + prompt_async's own 413 uses (see its description): blob `data` + is base64 and decoding it allocates, so an oversized body is + stopped at the read rather than after the cost is paid. Nothing + was durably accepted, so the caller's `seq` was not consumed. + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } "500": description: > The prompt.queued record was not durably accepted — a journal From 42f63d68ee11e851e9c9e290e18130b0ca9de9a0 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 14:15:28 -0400 Subject: [PATCH 50/95] fix(engine): emit a turn segment's deltas before its own message (#239) * fix(engine): emit a turn segment's deltas before its own message Andy, on a live console: one assistant reply rendered twice, and the duplicate cleared the moment the next tool call populated. That is this inversion, seen from the outside. consumeClaudeCodeStream emitted EventMessage FIRST and the segment's own text/reasoning/tool deltas after it. The native lane's contract is the other way round -- deltas stream while a turn is open, then the durable message finalizes what they built -- and a consumer's fold is written against exactly that order: deltas grow an open row, EventMessage replaces that row in place and adopts the durable id. Inverted, a consumer holding no open row appends the message as a finished row, and the deltas that follow open a SECOND row and rebuild the same reasoning and text inside it. One model response, on screen twice, verbatim. It self-corrects only when a LATER envelope's message overwrites the stranded row -- which is why the duplicate vanished when a tool call followed, and why a turn ENDING on its own text left the duplicate up until the viewer reloaded (a reload re-derives history from the journal, which never held the second copy). The journal is not involved and never was: it holds exactly one record either way, which is why this was invisible to every log-based investigation of the bug, including two of mine. The fix moves the EventMessage emit to AFTER the per-part delta loop. s.append stays first, so the durable record is still written before a consumer can see a token of it and a crash cannot leave a client holding content the journal lost. The reasoning-only buffering path already emitted its delta with no EventMessage at all, so it was already correct and is untouched. Verified: TestClaudeCodeDeltasPrecedeTheirMessage red-verifies against the previous order ("text.delta for \"Here is my answer.\" emitted at 3, AFTER its own EventMessage at 1"). Reproduced the consumer-side effect directly against the boxes console reducer before writing this: feeding message-then-deltas yields two assistant rows carrying identical text, one durable and one pending. go test -race ./engine/ for the claude-code argv, reasoning, and thinking-display tests is green; gofmt clean. * fix(engine): stream a buffered thinking block's delta exactly once Copilot review of #239: the delta loop re-emitted the buffered reasoning part. The buffering path already streams a reasoning-only envelope's delta the moment it arrives -- so live streaming is unaffected by the buffering -- and the merge then puts that same part at the FRONT of the next envelope's message, so a loop over the whole merged slice sent the thinking text a second time. A consumer that APPENDS deltas therefore rendered the thinking twice until the EventMessage that follows replaced the row. That predates this PR, but moving the deltas ahead of their message is what makes the duplicate visible for the whole of a turn segment rather than inside a row about to be discarded, so it belongs here. The loop now skips the leading parts the buffering path already streamed, counted at merge time. Verified: TestClaudeCodeBufferedReasoningStreamsOnce red-verifies against the pre-fix loop ("reasoning.delta for the buffered thinking block emitted 2 times, want exactly 1"). go test -race ./engine/ for the claude-code reasoning and ordering tests is green; gofmt clean. --------- Co-authored-by: andybons --- engine/claude_code_backend.go | 40 ++++++++++++- engine/claude_code_backend_test.go | 90 ++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+), 3 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 01dba267..4b48c56a 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -71,7 +71,9 @@ // two adjacent assistant messages a one-bubble-per-message console // would render as two separate turns. Appended via plain // Session.append (no usage — see the usage-mapping note below) and -// emitted as EventMessage, with one EventReasoningDelta per non-empty +// emitted as its own parts' deltas FOLLOWED BY EventMessage — the +// native lane's order, which a consumer's fold depends on (see the +// emission site's own comment), with one EventReasoningDelta per non-empty // thinking part (runClaudeCodeTurn asks for a summary with // --thinking-display summarized, which is what makes a non-empty one // the ordinary case rather than the impossible one — the emission stays @@ -1226,6 +1228,15 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( if len(msg.Parts) == 0 { continue } + // alreadyStreamed counts the LEADING parts whose delta this + // envelope must not repeat. The buffering path streams a + // reasoning-only envelope's delta the moment it arrives, so + // live streaming is unaffected by the buffering, and the merge + // below puts that very part at the front of msg.Parts — + // emitting the whole slice would send the same thinking text + // twice, and a consumer that APPENDS deltas would show it + // twice until EventMessage replaced the row. + alreadyStreamed := 0 if len(pendingReasoning) > 0 { if env.ParentToolUseID == pendingReasoningParent { // The common case: this envelope is the rest of the @@ -1237,6 +1248,7 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( merged = append(merged, pendingReasoning...) merged = append(merged, msg.Parts...) msg.Parts = merged + alreadyStreamed = len(pendingReasoning) } else { // A different parent thread interrupted the buffered // thinking block: flush it standalone rather than @@ -1262,9 +1274,30 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( } continue } + // Durable first, then the stream: append writes the record + // before any client can see a token of it, so a crash can + // never leave a consumer holding content the journal lost. s.append(msg) - s.emit(Event{Type: EventMessage, Message: &msg}) - for _, p := range msg.Parts { + // The DELTAS precede their own EventMessage. This is the + // native lane's contract -- deltas stream while a turn is + // open, and the durable message finalizes what they built -- + // and a consumer's fold is written against exactly that + // order: deltas grow an open row, EventMessage replaces that + // row IN PLACE and adopts the durable id. + // + // Emitting EventMessage first inverted it, and the inversion + // duplicated the turn on screen. A consumer that had no open + // row when the message arrived appended it as a finished row, + // then the deltas that followed opened a SECOND row and + // rebuilt the very same reasoning and text inside it -- one + // model response rendered twice, verbatim. It resolved only + // if a LATER envelope's own message happened to overwrite the + // stranded row, so a turn ending on its text (no tool call + // after it) left the duplicate on screen until the viewer + // reloaded. Reported twice against the boxes console + // (meetneptune/boxes#599 fixed a different orphan shape; this + // is the one that produced the plain-text repro). + for _, p := range msg.Parts[alreadyStreamed:] { switch part := p.(type) { case *message.Text: if part.Text != "" { @@ -1278,6 +1311,7 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( s.emit(Event{Type: EventToolStart, ToolCall: part}) } } + s.emit(Event{Type: EventMessage, Message: &msg}) finalMsg = &msg case "user": msg := claudeCodeToolResultMessage(env.Message, env.ParentToolUseID) diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 64f5f587..96589896 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -1119,6 +1119,96 @@ func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { } } +// TestClaudeCodeBufferedReasoningStreamsOnce proves a buffered thinking +// block's delta is emitted exactly once. +// +// The buffering path streams a reasoning-only envelope's delta the moment +// it arrives, so live streaming is unaffected by the buffering. The merge +// then puts that same part at the FRONT of the next envelope's message, so +// a delta loop over the whole merged slice sends the thinking text a second +// time — and a consumer that appends deltas renders it twice until the +// EventMessage that follows replaces the row. +func TestClaudeCodeBufferedReasoningStreamsOnce(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking") + + var events []Event + s.cfg.OnEvent = func(ev Event) { events = append(events, ev) } + + if _, err := s.Prompt(context.Background(), "think about it"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + var reasoningDeltas int + for _, ev := range events { + if ev.Type == EventReasoningDelta && ev.Text == "Let me reason about this." { + reasoningDeltas++ + } + } + if reasoningDeltas != 1 { + t.Errorf("reasoning.delta for the buffered thinking block emitted %d times, want exactly 1", reasoningDeltas) + } +} + +// TestClaudeCodeDeltasPrecedeTheirMessage proves every delta for a turn +// segment reaches the consumer BEFORE that segment's own EventMessage. +// +// This is the native lane's contract, and a consumer's fold is written +// against it: deltas grow an open row, and EventMessage replaces that row +// in place and adopts the durable id. Emitting EventMessage first inverted +// it, and the inversion duplicated the turn on screen — a consumer with no +// open row appended the message as a finished row, then the deltas that +// followed opened a SECOND row and rebuilt the same reasoning and text +// inside it. One model response, rendered twice, verbatim. It cleared only +// when a later envelope's message happened to overwrite the stranded row, +// so a turn ending on its own text left the duplicate up until reload. +func TestClaudeCodeDeltasPrecedeTheirMessage(t *testing.T) { + s, _ := claudeCodeTestSession(t, "thinking") + + var events []Event + s.cfg.OnEvent = func(ev Event) { events = append(events, ev) } + + if _, err := s.Prompt(context.Background(), "think about it"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + firstMessage := -1 + for i, ev := range events { + if ev.Type == EventMessage { + firstMessage = i + break + } + } + if firstMessage < 0 { + t.Fatalf("no EventMessage emitted: %+v", events) + } + // Both deltas belong to the message at firstMessage (one merged + // reasoning+text turn — see TestClaudeCodeThinkingBlockDecodesToReasoningPart), + // so both must sit ahead of it. + for _, want := range []struct { + kind string + text string + }{ + {EventReasoningDelta, "Let me reason about this."}, + {EventTextDelta, "Here is my answer."}, + } { + at := -1 + for i, ev := range events { + if ev.Type == want.kind && ev.Text == want.text { + at = i + break + } + } + if at < 0 { + t.Errorf("no %s carrying %q: %+v", want.kind, want.text, events) + continue + } + if at > firstMessage { + t.Errorf("%s for %q emitted at %d, AFTER its own EventMessage at %d; deltas must precede the message they build", + want.kind, want.text, at, firstMessage) + } + } +} + // TestClaudeCodeThinkingBlockDecodesToReasoningPart proves a "thinking" // content block — previously silently dropped (see claudeCodeContentBlock's // switch in claudeCodeAssistantMessage) — decodes into a message.Reasoning From 0495e61254cffe97ed5cd1cfb3b1caf4b8dbd155 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 14:21:05 -0400 Subject: [PATCH 51/95] fix(engine): assemble one upstream response into one message (#240) A response holding parallel tool calls arrived as two harness messages. The CLI streams each content block of one API response as its own "assistant" envelope, all repeating that response's message.id, so [thinking, tool_use, tool_use] became: assistant[Reasoning, ToolCall] tool[ToolResult] assistant[ToolCall] tool[ToolResult] One model response, split, with no way to put it back together: the upstream id was decoded nowhere and persisted nowhere. Reported from a live box, and reproduced against a real `claude` 2.1.251 binary -- one upstream id spanning two envelopes, with the first tool's result interleaved before the second tool_use. consumeClaudeCodeStream now assembles by upstream id. An envelope carrying the same id as the message being assembled extends it instead of starting a second one; a different id, a "result", or the stream ending closes it. An envelope with NO id -- an older CLI, an unrecognized shape -- journals immediately, exactly as before. Because the CLI runs the first tool and reports its result BEFORE it sends the second tool_use, grouping means holding the assistant message across that execution, and holding the interleaved tool_result behind it: a result must never be journaled ahead of the call it answers. THE COST, stated plainly: this widens the crash window. Today every envelope is journaled the moment it arrives, so a crash mid-execution still keeps the record of the call; with grouping, that crash loses the call record for a tool that already ran and had its side effects. The window is one response's own tool executions. Nothing else regresses -- every LIVE event (tool start, tool end, deltas) is still emitted the instant its envelope arrives, so no consumer waits on the group. Two boundaries the grouping does not cross, both from review: The id boundary is checked BEFORE the reasoning-only branch. A thinking block that opens the NEXT response carries a different id, but that branch buffers and continues, so the check behind it never ran. A tool_result from a DIFFERENT parent_tool_use_id closes the open response rather than joining its hold. Deferring only same-thread results and appending the rest at once -- the reviewer's suggestion -- made the test fail for a reason worth recording: journaling a subagent result while a main-thread response is still buffered puts that result AHEAD of a message the wire sent first. Verified: TestClaudeCodeGroupsParallelToolCallsByUpstreamID red-verified against the pre-fix code ("History() len = 6, want 5: [... assistant[Reasoning,ToolCall] tool[ToolResult] assistant[ToolCall] ...]"), and TestClaudeCodeGroupingRespectsResponseAndThreadBoundaries covers both boundaries above -- it caught the ordering inversion on its first run. Two fakeclaude modes reproduce the real binary's streaming: "parallel_tools" (shared id, interleaved result) and "parallel_tools_crossing" (a subagent's own call and result around a main-thread response, then a thinking-only envelope opening the next). go test -race ./engine/ green (whole package); gofmt clean. Co-authored-by: andybons --- engine/claude_code_backend.go | 179 +++++++++++++++++++++++++---- engine/claude_code_backend_test.go | 146 +++++++++++++++++++++++ engine/testdata/fakeclaude/main.go | 154 +++++++++++++++++++++++++ 3 files changed, 455 insertions(+), 24 deletions(-) diff --git a/engine/claude_code_backend.go b/engine/claude_code_backend.go index 4b48c56a..5419d19b 100644 --- a/engine/claude_code_backend.go +++ b/engine/claude_code_backend.go @@ -1148,6 +1148,71 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( var pendingReasoning message.Parts var pendingReasoningParent string + // pendingAssistant assembles the ONE message.Message for one upstream + // API response. The CLI streams each content block of a response as + // its own "assistant" envelope, all sharing that response's + // message.id, so a response holding two parallel tool_use blocks + // arrived as two envelopes and became two harness messages — one model + // response, split in half, with no way for any consumer to put it back + // together (the upstream id was decoded nowhere and persisted nowhere). + // + // The CLI runs the first tool and reports its result BEFORE it sends + // the second tool_use envelope, so grouping means holding the assistant + // message across that execution, and holding the interleaved + // tool_result behind it to keep a result after the call it answers. + // deferredToolResults is that hold. The COST is a wider crash window: + // today each envelope is journaled the moment it arrives, and a crash + // mid-execution keeps the call; with grouping, that crash loses the + // call record for a tool that already ran. Nothing else regresses -- + // every LIVE event (tool start, tool end, deltas) is still emitted the + // instant its envelope arrives, so no consumer waits on the group. + var pendingAssistant *message.Message + var pendingAssistantUpstream string + var pendingAssistantParent string + var deferredToolResults []message.Message + + // emitClaudeCodeParts streams one envelope's parts, always AHEAD of the + // message they belong to (see the EventMessage emit for why that order + // is load-bearing). + emitClaudeCodeParts := func(parts message.Parts) { + for _, p := range parts { + switch part := p.(type) { + case *message.Text: + if part.Text != "" { + s.emit(Event{Type: EventTextDelta, Text: part.Text}) + } + case *message.Reasoning: + if part.Text != "" { + s.emit(Event{Type: EventReasoningDelta, Text: part.Text}) + } + case *message.ToolCall: + s.emit(Event{Type: EventToolStart, ToolCall: part}) + } + } + } + + // flushPendingAssistant journals the assembled message and then every + // tool_result held behind it, in arrival order — the shape the wire + // would have had if the CLI had sent the whole response at once. + flushPendingAssistant := func() { + if pendingAssistant == nil { + return + } + msg := *pendingAssistant + pendingAssistant = nil + pendingAssistantUpstream = "" + pendingAssistantParent = "" + s.append(msg) + s.emit(Event{Type: EventMessage, Message: &msg}) + finalMsg = &msg + for i := range deferredToolResults { + result := deferredToolResults[i] + s.append(result) + s.emit(Event{Type: EventMessage, Message: &result}) + } + deferredToolResults = nil + } + // flushPendingReasoning appends any buffered reasoning as a // standalone assistant message. This is the uncommon path: it fires // only when a differently-parented envelope interrupts a buffered @@ -1259,6 +1324,19 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( pendingReasoning = nil pendingReasoningParent = "" } + // The id boundary is checked BEFORE the reasoning-only branch + // below: a thinking block opening the NEXT response would + // otherwise slip past it (that branch buffers and continues), + // leaving the previous response's message open across a + // boundary it has nothing to do with — a longer hold than the + // one this grouping justifies, and a different flush order on + // a truncated stream. + upstream := claudeCodeUpstreamID(env.Message) + if pendingAssistant != nil && + (upstream == "" || upstream != pendingAssistantUpstream || + env.ParentToolUseID != pendingAssistantParent) { + flushPendingAssistant() + } if reasoningOnlyParts(msg.Parts) { // Buffer it — do not append or emit EventMessage yet — // and wait for the envelope that completes this turn @@ -1274,10 +1352,16 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( } continue } - // Durable first, then the stream: append writes the record - // before any client can see a token of it, so a crash can - // never leave a consumer holding content the journal lost. - s.append(msg) + // One upstream response, one message: an envelope carrying + // the SAME upstream id as the one being assembled extends it + // rather than starting a second message. Its parts still + // stream immediately. Any mismatch already flushed above, so + // a surviving pendingAssistant here IS this envelope's own. + if pendingAssistant != nil && upstream != "" { + emitClaudeCodeParts(msg.Parts[alreadyStreamed:]) + pendingAssistant.Parts = append(pendingAssistant.Parts, msg.Parts...) + continue + } // The DELTAS precede their own EventMessage. This is the // native lane's contract -- deltas stream while a turn is // open, and the durable message finalizes what they built -- @@ -1297,29 +1381,23 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // reloaded. Reported twice against the boxes console // (meetneptune/boxes#599 fixed a different orphan shape; this // is the one that produced the plain-text repro). - for _, p := range msg.Parts[alreadyStreamed:] { - switch part := p.(type) { - case *message.Text: - if part.Text != "" { - s.emit(Event{Type: EventTextDelta, Text: part.Text}) - } - case *message.Reasoning: - if part.Text != "" { - s.emit(Event{Type: EventReasoningDelta, Text: part.Text}) - } - case *message.ToolCall: - s.emit(Event{Type: EventToolStart, ToolCall: part}) - } + emitClaudeCodeParts(msg.Parts[alreadyStreamed:]) + pendingAssistant = &msg + pendingAssistantUpstream = upstream + pendingAssistantParent = env.ParentToolUseID + if upstream == "" { + // Nothing to group on: journal it at once, exactly as this + // driver did before grouping existed. + flushPendingAssistant() } - s.emit(Event{Type: EventMessage, Message: &msg}) - finalMsg = &msg case "user": msg := claudeCodeToolResultMessage(env.Message, env.ParentToolUseID) if msg == nil { continue } - s.append(*msg) - s.emit(Event{Type: EventMessage, Message: msg}) + // The live tool end first, for the same reason a delta + // precedes its message: a consumer folds the result onto the + // tool card it already has open. for _, p := range msg.Parts { if tr, ok := p.(*message.ToolResult); ok { s.emit(Event{ @@ -1330,7 +1408,36 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( }) } } + if pendingAssistant != nil && env.ParentToolUseID != pendingAssistantParent { + // A DIFFERENT thread's result while a response is open. + // Holding it behind that response would reorder it for + // nothing (it answers a call already journaled), but + // journaling it while the response is still buffered would + // put it AHEAD of a message the wire sent first. Close the + // response instead: wire order survives, and the only cost + // is that a group interrupted by another thread's result + // ends there. + flushPendingAssistant() + } + if pendingAssistant != nil { + // Hold it behind the message that carries its call: the + // CLI reports this result BEFORE the response's remaining + // tool_use blocks, and a result must never be journaled + // ahead of the call it answers. + // + // Only for the SAME thread. A subagent's result answers a + // call in a message that is already journaled, so holding + // it behind an unrelated main-thread response would + // reorder it for nothing and hold it longer than the + // response whose calls are being grouped. + deferredToolResults = append(deferredToolResults, *msg) + continue + } + s.append(*msg) + s.emit(Event{Type: EventMessage, Message: msg}) case "result": + // Terminal: nothing more can join the open response. + flushPendingAssistant() usage := mapClaudeCodeUsage(env.Usage) s.applyClaudeCodeUsage(usage, env.TotalCostUSD) streamMillis := env.DurationMillis - env.TTFTMillis @@ -1398,8 +1505,10 @@ func (s *Session) consumeClaudeCodeStream(r io.Reader, model message.ModelRef) ( // beyond the four above) is inert activity, per the package doc. } // The scanner loop ended without ever reaching "result" (a crashed or - // truncated stream) — flush any buffered thinking block now rather - // than silently drop it. See flushPendingReasoning's own doc comment. + // truncated stream) — flush what is buffered now rather than silently + // drop it. See flushPendingReasoning's and flushPendingAssistant's own + // doc comments. + flushPendingAssistant() flushPendingReasoning() return finalMsg, started, turnErr } @@ -1599,7 +1708,14 @@ func mapClaudeCodeUsage(u *claudeCodeUsage) provider.Usage { // claudeCodeContentBlock — see decodeClaudeCodeContentBlocks, which // accepts both. type claudeCodeMessage struct { - Role string `json:"role"` + Role string `json:"role"` + // ID is the UPSTREAM Anthropic message id (msg_01...) — the id of the + // one API response this envelope carries a single content block of. + // Several consecutive "assistant" envelopes share it whenever a + // response holds more than one block, which is what + // consumeClaudeCodeStream groups on; harness mints its own id for the + // message it assembles and never persists this one. + ID string `json:"id,omitempty"` Content json.RawMessage `json:"content"` } @@ -1822,6 +1938,21 @@ func claudeCodeAppendSystemPrompt(segments []string) (string, bool) { return strings.Join(segments, "\n\n"), true } +// claudeCodeUpstreamID reads an envelope's upstream Anthropic message id. +// Empty for any envelope that carries none — an older CLI, a shape this +// decoder does not recognize — which consumeClaudeCodeStream treats as +// "cannot group", falling back to one harness message per envelope. +func claudeCodeUpstreamID(raw json.RawMessage) string { + if len(raw) == 0 { + return "" + } + var cm claudeCodeMessage + if err := json.Unmarshal(raw, &cm); err != nil { + return "" + } + return cm.ID +} + func claudeCodeAppendPromptArg(arg string) bool { return arg == "--append-system-prompt" || strings.HasPrefix(arg, "--append-system-prompt=") || diff --git a/engine/claude_code_backend_test.go b/engine/claude_code_backend_test.go index 96589896..58b67b8e 100644 --- a/engine/claude_code_backend_test.go +++ b/engine/claude_code_backend_test.go @@ -10,6 +10,7 @@ import ( "os" "os/exec" "path/filepath" + "slices" "strconv" "strings" "sync" @@ -1119,6 +1120,78 @@ func TestClaudeCodeDisallowsNativeSpawnTools(t *testing.T) { } } +// TestClaudeCodeGroupsParallelToolCallsByUpstreamID proves ONE upstream API +// response becomes ONE harness message, even when the CLI streams its +// content blocks as several envelopes with the first tool's result +// interleaved between them. +// +// A real `claude` binary sends one envelope per content block, every +// envelope repeating the response's own message.id, and it runs the first +// tool before it sends the second tool_use. Appending one message per +// envelope therefore split a single response holding two parallel tool +// calls into two adjacent assistant messages, and the upstream id was +// decoded nowhere, so nothing downstream could put them back together. +// +// The result order is the other half of the contract: a tool_result must +// never be journaled ahead of the call it answers, so the interleaved +// result is held behind the assembled message and lands after it. +func TestClaudeCodeGroupsParallelToolCallsByUpstreamID(t *testing.T) { + s, _ := claudeCodeTestSession(t, "parallel_tools") + if _, err := s.Prompt(context.Background(), "run both"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + hist := s.History() + // user prompt, assistant(reasoning + BOTH tool calls), tool(alpha), + // tool(beta), assistant(text) = 5 — not 6, the pre-fix shape with the + // second tool call stranded in its own message. + if len(hist) != 5 { + var shape []string + for _, m := range hist { + kinds := make([]string, 0, len(m.Parts)) + for _, p := range m.Parts { + kinds = append(kinds, fmt.Sprintf("%T", p)) + } + shape = append(shape, string(m.Role)+"["+strings.Join(kinds, ",")+"]") + } + t.Fatalf("History() len = %d, want 5: %v", len(hist), shape) + } + + asst := hist[1] + if asst.Role != message.RoleAssistant { + t.Fatalf("hist[1].Role = %q, want assistant", asst.Role) + } + var calls []string + for _, p := range asst.Parts { + if tc, ok := p.(*message.ToolCall); ok { + calls = append(calls, tc.CallID) + } + } + if want := []string{"toolu_alpha", "toolu_beta"}; !slices.Equal(calls, want) { + t.Errorf("assembled tool calls = %v, want %v (both blocks of one response, in order)", calls, want) + } + if _, ok := asst.Parts[0].(*message.Reasoning); !ok { + t.Errorf("hist[1].Parts[0] = %T, want the response's own Reasoning first", asst.Parts[0]) + } + + // Both results follow the message that carries their calls, in arrival + // order, and the NEXT response's own id ends the group rather than + // joining it. + for i, want := range []string{"toolu_alpha", "toolu_beta"} { + m := hist[2+i] + if m.Role != message.RoleTool { + t.Fatalf("hist[%d].Role = %q, want tool", 2+i, m.Role) + } + tr, ok := m.Parts[0].(*message.ToolResult) + if !ok || tr.CallID != want { + t.Errorf("hist[%d].Parts[0] = %+v, want a ToolResult for %q", 2+i, m.Parts[0], want) + } + } + if got := hist[4].Parts.Text(); got != "done" { + t.Errorf("hist[4] text = %q, want the separate response %q", got, "done") + } +} + // TestClaudeCodeBufferedReasoningStreamsOnce proves a buffered thinking // block's delta is emitted exactly once. // @@ -1149,6 +1222,79 @@ func TestClaudeCodeBufferedReasoningStreamsOnce(t *testing.T) { } } +// TestClaudeCodeGroupingRespectsResponseAndThreadBoundaries proves the two +// boundaries the grouping must not cross, both found by review of #240. +// +// A SUBAGENT tool_result arrives on a different parent_tool_use_id while a +// main-thread response is still being assembled. It answers a call the +// subagent's own earlier response already journaled, so holding it behind +// the unrelated main-thread response would reorder it for nothing — and +// journaling it while that response is still buffered would put it AHEAD +// of a message the wire sent first. The open response closes instead. +// +// A THINKING-ONLY envelope then opens the NEXT response. It carries a +// different upstream id, but the reasoning-buffer path buffers and +// continues, so the id boundary has to be checked BEFORE that branch or +// the previous response stays open across a boundary it has nothing to do +// with. +func TestClaudeCodeGroupingRespectsResponseAndThreadBoundaries(t *testing.T) { + s, _ := claudeCodeTestSession(t, "parallel_tools_crossing") + if _, err := s.Prompt(context.Background(), "cross the boundaries"); err != nil { + t.Fatalf("Prompt: %v", err) + } + + hist := s.History() + var shape []string + for _, m := range hist { + kinds := make([]string, 0, len(m.Parts)) + for _, p := range m.Parts { + kinds = append(kinds, fmt.Sprintf("%T", p)) + } + shape = append(shape, string(m.Role)+"["+strings.Join(kinds, ",")+"]") + } + // user, assistant[subagent tool_use], assistant[main tool_use], + // tool[subagent result], assistant[reasoning + text]. + if len(hist) != 5 { + t.Fatalf("History() len = %d, want 5: %v", len(hist), shape) + } + + // Wire order survives: the subagent's result lands AFTER the + // main-thread message the wire sent before it, never held behind it + // and never journaled ahead of it. + if hist[1].ParentToolUseID != "toolu_parent" { + t.Errorf("hist[1].ParentToolUseID = %q, want the subagent's own thread: %v", hist[1].ParentToolUseID, shape) + } + if hist[2].ParentToolUseID != "" { + t.Errorf("hist[2].ParentToolUseID = %q, want the main thread: %v", hist[2].ParentToolUseID, shape) + } + if hist[3].Role != message.RoleTool { + t.Fatalf("hist[3].Role = %q, want the subagent tool result after the main-thread message: %v", hist[3].Role, shape) + } + tr, ok := hist[3].Parts[0].(*message.ToolResult) + if !ok || tr.CallID != "toolu_child" { + t.Errorf("hist[3].Parts[0] = %+v, want a ToolResult for toolu_child", hist[3].Parts[0]) + } + + // The next response's thinking block closed the previous one rather + // than joining it: its reasoning belongs to the FINAL message. + last := hist[4] + if last.Role != message.RoleAssistant { + t.Fatalf("hist[4].Role = %q, want assistant: %v", last.Role, shape) + } + if _, ok := last.Parts[0].(*message.Reasoning); !ok { + t.Errorf("hist[4].Parts[0] = %T, want the second response's own Reasoning", last.Parts[0]) + } + if got := last.Parts.Text(); got != "done" { + t.Errorf("hist[4] text = %q, want %q", got, "done") + } + // Each earlier response kept exactly its own single tool call. + for _, i := range []int{1, 2} { + if len(hist[i].Parts) != 1 { + t.Errorf("hist[%d].Parts = %+v, want exactly that response's one tool call", i, hist[i].Parts) + } + } +} + // TestClaudeCodeDeltasPrecedeTheirMessage proves every delta for a turn // segment reaches the consumer BEFORE that segment's own EventMessage. // diff --git a/engine/testdata/fakeclaude/main.go b/engine/testdata/fakeclaude/main.go index 51bfeb23..6b711dd8 100644 --- a/engine/testdata/fakeclaude/main.go +++ b/engine/testdata/fakeclaude/main.go @@ -450,6 +450,160 @@ func main() { "duration_ms": 800, }) return + case "parallel_tools": + // One upstream API response carrying a thinking block and TWO + // parallel tool_use blocks, streamed the way a real `claude` + // 2.1.251 binary streams it (verified live): one envelope per + // content block, every envelope repeating the SAME upstream + // message.id, and the FIRST tool's result interleaved before the + // second tool_use envelope is sent. A separate response with its + // own id closes the turn, so the test can prove the grouping ends + // at the id boundary rather than swallowing everything. + const parallelID = "msg_011FAKEPARALLEL" + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": parallelID, + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Two commands, one response.", "signature": "sig-par"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": parallelID, + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_alpha", "name": "Bash", "input": map[string]any{"command": "echo alpha"}}, + }, + }, + }) + emit(map[string]any{ + "type": "user", + "message": map[string]any{ + "role": "user", + "content": []map[string]any{ + {"type": "tool_result", "tool_use_id": "toolu_alpha", "content": "alpha"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": parallelID, + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_beta", "name": "Bash", "input": map[string]any{"command": "echo beta"}}, + }, + }, + }) + emit(map[string]any{ + "type": "user", + "message": map[string]any{ + "role": "user", + "content": []map[string]any{ + {"type": "tool_result", "tool_use_id": "toolu_beta", "content": "beta"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": "msg_011FAKEFINAL", + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "done"}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "done", + "usage": map[string]any{ + "input_tokens": 20, + "output_tokens": 10, + }, + }) + return + case "parallel_tools_crossing": + // The two boundary cases the grouping must not cross, in one + // stream. A SUBAGENT response calls a tool on its own + // parent_tool_use_id thread and is journaled when the main + // thread's response opens; its tool_result then arrives while + // that main-thread response is still being assembled, so it + // answers a call that IS already journaled and must not be held + // behind an unrelated response. Then a THINKING-ONLY envelope + // opens the next response: its id differs, so it must close the + // open one rather than slip past on the reasoning-buffer path. + const subagentID = "msg_011FAKECROSSSUB" + const firstID = "msg_011FAKECROSSA" + const secondID = "msg_011FAKECROSSB" + emit(map[string]any{ + "type": "assistant", + "parent_tool_use_id": "toolu_parent", + "message": map[string]any{ + "id": subagentID, + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_child", "name": "Bash", "input": map[string]any{"command": "echo child"}}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": firstID, + "role": "assistant", + "content": []map[string]any{ + {"type": "tool_use", "id": "toolu_main", "name": "Bash", "input": map[string]any{"command": "echo main"}}, + }, + }, + }) + emit(map[string]any{ + "type": "user", + "parent_tool_use_id": "toolu_parent", + "message": map[string]any{ + "role": "user", + "content": []map[string]any{ + {"type": "tool_result", "tool_use_id": "toolu_child", "content": "child"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": secondID, + "role": "assistant", + "content": []map[string]any{ + {"type": "thinking", "thinking": "Next response opens with thinking.", "signature": "sig-cross"}, + }, + }, + }) + emit(map[string]any{ + "type": "assistant", + "message": map[string]any{ + "id": secondID, + "role": "assistant", + "content": []map[string]any{ + {"type": "text", "text": "done"}, + }, + }, + }) + emit(map[string]any{ + "type": "result", + "subtype": "success", + "is_error": false, + "result": "done", + "usage": map[string]any{ + "input_tokens": 20, + "output_tokens": 10, + }, + }) + return case "thinking_interleaved": // Proves the reasoning-buffer merge (claude_code_backend.go's // pendingReasoning) attaches ONLY forward, to the envelope that From e21f4153899e0a5797a28cd75e5564b3d6b07437 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 14:33:12 -0400 Subject: [PATCH 52/95] feat(engine): add compaction.started event for live progress (#241) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A client watching a session's event stream sees a compaction only after it settles: history.compacted on success, compaction.failed on error. There is no signal that a compaction is IN PROGRESS, so a UI cannot show a "compacting now" indicator during the summarization call, which is synchronous and can take several seconds. Session.Compact now emits EventCompactionStarted immediately before the blocking runCompactionSummary call, once every early-return skip (not-enough-turns, lone-existing-summary) and journal-boundary error above it has already returned — the emit only fires when Compact is committed to attempting a summary. It carries the same CompactFirstID/CompactLastID/CompactTurnsFolded fields the eventual EventHistoryCompacted carries (computed from the same fold bounds, before the summary exists), so a client can correlate the start with its settlement; it never carries CompactSummaryID, which does not exist yet. Like EventCompactionFailed, it is live-only and never journaled: a "started" that never resolves has nothing durable to reconcile against on replay. The server wires it through Publish as a new live-only "compaction.started" SSE event, mirroring the existing "compaction.failed" case, and server/openapi.yaml's Event schema documents the new type and its fields. Two new tests in engine/compact_test.go cover the pairing invariant directly: TestCompactionStartedPrecedesSummaryAndHistoryCompacted asserts compaction.started fires exactly once, strictly before both the summary's message event and history.compacted, carrying matching fold-range fields; TestCompactionStartedNeverOrphanedOnFailure asserts that when the summarization call itself fails, started is still always followed by compaction.failed. A third test, TestCompactionStartedNotEmittedOnEarlyReturnSkip, asserts the event is absent on a skip that never attempts a summary. All three were red-verified against the exact emit statement before it existed. Verified: go build ./..., go vet ./..., gofmt -l . (clean), and go test -race ./engine/... ./server/... (all pass). --- docs/design/context-compaction.md | 43 +++++--- engine/compact.go | 32 ++++++ engine/compact_test.go | 156 ++++++++++++++++++++++++++++++ engine/engine.go | 8 +- server/journal.go | 25 +++++ server/openapi.yaml | 21 +++- 6 files changed, 265 insertions(+), 20 deletions(-) diff --git a/docs/design/context-compaction.md b/docs/design/context-compaction.md index be940ddc..0b2030f8 100644 --- a/docs/design/context-compaction.md +++ b/docs/design/context-compaction.md @@ -330,21 +330,34 @@ against within that section either. ### Live event surface Anything tailing the event stream (`GET /event`, SSE) must see the -compaction, not just readers of durable state: a successful compaction -emits TWO things, in order. First the summary itself flows through the -ordinary message-event path (`EventMessage` → server journal, the same -route every other message takes), so an `events.jsonl` tailer receives the -summary CONTENT — the durable `compact` record carries the summary inline -rather than as a `recMessage`, so without this emission a tailer would -hold a dangling id for a message it never received. Then a -`history.compacted` engine event (journaled via the server's `emitDurable` -path like `session.status`) carrying `{first_id, last_id, turns_folded, -summary_id}`, where `summary_id` refers to the message the tailer just -saw. A tailer replaying from a `from` cursor older than the compaction -sees the original messages, the summary message, and the compaction event -— the event is the reconciliation signal telling it which prefix the -summary replaced. The `compaction.failed` event (above) is its -fire-and-forget counterpart. +compaction, not just readers of durable state. A live client also wants an +IN-PROGRESS signal, not just a settled one: `compaction.started` fires +exactly once, immediately before the blocking summarization call begins — +after `Compact` has committed to attempting a summary (past every +early-return skip and every journal-boundary error), but before the +summary exists. It carries `{first_id, last_id, turns_folded}` — the same +fold-range fields the eventual settlement carries, computed from the same +fold bounds, so a client can correlate "compacting N turns now" with that +settlement — but no `summary_id`, which does not exist yet. Live only, +like `compaction.failed` below: never journaled, since a `started` that +never resolves has nothing durable to reconcile against on replay. It is +always followed by exactly one of `history.compacted` or +`compaction.failed`, never left orphaned. + +A successful compaction then emits TWO more things, in order. First the +summary itself flows through the ordinary message-event path +(`EventMessage` → server journal, the same route every other message +takes), so an `events.jsonl` tailer receives the summary CONTENT — the +durable `compact` record carries the summary inline rather than as a +`recMessage`, so without this emission a tailer would hold a dangling id +for a message it never received. Then a `history.compacted` engine event +(journaled via the server's `emitDurable` path like `session.status`) +carrying `{first_id, last_id, turns_folded, summary_id}`, where +`summary_id` refers to the message the tailer just saw. A tailer replaying +from a `from` cursor older than the compaction sees the original messages, +the summary message, and the compaction event — the event is the +reconciliation signal telling it which prefix the summary replaced. The +`compaction.failed` event (above) is its fire-and-forget counterpart. ## 5. Non-goals diff --git a/engine/compact.go b/engine/compact.go index 45137e4e..54a5c9d7 100644 --- a/engine/compact.go +++ b/engine/compact.go @@ -21,6 +21,24 @@ import ( const ( EventHistoryCompacted = "history.compacted" EventCompactionFailed = "compaction.failed" + // EventCompactionStarted fires exactly once, immediately before the + // blocking runCompactionSummary call begins (see Session.Compact) — + // only once Compact is fully committed to attempting a summary, past + // every early-return skip (not-enough-turns, lone-existing-summary) + // and every journal-boundary error above it. It is therefore always + // followed by exactly one of EventHistoryCompacted (success) or + // EventCompactionFailed (any failure, including the benign + // empty-summary skip, which still fires EventCompactionFailed for + // observability) — never left orphaned. It carries the same + // CompactFirstID/CompactLastID/CompactTurnsFolded a following + // EventHistoryCompacted will carry, computed from the same fold + // bounds before the summary call runs, so a client can correlate + // "compacting N turns now" with the eventual settlement — but no + // CompactSummaryID, which does not exist yet at this point. Live + // only, like EventCompactionFailed: not journaled, since a "started" + // that never resolves has nothing durable to reconcile against on + // replay. + EventCompactionStarted = "compaction.started" ) // defaultCompactionThreshold is Config.CompactionThreshold's zero-fills-a- @@ -351,6 +369,20 @@ func (s *Session) Compact(ctx context.Context, opts CompactOptions) (CompactResu model = s.Model() } + // Past this point Compact is committed to attempting a summary: every + // early-return skip (not-enough-turns, lone-existing-summary) and every + // journal-boundary error above have already returned. Emit the started + // signal now, immediately before the blocking summary call, so a live + // client can show a "compacting now" indicator — see + // EventCompactionStarted's doc comment for why this is always paired + // with a following EventHistoryCompacted or EventCompactionFailed. + s.emit(Event{ + Type: EventCompactionStarted, + CompactFirstID: journaledFirstID, + CompactLastID: journaledLastID, + CompactTurnsFolded: foldTurns, + }) + summaryText, usage, err := s.runCompactionSummary(ctx, model, history[foldStart:foldEnd+1]) if err != nil { s.emit(Event{Type: EventCompactionFailed, Text: err.Error()}) diff --git a/engine/compact_test.go b/engine/compact_test.go index 0604da28..f83aa235 100644 --- a/engine/compact_test.go +++ b/engine/compact_test.go @@ -857,6 +857,162 @@ func TestCompactSummaryFlowsThroughEventMessageBeforeHistoryCompacted(t *testing } } +// TestCompactionStartedPrecedesSummaryAndHistoryCompacted is the red-first +// test for EventCompactionStarted (docs/design/context-compaction.md §4 +// "Live event surface"): on a successful, triggered compaction, exactly one +// compaction.started fires, strictly before both the summary's EventMessage +// and the closing EventHistoryCompacted — i.e. before the summary result is +// even available — and it carries the same CompactFirstID/CompactLastID/ +// CompactTurnsFolded the eventual EventHistoryCompacted carries, so a client +// can correlate "compacting now" with "compaction settled". +func TestCompactionStartedPrecedesSummaryAndHistoryCompacted(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + compactSummaryTurn("gist", provider.Usage{InputTokens: 5}), + }} + var evs []Event + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + OnEvent: func(ev Event) { evs = append(evs, ev) }, + }) + runTurns(t, s, 2) + evs = nil // discard the two ordinary turns' events + + res, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}) + if err != nil { + t.Fatal(err) + } + + var startedIdx, messageIdx, compactedIdx = -1, -1, -1 + var startedCount int + for i, ev := range evs { + switch ev.Type { + case EventCompactionStarted: + startedCount++ + startedIdx = i + case EventMessage: + if ev.Message != nil && ev.Message.ID == res.Summary.ID { + messageIdx = i + } + case EventHistoryCompacted: + compactedIdx = i + } + } + if startedCount != 1 { + t.Fatalf("EventCompactionStarted count = %d, want exactly 1", startedCount) + } + if messageIdx == -1 { + t.Fatal("no EventMessage carrying the summary was emitted") + } + if compactedIdx == -1 { + t.Fatal("no EventHistoryCompacted was emitted") + } + if startedIdx >= messageIdx { + t.Errorf("EventCompactionStarted at %d, summary EventMessage at %d; want started strictly before the summary is even available", startedIdx, messageIdx) + } + if startedIdx >= compactedIdx { + t.Errorf("EventCompactionStarted at %d, EventHistoryCompacted at %d; want started strictly before", startedIdx, compactedIdx) + } + + started := evs[startedIdx] + if started.CompactFirstID != res.FirstID || started.CompactLastID != res.LastID || started.CompactTurnsFolded != res.TurnsFolded { + t.Errorf("EventCompactionStarted = %+v, want CompactFirstID=%q CompactLastID=%q CompactTurnsFolded=%d (matching the eventual result)", + started, res.FirstID, res.LastID, res.TurnsFolded) + } + if started.CompactSummaryID != "" { + t.Errorf("EventCompactionStarted.CompactSummaryID = %q, want empty (the summary does not exist yet when started fires)", started.CompactSummaryID) + } +} + +// TestCompactionStartedNeverOrphanedOnFailure is the red-first test for +// EventCompactionStarted's pairing invariant on the failure path: a +// compaction that fails AFTER starting (the summarization call itself +// errors) still emits exactly one EventCompactionStarted, and it is always +// followed by an EventCompactionFailed — started must never be left +// dangling with no terminal event. Reuses +// TestCompactFailureNoJournalNoMutation's setup: only two turns are +// scripted, so the third provider.Stream call (the summarization call) +// exhausts p.turns and fails. +func TestCompactionStartedNeverOrphanedOnFailure(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + compactTurn("two", provider.Usage{InputTokens: 10}), + }} + var evs []Event + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + SessionDir: t.TempDir(), + OnEvent: func(ev Event) { evs = append(evs, ev) }, + }) + runTurns(t, s, 2) + evs = nil // discard the two ordinary turns' events + + _, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}) + if err == nil { + t.Fatal("Compact succeeded, want an error (provider call exhausted)") + } + + var startedIdx, failedIdx = -1, -1 + var startedCount, failedCount int + for i, ev := range evs { + switch ev.Type { + case EventCompactionStarted: + startedCount++ + startedIdx = i + case EventCompactionFailed: + failedCount++ + failedIdx = i + } + } + if startedCount != 1 { + t.Fatalf("EventCompactionStarted count = %d, want exactly 1", startedCount) + } + if failedCount != 1 { + t.Fatalf("EventCompactionFailed count = %d, want exactly 1 (started must never be orphaned)", failedCount) + } + if startedIdx >= failedIdx { + t.Errorf("EventCompactionStarted at %d, EventCompactionFailed at %d; want started strictly before failed", startedIdx, failedIdx) + } +} + +// TestCompactionStartedNotEmittedOnEarlyReturnSkip is the red-first test +// for EventCompactionStarted's other half of its pairing invariant: a +// compact call that skips BEFORE ever committing to a summary attempt +// (fewer than the effective keep-turns floor's worth of complete turns — +// SkipReasonNotEnoughTurns, the cheapest of the two early-return skips — +// never calls the provider) must not emit EventCompactionStarted at all — +// only a call that is actually going to attempt a summary ever fires it. +func TestCompactionStartedNotEmittedOnEarlyReturnSkip(t *testing.T) { + prov := &scriptedProvider{name: "test", turns: [][]provider.Event{ + compactTurn("one", provider.Usage{InputTokens: 10}), + }} + var evs []Event + s := NewSession(Config{ + Providers: provider.Registry{"test": prov}, + Model: message.ModelRef{Provider: "test", Model: "m1"}, + OnEvent: func(ev Event) { evs = append(evs, ev) }, + }) + runTurns(t, s, 1) + evs = nil // discard the one ordinary turn's events + + res, err := s.Compact(context.Background(), CompactOptions{KeepTurns: 1}) + if err != nil { + t.Fatal(err) + } + if res.SkipReason != SkipReasonNotEnoughTurns { + t.Fatalf("SkipReason = %q, want %q", res.SkipReason, SkipReasonNotEnoughTurns) + } + + for _, ev := range evs { + if ev.Type == EventCompactionStarted { + t.Fatalf("EventCompactionStarted emitted on a not-enough-turns skip, want none (the provider was never called)") + } + } +} + // TestCompactSurvivesReload is the red-first restart test for §2's // "LoadSession replay": a reloaded session replays the compact record and // the trimmed history — the summary lands exactly where it did live, and diff --git a/engine/engine.go b/engine/engine.go index 817c9918..401bf1ee 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -201,8 +201,12 @@ type Event struct { // EventHistoryCompacted: CompactFirstID/CompactLastID name the folded // range, CompactTurnsFolded is the fold count, and CompactSummaryID // names the summary message (already delivered via a preceding - // EventMessage — see Session.Compact). EventCompactionFailed carries - // only Text (the error detail). + // EventMessage — see Session.Compact). EventCompactionStarted carries + // the same CompactFirstID/CompactLastID/CompactTurnsFolded (computed + // before the summary call, so it always precedes and matches the + // EventHistoryCompacted or EventCompactionFailed that follows it), but + // never CompactSummaryID — the summary does not exist yet when it + // fires. EventCompactionFailed carries only Text (the error detail). CompactFirstID string `json:"compact_first_id,omitempty"` CompactLastID string `json:"compact_last_id,omitempty"` CompactTurnsFolded int `json:"compact_turns_folded,omitempty"` diff --git a/server/journal.go b/server/journal.go index 1b090e05..83d33eb4 100644 --- a/server/journal.go +++ b/server/journal.go @@ -125,6 +125,10 @@ type Event struct { // CompactSummaryID names the summary message — already delivered via a // preceding evtMessage record (see Publish/publishHistoryCompacted). // evtCompactionFailed (live only, never journaled) carries only Error. + // evtCompactionStarted (also live only, never journaled) fires before + // evtHistoryCompacted/evtCompactionFailed and carries the same + // CompactFirstID/CompactLastID/CompactTurnsFolded, but never + // CompactSummaryID — the summary does not exist yet when it fires. CompactFirstID string `json:"compact_first_id,omitempty"` CompactLastID string `json:"compact_last_id,omitempty"` CompactTurnsFolded int `json:"compact_turns_folded,omitempty"` @@ -251,6 +255,14 @@ const ( // counterpart — live only, never journaled (a failed compaction never // mutates durable state, so there is nothing to reconcile on replay). evtCompactionFailed = "compaction.failed" + // evtCompactionStarted mirrors engine.EventCompactionStarted: fired + // once, immediately before Compact's blocking summarization call + // begins, so a live client can show a "compacting now" indicator — see + // that constant's doc comment for why it is always followed by exactly + // one of evtHistoryCompacted or evtCompactionFailed. Live only, like + // evtCompactionFailed above — never journaled, since a "started" that + // never resolves has nothing durable to reconcile on replay. + evtCompactionStarted = "compaction.started" ) const journalName = "events.jsonl" @@ -395,6 +407,19 @@ func (s *Server) Publish(ev engine.Event) { s.publishHistoryCompacted(ev) case engine.EventCompactionFailed: s.publishLive(Event{Type: evtCompactionFailed, SessionID: ev.SessionID, Error: ev.Text}) + case engine.EventCompactionStarted: + // Live only, like evtCompactionFailed above — see + // engine.EventCompactionStarted's doc comment. Carries the same + // fold-range fields the paired evtHistoryCompacted/ + // evtCompactionFailed will carry, minus CompactSummaryID (the + // summary does not exist yet). + s.publishLive(Event{ + Type: evtCompactionStarted, + SessionID: ev.SessionID, + CompactFirstID: ev.CompactFirstID, + CompactLastID: ev.CompactLastID, + CompactTurnsFolded: ev.CompactTurnsFolded, + }) } } diff --git a/server/openapi.yaml b/server/openapi.yaml index 4c31a292..e372aa09 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -1152,6 +1152,7 @@ components: - workdir.worktree_removed - history.compacted - compaction.failed + - compaction.started - text.delta - reasoning.delta - tool.start @@ -1307,6 +1308,14 @@ components: compaction.failed is compaction's fire-and-forget failure counterpart (never journaled — a failed compaction never mutates durable state), carrying the failure detail in `error`. + compaction.started fires once, immediately before the blocking + summarization call begins — live only, never journaled — and + carries the same compact_first_id/compact_last_id/ + compact_turns_folded a following history.compacted or + compaction.failed will carry (but no compact_summary_id, which + does not exist yet). It is always followed by exactly one of + those two, so a client can pair it with the eventual settlement + to show a "compacting now" indicator. session_id: { type: string } seq: type: integer @@ -1497,13 +1506,19 @@ components: workdir.worktree_removed). compact_first_id: type: string - description: The folded range's first message id (history.compacted). + description: > + The folded range's first message id (history.compacted, + compaction.started). compact_last_id: type: string - description: The folded range's last message id (history.compacted). + description: > + The folded range's last message id (history.compacted, + compaction.started). compact_turns_folded: type: integer - description: Number of whole turns folded (history.compacted). + description: > + Number of whole turns folded (history.compacted, + compaction.started). compact_summary_id: type: string description: > From f87a6c66db8dfd0d1193a292d7ffe37b6b5325ab Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 15:41:08 -0400 Subject: [PATCH 53/95] fix(engine): preserve claude-code session state across snapshot loads (#245) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem: a snapshot-based cold load (hibernate -> wake) dropped a delegated (claude-code) session's per-session state: the CLI's own --resume id, the history watermark, and the cumulative dollar cost of every completed delegated turn. The next delegated turn started a fresh CLI session, ran a needless get_conversation_history replay, and reported the session's claude-code cost as $0/unset even though prior turns had already run. Session.claudeCodeCLISessionID, claudeCodeHistoryWatermark, claudeCodeSessionCostUSD, and haveClaudeCodeCost are all set only by a journal-replay fold (recClaudeCodeSessionID, recClaudeCodeHistoryWatermark, and recClaudeCodeUsage, store.go). sessionSnapshot had no field for any of them, and the snapshot path never decodes a record at or before its anchor, so none of those folds ran on a snapshot-restored session. snapshotOnIdle fires right after every delegated turn, right after those records land, so the anchor almost always covered them: the common case, not an edge case. An audit of how this shipped found the real gap one level up: TestSnapshotCarriesEveryFoldedField only checks the snapshot against foldState, a hand-maintained struct literal in the test itself — a field a fold sets that is missing from BOTH sessionSnapshot and foldState passes that test silently, which is exactly how these four fields went missing without a failing test anywhere. Design: add ClaudeCodeCLISessionID, ClaudeCodeHistoryWatermark, ClaudeCodeSessionCostUSD, and HaveClaudeCodeCost to sessionSnapshot, populate them in captureSnapshotLocked from the live session fields, and restore them in restoreSnapshot, mirroring each journal fold's own unconditional assignment. An old snapshot written before these fields existed decodes them to their zero value, which reproduces exactly today's (buggy) behavior rather than failing to load, so no version bump is needed. To close the gap that let the fields go missing, rather than only the fields themselves: engine/snapshot_field_coverage_test.go adds TestEverySessionFieldIsClassifiedForSnapshotting, a reflection-based guard over the Session struct (engine.go). It requires every field reflect.TypeOf(Session{}) reports to appear in exactly one of two explicit sets: snapshottedFields (round-tripped by captureSnapshotLocked/restoreSnapshot) or snapshotExcludedFields (deliberately not snapshotted, each with a one-line reason — session identity/config, a runtime-only handle or lock, journal bookkeeping the loader computes directly, a lazy disk cache, or state a full replay itself never reconstructs either). A field in neither set fails the test immediately, forcing an explicit "snapshot it or exclude it" decision on every new Session field, present and future — fail-closed, rather than depending on someone remembering to extend a hand-maintained comparison struct. Pointer comments at sessionSnapshot's own doc comment and at store.go's LoadSession fold switch point future readers at the guard. Semantic change: a session reloaded through a snapshot now resumes the same claude-code CLI session, carries an accurate history watermark, and reports an accurate cumulative claude-code dollar cost, instead of silently starting a fresh CLI session, resetting the watermark to 0, and reporting cost as unset. A future Session field left unclassified now fails a test instead of silently repeating this bug class. Verification: TestSnapshotCarriesClaudeCodeSessionID drives one delegated turn through the fakeclaude test double, snapshots at the resulting head, reloads (asserting the snapshot path actually ran, not a fallback full replay), and checks the reloaded session's CLI session id, history watermark, and that its next turn's argv carries --resume with that id. TestSnapshotCarriesClaudeCodeCost sets a non-zero cost directly on a session, captures a snapshot, restores it into a fresh session, and checks the cost and its have-bit survive. Both extended foldState (TestSnapshotRoundTrip's own family) with the new fields. Confirmed red against the pre-fix snapshot.go for both tests (empty id, watermark 0, no --resume, and separately cost 0/have false) and green after the fix. TestEverySessionFieldIsClassifiedForSnapshotting was confirmed to fail closed by temporarily adding an unclassified dummy field to Session, observing the test fail with a message naming the field and explaining the required classification, then removing the dummy field (git diff was clean afterward) and confirming the test passes again. go build ./..., go vet ./..., gofmt -l ., and go test -race ./engine/... all pass. --- engine/snapshot.go | 48 +++++ engine/snapshot_field_coverage_test.go | 193 +++++++++++++++++++++ engine/snapshot_test.go | 231 +++++++++++++++++++------ engine/store.go | 11 ++ 4 files changed, 429 insertions(+), 54 deletions(-) create mode 100644 engine/snapshot_field_coverage_test.go diff --git a/engine/snapshot.go b/engine/snapshot.go index 1949fcc7..3f184dfd 100644 --- a/engine/snapshot.go +++ b/engine/snapshot.go @@ -97,6 +97,18 @@ const defaultSnapshotEveryRecords = 64 // whole file is built around, and making an observable field // (session_info's turn count) depend on whether a snapshot happened to // exist. The invariant governs; the field list does not. +// +// Every Session field (engine.go), not only these two, must be classified +// as either round-tripped here or deliberately excluded — see +// snapshottedSessionFields/snapshotExcludedSessionFields and +// TestEverySessionFieldIsClassifiedForSnapshotting in +// engine/snapshot_field_coverage_test.go. A new Session field the author +// forgets to add to this struct (and forgets to wire into +// captureSnapshotLocked/restoreSnapshot below) fails that test, closed by +// construction rather than by remembering to update it: this is the guard +// that would have caught claudeCodeCLISessionID/claudeCodeHistoryWatermark/ +// claudeCodeSessionCostUSD/haveClaudeCodeCost going missing before they +// shipped missing. type sessionSnapshot struct { Version int `json:"version"` ID string `json:"id"` @@ -147,6 +159,27 @@ type sessionSnapshot struct { TurnUnsettled bool `json:"turn_unsettled,omitempty"` CommittedOutcome *taskNotification `json:"committed_outcome,omitempty"` + + // ClaudeCodeCLISessionID, ClaudeCodeHistoryWatermark, + // ClaudeCodeSessionCostUSD and HaveClaudeCodeCost mirror + // Session.claudeCodeCLISessionID/claudeCodeHistoryWatermark/ + // claudeCodeSessionCostUSD/haveClaudeCodeCost — see their own doc + // comments (engine.go). All four are set ONLY by a fold + // (recClaudeCodeSessionID/recClaudeCodeHistoryWatermark/ + // recClaudeCodeUsage, store.go), so without a snapshot field for them + // a snapshot-anchored load silently drops whichever of those records + // fell at or before the anchor: --resume is never passed on the next + // delegated turn (a needless fresh CLI session), the watermark resets + // to 0 (a needless get_conversation_history directive), and the + // cumulative claude-code dollar cost resets to 0/unset (as if no + // delegated turn had ever completed). Empty/zero on an OLD snapshot + // written before these fields existed — the same pre-fix behavior, + // not a load failure. + ClaudeCodeCLISessionID string `json:"claude_code_cli_session_id,omitempty"` + ClaudeCodeHistoryWatermark int `json:"claude_code_history_watermark,omitempty"` + + ClaudeCodeSessionCostUSD float64 `json:"claude_code_session_cost_usd,omitempty"` + HaveClaudeCodeCost bool `json:"have_claude_code_cost,omitempty"` } // sessionSnapshotFile is the on-disk wrapper: the snapshot bytes plus a @@ -432,6 +465,12 @@ func (s *Session) captureSnapshotLocked() *sessionSnapshot { ToolResultBytes: s.toolResultBytes, SpawnedChildIDs: append([]string(nil), s.spawnedChildIDs...), TurnUnsettled: s.turnUnsettled, + + ClaudeCodeCLISessionID: s.claudeCodeCLISessionID, + ClaudeCodeHistoryWatermark: s.claudeCodeHistoryWatermark, + + ClaudeCodeSessionCostUSD: s.claudeCodeSessionCostUSD, + HaveClaudeCodeCost: s.haveClaudeCodeCost, } if len(s.toolResults) > 0 { snap.ToolResults = make(map[string]toolResultMeta, len(s.toolResults)) @@ -515,6 +554,15 @@ func (s *Session) restoreSnapshot(snap *sessionSnapshot) { s.spawnedChildIDs = append([]string(nil), snap.SpawnedChildIDs...) s.taskNotifications = append([]taskNotification(nil), snap.TaskNotifications...) s.turnUnsettled = snap.TurnUnsettled + // Mirrors recClaudeCodeSessionID/recClaudeCodeHistoryWatermark/ + // recClaudeCodeUsage's own unconditional folds (store.go) — an + // empty/zero snapshot value (an old snapshot predating these fields, + // or a session never delegated) restores to exactly the zero value a + // full replay would also leave. + s.claudeCodeCLISessionID = snap.ClaudeCodeCLISessionID + s.claudeCodeHistoryWatermark = snap.ClaudeCodeHistoryWatermark + s.claudeCodeSessionCostUSD = snap.ClaudeCodeSessionCostUSD + s.haveClaudeCodeCost = snap.HaveClaudeCodeCost if snap.CommittedOutcome != nil { oc := *snap.CommittedOutcome s.committedOutcome = &oc diff --git a/engine/snapshot_field_coverage_test.go b/engine/snapshot_field_coverage_test.go new file mode 100644 index 00000000..fa134df9 --- /dev/null +++ b/engine/snapshot_field_coverage_test.go @@ -0,0 +1,193 @@ +package engine + +import ( + "reflect" + "sort" + "strings" + "testing" +) + +// This file is the fail-closed guard TestSnapshotCarriesClaudeCodeSessionID +// and TestSnapshotCarriesClaudeCodeCost exist beside: those two tests each +// pin ONE fold-only field the snapshot forgot (Session.claudeCodeCLISessionID/ +// claudeCodeHistoryWatermark, then claudeCodeSessionCostUSD/haveClaudeCodeCost) +// after a live audit found them missing. But TestSnapshotCarriesEveryFoldedField +// (snapshot_test.go) only checks against foldState, a HAND-MAINTAINED struct +// literal — a field a fold sets that is missing from BOTH sessionSnapshot and +// foldState passes that test silently, which is exactly how the claude-code +// fields slipped through in the first place. +// +// TestEverySessionFieldIsClassifiedForSnapshotting closes that gap +// structurally instead of by remembering to update another hand-maintained +// list: it reflects over the Session struct itself (engine.go) and requires +// EVERY field to appear in EXACTLY ONE of the two classification sets below. +// A field in neither is a compile-clean, test-red failure — the author must +// explicitly decide "snapshot it" or "exclude it, and say why" before the +// field can exist at all. It cannot prove a field in snapshottedSessionFields +// is actually wired correctly into captureSnapshotLocked/restoreSnapshot +// (that correctness is what TestSnapshotCarriesEveryFoldedField and the two +// targeted claude-code tests are for) — it only proves no field was left +// unclassified, which is the specific gap that let two fields go missing +// silently. + +// snapshottedSessionFields is every Session field (engine.go) that +// snapshot.go's captureSnapshotLocked/restoreSnapshot round-trip through a +// sessionSnapshot. Adding a field here without also wiring it into both of +// those functions leaves TestSnapshotCarriesEveryFoldedField (and, for the +// four claude-code fields specifically, TestSnapshotCarriesClaudeCodeSessionID/ +// TestSnapshotCarriesClaudeCodeCost) to catch the omission — this map only +// asserts the field was CONSIDERED, not that the wiring is correct. +var snapshottedSessionFields = map[string]bool{ + "model": true, + "effort": true, + "serviceTier": true, + "history": true, + "usage": true, + "lastUsage": true, + "haveLastUsage": true, + "goalActive": true, + "goalCondition": true, + "compactCount": true, + "lastCompactedAt": true, + "promptQueue": true, + "promptQueueNextID": true, + "enqueueSeq": true, + "toolResults": true, + "toolResultNextID": true, + "toolResultBytes": true, + "mcpSelected": true, + "spawnedChildIDs": true, + "taskNotifications": true, + "turnUnsettled": true, + "committedOutcome": true, + "claudeCodeCLISessionID": true, + "claudeCodeHistoryWatermark": true, + "claudeCodeSessionCostUSD": true, + "haveClaudeCodeCost": true, +} + +// snapshotExcludedSessionFields is every other Session field, each mapped +// to a one-line reason it is deliberately NOT part of the snapshot round +// trip. A new field lands here only when it genuinely belongs to one of +// these categories — session identity/config, a runtime-only handle or +// lock, journal/snapshot bookkeeping the loader computes directly, a lazy +// disk cache, or state a full replay itself never reconstructs either (so +// snapshotting it would violate the snapshot-equals-full-replay invariant +// sessionSnapshot's own doc comment states, not merely omit an +// optimization). When in doubt, it belongs in snapshottedSessionFields +// instead. +var snapshotExcludedSessionFields = map[string]string{ + "ID": "session identity, set directly by NewSession/LoadSession before any header/fold/restore runs", + "cfg": "Config value: header-derived subfields (WorkDir, ParentSession, TaskParentID, ...) are replayed unconditionally from the recSession header regardless of anchor; the rest is construction-time config (live callbacks, SessionDir, ...), not fold state", + "tools": "the registered tool set, (re)constructed by session setup from cfg and runtime capability checks, not a journal fold target", + "mu": "sync.Mutex, runtime-only", + "createdAt": "header-derived; replayed unconditionally from the recSession header regardless of anchor (see sessionSnapshot's own doc comment, \"Header-derived state ... is NOT here\")", + "subscriptionUsage": "explicitly process-local only per its own doc comment; never folded into cumulative state or replayed by LoadSession", + "logFile": "*os.File, runtime-only handle", + "logStarted": "runtime bookkeeping about whether the log file exists; set directly by LoadSession/ensureLog, not a fold", + "lastPersistErr": "runtime error cache; never durable state, a fresh load starts with none", + "recordsWritten": "journal-head bookkeeping the loader computes directly from the journal's own line count, not fold state a record's payload carries", + "snapshotSeq": "the snapshot writer's own anchor bookkeeping; set directly by the loader/writer, not restored from a snapshot payload", + "snapshotting": "coalescing flag for the in-flight snapshot write, runtime-only", + "lastSnapshotErr": "runtime error cache for the snapshot writer itself", + "snapshotWG": "sync.WaitGroup, runtime-only", + "snapshotWrites": "atomic counter, runtime-only diagnostics", + "snapshotInFlight": "atomic counter, runtime-only diagnostics", + "snapshotConcurrentPeak": "atomic counter, runtime-only diagnostics", + "replayedRecords": "the loader's own decoded-record counter, not fold state", + "durableDebt": "in-flight deferred-durable-write counter; snapshotSafeLocked refuses to capture while it is non-zero, so it is always 0 at any actual anchor", + "index": "the metadata-index fold; LoadSession marks it broken on a snapshot load and lets it self-heal on the next write (see LoadSession's own comment: \"Snapshotting the index fold itself is a possible follow-up; nothing here may guess at it\")", + "logSize": "journal byte-length bookkeeping, set by ensureLog/the loader, not fold state", + "indexFile": "*os.File, runtime-only handle", + "lastIndexErr": "runtime error cache for the index sidecar writer", + "instrLoaded": "lazy load-once cache gate, populated from disk on the first Prompt, not journal state", + "instrSeg": "lazy load-once cache payload, same pattern as instrLoaded", + "instrErr": "lazy load-once cache error, same pattern as instrLoaded", + "instrPath": "lazy load-once cache payload, same pattern as instrLoaded", + "turn": "explicitly excluded by sessionSnapshot's own doc comment: no record carries it, so a full replay reports turn=0 too — capturing it would violate the snapshot-equals-full-replay invariant, not merely skip an optimization", + "lastSystem": "same explicit exclusion as turn, same doc comment, same reasoning", + "pendingContinuationNudge": "one-shot scratch state for the single in-flight Prompt call, cleared before that call returns; never persisted", + "skills": "lazy discovery cache, same load-once pattern as instrLoaded", + "skillsLoaded": "lazy discovery cache gate, same pattern as instrLoaded", + "skillsSeg": "lazy discovery cache payload, same pattern as instrLoaded", + "skillsErr": "lazy discovery cache error, same pattern as instrLoaded", + "goalGen": "explicitly documented \"Deliberately runtime-only: never persisted ... never restored on LoadSession\"", + "goalParked": "explicitly documented \"Deliberately runtime-only: never persisted, never folded by LoadSession\"", + "goalParkedReason": "same explicit exclusion as goalParked, same doc comment", + "goalParkedAttempts": "same explicit exclusion as goalParked, same doc comment", + "toolExecCount": "runtime-only retry-safety counter for the current goal-loop attempt; not persisted or folded by LoadSession", + "compactHysteresis": "explicitly documented \"Deliberately NOT persisted: a reload re-evaluates from scratch\"", + "contextWindowExplicit": "derived once at construction from cfg.ContextWindowTokens/the model, re-derived identically by newSession/LoadSession on every load path; not fold state", + "contextWindowSource": "same derivation as contextWindowExplicit, same reasoning", + "contextWindowErr": "same derivation as contextWindowExplicit; set/cleared by construction and SetModel, recomputed the same way on any load", + "toolConcurrency": "resolved once in newSession from Config.ToolConcurrency, read-only afterward; not fold state", + "readBudget": "resolved once in newSession from Config.ToolReadBudgetBytes; not fold state", + "deferredQueueRecords": "in-flight deferred-durable-write buffer; snapshotSafeLocked refuses to capture while it is non-empty, so it is always empty at any actual anchor", + "claudeCodeQueueWake": "atomic.Pointer wake channel for a currently-running turn's stdin pump; nil after any load, never persisted", + "readHashes": "explicitly documented \"Deliberately in-memory and per-live-Session only: never persisted, never folded by LoadSession\"", + "taskNotificationsInFlight": "in-turn checkout state; nil after ANY load (a full replay never populates it either, since checkout only happens during a live turn) — the snapshot's own TaskNotifications field already carries these entries back into the plain taskNotifications queue on restore", + "agentDefsLoaded": "lazy discovery cache (triggered by the task tool's first call), same load-once pattern as instrLoaded/skillsLoaded", + "agentDefs": "lazy discovery cache payload, same pattern as agentDefsLoaded", + "agentDefsErr": "lazy discovery cache error, same pattern as agentDefsLoaded", +} + +// TestEverySessionFieldIsClassifiedForSnapshotting is the fail-closed net: +// every field reflect.TypeOf(Session{}) reports must be in EXACTLY ONE of +// snapshottedSessionFields or snapshotExcludedSessionFields. A newly added +// Session field that is in neither fails this test immediately, forcing +// the author to make an explicit "snapshot it or exclude it, and say why" +// decision — the exact decision that was skipped for claudeCodeCLISessionID/ +// claudeCodeHistoryWatermark/claudeCodeSessionCostUSD/haveClaudeCodeCost +// before this guard existed. +func TestEverySessionFieldIsClassifiedForSnapshotting(t *testing.T) { + typ := reflect.TypeOf(Session{}) + + live := make(map[string]bool, typ.NumField()) + var unclassified, both []string + for i := 0; i < typ.NumField(); i++ { + name := typ.Field(i).Name + live[name] = true + _, snapped := snapshottedSessionFields[name] + _, excluded := snapshotExcludedSessionFields[name] + switch { + case snapped && excluded: + both = append(both, name) + case !snapped && !excluded: + unclassified = append(unclassified, name) + } + } + + if len(both) > 0 { + sort.Strings(both) + t.Errorf("Session field(s) %s are in BOTH snapshottedSessionFields and snapshotExcludedSessionFields (engine/snapshot_field_coverage_test.go) — a field is either snapshotted or excluded, never both", + strings.Join(both, ", ")) + } + if len(unclassified) > 0 { + sort.Strings(unclassified) + t.Errorf("Session field(s) %s are not classified in snapshottedSessionFields or snapshotExcludedSessionFields (engine/snapshot_field_coverage_test.go). "+ + "Add the new field to snapshottedSessionFields if captureSnapshotLocked/restoreSnapshot (snapshot.go) must round-trip it through a snapshot-anchored load, "+ + "or to snapshotExcludedSessionFields with a one-line reason if it is deliberately not snapshot state (runtime-only, header-derived, a lazy cache, journal bookkeeping, ...). "+ + "This is the guard against the bug class that shipped without a snapshot field for claudeCodeCLISessionID/claudeCodeHistoryWatermark/claudeCodeSessionCostUSD/haveClaudeCodeCost.", + strings.Join(unclassified, ", ")) + } + + // The reverse direction: a classification entry naming a field that no + // longer exists on Session (a rename, or a field removed outright) + // would otherwise sit there forever, silently vacuous. + var stale []string + for name := range snapshottedSessionFields { + if !live[name] { + stale = append(stale, name) + } + } + for name := range snapshotExcludedSessionFields { + if !live[name] { + stale = append(stale, name) + } + } + if len(stale) > 0 { + sort.Strings(stale) + t.Errorf("classification names field(s) %s that no longer exist on Session (engine/snapshot_field_coverage_test.go) — remove the stale entry", + strings.Join(stale, ", ")) + } +} diff --git a/engine/snapshot_test.go b/engine/snapshot_test.go index 4c44ebac..383311ca 100644 --- a/engine/snapshot_test.go +++ b/engine/snapshot_test.go @@ -21,33 +21,37 @@ import ( // re-marshaling the snapshot: a test that compared snapshots to snapshots // would pass for a field the snapshot drops on the floor. type foldState struct { - History json.RawMessage - Model message.ModelRef - Effort message.Effort - Usage provider.Usage - LastUsage provider.Usage - HaveLastUsage bool - GoalActive bool - GoalCondition string - CompactCount int - LastCompactedAt string - Queue []QueuedPrompt - QueueNextID int64 - EnqueueSeq int64 - ToolResults map[string]toolResultMeta - ToolResultNext int64 - ToolResultBytes int - MCPSelected map[string]bool - SpawnedChildren []string - Notifications []taskNotification - TurnUnsettled bool - Committed *taskNotification - CreatedAt string - WorkDir string - ParentSession string - TaskParentID string - TaskAgentType string - TaskDepth int + History json.RawMessage + Model message.ModelRef + Effort message.Effort + Usage provider.Usage + LastUsage provider.Usage + HaveLastUsage bool + GoalActive bool + GoalCondition string + CompactCount int + LastCompactedAt string + Queue []QueuedPrompt + QueueNextID int64 + EnqueueSeq int64 + ToolResults map[string]toolResultMeta + ToolResultNext int64 + ToolResultBytes int + MCPSelected map[string]bool + SpawnedChildren []string + Notifications []taskNotification + TurnUnsettled bool + Committed *taskNotification + ClaudeCodeCLISessionID string + ClaudeCodeHistoryWatermark int + ClaudeCodeSessionCostUSD float64 + HaveClaudeCodeCost bool + CreatedAt string + WorkDir string + ParentSession string + TaskParentID string + TaskAgentType string + TaskDepth int } func foldStateOf(t *testing.T, s *Session) string { @@ -59,33 +63,37 @@ func foldStateOf(t *testing.T, s *Session) string { t.Fatalf("marshal history: %v", err) } st := foldState{ - History: h, - Model: s.model, - Effort: s.effort, - Usage: s.usage, - LastUsage: s.lastUsage, - HaveLastUsage: s.haveLastUsage, - GoalActive: s.goalActive, - GoalCondition: s.goalCondition, - CompactCount: s.compactCount, - LastCompactedAt: s.lastCompactedAt.UTC().String(), - Queue: s.promptQueue, - QueueNextID: s.promptQueueNextID, - EnqueueSeq: s.enqueueSeq, - ToolResults: s.toolResults, - ToolResultNext: s.toolResultNextID, - ToolResultBytes: s.toolResultBytes, - MCPSelected: s.mcpSelected, - SpawnedChildren: s.spawnedChildIDs, - Notifications: s.taskNotifications, - TurnUnsettled: s.turnUnsettled, - Committed: s.committedOutcome, - CreatedAt: s.createdAt.UTC().String(), - WorkDir: s.cfg.WorkDir, - ParentSession: s.cfg.ParentSession, - TaskParentID: s.cfg.TaskParentID, - TaskAgentType: s.cfg.TaskAgentType, - TaskDepth: s.cfg.TaskDepth, + History: h, + Model: s.model, + Effort: s.effort, + Usage: s.usage, + LastUsage: s.lastUsage, + HaveLastUsage: s.haveLastUsage, + GoalActive: s.goalActive, + GoalCondition: s.goalCondition, + CompactCount: s.compactCount, + LastCompactedAt: s.lastCompactedAt.UTC().String(), + Queue: s.promptQueue, + QueueNextID: s.promptQueueNextID, + EnqueueSeq: s.enqueueSeq, + ToolResults: s.toolResults, + ToolResultNext: s.toolResultNextID, + ToolResultBytes: s.toolResultBytes, + MCPSelected: s.mcpSelected, + SpawnedChildren: s.spawnedChildIDs, + Notifications: s.taskNotifications, + TurnUnsettled: s.turnUnsettled, + Committed: s.committedOutcome, + ClaudeCodeCLISessionID: s.claudeCodeCLISessionID, + ClaudeCodeHistoryWatermark: s.claudeCodeHistoryWatermark, + ClaudeCodeSessionCostUSD: s.claudeCodeSessionCostUSD, + HaveClaudeCodeCost: s.haveClaudeCodeCost, + CreatedAt: s.createdAt.UTC().String(), + WorkDir: s.cfg.WorkDir, + ParentSession: s.cfg.ParentSession, + TaskParentID: s.cfg.TaskParentID, + TaskAgentType: s.cfg.TaskAgentType, + TaskDepth: s.cfg.TaskDepth, } b, err := json.Marshal(st) if err != nil { @@ -650,3 +658,118 @@ func TestSnapshotAnchorSurvivesTornTail(t *testing.T) { t.Errorf("history = %d messages, want %d", got, want) } } + +// TestSnapshotCarriesClaudeCodeSessionID pins the fix for a snapshot-based +// cold load (hibernate -> wake) losing the Claude Code CLI's own session +// id. Before the fix, sessionSnapshot had no field for +// Session.claudeCodeCLISessionID or claudeCodeHistoryWatermark, both of +// which are set ONLY by a journal-replay fold (recClaudeCodeSessionID/ +// recClaudeCodeHistoryWatermark, store.go) — a fold the snapshot path +// skips for every record at or before its anchor. snapshotOnIdle fires +// right after each delegated turn, right after those records are written, +// so the anchor almost always covers them: the common case, not an edge +// case. The result was a snapshot-loaded session with an empty CLI +// session id, so its next delegated turn dropped --resume and started a +// fresh CLI session (paying a needless get_conversation_history replay +// too, since the watermark also reset to 0). +func TestSnapshotCarriesClaudeCodeSessionID(t *testing.T) { + s, logPath := claudeCodeTestSession(t, "normal") + s.cfg.SnapshotEveryRecords = idleOnly + + if _, err := s.Prompt(context.Background(), "first turn"); err != nil { + t.Fatalf("first Prompt: %v", err) + } + if err := s.PersistErr(); err != nil { + t.Fatalf("PersistErr = %v", err) + } + + wantSessionID := s.claudeCodeSessionID() + if wantSessionID == "" { + t.Fatal("claudeCodeSessionID() empty after first delegated turn") + } + wantWatermark := s.claudeCodeHistoryWatermarkCount() + if wantWatermark == 0 { + t.Fatal("claudeCodeHistoryWatermarkCount() zero after first delegated turn") + } + + // Snapshot at the current journal head. With the default on-idle + // cadence, this anchor covers the recClaudeCodeSessionID/ + // recClaudeCodeHistoryWatermark records this turn just wrote — + // exactly the common case snapshotOnIdle hits right after every + // delegated turn (engine.go's own defer). + s.snapshotOnIdle() + s.waitSnapshots() + + reloaded, err := LoadSession(Config{ + SessionDir: s.cfg.SessionDir, + ClaudeCode: s.cfg.ClaudeCode, + }, s.ID) + if err != nil { + t.Fatalf("LoadSession: %v", err) + } + if reloaded.replayedRecords >= reloaded.recordsWritten { + t.Fatalf("replayed %d of %d records — the snapshot was not used, this test proves nothing", + reloaded.replayedRecords, reloaded.recordsWritten) + } + + if got := reloaded.claudeCodeSessionID(); got != wantSessionID { + t.Errorf("reloaded claudeCodeSessionID() = %q, want %q (snapshot must carry the CLI session id, not rely on a skipped tail replay)", got, wantSessionID) + } + if got := reloaded.claudeCodeHistoryWatermarkCount(); got != wantWatermark { + t.Errorf("reloaded claudeCodeHistoryWatermarkCount() = %d, want %d", got, wantWatermark) + } + + // The next delegated turn must --resume the CLI session the snapshot + // restored, not silently start a fresh one. + if _, err := reloaded.Prompt(context.Background(), "second turn"); err != nil { + t.Fatalf("second Prompt (post-reload): %v", err) + } + invocations := readInvocations(t, logPath) + if len(invocations) != 2 { + t.Fatalf("invocations = %d, want 2: %+v", len(invocations), invocations) + } + resumeID, ok := argvValueAfter(invocations[1], "--resume") + if !ok || resumeID == "" { + t.Fatalf("post-reload invocation argv has no non-empty --resume: %v", invocations[1]) + } + if resumeID != wantSessionID { + t.Errorf("--resume value = %q, want %q", resumeID, wantSessionID) + } +} + +// TestSnapshotCarriesClaudeCodeCost pins the fix for the sibling bug in the +// same class: sessionSnapshot had no field for +// Session.claudeCodeSessionCostUSD/haveClaudeCodeCost, both of which are +// set ONLY by the recClaudeCodeUsage fold (store.go) — a fold a +// snapshot-anchored load skips for every record at or before its anchor, +// exactly like the CLI session id bug above. The result was a +// snapshot-loaded session reporting $0/unset cumulative claude-code cost +// even when delegated turns actually ran before the snapshot anchor. +// +// This drives captureSnapshotLocked/restoreSnapshot directly, rather than +// through a full LoadSession, so it isolates the snapshot round trip from +// the journal fold entirely: "restore into a fresh session from ONLY the +// snapshot" is exactly what a snapshot-anchored load does for any record +// at or before the anchor. +func TestSnapshotCarriesClaudeCodeCost(t *testing.T) { + s := NewSession(Config{}) + s.mu.Lock() + s.claudeCodeSessionCostUSD = 1.2345 + s.haveClaudeCodeCost = true + snap := s.captureSnapshotLocked() + s.mu.Unlock() + + fresh := NewSession(Config{}) + fresh.restoreSnapshot(snap) + + fresh.mu.Lock() + gotCost, gotHave := fresh.claudeCodeSessionCostUSD, fresh.haveClaudeCodeCost + fresh.mu.Unlock() + + if !gotHave { + t.Error("haveClaudeCodeCost = false after restoreSnapshot, want true") + } + if gotCost != 1.2345 { + t.Errorf("claudeCodeSessionCostUSD = %v after restoreSnapshot, want 1.2345", gotCost) + } +} diff --git a/engine/store.go b/engine/store.go index a76a23c7..3797884d 100644 --- a/engine/store.go +++ b/engine/store.go @@ -1486,6 +1486,17 @@ func LoadSession(cfg Config, id string) (*Session, error) { // two can never drift on the torn-write and ID-burn rules it holds. qf := promptQueueFold{queue: s.promptQueue, nextID: s.promptQueueNextID, seq: s.enqueueSeq} + // apply is the switch every fold below writes into a Session field + // through. A snapshot-anchored load (snapshotStartAfter above) SKIPS + // this switch for every record at or before the anchor, so any Session + // field only ever set here (never restored elsewhere) must also be + // captured/restored by snapshot.go's captureSnapshotLocked/ + // restoreSnapshot — see sessionSnapshot's own doc comment and + // TestEverySessionFieldIsClassifiedForSnapshotting + // (engine/snapshot_field_coverage_test.go), the fail-closed guard that + // requires every Session field to be classified as snapshotted or + // deliberately excluded, so a new case added here cannot silently ship + // without that decision being made. apply := func(rec record, line int, isLast bool) error { // Seed the session's metadata-index fold from the same records // (index.go). A resumed session keeps writing that index through From f633386fc9c6950264dac7b9b91036a972adcdaf Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 17:17:30 -0400 Subject: [PATCH 54/95] docs(agents): scope mandatory TDD and set the comment default (#244) * docs(agents): scope mandatory TDD and set the comment default The Testing section required a failing test for every behavior change and set no limit, so a change with no assertion available except a copy of its own implementation still had to grow one. Such a test states that the change was made, not that it works, and the next refactor breaks it. The rule now starts where the value is: name the failure, then write the test that catches it. That filter is general. It rejects an assertion that copies a constant into its own oracle and a mock that asserts its own calls for the same reason -- neither can fail for a reason a reader can state. The list under it names the check that replaces the test for each kind of change, so the exemption cannot read as "skip the work". The red step keeps one more job here, now stated: it proves that the agent did the work that it reports. The Writing style section ruled repository prose only, so it never said what earns a code comment. The rule sets the default at no comment, points at the two cheaper fixes first -- a clearer name and a smaller function -- and names the failure mode that a reader can spot: "no longer", "now", and "instead of" mark a comment that is really a commit message. Prose only. It moves no existing rule and changes no command. * docs(agents): name the failing test instead of the red step Two review findings. "The red step" is TDD jargon in a rule sentence; the sentence before it already names the failing test, so it now says "that failing test". The existing "Red-verify" bullet keeps its own term. "Default to no comment" sits in the Writing style section, where a comment could read as a review comment. It now says "no code comment". --------- Co-authored-by: andybons --- AGENTS.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index e7898741..22bfff8c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,8 +113,22 @@ a repository-wide or concurrency-sensitive change. ## Testing +Name the failure before you write the test. State the input, the state, and the +wrong output. A test that cannot fail for one named reason has no value. + For behavior-changing code, add and confirm the failing test first. Then implement -the change. For prose-only changes, validate links, formatting, and loaders. +the change. That failing test also proves that the agent did the work. For prose-only +changes, validate links, formatting, and loaders. + +Do not write a test that can only restate the implementation. Use the check that +fits the change instead. + +- Wiring and composition: the build and a startup path cover them. +- A rule that the type system enforces: the compiler is the test. +- A thin adapter over an external system: test the contract with a fake. +- An exploratory design: spike, delete the spike, then test what you keep. + +Keep a test that pins a named, reported regression. - Run Go tests with `-race`. - Red-verify each regression test against the exact mechanism it names. @@ -175,6 +189,16 @@ Use active voice, common words, and one stable term for each concept. Keep instructions at 20 words or fewer when practical. Name code elements instead of using vague references. Quote identifiers and error strings exactly. +Default to no code comment. Prefer a clearer name or a smaller function over an +explanation. Comment only what the code cannot show: a constraint, a hazard, a +rejected alternative, or a non-obvious reason. Keep it to the shortest form that +carries the reason. + +Do not write a comment that describes the change that you made. `git blame` and +the commit body hold that history, and the comment goes stale at the next edit. +A comment that says "no longer", "now", or "instead of" belongs in the commit +message. + Use standard Go style. Run `gofmt` and `go vet`. Prefer explicit exported types and small interfaces. From af098c09988e7d1c620b2f9cb3894179fe2c0e84 Mon Sep 17 00:00:00 2001 From: Andy Bonventre Date: Wed, 2 Sep 2026 17:46:05 -0400 Subject: [PATCH 55/95] chore(monitor): remove the embedded session monitor (#246) The single-box monitor UI has no consumer. Boxes, the one product that runs `harness serve` at scale, never calls it: every "monitor" reference in that repository is a comment citing this package as a design precedent for its own embedded console, not a call, proxy, or link. Removing it deletes 8333 lines and one unauthenticated browser surface. `GET /monitor`, `GET /monitor/`, and the bare-root redirect are gone, so a `harness serve` host with no path now 404s like any other unmatched path. `server.Options.MonitorPage` is gone with them, which removes the server's only reason to accept a page of HTML it never parsed. The `serve start` log line no longer advertises a monitor_url, and the tty-gated hint that printed a tokenized monitor link is gone with its `stderrIsTerminal` helper -- one fewer path that writes a run token to a terminal. server_test.go keeps the contract as TestNoMonitorOrRootRoute: /monitor, /monitor/, and / must 404. The root is worth pinning on its own, because the removed route was anchored `GET /{$}` precisely so it could not become a catch-all that swallows every unmatched path's 404. docs/design/monitor-mockup.html goes too: it was the visual specification for the deleted page and nothing else referenced it. docs/plans stays -- a dated plan is a record of what was decided then, not a live reference. Verification: `go build ./...`, `go vet ./...`, and `go test -race ./...` all pass. The flaky tools/monitor/e2e suite leaves with the package. CI keeps its inspector and hub node steps. --- .github/workflows/ci.yml | 8 - cmd/harness/AGENTS.md | 7 +- cmd/harness/main.go | 100 +- cmd/harness/main_test.go | 29 - docs/README.md | 3 +- docs/design/monitor-mockup.html | 418 ---- docs/development-interfaces.md | 127 +- server/AGENTS.md | 15 +- server/handlers.go | 68 - server/openapi.yaml | 52 - server/server.go | 30 +- server/server_test.go | 101 +- tools/AGENTS.md | 35 +- tools/monitor/e2e/.gitignore | 1 - tools/monitor/e2e/README.md | 90 - tools/monitor/e2e/e2e_test.go | 151 -- tools/monitor/e2e/package-lock.json | 516 ----- tools/monitor/e2e/package.json | 12 - tools/monitor/e2e/real_e2e.mjs | 951 -------- tools/monitor/e2e/stub.go | 780 ------- tools/monitor/embed.go | 25 - tools/monitor/index.html | 3169 --------------------------- tools/monitor/monitor_test.mjs | 1679 -------------- 23 files changed, 18 insertions(+), 8349 deletions(-) delete mode 100644 docs/design/monitor-mockup.html delete mode 100644 tools/monitor/e2e/.gitignore delete mode 100644 tools/monitor/e2e/README.md delete mode 100644 tools/monitor/e2e/e2e_test.go delete mode 100644 tools/monitor/e2e/package-lock.json delete mode 100644 tools/monitor/e2e/package.json delete mode 100644 tools/monitor/e2e/real_e2e.mjs delete mode 100644 tools/monitor/e2e/stub.go delete mode 100644 tools/monitor/embed.go delete mode 100644 tools/monitor/index.html delete mode 100644 tools/monitor/monitor_test.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4858e393..38dd64d1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,14 +47,6 @@ jobs: # test file, which the runner then runs directly. run: node --test tools/inspector/*_test.mjs - - name: Monitor unit tests - # tools/monitor is the same pattern as the inspector above: a single - # build-free HTML file whose pure helpers (SSE parser, activity - # reducer, transcript fold, route codec, formatters) are extracted - # from a TESTABLE region and unit-tested with node's built-in test - # runner. Same glob-not-directory rationale as the inspector step. - run: node --test tools/monitor/*_test.mjs - - name: Hub unit tests # tools/hub's pure logic has been unit-tested by hub_test.mjs since # its introduction (AGENTS.md's "Development hub" section documents diff --git a/cmd/harness/AGENTS.md b/cmd/harness/AGENTS.md index 95205216..aecaa4e7 100644 --- a/cmd/harness/AGENTS.md +++ b/cmd/harness/AGENTS.md @@ -49,18 +49,15 @@ Responses clients so opaque data cannot cross endpoints. Do not validate credentials during registry construction. The first provider request owns credential validation. -## Monitor and hub composition +## Hub composition -`cmd/harness` may import `tools/hub` and `tools/monitor`. The server may +`cmd/harness` may import `tools/hub`. The server may not. Only allow empty-token unauthenticated service when `resolveUnauthenticated` proves loopback or receives the explicit non-loopback opt-in. Keep the two warning messages distinct. -Print tokenized monitor URLs only to an interactive terminal. Do not write a -tokenized URL to piped production logs. - ## GC and pprof diagnostics Use `runtime/metrics` for GC pause observation. Do not use diff --git a/cmd/harness/main.go b/cmd/harness/main.go index eba5c7e1..eefe7d4c 100644 --- a/cmd/harness/main.go +++ b/cmd/harness/main.go @@ -37,7 +37,6 @@ import ( "github.com/majorcontext/harness/provider/openaicompat" "github.com/majorcontext/harness/server" "github.com/majorcontext/harness/tools/hub" - "github.com/majorcontext/harness/tools/monitor" ) // defaultOpenRouterName is the providers map key that gets a built-in @@ -1333,69 +1332,6 @@ func serveURLForAddr(addr string) string { return "http://" + net.JoinHostPort(host, port) } -// stderrIsTerminal reports whether os.Stderr is an interactive terminal -// (isatty), stdlib-only: os.ModeCharDevice on the file mode is the -// established Go idiom for this check (no golang.org/x/term or other -// dependency — this repo's zero-dep rule for production code). Used solely -// to gate serveCmd's tokenized monitor URL print (monitorTerminalHint) -// below: piped/redirected/production stderr (a file, a pipe into a log -// collector, /dev/null) is never a character device, so this is false -// there and true only for a human's own terminal session. -// -// NOT unit-tested: there is no PTY available in a plain `go test` process, -// and pulling in a PTY library only to exercise this one syscall wrapper -// would violate the zero-dependency rule for a trivial, well-established -// stdlib idiom. monitorTerminalHint below is factored out specifically so -// everything ELSE about the print (the gating logic, the exact format) IS -// unit-tested with an explicit bool in place of this function's real -// result — this is the one piece left manually verified: run -// `HARNESS_RUN_TOKEN=x harness serve` from an actual terminal and confirm -// the "monitor: ...#t=..." line appears; run it with stderr piped (e.g. -// `2>&1 | cat`) and confirm it does not. -func stderrIsTerminal() bool { - info, err := os.Stderr.Stat() - if err != nil { - return false - } - return info.Mode()&os.ModeCharDevice != 0 -} - -// monitorTerminalHint writes the tty-gated, click-ready monitor URL line to -// w when BOTH monitorEnabled (this box actually has MonitorPage configured) -// and isTTY (see stderrIsTerminal's doc comment for why that half is not -// itself unit-tested here) are true; a no-op otherwise. Two shapes, -// depending on token: -// - token != "" (the normal, authenticated case): "monitor: -// http://host:port/monitor#t=" — a capability URL index.html's -// own extractFragmentToken adopts on load with no manual typing. -// - token == "" (the loopback-Unauthenticated case — see server.Options. -// Unauthenticated): plain "monitor: http://host:port/monitor", no -// "#t=" at all, since there is no token to carry and appending an -// empty "#t=" would be misleading (it would round-trip through -// extractFragmentToken as "no token", but LOOKS like a credential is -// present). -// -// A tokenized URL is a credential riding the URL: gating it out of every -// non-interactive destination (piped/redirected/production stderr) is what -// keeps it off a log aggregator or a captured file — see the call site's -// own comment for the full reasoning (the loopback-Unauthenticated case has -// no credential to leak, but stays behind the SAME tty gate for one -// uniform rule rather than a special case an operator has to remember). -// Factored out of serveCmd so the DECISION (what to print, and the exact -// format) is unit-testable with a bytes.Buffer and explicit bools, -// independent of the real os.Stderr/os.Stderr.Stat() this function never -// touches itself. -func monitorTerminalHint(w io.Writer, monitorEnabled, isTTY bool, addr, token string) { - if !monitorEnabled || !isTTY { - return - } - url := serveURLForAddr(addr) + "/monitor" - if token != "" { - url += "#t=" + token - } - fmt.Fprintf(w, "monitor: %s\n", url) -} - // serveCmd starts the HTTP+SSE session API. The run token comes from // HARNESS_RUN_TOKEN, required on any bind unless the operator explicitly // opts out via -unauthenticated/HARNESS_UNAUTHENTICATED (non-loopback) or the @@ -1457,7 +1393,7 @@ func serveCmd(args []string) error { if unauthenticated { if isLoopbackAddr(addr) { // A clear, impossible-to-miss line: this process is about to - // serve its full API (not just /health/monitor) with no bearer + // serve its full API (not just /health) with no bearer // token check at all. Loopback-only makes this safe (see // isLoopbackAddr's doc comment), but it is still a deviation // from this binary's normal secure-by-default behavior, worth a @@ -1723,12 +1659,6 @@ func serveCmd(args []string) error { }(), } } - // monitorPage is a named local (rather than inlining monitor.Page below) - // so serveCmd's tty-gated tokenized-URL print further down can gate on - // the SAME "is the monitor actually enabled" condition the Options - // literal itself uses, without repeating the tools/monitor.Page - // reference or risking the two silently drifting apart. - monitorPage := monitor.Page srv, err = server.New(server.Options{ SessionDir: sesDir, RunToken: token, @@ -1754,13 +1684,6 @@ func serveCmd(args []string) error { // Session to ask, so the process's plugin list is supplied here // directly — see server.Options.Plugins. Plugins: pluginInfoFn(pluginHost), - // MonitorPage: every `harness serve` box offers its own same-origin - // monitor at GET /monitor — no CORS/-cors-origin dance, no - // separately hosted copy required (see tools/AGENTS.md's "Session - // monitor" section). tools/monitor.Page embeds the exact committed - // tools/monitor/index.html; the static/file:// hosting path it - // documents keeps working unchanged alongside this. - MonitorPage: monitorPage, // Unauthenticated: set only in case (b) above (empty token + // loopback bind) — see server.Options.Unauthenticated's own doc // comment for why this is the ONE place that ever sets it (server @@ -1797,26 +1720,7 @@ func serveCmd(args []string) error { errc := make(chan error, 1) go func() { errc <- httpSrv.ListenAndServe() }() - // monitor_url logs the same host:port serveURLForAddr already resolves - // -addr against (e.g. 0.0.0.0 -> 127.0.0.1) for the plugin-host URL - // above, so the two log lines never disagree about how to reach this - // same process. - logger.Info("serve start", "addr", addr, "version", version, "monitor_url", serveURLForAddr(addr)+"/monitor") - // A second, CLICK-READY monitor URL (monitorTerminalHint — see its own - // doc comment for the two shapes: tokenized via index.html's #t= - // capability-URL adoption, or plain when this process is running - // loopback-Unauthenticated) is convenient — no typing a run token, or - // no token needed at all — but a tokenized one IS a credential riding - // the URL: printing it into the structured "serve start" log line - // above would ship it into whatever piped/redirected destination this - // process's stderr normally lands in (a log aggregator, a captured - // file, a terminal multiplexer's scrollback in a shared session) — a - // credential leak, not a convenience, once it's off an operator's own - // screen. Gating this SEPARATE, plain (non-JSON) line on - // stderrIsTerminal confines it to an actual interactive terminal: - // piped/production stderr gets nothing extra here, only the tokenless - // monitor_url already logged above. - monitorTerminalHint(os.Stderr, monitorPage != nil, stderrIsTerminal(), addr, token) + logger.Info("serve start", "addr", addr, "version", version) select { case err := <-errc: diff --git a/cmd/harness/main_test.go b/cmd/harness/main_test.go index 4fe21af9..cbf7cad4 100644 --- a/cmd/harness/main_test.go +++ b/cmd/harness/main_test.go @@ -142,35 +142,6 @@ func TestEnvUnauthenticated(t *testing.T) { } } -// TestMonitorTerminalHint covers monitorTerminalHint's full decision table -// (the untestable half — the real os.Stderr.Stat() tty check — is -// deliberately excluded via the explicit isTTY bool; see stderrIsTerminal's -// own doc comment for why that piece is manually verified instead). -func TestMonitorTerminalHint(t *testing.T) { - cases := []struct { - name string - monitorEnabled bool - isTTY bool - token string - want string - }{ - {"tty, monitor enabled, token set: tokenized URL", true, true, "secret-tok", "monitor: http://localhost:4096/monitor#t=secret-tok\n"}, - {"tty, monitor enabled, NO token (loopback-Unauthenticated): plain URL, no #t= at all", true, true, "", "monitor: http://localhost:4096/monitor\n"}, - {"not a tty: nothing printed, even with a token", true, false, "secret-tok", ""}, - {"monitor not enabled: nothing printed, even on a tty with a token", false, true, "secret-tok", ""}, - {"neither tty nor monitor enabled: nothing printed", false, false, "secret-tok", ""}, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - var buf strings.Builder - monitorTerminalHint(&buf, tc.monitorEnabled, tc.isTTY, "localhost:4096", tc.token) - if got := buf.String(); got != tc.want { - t.Errorf("monitorTerminalHint(...) wrote %q, want %q", got, tc.want) - } - }) - } -} - func TestSessionDir(t *testing.T) { t.Run("no-save disables persistence", func(t *testing.T) { t.Setenv("HARNESS_SESSION_DIR", "/somewhere") diff --git a/docs/README.md b/docs/README.md index ed4837a7..6046fd06 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,7 @@ remain authoritative when historical material describes an earlier behavior. | [models-and-providers.md](models-and-providers.md) | Model state, effort, cache affinity, and adapters | | [mcp-tool-loading.md](mcp-tool-loading.md) | Deferred MCP schemas and stable tool ordering | | [plugins-and-protocols.md](plugins-and-protocols.md) | Plugin lifecycle and external protocol boundaries | -| [development-interfaces.md](development-interfaces.md) | Hub and monitor behavior | +| [development-interfaces.md](development-interfaces.md) | Hub behavior | | [fleet-and-serve.md](fleet-and-serve.md) | Fleet state, lineage, exhaustion, and diagnostics | | [deploy-modal.md](deploy-modal.md) | Deployment modal behavior | @@ -36,4 +36,3 @@ documents above and keep superseded chronology in `history/` or `plans/`. | [managed-processes.md](design/managed-processes.md) | Box-scoped managed process lifecycle | | [mcp-lazy-tools.md](design/mcp-lazy-tools.md) | Deferred MCP schema design | | [nested-instruction-loading.md](design/nested-instruction-loading.md) | Project instruction discovery and truncation | -| [monitor-mockup.html](design/monitor-mockup.html) | Monitor visual reference | diff --git a/docs/design/monitor-mockup.html b/docs/design/monitor-mockup.html deleted file mode 100644 index 127457e3..00000000 --- a/docs/design/monitor-mockup.html +++ /dev/null @@ -1,418 +0,0 @@ -Harness Monitor — Design Mockup - - -
-

Design mockup, sample data. Click the first session row to see the detail view; elapsed times tick; theme follows your OS.

- - - -
- -
- ses_01ky9fjq2wexvq8rn0m4tdq2m - - 142 msgs - turn 2m 34s - 17 tool calls - - tool running -
- -
- -
turn 41 · 6m ago
- -
- operator -

run the race suite on the server package and fix anything flaky before we tag

-
- -
- assistant -
-

Running the race-enabled suite first to get a baseline — if anything is flaky it usually shows within a few hundred iterations.

-
-
- -
- tool -
-
- -
Bash · gofmt -l . show output0.4s
- -
(no output — formatting clean)
-
-
-
- -
current turn · 2m 34s
- -
- assistant -

Formatting is clean. Starting the race suite — this takes about 45 seconds on this box.

-
- -
- tool -
-
- -
Bash · go test ./server/ -race running · 41s
- -
=== RUN TestQueuedPromptDispatchesOnDrain ---- PASS: TestQueuedPromptDispatchesOnDrain (0.02s) -=== RUN TestEnqueueBusyQueuesAndDeduplicates ---- PASS: TestEnqueueBusyQueuesAndDeduplicates (0.01s) -=== RUN TestClaimForPromptSurvivesEvictionRace
-
-
-
- -
- -
-
- - -
-

Enter to send · Shift+Enter for a new line

-
-
-
- - diff --git a/docs/development-interfaces.md b/docs/development-interfaces.md index 3bb7cc5e..920cc7e9 100644 --- a/docs/development-interfaces.md +++ b/docs/development-interfaces.md @@ -1,6 +1,6 @@ # Development interfaces -This document describes the local hub and session monitor. +This document describes the local development hub. ## Development hub @@ -100,128 +100,3 @@ which still wears the older soft theme — follows these rules: - **Selectors are load-bearing**: the renderers create elements by class name (`.sess`, `.box-card`, `.dot`, `.goalnarr`, …). Restyle classes; never rename them in a styling pass. - -## Session monitor - -`tools/monitor` (`tools/monitor/index.html`) is the single-instance -counterpart to the hub above: where the hub answers "what are my boxes doing" -across a FLEET, the monitor answers "what is THIS `harness serve` instance -doing right now" — a live board of every session on that one box (phase, -current tool, staleness), a per-session detail view with a scrolling -transcript, and a composer to speak into a session (`prompt_async`). Like the -hub and the inspector, it is a build-free, dependency-free single HTML file -with no Go-side handler of its own. - -- **How to run it**: open the file directly (`file://`) or serve it from any - static host — nothing box-specific is baked in. The target box must be - started with `-cors-origin` covering the monitor's origin (or `*` for local - hacking), exactly like the hub's requirement above; a box without it looks - permanently unreachable. The base URL and run token are entered in the - page itself and persisted to `localStorage` in plaintext (same documented - tradeoff as the inspector — a dev tool, not for a shared origin with a - long-lived token) so a reload can reconnect automatically. Routing lives in - the URL fragment as small explicit params, not the hub's base64 blob: - `#b=` (box base URL) and `#s=` (open detail view) — both - bookmarkable, encoded/decoded by a tested pure helper. -- **Embedded serving, frictionless local**: every `harness serve` box also - offers its own copy same-origin, at `GET /monitor` — by default - `http://localhost:4096/monitor` (the port follows `-addr`, default - `localhost:4096`; the exact URL, with any `#t=` capability suffix, is - printed to the terminal on startup — `monitorTerminalHint`). The bare root - is a convenience redirect: `GET /` 302s to `/monitor` (via the `GET /{$}` - route — `{$}`-anchored to the root path only, never a catch-all — registered - under the same `MonitorPage` guard, so a pure-API box keeps `/` a clean 404). - `tools/monitor` - (package `monitor`, `embed.go`) `//go:embed`s the exact committed - `index.html`; `cmd/harness`'s `serveCmd` wires it into - `server.Options.MonitorPage`, which the server serves unauthenticated - (like `/health` — the page itself carries no secrets) with a - same-origin-scoped `Content-Security-Policy` (`connect-src 'self'`). - `server` itself never imports `tools/monitor` (layering: `server` must - not import `tools/*`); only `cmd/harness` does, the same pattern - `harness hub` already uses. The `file://`/static-host path is unaffected - — this is additive, and stays the only option for monitoring a box from a - different origin (the embedded route's CSP deliberately does not permit - that). - - **Unauthenticated-on-loopback** (`server.Options.Unauthenticated`, an - EXPLICIT opt-in never inferred from an empty `RunToken` on its own — - `New` still fails closed otherwise): `serveCmd` classifies `-addr` - (`isLoopbackAddr` — `localhost`, `127.0.0.1`/`::1`, any - `net.IP.IsLoopback()` address; a bare `:port`, `0.0.0.0`, `::`, or any - other routable address is NOT loopback). `HARNESS_RUN_TOKEN` unset + - loopback bind runs the box fully unauthenticated (every route, not just - `/health`/`/monitor`) and logs a clear warning; unset + non-loopback - still hard-errors `HARNESS_RUN_TOKEN is required` exactly as before. The - token guards network reachability, and loopback is a server-verifiable - proxy for that (unlike, say, `Origin`, which a client controls). - - **Unauthenticated on a non-loopback bind is also possible, but ONLY via - a SECOND, separate, EXPLICIT opt-in** — the `-unauthenticated` serve - flag, or `HARNESS_UNAUTHENTICATED=1` (`envUnauthenticated`, parsed with - `strconv.ParseBool`; an unset or unparsable value is false, fail-closed - on a malformed setting). `resolveUnauthenticated` (`cmd/harness/main.go`) - is the single decision point both serve flags feed: a non-empty token - always wins (token path unaffected, any bind); an empty token on a - loopback bind is unchanged from above; an empty token on a non-loopback - bind needs this opt-in or it still hard-errors exactly as before — the - opt-in is never inferred from the empty token alone, only from the flag - or env var. This is for a deployment where a trusted external gate - already restricts reachability (e.g. a Cloudflare Access-gated tunnel, - or a sandboxed network boundary) so the token is redundant with that - gate — `server.Options.Unauthenticated` itself is bind-address-agnostic - (see its own doc comment); the safety property lives entirely in - `cmd/harness` deciding WHEN to set it. Landing this on a non-loopback - bind logs a SEPARATE, distinctly worded loud warning ("serving - unauthenticated on a non-loopback bind") from the loopback one above, so - the two are distinguishable in a log search. - - **Same-origin auto-connect** (`embeddedConnectPlan`, index.html): opening - a box's own `/monitor` attempts the connection immediately against - `location.origin`, using whatever token is available (`#t=` fragment, - then a stored one, then none) — success lands straight on the board, no - panel, nothing typed (covers both the unauthenticated-loopback case and - a valid capability URL/returning operator with the SAME call). Only a - failed attempt (a real token is required and none was available) falls - back to a minimal token-only panel — host is already known, so the base - field never reappears. A `#t=` fragment (mirroring `tools/hub`'s - "run tokens ride the URL by design" precedent, `extractFragmentToken`) - is adopted into the SAME `localStorage` key manual entry uses and - immediately scrubbed from the visible URL via `history.replaceState`; a - fragment `#b=` naming a DIFFERENT origin than this box's own — which the - embedded route's CSP would block outright if tried — surfaces a - plain-text notice instead of attempting it, composing with (never - blocking) the own-origin auto-connect. None of this weakens the auth - model itself: a token is still required and still checked exactly as a - hand-typed one would be, on every box that hasn't explicitly opted into - `Unauthenticated`. - - On a TTY, `serveCmd` also prints a click-ready line to stderr after - "serve start" — `monitor: http:///monitor#t=` (or, when - running unauthenticated-loopback, the plain URL with no `#t=` at all, - since there's no credential to carry) — gated on `stderrIsTerminal` - (`os.ModeCharDevice`, stdlib-only) so a tokenized URL never lands in - piped/production stderr, only an interactive operator's own terminal. -- **Test layers.** Pure helpers (SSE parser, activity reducer, transcript - fold, route codec, formatters) live inside index.html's `/* TESTABLE-BEGIN - */ … /* TESTABLE-END */` region and are covered by - `tools/monitor/monitor_test.mjs` (run: `node --test tools/monitor/*_test.mjs`), - using the same extraction-and-`vm`-evaluate pattern as the inspector's - `inspector_test.mjs` (and now the hub's, above) — no build step, so the - tests read the region straight out of the committed HTML. End-to-end, - against a real backend, is `tools/monitor/e2e` (see its README): a `go - test` subtree that starts a real `server.Server` plus a plain static file - server for the actual committed `index.html`, and drives it with Node + - jsdom and an unmocked `fetch` — mirroring `tools/hub/e2e`'s structure and - conventions. A `window.__monitorTuning = {QUIET_MS, STALL_MS}` seam (set - via jsdom's `beforeParse` before the page's own script runs, a no-op in - production since nothing else ever sets it) lets both the unit and e2e - suites shrink the staleness thresholds so `quiet`/`stalled` transitions are - observable in test time instead of real minutes. -- **UI design language.** The monitor deliberately does NOT inherit the - hub's committed dark brutalist archetype above. It carries its own - "instrument sheet" language: light-first with a dark variant, both driven - by one OKLCH token set (`--surface`, `--text-1..3`, `--separator`, etc.); - semantic green/amber/red (`--ok`/`--warn`/`--critical`) are reserved - strictly for session/staleness state, never decoration; a single accent - color is owned by interaction (the composer's send affordance is the - page's only filled-accent control). `docs/design/monitor-mockup.html` is - the user-approved visual spec — its tokens, grid template, and markup - shapes are binding; restyle within that spec rather than importing the - hub's theme onto it. diff --git a/server/AGENTS.md b/server/AGENTS.md index d5bcf611..d46b88ed 100644 --- a/server/AGENTS.md +++ b/server/AGENTS.md @@ -99,19 +99,8 @@ spawn a replacement. Fail closed unless the CLI explicitly selects an allowed unauthenticated mode. Do not infer unauthenticated service from an empty token inside `server.New`. -Keep monitor HTML unauthenticated because it contains no secret. Keep API -routes under the normal auth policy. Apply CORS only from configured origins. - -## Session monitor - -`GET /monitor` serves bytes supplied through `Options.MonitorPage`. -`GET /{$}` redirects only when that page exists. Do not add a catch-all -route. - -Keep the embedded page's CSP same-origin. Cross-origin monitoring uses a -separately hosted page. - -Read `tools/AGENTS.md` before changing monitor behavior. +Keep `/health` unauthenticated. Keep every other route under the normal +auth policy. Apply CORS only from configured origins. ## Serve-mode latency diagnostics diff --git a/server/handlers.go b/server/handlers.go index 1d0d6838..a2d15715 100644 --- a/server/handlers.go +++ b/server/handlers.go @@ -612,74 +612,6 @@ func (s *Server) handleHealth(w http.ResponseWriter, _ *http.Request) { }) } -// monitorContentSecurityPolicy hardens the embedded monitor page, mirroring -// tools/hub/hub.go's contentSecurityPolicy (same reasoning: a single inline -// file with no build step, so a per-response nonce/hash is not viable — -// 'unsafe-inline' is the only way to permit its own inline script/style) -// adapted to what THIS page actually needs when served from the box it -// monitors: connect-src 'self', not '*'. The hub's page is deliberately -// origin-agnostic (it drives arbitrary, operator-added box origins it keeps -// no state about); the embedded monitor is the opposite — GET /monitor -// exists specifically so a box can offer a same-origin, zero-CORS way to -// watch ITSELF (see docs/development-interfaces.md's "Session monitor" -// section), so this CSP -// scopes fetch/EventSource targets to that one origin, blocking it from -// being used to reach anywhere else even if the served copy were somehow -// pointed at a different base URL. An operator who genuinely wants -// cross-origin monitoring still has the unrestricted, unauthenticated -// file://-or-any-static-host path index.html's own header comment -// documents — this route is additive, not a replacement. -const monitorContentSecurityPolicy = "default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; connect-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'" - -// handleMonitor serves the embedded tools/monitor/index.html verbatim at -// GET /monitor (and /monitor/ — see routes()), registered only when -// Options.MonitorPage is non-nil (cmd/harness's serveCmd is the only -// caller that sets it, via tools/monitor.Page). Deliberately UNAUTHENTICATED -// — see MonitorPage's own doc comment for why that is correct here — and, -// unlike every other handler in this file, reached WITHOUT s.auth wrapping -// it (see routes()). GET or HEAD only, matching tools/hub/hub.go's -// handleIndex precedent for its own embedded page. -func (s *Server) handleMonitor(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet && r.Method != http.MethodHead { - w.Header().Set("Allow", "GET, HEAD") - http.Error(w, "method not allowed", http.StatusMethodNotAllowed) - return - } - w.Header().Set("Content-Type", "text/html; charset=utf-8") - w.Header().Set("Content-Security-Policy", monitorContentSecurityPolicy) - // no-cache: the page is embedded at build time, so a box that respawns - // on a newer image serves a newer page — but browsers heuristically - // cache an HTML response with no freshness headers, so an operator who - // revisits the same box URL would keep seeing a stale page across a pin - // bump (or a local dev rebuild) until a hard reload. no-cache forces a - // revalidation on every load; the page itself is a few KB, so the cost - // is negligible and correctness (always the current build) wins. - w.Header().Set("Cache-Control", "no-cache") - w.WriteHeader(http.StatusOK) - if r.Method == http.MethodGet { - w.Write(s.opts.MonitorPage) //nolint:errcheck - } -} - -// handleRoot 302-redirects the bare root path to the canonical monitor URL -// at /monitor, so visiting a box's host with no path lands on the monitor -// instead of a bare 404. Registered only when Options.MonitorPage is non-nil -// (see routes(), which anchors it with the GET /{$} pattern so it matches "/" -// EXACTLY — a plain "GET /" would be a catch-all swallowing every otherwise- -// unmatched path's 404 and redirecting it here instead). A pure-API box that -// never sets MonitorPage keeps / as a clean 404, not a redirect to a route it -// doesn't serve. Unauthenticated and outside s.auth, same as handleMonitor: -// the redirect carries no secret, and a browser preserves any #t= -// fragment across the 302 (the target carries none of its own — see -// tools/monitor's capability-URL handling), so the capability flow still -// works from the bare host. 302 (not 301) keeps / uncacheable as a permanent -// alias: /monitor is the one canonical URL (printed by monitorTerminalHint, -// carried in the capability link), and / stays a convenience we're free to -// repurpose later. -func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) { - http.Redirect(w, r, "/monitor", http.StatusFound) -} - // handleGoroutines writes the full, all-goroutine stack dump (the exact // text Go's default SIGQUIT handler prints) as a diagnostic HTTP surface — // for a box wedged badly enough that even exec is awkward (or unavailable, diff --git a/server/openapi.yaml b/server/openapi.yaml index e372aa09..484c3182 100644 --- a/server/openapi.yaml +++ b/server/openapi.yaml @@ -3162,58 +3162,6 @@ paths: application/json: schema: { $ref: "#/components/schemas/Error" } - /: - get: - operationId: monitorRootRedirect - summary: Convenience redirect from the bare root to the monitor UI. - description: > - Present only when this instance was started with the monitor page - embedded (same `MonitorPage` guard as `/monitor` below); a box that - omits it 404s here exactly as any other unmatched path. 302-redirects - to the canonical `/monitor` so visiting the box's host with no path - lands on the monitor. UNAUTHENTICATED like `/monitor` — the redirect - carries no secret, and a browser preserves any `#t=` fragment - across the 302 (fragments are never sent to the server). Anchored to - the root path ONLY (`GET /{$}`), so it never shadows another route's - 404 — see server/handlers.go's handleRoot. - security: [] - responses: - "302": - description: > - Redirect to /monitor. `Location: /monitor`. - headers: - Location: - schema: { type: string, example: /monitor } - "404": - description: This instance was not started with the monitor page embedded. - - /monitor: - get: - operationId: monitorPage - summary: The embedded, single-file session monitor UI. - description: > - Present only when this instance was started with the monitor page - embedded (cmd/harness's `serve` always does this; see - docs/development-interfaces.md); a box that omits it 404s here - exactly as it always has. Serves tools/monitor/index.html verbatim, - byte for byte — the same static, build-free, credential-free page - that can also be opened via file:// or hosted separately. - UNAUTHENTICATED like /health: the page itself carries no secrets, - and every API call it makes is still authenticated normally by the - browser entering a run token into the page. A - Content-Security-Policy header scopes it to same-origin - (connect-src 'self') — see server/handlers.go's - monitorContentSecurityPolicy. - security: [] - responses: - "200": - description: OK — the monitor's index.html. - content: - text/html: - schema: { type: string } - "404": - description: This instance was not started with the monitor page embedded. - # Candidate for v1.1, deliberately deferred: GET /session/{id}/diff # (working-tree diff for orchestrator PR-preview rendering). Needs a # position on git ownership inside the engine first. diff --git a/server/server.go b/server/server.go index c9b5f7d1..797811aa 100644 --- a/server/server.go +++ b/server/server.go @@ -274,25 +274,8 @@ type Options struct { // per-session. Nil disables the /process endpoints entirely (they // 404), matching a nil engine.Config.Processes. Processes engine.ProcessRegistry - // MonitorPage, when non-nil, is served verbatim at GET /monitor (and - // /monitor/) — the single-file board+detail+composer UI documented in - // docs/development-interfaces.md's "Session monitor" section, letting a box serve its own - // copy same-origin instead of requiring a separately hosted one. Nil - // (the default) registers no route at all: GET /monitor 404s exactly - // as it always has, so an existing deployment that never sets this is - // completely unaffected. Deliberately UNAUTHENTICATED, like /health: - // the page is public, static, credential-free code (same "byte-for- - // byte, no build step" file this package never parses or executes) — - // every actual API call it makes still goes through s.auth like any - // other client, exactly as when the same file is opened via file:// or - // served from an unrelated static host. cmd/harness's serveCmd is the - // only place that sets this (via tools/monitor.Page) — server itself - // never imports tools/monitor, keeping this package's only coupling to - // the page a plain []byte it neither inspects nor depends on the - // shape of. - MonitorPage []byte - // Unauthenticated, when true, serves EVERY route (not just /health and - // MonitorPage) without requiring a bearer token — see authorized(), + // Unauthenticated, when true, serves EVERY route (not just /health) + // without requiring a bearer token — see authorized(), // which returns true unconditionally in this mode. This is a // deliberate, EXPLICIT opt-in, never inferred from RunToken=="" on its // own: an empty RunToken with Unauthenticated left false still fails @@ -993,15 +976,6 @@ func (s *Server) routes() { if s.opts.PProf { registerPProf(mux, s.auth) } - if s.opts.MonitorPage != nil { - mux.HandleFunc("GET /monitor", s.handleMonitor) - mux.HandleFunc("GET /monitor/", s.handleMonitor) - // Bare root -> the monitor. The {$} anchor matches "/" EXACTLY; a - // plain "GET /" would be a catch-all matching every otherwise- - // unmatched path, turning the whole API's 404s into redirects. See - // handleRoot. - mux.HandleFunc("GET /{$}", s.handleRoot) - } s.mux = mux } diff --git a/server/server_test.go b/server/server_test.go index b4ef0302..a68ecfdf 100644 --- a/server/server_test.go +++ b/server/server_test.go @@ -612,98 +612,12 @@ func TestHealthSessionSyncVolumeMode(t *testing.T) { } } -// TestMonitorPageServed covers cmd/harness's normal case (server.Options. -// MonitorPage set — see tools/monitor.Page): GET /monitor and GET /monitor/ -// both serve the configured bytes verbatim, unauthenticated (no Authorization -// header sent, same as /health), with the correct Content-Type and the -// same-origin-scoped Content-Security-Policy header (monitorContentSecurityPolicy). -func TestMonitorPageServed(t *testing.T) { - dir := t.TempDir() - const page = "fake monitor page" - srv := newServer(t, dir, &scriptedProvider{name: "test"}, 0, func(o *Options) { - o.MonitorPage = []byte(page) - }) - ts := httptest.NewServer(srv) - t.Cleanup(ts.Close) - - for _, path := range []string{"/monitor", "/monitor/"} { - req, _ := http.NewRequest("GET", ts.URL+path, nil) // no auth header - resp, err := ts.Client().Do(req) - if err != nil { - t.Fatal(err) - } - body, _ := io.ReadAll(resp.Body) - resp.Body.Close() - if resp.StatusCode != http.StatusOK { - t.Fatalf("GET %s status = %d, want 200 (unauthenticated, like /health)", path, resp.StatusCode) - } - if string(body) != page { - t.Errorf("GET %s body = %q, want the configured MonitorPage verbatim: %q", path, body, page) - } - if ct := resp.Header.Get("Content-Type"); ct != "text/html; charset=utf-8" { - t.Errorf("GET %s Content-Type = %q", path, ct) - } - if csp := resp.Header.Get("Content-Security-Policy"); csp != monitorContentSecurityPolicy { - t.Errorf("GET %s Content-Security-Policy = %q, want %q", path, csp, monitorContentSecurityPolicy) - } - } -} - -// TestMonitorRootRedirects covers the bare-host convenience: with -// Options.MonitorPage set, GET / (the root path only) 302-redirects to the -// canonical /monitor, unauthenticated (no Authorization header), so visiting a -// box's host with no path lands on the monitor. The {$}-anchored route must -// NOT be a catch-all — an unmatched non-root path still 404s, it does not -// redirect here. -func TestMonitorRootRedirects(t *testing.T) { - dir := t.TempDir() - const page = "fake monitor page" - srv := newServer(t, dir, &scriptedProvider{name: "test"}, 0, func(o *Options) { - o.MonitorPage = []byte(page) - }) - ts := httptest.NewServer(srv) - t.Cleanup(ts.Close) - - // Assert the redirect itself rather than following it through. - client := ts.Client() - client.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse } - - req, _ := http.NewRequest("GET", ts.URL+"/", nil) // no auth header - resp, err := client.Do(req) - if err != nil { - t.Fatal(err) - } - resp.Body.Close() - if resp.StatusCode != http.StatusFound { - t.Fatalf("GET / status = %d, want 302 to /monitor", resp.StatusCode) - } - if loc := resp.Header.Get("Location"); loc != "/monitor" { - t.Errorf("GET / Location = %q, want %q", loc, "/monitor") - } - - // The {$} anchor must keep root from becoming a catch-all: an unmatched - // path still 404s, it is not redirected to the monitor. - req2, _ := http.NewRequest("GET", ts.URL+"/no-such-path", nil) - resp2, err := client.Do(req2) - if err != nil { - t.Fatal(err) - } - resp2.Body.Close() - if resp2.StatusCode != http.StatusNotFound { - t.Errorf("GET /no-such-path status = %d, want 404 (the root redirect must not be a catch-all)", resp2.StatusCode) - } -} - -// TestMonitorPageNotConfigured guards the "existing deployments unchanged" -// contract: a server that never sets Options.MonitorPage (the zero value, -// same as every server.New call before this field existed) must 404 at -// GET /monitor exactly as it always has — no route registered at all, not -// merely an empty 200 — and the bare root stays a 404 too (the root redirect -// is registered under the SAME MonitorPage guard, so a pure-API box never -// redirects / to a route it doesn't serve). -func TestMonitorPageNotConfigured(t *testing.T) { +// contract: neither GET /monitor nor the bare root is a route this server +// serves. The root matters on its own: it must stay a clean 404 and never +// become a catch-all, which would swallow every unmatched path's 404. +func TestNoMonitorOrRootRoute(t *testing.T) { h := newHarness(t, &scriptedProvider{name: "test"}) - for _, path := range []string{"/monitor", "/"} { + for _, path := range []string{"/monitor", "/monitor/", "/"} { req, _ := http.NewRequest("GET", h.ts.URL+path, nil) resp, err := h.ts.Client().Do(req) if err != nil { @@ -711,7 +625,7 @@ func TestMonitorPageNotConfigured(t *testing.T) { } resp.Body.Close() if resp.StatusCode != http.StatusNotFound { - t.Fatalf("GET %s status = %d, want 404 when MonitorPage is not configured", path, resp.StatusCode) + t.Fatalf("GET %s status = %d, want 404", path, resp.StatusCode) } } } @@ -719,8 +633,7 @@ func TestMonitorPageNotConfigured(t *testing.T) { // TestUnauthenticatedServesWithoutToken covers cmd/harness's loopback- // unauthenticated path (server.Options.Unauthenticated): every route // serves successfully with NO Authorization header at all — not just -// /health/MonitorPage, which were already unauthenticated before this -// field existed. +// /health, which was already unauthenticated before this field existed. func TestUnauthenticatedServesWithoutToken(t *testing.T) { dir := t.TempDir() srv := newServer(t, dir, &scriptedProvider{name: "test"}, 0, func(o *Options) { diff --git a/tools/AGENTS.md b/tools/AGENTS.md index ecfdf3ab..ae0915b2 100644 --- a/tools/AGENTS.md +++ b/tools/AGENTS.md @@ -4,7 +4,7 @@ These rules apply to `tools/`. Harness does not merge ancestor files. If root guidance is not active, locate the Git root and read `/AGENTS.md`. Resolve repository paths and commands from that root. -The hub, monitor, and inspector are operator tools. They are not deployed +The hub and inspector are operator tools. They are not deployed multi-user products. Keep each page build-free and dependency-free unless a separate design changes that constraint. @@ -58,39 +58,6 @@ The hub uses a dark tactical-telemetry style. - Do not add gradients, soft shadows, rounded corners, or decorative metadata. - Do not add emoji or em dashes to hub UI strings. -## Session monitor - -The monitor is a single-box live board. - -- Keep the standalone `file://` and static-host path working. -- Keep `GET /monitor` as the same-origin embedded path. -- Store manual connection settings in the documented local-storage keys. -- Keep route state in explicit fragment parameters. -- Adopt a `#t=` token into storage, then scrub it from the visible URL. -- Do not connect an embedded page to a different origin under its same-origin - CSP. -- Keep the testable helper region stable. - -Run: - -```bash -node --test tools/monitor/*_test.mjs -go test -race ./tools/monitor/... -``` - -Read `tools/monitor/e2e/README.md`, -`docs/development-interfaces.md`, and the approved monitor -mockup before a layout change. - -### Monitor UI design language - -The monitor uses the instrument-sheet design, not the hub design. - -- Preserve the light-first OKLCH token system and its dark variant. -- Reserve green, amber, and red for state. -- Reserve the filled accent for the send action. -- Keep `docs/design/monitor-mockup.html` as the visual specification. - ## Inspector Follow the same build-free, pure-helper, and no-fixed-sleep rules. Do not apply diff --git a/tools/monitor/e2e/.gitignore b/tools/monitor/e2e/.gitignore deleted file mode 100644 index c2658d7d..00000000 --- a/tools/monitor/e2e/.gitignore +++ /dev/null @@ -1 +0,0 @@ -node_modules/ diff --git a/tools/monitor/e2e/README.md b/tools/monitor/e2e/README.md deleted file mode 100644 index 30ed76d0..00000000 --- a/tools/monitor/e2e/README.md +++ /dev/null @@ -1,90 +0,0 @@ -# tools/monitor/e2e — real-backend verification for the monitor page - -`tools/monitor/index.html` is a single, build-free HTML file with zero -dependencies (see its own header comment) — that does not change here. This -directory is a separate, isolated, npm-based verification *tool* that proves -the page's board/detail/composer behavior against a **real** running harness -backend, not hand-rolled mocks. Mirrors `tools/hub/e2e`'s structure and -conventions throughout. - -## What it checks - -`real_e2e.mjs`, driven by `e2e_test.go`, starts a real `server.Server` -(`stub.go`'s `Start`, the same wiring as `harness serve`) backed by a -handful of small scripted providers (no API key needed — including a -**real** `bash` tool call, a short `sleep`, not a simulated delay), plus a -plain static file server for the *actual* `tools/monitor/index.html`, then -loads that real page in [jsdom](https://github.com/jsdom/jsdom) with Node's -own, **unmocked** `fetch` — real HTTP requests, real SSE streams, real -engine turns. It confirms: - -1. the static server serves `tools/monitor/index.html` byte-for-byte - (production wiring, not a stale copy); -2. a wrong run token surfaces an inline connect error (from `/session`'s - 401, never from the unauthenticated `/health`); a correct one renders a - real `/health` identity line (version, `session_sync`) and an empty - board; -3. a real scripted turn — streaming text deltas AND a real, briefly-blocking - `bash` tool call — drives a board row through the streaming/tool phases - and back to idle with outcome `completed`; -4. a session left running long enough (against shrunk, test-only staleness - thresholds — see `index.html`'s `window.__monitorTuning` seam) crosses - the `quiet` and `stalled` tiers live; -5. opening a session's detail view via a **real row click** renders its - durable history — operator/assistant/tool entries, a completed tool fold - starting collapsed — and a fold that is still genuinely running at the - moment it's observed renders open, then settles to completed live - without ever being force-collapsed; -6. the composer's `prompt.queued` optimistic entry appears for a send into - a **busy** session and is replaced (not duplicated) once the durable, - template-wrapped message lands; a send into an idle session runs a - normal turn (a `message` event, not `prompt.queued`); a send against an - unknown session id surfaces the server's real non-2xx error text inline; - a composer submit also renders the operator's own text **synchronously**, - before the POST even resolves, settling to exactly one entry once the - real message lands; a real, currently-running turn with no content yet - shows a quiet "Thinking…" pending indicator, dismissed the instant real - streaming content arrives; -7. killing the box's HTTP layer **server-side** flips the header to - "reconnecting…", and restarting it resumes the stream — proven live by a - brand-new session created after the restart still arriving via SSE. - -## The staleness test seam - -Production `QUIET_MS`/`STALL_MS` are 15s/60s — far too slow for a CI-sane -test. `index.html` reads `window.__monitorTuning = { QUIET_MS, STALL_MS }` -(set here via jsdom's `beforeParse`, so it lands before the page's inline -script ever runs) to override them; nothing in production ever sets that -global, so it is a no-op outside this harness. See `index.html`'s comment -just after `TESTABLE-END` and `real_e2e.mjs`'s `TUNING` constant (which also -explains why the QUIET/STALL gap must stay wider than the board's own 1s -ticker, or the "quiet" tier can fall entirely between two samples). - -## Running it - -No manual setup step is required. Just run the same command already used to -verify this repo: - -```sh -go test -race ./... # or narrower: go test ./tools/monitor/e2e/... -``` - -`TestRealEndToEnd` installs its own dependency (`npm ci`, using the -package-lock.json committed here) the first time it runs if jsdom isn't -already present in this directory, then drives the real check. `node` (and -therefore `npm`, which ships with it) is already a hard requirement of this -repo's `node --test tools/monitor/*_test.mjs` check, so this test only skips -in the one case where that other required command would ALSO be unrunnable -— no Node toolchain on `PATH` at all. It fails loudly (not a silent skip) if -`node`/`npm` ARE present but the dependency install itself fails (e.g. no -network access to npm's registry on first run). - -To drive it by hand instead (e.g. to poke at the real backend from an -actual browser), write a small one-off `main` package that calls -`e2e.Start()` and prints the returned `Stub`'s `BoxBase`/`MonitorBase`/ -`Token` (see `stub.go`), then: - -```sh -node tools/monitor/e2e/real_e2e.mjs -# or open monitor_base in a real browser and connect with box_base + token by hand -``` diff --git a/tools/monitor/e2e/e2e_test.go b/tools/monitor/e2e/e2e_test.go deleted file mode 100644 index 34a42087..00000000 --- a/tools/monitor/e2e/e2e_test.go +++ /dev/null @@ -1,151 +0,0 @@ -package e2e - -import ( - "bytes" - "context" - "os" - "os/exec" - "path/filepath" - "runtime" - "testing" - "time" -) - -// TestRealEndToEnd starts a REAL box server + a REAL static file server for -// the ACTUAL tools/monitor/index.html (see stub.go's Start) and drives that -// page with Node + jsdom, using Node's own unmocked fetch — real HTTP, real -// SSE, real engine turns (including a REAL "bash" tool execution — see -// stub.go's toolCallPart). This is the automated counterpart to -// index.html's header-comment hand-test checklist: it exists so plain -// `go test -race ./...` — the exact command already used to verify this -// repo, no extra step required — checks, without any manual browser -// session, that: -// - connecting with a wrong token surfaces an inline error, and a correct -// one renders a real /health identity line and an empty board; -// - a real scripted turn (streaming text + a real, briefly-blocking bash -// tool call) drives a board row through the streaming/tool phases and -// back to idle with outcome "completed"; -// - a session left running long enough (against shrunk, test-only -// staleness thresholds — see index.html's window.__monitorTuning seam) -// crosses the quiet and stalled tiers; -// - opening a session's detail view (via a real row click) renders its -// durable history, including a completed tool fold and a fold that is -// still genuinely running at the moment it's observed; -// - the composer's prompt.queued optimistic entry appears for a send into -// a BUSY session and is replaced (not duplicated) once the durable -// message lands; a send into an idle session runs a normal turn; a send -// against an unknown session id surfaces the server's real non-2xx -// error text inline; a composer submit renders the operator's own text -// SYNCHRONOUSLY, before the POST even resolves, settling to exactly one -// entry once the real message lands; a real, currently-running turn -// with no content yet shows a quiet "Thinking…" pending indicator, -// dismissed the instant real streaming content arrives; an idle-send's -// optimistic operator entry precedes — in actual DOM order, not merely -// "both exist" — that turn's own pending indicator and streaming -// assistant reply, never the other way around; -// - a durable message's reasoning part and text part (bundled together -// ahead of a real tool call) render as two distinct, correctly-labeled -// entries — never merged onto one DOM node showing the wrong label; -// - killing the box's HTTP layer server-side flips the header to -// "reconnecting…", and restarting it resumes the stream; -// - a real provider stream failure renders a critical transcript error -// entry with the chip settling to idle promptly (no poll dependency); -// - detailState.liveEvents crosses a tuned cap and reconcileDetail trims -// it back down; a reconnect gap (pollOnce advancing state.lastSeq past -// what the page's own stream actually delivered) heals via the SAME -// reconcileDetail, backfilling a turn the detail view never observed -// live; -// - embeddedConnectPlan's "frictionless local" behavior against the -// box's REAL GET /monitor route on fresh page loads: an Unauthenticated -// box auto-connects with zero typing; a "#t=" capability URL -// auto-connects a tokened box and scrubs the token from the visible -// URL; a tokened box with no token anywhere falls back to a usable, -// token-only panel (host absent). -// -// Dependency setup is automatic, not a documented manual prerequisite: if -// jsdom isn't already installed in this directory, the test runs `npm ci` -// (falling back to `npm install`) itself before driving real_e2e.mjs, using -// the package.json/package-lock.json committed alongside this file — same -// pattern as tools/hub/e2e/e2e_test.go, including the one skip condition -// (no Node toolchain on PATH at all — the one case where this repo's other -// required check, `node --test tools/monitor/*_test.mjs`, would ALSO be -// unrunnable). -func TestRealEndToEnd(t *testing.T) { - nodePath, err := exec.LookPath("node") - if err != nil { - t.Skip("node not found on PATH — this environment could not run `node --test tools/monitor/*_test.mjs` either; skipping real end-to-end monitor verification") - } - npmPath, err := exec.LookPath("npm") - if err != nil { - t.Skip("npm not found on PATH (normally ships with node); skipping real end-to-end monitor verification") - } - - _, thisFile, _, ok := runtime.Caller(0) - if !ok { - t.Fatal("could not determine tools/monitor/e2e directory") - } - dir := filepath.Dir(thisFile) - script := filepath.Join(dir, "real_e2e.mjs") - - if _, err := os.Stat(filepath.Join(dir, "node_modules", "jsdom")); err != nil { - installDeps(t, npmPath, dir) - } - - stub, err := Start() - if err != nil { - t.Fatalf("starting the real box/monitor stub servers: %v", err) - } - defer stub.Close() - - ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) - defer cancel() - - start := time.Now() - cmd := exec.CommandContext(ctx, nodePath, script, stub.BoxBase, stub.MonitorBase, stub.Token, stub.UnauthBase) - cmd.Dir = dir - var out bytes.Buffer - cmd.Stdout = &out - cmd.Stderr = &out - runErr := cmd.Run() - t.Logf("real_e2e.mjs runtime: %s", time.Since(start)) - t.Log(out.String()) - if runErr != nil { - t.Fatalf("real_e2e.mjs failed: %v", runErr) - } -} - -// installDeps runs `npm ci` (a clean, lockfile-exact install, using the -// package-lock.json committed in this directory) to fetch jsdom before the -// real end-to-end check needs it, so a fresh clone requires no manual setup -// step beyond having node/npm on PATH. Falls back to `npm install` if `npm -// ci` itself is unavailable in this npm version (older npm predates it). -// Requires network access to npm's registry; a genuinely offline CI run -// fails loudly here (t.Fatalf) rather than silently skipping the real -// check — an offline environment is a real gap in verification, not a -// reason to pretend everything passed. Copied from tools/hub/e2e/e2e_test.go. -func installDeps(t *testing.T, npmPath, dir string) { - t.Helper() - t.Logf("jsdom not present in %s; running npm ci to install it (see package.json/package-lock.json)", dir) - ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) - defer cancel() - cmd := exec.CommandContext(ctx, npmPath, "ci", "--no-audit", "--no-fund") - cmd.Dir = dir - var out bytes.Buffer - cmd.Stdout = &out - cmd.Stderr = &out - if err := cmd.Run(); err != nil { - t.Logf("npm ci failed (%v), output:\n%s\nfalling back to npm install", err, out.String()) - cmd = exec.CommandContext(ctx, npmPath, "install", "--no-audit", "--no-fund") - cmd.Dir = dir - out.Reset() - cmd.Stdout = &out - cmd.Stderr = &out - if err := cmd.Run(); err != nil { - t.Fatalf("npm install failed too (%v); real end-to-end monitor verification requires network access to npm's registry on first run:\n%s", err, out.String()) - } - } - t.Log(out.String()) - if _, err := os.Stat(filepath.Join(dir, "node_modules", "jsdom")); err != nil { - t.Fatalf("jsdom still missing from %s/node_modules after npm install; something is wrong with the dependency install", dir) - } -} diff --git a/tools/monitor/e2e/package-lock.json b/tools/monitor/e2e/package-lock.json deleted file mode 100644 index 6fe1f117..00000000 --- a/tools/monitor/e2e/package-lock.json +++ /dev/null @@ -1,516 +0,0 @@ -{ - "name": "harness-monitor-e2e", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "harness-monitor-e2e", - "dependencies": { - "jsdom": "^29.1.1" - } - }, - "node_modules/@asamuzakjp/css-color": { - "version": "5.1.11", - "resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-5.1.11.tgz", - "integrity": "sha512-KVw6qIiCTUQhByfTd78h2yD1/00waTmm9uy/R7Ck/ctUyAPj+AEDLkQIdJW0T8+qGgj3j5bpNKK7Q3G+LedJWg==", - "license": "MIT", - "dependencies": { - "@asamuzakjp/generational-cache": "^1.0.1", - "@csstools/css-calc": "^3.2.0", - "@csstools/css-color-parser": "^4.1.0", - "@csstools/css-parser-algorithms": "^4.0.0", - "@csstools/css-tokenizer": "^4.0.0" - }, - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/@asamuzakjp/dom-selector": { - "version": "7.1.1", - "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-7.1.1.tgz", - "integrity": "sha512-67RZDnYRc8H/8MLDgQCDE//zoqVFwajkepHZgmXrbwybzXOEwOWGPYGmALYl9J2DOLfFPPs6kKCqmbzV895hTQ==", - "license": "MIT", - "dependencies": { - "@asamuzakjp/generational-cache": "^1.0.1", - "@asamuzakjp/nwsapi": "^2.3.9", - "bidi-js": "^1.0.3", - "css-tree": "^3.2.1", - "is-potential-custom-element-name": "^1.0.1" - }, - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/@asamuzakjp/generational-cache": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/@asamuzakjp/generational-cache/-/generational-cache-1.0.1.tgz", - "integrity": "sha512-wajfB8KqzMCN2KGNFdLkReeHncd0AslUSrvHVvvYWuU8ghncRJoA50kT3zP9MVL0+9g4/67H+cdvBskj9THPzg==", - "license": "MIT", - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/@asamuzakjp/nwsapi": { - "version": "2.3.9", - "resolved": "https://registry.npmjs.org/@asamuzakjp/nwsapi/-/nwsapi-2.3.9.tgz", - "integrity": "sha512-n8GuYSrI9bF7FFZ/SjhwevlHc8xaVlb/7HmHelnc/PZXBD2ZR49NnN9sMMuDdEGPeeRQ5d0hqlSlEpgCX3Wl0Q==", - "license": "MIT" - }, - "node_modules/@bramus/specificity": { - "version": "2.4.2", - "resolved": "https://registry.npmjs.org/@bramus/specificity/-/specificity-2.4.2.tgz", - "integrity": "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==", - "license": "MIT", - "dependencies": { - "css-tree": "^3.0.0" - }, - "bin": { - "specificity": "bin/cli.js" - } - }, - "node_modules/@csstools/color-helpers": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-6.1.0.tgz", - "integrity": "sha512-064IFJdjTfUqnjpCVpMOdbr8FLQBhinbZj6yRv2An2E41O/pLEXqfFRWqGq/SxlE5PEUYTlvWsG2r8MswAVvkg==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT-0", - "engines": { - "node": ">=20.19.0" - } - }, - "node_modules/@csstools/css-calc": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-3.3.0.tgz", - "integrity": "sha512-c5ihYsPkdG6JCkU2zTMm4+k6r7RXuGxtWYhu5DHMIiF1FHzrfmHL5so11AoFpUv/tu61xfcmT4AmKoFfMPoqdQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT", - "engines": { - "node": ">=20.19.0" - }, - "peerDependencies": { - "@csstools/css-parser-algorithms": "^4.0.0", - "@csstools/css-tokenizer": "^4.0.0" - } - }, - "node_modules/@csstools/css-color-parser": { - "version": "4.1.10", - "resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-4.1.10.tgz", - "integrity": "sha512-UZhQLIUyJaaMepqehrCODwCg2KW25vFvLWBmqYFaPclYvvxzj/sG8LBOhBFCp11i9uE7t1EyS+RAoV9tztPFyw==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT", - "dependencies": { - "@csstools/color-helpers": "^6.1.0", - "@csstools/css-calc": "^3.3.0" - }, - "engines": { - "node": ">=20.19.0" - }, - "peerDependencies": { - "@csstools/css-parser-algorithms": "^4.0.0", - "@csstools/css-tokenizer": "^4.0.0" - } - }, - "node_modules/@csstools/css-parser-algorithms": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-4.0.0.tgz", - "integrity": "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT", - "engines": { - "node": ">=20.19.0" - }, - "peerDependencies": { - "@csstools/css-tokenizer": "^4.0.0" - } - }, - "node_modules/@csstools/css-syntax-patches-for-csstree": { - "version": "1.1.7", - "resolved": "https://registry.npmjs.org/@csstools/css-syntax-patches-for-csstree/-/css-syntax-patches-for-csstree-1.1.7.tgz", - "integrity": "sha512-fQ+05118eQS1cofO3aJpB5efgpBZMvIzwr/sbC8kDLVA5XLG8q1kJV5yzrUAI1f7lvhPnm8fgIjzFB8/O/5Dig==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT-0", - "peerDependencies": { - "css-tree": "^3.2.1" - }, - "peerDependenciesMeta": { - "css-tree": { - "optional": true - } - } - }, - "node_modules/@csstools/css-tokenizer": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-4.0.0.tgz", - "integrity": "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/csstools" - }, - { - "type": "opencollective", - "url": "https://opencollective.com/csstools" - } - ], - "license": "MIT", - "engines": { - "node": ">=20.19.0" - } - }, - "node_modules/@exodus/bytes": { - "version": "1.15.1", - "resolved": "https://registry.npmjs.org/@exodus/bytes/-/bytes-1.15.1.tgz", - "integrity": "sha512-S6mL0yNB/Abt9Ei4tq8gDhcczc4S3+vQ4ra7vxnAf+YHC02srtqxKKZghx2Dq6p0e66THKwR6r8N6P95wEty7Q==", - "license": "MIT", - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - }, - "peerDependencies": { - "@noble/hashes": "^1.8.0 || ^2.0.0" - }, - "peerDependenciesMeta": { - "@noble/hashes": { - "optional": true - } - } - }, - "node_modules/bidi-js": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz", - "integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==", - "license": "MIT", - "dependencies": { - "require-from-string": "^2.0.2" - } - }, - "node_modules/css-tree": { - "version": "3.2.1", - "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz", - "integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==", - "license": "MIT", - "dependencies": { - "mdn-data": "2.27.1", - "source-map-js": "^1.2.1" - }, - "engines": { - "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0" - } - }, - "node_modules/data-urls": { - "version": "7.0.0", - "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz", - "integrity": "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA==", - "license": "MIT", - "dependencies": { - "whatwg-mimetype": "^5.0.0", - "whatwg-url": "^16.0.0" - }, - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/decimal.js": { - "version": "10.6.0", - "resolved": "https://registry.npmjs.org/decimal.js/-/decimal.js-10.6.0.tgz", - "integrity": "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==", - "license": "MIT" - }, - "node_modules/entities": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/entities/-/entities-8.0.0.tgz", - "integrity": "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=20.19.0" - }, - "funding": { - "url": "https://github.com/fb55/entities?sponsor=1" - } - }, - "node_modules/html-encoding-sniffer": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz", - "integrity": "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==", - "license": "MIT", - "dependencies": { - "@exodus/bytes": "^1.6.0" - }, - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/is-potential-custom-element-name": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/is-potential-custom-element-name/-/is-potential-custom-element-name-1.0.1.tgz", - "integrity": "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==", - "license": "MIT" - }, - "node_modules/jsdom": { - "version": "29.1.1", - "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-29.1.1.tgz", - "integrity": "sha512-ECi4Fi2f7BdJtUKTflYRTiaMxIB0O6zfR1fX0GXpUrf6flp8QIYn1UT20YQqdSOfk2dfkCwS8LAFoJDEppNK5Q==", - "license": "MIT", - "dependencies": { - "@asamuzakjp/css-color": "^5.1.11", - "@asamuzakjp/dom-selector": "^7.1.1", - "@bramus/specificity": "^2.4.2", - "@csstools/css-syntax-patches-for-csstree": "^1.1.3", - "@exodus/bytes": "^1.15.0", - "css-tree": "^3.2.1", - "data-urls": "^7.0.0", - "decimal.js": "^10.6.0", - "html-encoding-sniffer": "^6.0.0", - "is-potential-custom-element-name": "^1.0.1", - "lru-cache": "^11.3.5", - "parse5": "^8.0.1", - "saxes": "^6.0.0", - "symbol-tree": "^3.2.4", - "tough-cookie": "^6.0.1", - "undici": "^7.25.0", - "w3c-xmlserializer": "^5.0.0", - "webidl-conversions": "^8.0.1", - "whatwg-mimetype": "^5.0.0", - "whatwg-url": "^16.0.1", - "xml-name-validator": "^5.0.0" - }, - "engines": { - "node": "^20.19.0 || ^22.13.0 || >=24.0.0" - }, - "peerDependencies": { - "canvas": "^3.0.0" - }, - "peerDependenciesMeta": { - "canvas": { - "optional": true - } - } - }, - "node_modules/lru-cache": { - "version": "11.5.2", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.2.tgz", - "integrity": "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==", - "license": "BlueOak-1.0.0", - "engines": { - "node": "20 || >=22" - } - }, - "node_modules/mdn-data": { - "version": "2.27.1", - "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz", - "integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==", - "license": "CC0-1.0" - }, - "node_modules/parse5": { - "version": "8.0.1", - "resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.1.tgz", - "integrity": "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw==", - "license": "MIT", - "dependencies": { - "entities": "^8.0.0" - }, - "funding": { - "url": "https://github.com/inikulin/parse5?sponsor=1" - } - }, - "node_modules/punycode": { - "version": "2.3.1", - "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", - "integrity": "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, - "node_modules/require-from-string": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", - "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/saxes": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz", - "integrity": "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==", - "license": "ISC", - "dependencies": { - "xmlchars": "^2.2.0" - }, - "engines": { - "node": ">=v12.22.7" - } - }, - "node_modules/source-map-js": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", - "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", - "license": "BSD-3-Clause", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/symbol-tree": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz", - "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==", - "license": "MIT" - }, - "node_modules/tldts": { - "version": "7.4.9", - "resolved": "https://registry.npmjs.org/tldts/-/tldts-7.4.9.tgz", - "integrity": "sha512-3kZ8wQQ/k5DrChD4X4FVvr2D7E5uoRgAqkPyLpSCGUvqOvqu+JEdr3mwMUaVWb+vMHZaKhF5fp2PBigKsui7hA==", - "license": "MIT", - "dependencies": { - "tldts-core": "^7.4.9" - }, - "bin": { - "tldts": "bin/cli.js" - } - }, - "node_modules/tldts-core": { - "version": "7.4.9", - "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-7.4.9.tgz", - "integrity": "sha512-DxKfPBI52p2msTEu7MPhdpdDTBhhVQg1a/8PjQckeyAvO13eMYElX545grIp6nnTGIMZlRvFZPvFhvI/WIz2Vg==", - "license": "MIT" - }, - "node_modules/tough-cookie": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-6.0.2.tgz", - "integrity": "sha512-exgYmnmL/sJpR3upZfXG5PoatXQii55xAiXGXzY+sROLZ/Y+SLcp9PgJNI9Vz37HpQ74WvDcLT8eqm+kV3FzrA==", - "license": "BSD-3-Clause", - "dependencies": { - "tldts": "^7.0.5" - }, - "engines": { - "node": ">=16" - } - }, - "node_modules/tr46": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz", - "integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==", - "license": "MIT", - "dependencies": { - "punycode": "^2.3.1" - }, - "engines": { - "node": ">=20" - } - }, - "node_modules/undici": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/undici/-/undici-7.29.0.tgz", - "integrity": "sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==", - "license": "MIT", - "engines": { - "node": ">=20.18.1" - } - }, - "node_modules/w3c-xmlserializer": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz", - "integrity": "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==", - "license": "MIT", - "dependencies": { - "xml-name-validator": "^5.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/webidl-conversions": { - "version": "8.0.1", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz", - "integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=20" - } - }, - "node_modules/whatwg-mimetype": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz", - "integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==", - "license": "MIT", - "engines": { - "node": ">=20" - } - }, - "node_modules/whatwg-url": { - "version": "16.0.1", - "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz", - "integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==", - "license": "MIT", - "dependencies": { - "@exodus/bytes": "^1.11.0", - "tr46": "^6.0.0", - "webidl-conversions": "^8.0.1" - }, - "engines": { - "node": "^20.19.0 || ^22.12.0 || >=24.0.0" - } - }, - "node_modules/xml-name-validator": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-5.0.0.tgz", - "integrity": "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==", - "license": "Apache-2.0", - "engines": { - "node": ">=18" - } - }, - "node_modules/xmlchars": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/xmlchars/-/xmlchars-2.2.0.tgz", - "integrity": "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==", - "license": "MIT" - } - } -} diff --git a/tools/monitor/e2e/package.json b/tools/monitor/e2e/package.json deleted file mode 100644 index db34bc87..00000000 --- a/tools/monitor/e2e/package.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "name": "harness-monitor-e2e", - "private": true, - "description": "Isolated verification tooling for tools/monitor/index.html's real-backend end-to-end check (real_e2e.mjs, driven by e2e_test.go). Deliberately kept in its own package.json, separate from the shipped monitor page itself, which stays a single build-free HTML file with zero dependencies — see tools/monitor/index.html's header comment. Mirrors tools/hub/e2e's structure exactly.", - "type": "module", - "scripts": { - "e2e": "node real_e2e.mjs" - }, - "dependencies": { - "jsdom": "^29.1.1" - } -} diff --git a/tools/monitor/e2e/real_e2e.mjs b/tools/monitor/e2e/real_e2e.mjs deleted file mode 100644 index 5a664bc0..00000000 --- a/tools/monitor/e2e/real_e2e.mjs +++ /dev/null @@ -1,951 +0,0 @@ -// REAL end-to-end verification of tools/monitor/index.html (see its own -// header comment's hand-test checklist for the behaviors this automates, and -// AGENTS.md for the monitor's role): drives the ACTUAL page — byte-for-byte -// the same file committed at tools/monitor/index.html (checked below), no -// mock DOM shortcuts — against a REAL running harness box (tools/monitor/ -// e2e's Stub — same wiring as `harness serve`, plus a couple of scripted -// providers so turns don't need a real model API key), using jsdom + Node's -// own, UNMOCKED fetch. Nothing in this file simulates HTTP/SSE traffic; -// every request below is a real socket round-trip to the servers -// e2e_test.go started, including a REAL "bash" tool call (a short `sleep`) -// executed by the real engine — not a mocked delay. -// -// Expects three arguments: (see -// tools/monitor/e2e/stub.go's Start). Exits non-zero on any failed -// assertion, printing the failure to stderr. Requires "jsdom" (see -// tools/monitor/e2e/package.json). Run directly with: -// go run ./tools/monitor/e2e/... is not provided (unlike tools/hub/e2e's -// hubverify) — this package's only entry point is e2e_test.go, which -// starts the stub and drives this script itself. To poke at it by hand, -// write a small one-off `main` calling e2e.Start(), print the returned -// Stub fields, then: -// node tools/monitor/e2e/real_e2e.mjs -// -// Mirrors tools/hub/e2e/real_e2e.mjs's structure and conventions throughout -// (the jsdom setup, the fetch/AbortController polyfilling, the byte-for-byte -// served-file check, the PASS/console.error-per-assertion narration, the -// forced process.exit at the end). -import { JSDOM } from "jsdom"; -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, join } from "node:path"; - -const [, , boxBase, monitorBase, token, unauthBase] = process.argv; -if (!boxBase || !monitorBase || !token || !unauthBase) { - console.error("usage: node real_e2e.mjs "); - process.exit(2); -} -console.error("box:", boxBase, "monitor:", monitorBase, "unauth:", unauthBase); - -// releaseTurn opens a GATED scripted turn (stub.go's turnGates): the -// ProvPendingThink scenarios below hold their turn's first event until the -// test has finished observing the "Thinking…" indicator's -// busy-with-no-content window. The window is therefore bounded by the -// observation itself, never by a clock the test has to outrun — a fixed -// delay here flaked on CI even after being widened once. -async function releaseTurn(sessionID) { - const r = await fetch(monitorBase + "/__control/release-turn?session=" + encodeURIComponent(sessionID), { method: "POST" }); - assert.equal(r.status, 200, "control-plane release-turn for " + sessionID); -} - -const here = dirname(fileURLToPath(import.meta.url)); -const committedIndexHTML = readFileSync(join(here, "..", "index.html"), "utf8"); - -// Provider keys — must match the consts in stub.go exactly (see its Prov* -// block); duplicated here as plain strings rather than parsed out of the Go -// source because this script has no Go tooling available to it. -const ProvQuickIdle = "e2e-quick-idle"; -const ProvToolBoard = "e2e-tool-board"; -const ProvToolDetail = "e2e-tool-detail"; -const ProvStallStale = "e2e-stall-stale"; -const ProvStallDedup = "e2e-stall-dedup"; -const ProvStreamError = "e2e-stream-error"; -const ProvReconnectGap = "e2e-reconnect-gap"; -const ProvLiveCap = "e2e-live-cap"; -const ProvPendingThink = "e2e-pending-think"; -// Must match stub.go's StreamErrorText/ReconnectGapReply/PendingThinkReply -// exactly — same duplication-by-hand reasoning as the Prov* keys above (this -// script has no Go tooling). -const STREAM_ERROR_TEXT = "simulated upstream failure: connection reset by peer"; -const RECONNECT_GAP_REPLY = "reconnect-gap turn landed"; -const PENDING_THINK_REPLY = "pending-think reply landed"; - -// TUNING shrinks the monitor's staleness thresholds (production QUIET_MS/ -// STALL_MS are 15000/60000 — see index.html) down to something a real, -// bounded bash `sleep` can cross inside a CI-sane test budget, shrinks -// DETAIL_LIVE_EVENTS_CAP (production 500) down to something a single -// scripted turn's handful of live events comfortably crosses, and WIDENS -// BACKOFF_MIN (production 500ms) so the reconnect-gap-heal scenario has a -// deterministically generous window between a real server-side kill/ -// restart and the page's own next reconnect attempt, rather than depending -// on exact wall-clock luck to land its race. Read by index.html's -// window.__monitorTuning seam, set below via JSDOM's beforeParse so it -// lands before the page's inline - - diff --git a/tools/monitor/monitor_test.mjs b/tools/monitor/monitor_test.mjs deleted file mode 100644 index 0e5f09c8..00000000 --- a/tools/monitor/monitor_test.mjs +++ /dev/null @@ -1,1679 +0,0 @@ -// Unit tests for the pure helpers in tools/monitor/index.html. -// -// The monitor is a single self-contained HTML file with no build step, so -// there is nothing to import. Instead we read index.html, extract the region -// between the /* TESTABLE-BEGIN */ and /* TESTABLE-END */ markers, and -// evaluate it in a node:vm sandbox exposing only Date and JSON. This keeps -// the page build-free while making its parser + helpers reproducibly -// testable. Copied from tools/inspector/inspector_test.mjs's extraction -// preamble, adjusted to this file's path. -// -// Run: node --test tools/monitor/ - -import test from "node:test"; -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, join } from "node:path"; -import vm from "node:vm"; - -const here = dirname(fileURLToPath(import.meta.url)); -const html = readFileSync(join(here, "index.html"), "utf8"); - -const begin = "/* TESTABLE-BEGIN"; -const end = "/* TESTABLE-END */"; -const bi = html.indexOf(begin); -const ei = html.indexOf(end); -assert.ok(bi >= 0 && ei > bi, "TESTABLE markers must be present in index.html"); -// Start extraction after the BEGIN comment's closing */ so the comment body -// (which itself contains no code) is not part of the evaluated source. -const afterBegin = html.indexOf("*/", bi) + 2; -const source = html.slice(afterBegin, ei); - -// Function declarations (and `var`-declared bindings, unlike `const`/`let`, -// which live in the vm context's lexical environment rather than as -// properties of it) at the top level of a vm script become properties of the -// sandbox's global object; read them straight off the context. -const sandbox = { Date, JSON }; -vm.createContext(sandbox); -vm.runInContext(source, sandbox); -const { - createSSEParser, - maxSeq, - partsText, - summarizeArgs, - toolLabel, - fmtElapsed, - fmtAgo, - hostLabel, - route, - unroute, - sameOriginDefaultBase, - embeddedConnectPlan, - extractFragmentToken, - QUIET_MS, - STALL_MS, - staleness, - reduceActivity, - seedActivity, - boardModel, - transcriptModel, - HISTORY_WINDOW, - historyWindow, - adaptHistory, - countKind, - liveMessageCount, - detailUnderpopulated, - entryKey, - turnMarkAgoText, - keepsLiveEventAfterReconcile, -} = sandbox; - -// collect gathers every frame the parser dispatches for the given chunks. -// Frames are rebuilt as plain objects in this realm: the parser creates them -// inside the vm sandbox, and deepStrictEqual rejects cross-realm objects even -// when their structure is identical. -function collect(chunks) { - const frames = []; - const feed = createSSEParser(f => frames.push({ id: f.id, data: f.data })); - for (const c of chunks) feed(c); - return frames; -} - -// reify deep-clones a value that crossed the vm sandbox boundary back into -// this realm's plain Object/Array graph. node:assert's strict deepEqual -// treats same-shaped values from different realms as unequal (their -// [[Prototype]]s differ) — the same cross-realm gotcha `collect` above works -// around for SSE frames. JSON-safe here: every helper under test returns -// plain data (strings/numbers/booleans/null/arrays/objects), so a -// stringify/parse round trip reconstructs it natively in this realm. -function reify(v) { - return v === undefined ? v : JSON.parse(JSON.stringify(v)); -} - -/* ---------- createSSEParser (copied base cases from inspector) ---------- */ - -test("SSE parser: single frame with data", () => { - const f = collect(["data: hello\n\n"]); - assert.deepEqual(f, [{ id: null, data: "hello" }]); -}); - -test("SSE parser: multi-line data joined with \\n", () => { - const f = collect(["data: a\ndata: b\ndata: c\n\n"]); - assert.deepEqual(f, [{ id: null, data: "a\nb\nc" }]); -}); - -test("SSE parser: id line is captured", () => { - const f = collect(["id: 42\ndata: x\n\n"]); - assert.deepEqual(f, [{ id: "42", data: "x" }]); -}); - -test("SSE parser: comment / heartbeat lines are ignored", () => { - const f = collect([": keep-alive\ndata: x\n\n"]); - assert.deepEqual(f, [{ id: null, data: "x" }]); -}); - -test("SSE parser: only a comment dispatches nothing", () => { - assert.deepEqual(collect([": ping\n\n"]), []); -}); - -test("SSE parser: heartbeat comment interleaved with real frames", () => { - const f = collect(["data: one\n\n: keep-alive\n\ndata: two\n\n"]); - assert.deepEqual(f, [ - { id: null, data: "one" }, - { id: null, data: "two" }, - ]); -}); - -test("SSE parser: CRLF line endings are handled", () => { - const f = collect(["id: 7\r\ndata: hi\r\n\r\n"]); - assert.deepEqual(f, [{ id: "7", data: "hi" }]); -}); - -test("SSE parser: colons inside JSON values survive", () => { - const payload = '{"type":"tool.start","url":"http://x:8080"}'; - const f = collect(["data: " + payload + "\n\n"]); - assert.equal(f.length, 1); - assert.deepEqual(JSON.parse(f[0].data), { type: "tool.start", url: "http://x:8080" }); -}); - -test("SSE parser: chunk boundary splits mid-frame", () => { - const f = collect(["data: hel", "lo\n\n"]); - assert.deepEqual(f, [{ id: null, data: "hello" }]); -}); - -test("SSE parser: id persists to a subsequent id-less frame", () => { - const f = collect(["id: 100\ndata: first\n\n", "data: second\n\n"]); - assert.deepEqual(f, [ - { id: "100", data: "first" }, - { id: "100", data: "second" }, - ]); -}); - -/* ---------- maxSeq ---------- */ - -test("maxSeq returns the largest numeric seq, else 0", () => { - assert.equal(maxSeq([{ seq: 3 }, { seq: 9 }, { seq: 5 }]), 9); - assert.equal(maxSeq([{ seq: 3 }, {}, { seq: "x" }]), 3); - assert.equal(maxSeq([]), 0); -}); - -/* ---------- partsText / summarizeArgs / toolLabel ---------- */ - -test("partsText joins text parts and ignores non-text / non-arrays", () => { - assert.equal( - partsText([{ type: "text", text: "one" }, { type: "image" }, { type: "text", text: "two" }, null]), - "one\ntwo", - ); - assert.equal(partsText("nope"), ""); - assert.equal(partsText([]), ""); -}); - -test("summarizeArgs extracts a recognizable field from object or JSON-string arguments", () => { - assert.equal(summarizeArgs({ command: "go test ./server/ -race" }), "go test ./server/ -race"); - assert.equal(summarizeArgs('{"command":"gofmt -l ."}'), "gofmt -l ."); - assert.equal(summarizeArgs({ pattern: "TODO" }), "TODO"); - assert.equal(summarizeArgs({ nothingRecognizable: 1 }), ""); - assert.equal(summarizeArgs("not json"), ""); - assert.equal(summarizeArgs(null), ""); - assert.equal(summarizeArgs([1, 2]), ""); -}); - -test("toolLabel joins name and summary, or falls back to name / generic tool", () => { - assert.equal(toolLabel("Bash", "go test ./..."), "Bash · go test ./..."); - assert.equal(toolLabel("Bash", ""), "Bash"); - assert.equal(toolLabel("", ""), "tool"); -}); - -/* ---------- fmtElapsed / fmtAgo ---------- */ - -test("fmtElapsed formats seconds/minutes/hours like the mockup ticker", () => { - assert.equal(fmtElapsed(0), "0s"); - assert.equal(fmtElapsed(41), "41s"); - assert.equal(fmtElapsed(59), "59s"); - assert.equal(fmtElapsed(60), "1m 00s"); - assert.equal(fmtElapsed(154), "2m 34s"); - assert.equal(fmtElapsed(3600), "1h 0m"); - assert.equal(fmtElapsed(8047), "2h 14m"); - assert.equal(fmtElapsed(null), "—"); -}); - -test("fmtAgo renders a coarse ago string", () => { - assert.equal(fmtAgo(5000), "5s ago"); - assert.equal(fmtAgo(12 * 60 * 1000), "12m ago"); - assert.equal(fmtAgo((60 * 60 + 3 * 60) * 1000), "1h 3m ago"); - assert.equal(fmtAgo(2 * 60 * 60 * 1000), "2h ago"); - assert.equal(fmtAgo(null), "—"); -}); - -/* ---------- hostLabel ---------- */ - -test("hostLabel strips the scheme and cuts at the first path/query/fragment", () => { - assert.equal(hostLabel("http://localhost:4096"), "localhost:4096"); - assert.equal(hostLabel("https://box.example.com/"), "box.example.com"); - assert.equal(hostLabel("http://127.0.0.1:4096/session?x=1#y"), "127.0.0.1:4096"); -}); - -test("hostLabel tolerates missing scheme, and empty/missing input", () => { - assert.equal(hostLabel("localhost:4096"), "localhost:4096"); - assert.equal(hostLabel(""), ""); - assert.equal(hostLabel(null), ""); - assert.equal(hostLabel(undefined), ""); -}); - -/* ---------- route / unroute ---------- */ - -test("route/unroute round-trip base + session id", () => { - const state = { base: "http://localhost:4096", sessionId: "ses_01ky9fjq2wexvq8rn0m4tdq2m" }; - const hash = route(state); - assert.equal(hash, "#b=http%3A%2F%2Flocalhost%3A4096&s=ses_01ky9fjq2wexvq8rn0m4tdq2m"); - assert.deepEqual(reify(unroute(hash)), state); -}); - -test("route omits absent fields; unroute of '#' yields nulls", () => { - assert.equal(route({}), "#"); - assert.deepEqual(reify(unroute("#")), { base: null, sessionId: null }); - assert.deepEqual(reify(unroute("")), { base: null, sessionId: null }); - assert.equal(route({ base: "http://x" }), "#b=http%3A%2F%2Fx"); -}); - -test("unroute tolerates junk hashes without throwing", () => { - assert.deepEqual(reify(unroute(undefined)), { base: null, sessionId: null }); - assert.deepEqual(reify(unroute(null)), { base: null, sessionId: null }); - assert.deepEqual(reify(unroute("not-a-hash-at-all")), { base: null, sessionId: null }); - assert.deepEqual(reify(unroute("#&&&")), { base: null, sessionId: null }); - assert.deepEqual(reify(unroute("#x=1&y")), { base: null, sessionId: null }); - // Malformed percent-encoding must not throw; it degrades to the raw text. - assert.deepEqual(reify(unroute("#b=%zz&s=ok")), { base: "%zz", sessionId: "ok" }); -}); - -/* ---------- sameOriginDefaultBase (embedded GET /monitor same-origin - default) ---------- */ - -test("sameOriginDefaultBase: pathname '/monitor' defaults the base URL to the page's own origin", () => { - assert.equal(sameOriginDefaultBase({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }), "http://127.0.0.1:4096"); -}); - -test("sameOriginDefaultBase: a trailing-slash pathname ('/monitor/') also matches", () => { - assert.equal(sameOriginDefaultBase({ pathname: "/monitor/", origin: "http://127.0.0.1:4096" }), "http://127.0.0.1:4096"); -}); - -test("sameOriginDefaultBase: any other pathname (file://, a static host's own path, the board root) returns null", () => { - assert.equal(sameOriginDefaultBase({ pathname: "/", origin: "http://127.0.0.1:4096" }), null); - assert.equal(sameOriginDefaultBase({ pathname: "/tools/monitor/index.html", origin: "https://cdn.example.com" }), null); - assert.equal(sameOriginDefaultBase({ pathname: "/Users/dev/harness/tools/monitor/index.html", origin: "null" }), null); - // "monitoring" shares "/monitor" as a prefix but is a DIFFERENT path — - // must not false-positive on a substring match. - assert.equal(sameOriginDefaultBase({ pathname: "/monitoring", origin: "http://127.0.0.1:4096" }), null); -}); - -test("sameOriginDefaultBase: tolerates missing/malformed input without throwing", () => { - assert.equal(sameOriginDefaultBase(null), null); - assert.equal(sameOriginDefaultBase(undefined), null); - assert.equal(sameOriginDefaultBase({}), null); - assert.equal(sameOriginDefaultBase({ pathname: "/monitor" }), null); // no origin - assert.equal(sameOriginDefaultBase({ pathname: "/monitor", origin: "" }), null); // empty origin -}); - -/* ---------- embeddedConnectPlan (RED-FIRST — same-origin auto-connect: - opening a local box's /monitor drops the operator straight on the board, - no panel, no typing, unless/until an attempt actually fails) ---------- */ - -test("embeddedConnectPlan: not embedded (file:// / static host) — show-full-panel, no auto-attempt, no notice", () => { - const plan = embeddedConnectPlan({ pathname: "/", origin: "http://127.0.0.1:4096" }, null, "", "http://elsewhere:9000"); - assert.deepEqual(reify(plan), { embedded: false, base: null, autoToken: null, showBaseField: true, crossBoxNotice: null }); -}); - -test("embeddedConnectPlan: RED-FIRST — embedded with NO token anywhere still signals auto-attempt-same-origin (autoToken null, not a reason to skip trying — covers the loopback-Unauthenticated case)", () => { - const plan = embeddedConnectPlan({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }, null, "", null); - assert.equal(plan.embedded, true); - assert.equal(plan.base, "http://127.0.0.1:4096"); - assert.equal(plan.autoToken, null); - assert.equal(plan.showBaseField, false); - assert.equal(plan.crossBoxNotice, null); -}); - -test("embeddedConnectPlan: an explicit fragment token (#t=) wins as autoToken over a stored one — the fresher credential", () => { - const plan = embeddedConnectPlan({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }, "fresh-tok", "stale-tok", null); - assert.equal(plan.autoToken, "fresh-tok"); -}); - -test("embeddedConnectPlan: falls back to the stored token when no fragment token is present", () => { - const plan = embeddedConnectPlan({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }, null, "stale-tok", null); - assert.equal(plan.autoToken, "stale-tok"); -}); - -test("embeddedConnectPlan: RED-FIRST — a fragment base naming a DIFFERENT origin surfaces show-cross-box-notice, composing with (not replacing) the auto-attempt", () => { - const plan = embeddedConnectPlan({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }, "tok", "", "http://other-box:4096"); - assert.equal(plan.embedded, true, "the own-origin auto-attempt must still be signaled"); - assert.equal(plan.base, "http://127.0.0.1:4096", "the own-origin base must still be usable despite the notice"); - assert.equal(plan.autoToken, "tok", "the auto-attempt's token is unaffected by the notice"); - assert.ok(plan.crossBoxNotice, "expected a cross-box notice"); - assert.ok(plan.crossBoxNotice.includes("other-box:4096"), "notice must name the OTHER box: " + plan.crossBoxNotice); - assert.ok(plan.crossBoxNotice.includes("/monitor"), "notice should point at the other box's own /monitor: " + plan.crossBoxNotice); -}); - -test("embeddedConnectPlan: a fragment base that AGREES with this origin is not a conflict — no notice", () => { - const plan = embeddedConnectPlan({ pathname: "/monitor", origin: "http://127.0.0.1:4096" }, null, "", "http://127.0.0.1:4096"); - assert.equal(plan.crossBoxNotice, null); -}); - -test("embeddedConnectPlan: show-token-panel's static half — embedded always signals showBaseField false (base fixed/known), regardless of whether a token is available yet", () => { - // The DYNAMIC half (whether a panel is ever actually shown) depends on - // the auto-attempt's real result, which this pure function cannot know - // — see its own doc comment. bootstrap() only reveals a panel when - // attemptConnect resolves false; when it does, THIS is what that panel - // renders as. - assert.equal(embeddedConnectPlan({ pathname: "/monitor", origin: "http://x" }, null, "", null).showBaseField, false); - assert.equal(embeddedConnectPlan({ pathname: "/monitor", origin: "http://x" }, "tok", "", null).showBaseField, false); -}); - -test("embeddedConnectPlan: tolerates missing/malformed input without throwing", () => { - assert.equal(embeddedConnectPlan(null, null, null, null).embedded, false); - assert.equal(embeddedConnectPlan({ pathname: "/monitor", origin: "http://x" }, undefined, undefined, undefined).autoToken, null); - assert.equal(embeddedConnectPlan({ pathname: "/monitor", origin: "http://x" }, "", "", "").crossBoxNotice, null); // empty strings are "absent", not conflicts -}); - -/* ---------- extractFragmentToken (RED-FIRST — #t= capability URL: - zero-typing access without weakening auth) ---------- */ - -test("extractFragmentToken: adopts a plain '#t=' and scrubs it, leaving a bare '#'", () => { - const r = extractFragmentToken("#t=abc123"); - assert.equal(r.token, "abc123"); - assert.equal(r.cleanedHash, "#"); -}); - -test("extractFragmentToken: RED-FIRST — scrubbing preserves OTHER recognized params (s=) intact", () => { - const r = extractFragmentToken("#t=abc123&s=ses_01ky9fjq2wexvq8rn0m4tdq2m"); - assert.equal(r.token, "abc123"); - assert.equal(r.cleanedHash, "#s=ses_01ky9fjq2wexvq8rn0m4tdq2m"); -}); - -test("extractFragmentToken: preserves the base param (b=) too, in either param order", () => { - const r1 = extractFragmentToken("#b=http%3A%2F%2Flocalhost%3A4096&t=abc123"); - assert.equal(r1.token, "abc123"); - assert.equal(r1.cleanedHash, "#b=http%3A%2F%2Flocalhost%3A4096"); - const r2 = extractFragmentToken("#t=abc123&b=http%3A%2F%2Flocalhost%3A4096&s=xyz"); - assert.equal(r2.token, "abc123"); - assert.equal(r2.cleanedHash, "#b=http%3A%2F%2Flocalhost%3A4096&s=xyz"); -}); - -test("extractFragmentToken: no 't' param — token null, cleanedHash unchanged (modulo re-encoding)", () => { - const r = extractFragmentToken("#s=xyz"); - assert.equal(r.token, null); - assert.equal(r.cleanedHash, "#s=xyz"); - const empty = extractFragmentToken("#"); - assert.equal(empty.token, null); - assert.equal(empty.cleanedHash, "#"); -}); - -test("extractFragmentToken: an explicitly empty '#t=' is null (nothing to adopt), not an empty-string credential", () => { - const r = extractFragmentToken("#t="); - assert.equal(r.token, null); -}); - -test("extractFragmentToken: percent-decodes the token value, tolerating malformed encoding (degrades to raw text)", () => { - assert.equal(extractFragmentToken("#t=a%20b").token, "a b"); - assert.equal(extractFragmentToken("#t=%zz").token, "%zz"); -}); - -test("extractFragmentToken: a repeated 't' param uses the LAST occurrence (matches unroute's own repeated-param handling)", () => { - const r = extractFragmentToken("#t=first&t=second"); - assert.equal(r.token, "second"); -}); - -test("extractFragmentToken: tolerates junk hashes without throwing", () => { - assert.deepEqual(reify(extractFragmentToken(undefined)), { token: null, cleanedHash: "#" }); - assert.deepEqual(reify(extractFragmentToken(null)), { token: null, cleanedHash: "#" }); - assert.deepEqual(reify(extractFragmentToken("not-a-hash-at-all")), { token: null, cleanedHash: "#" }); - assert.deepEqual(reify(extractFragmentToken("#&&&")), { token: null, cleanedHash: "#" }); -}); - -/* ---------- staleness / QUIET_MS / STALL_MS ---------- */ - -test("QUIET_MS and STALL_MS are exported with the documented values", () => { - assert.equal(QUIET_MS, 15000); - assert.equal(STALL_MS, 60000); -}); - -test("staleness: idle phase is never quiet or stalled, regardless of silence", () => { - const idleActivity = { phase: "idle", tool: null, turn: null, lastEventAt: 0, lastOutcome: null }; - assert.equal(staleness(idleActivity, 10_000_000), "idle"); - assert.equal(staleness(null, 1000), "idle"); -}); - -test("staleness: boundaries are pinned exactly at 15000ms and 60000ms", () => { - const base = { phase: "tool", tool: { name: "Bash", sinceAt: 0 }, turn: { startedAt: 0, toolCalls: 1 }, lastEventAt: 0, lastOutcome: null }; - assert.equal(staleness(base, 14_999), "live"); - assert.equal(staleness(base, 15_000), "quiet"); // exactly QUIET_MS: quiet, not live - assert.equal(staleness(base, 59_999), "quiet"); - assert.equal(staleness(base, 60_000), "stalled"); // exactly STALL_MS: stalled, not quiet - assert.equal(staleness(base, 60_001), "stalled"); -}); - -/* ---------- reduceActivity (RED-FIRST: written before the implementation) ---------- */ - -test("reduceActivity: null prev is safe and starts from an idle base", () => { - const a = reduceActivity(null, { type: "session.status", status: "busy" }, 1000); - assert.equal(a.phase, "between"); - assert.deepEqual(reify(a.turn), { startedAt: 1000, toolCalls: 0 }); - assert.equal(a.tool, null); - assert.equal(a.lastEventAt, 1000); -}); - -test("reduceActivity: an unrecognized event bumps lastEventAt and otherwise passes prev through", () => { - const prev = { phase: "streaming", tool: null, turn: { startedAt: 0, toolCalls: 0 }, lastEventAt: 0, lastOutcome: null }; - const a = reduceActivity(prev, { type: "goal.eval" }, 500); - assert.equal(a.phase, "streaming"); - assert.equal(a.lastEventAt, 500); -}); - -test("reduceActivity: text.delta and reasoning.delta set phase streaming and open a turn if none is open", () => { - let a = reduceActivity(null, { type: "text.delta", text: "hi" }, 100); - assert.equal(a.phase, "streaming"); - assert.deepEqual(reify(a.turn), { startedAt: 100, toolCalls: 0 }); - a = reduceActivity(a, { type: "reasoning.delta", text: "thinking" }, 150); - assert.equal(a.phase, "streaming"); - assert.deepEqual(reify(a.turn), { startedAt: 100, toolCalls: 0 }); // unchanged, not re-opened -}); - -test("reduceActivity: tool.start sets phase tool, records name/argsSummary/sinceAt, increments toolCalls", () => { - const busy = reduceActivity(null, { type: "session.status", status: "busy" }, 0); - const a = reduceActivity(busy, { - type: "tool.start", - tool_call: { call_id: "call_1", name: "Bash", arguments: { command: "go test ./server/ -race" } }, - }, 41_000); - assert.equal(a.phase, "tool"); - assert.deepEqual(reify(a.tool), { name: "Bash", argsSummary: "go test ./server/ -race", sinceAt: 41_000 }); - assert.equal(a.turn.toolCalls, 1); - assert.equal(a.turn.startedAt, 0); // the turn's own start is untouched by the tool starting -}); - -test("reduceActivity: tool.end closes the tool and returns to the between phase", () => { - const busy = reduceActivity(null, { type: "session.status", status: "busy" }, 0); - const withTool = reduceActivity(busy, { type: "tool.start", tool_call: { call_id: "c1", name: "Bash", arguments: {} } }, 10); - const after = reduceActivity(withTool, { type: "tool.end", tool_call: { call_id: "c1" }, output: [], is_error: false }, 20); - assert.equal(after.phase, "between"); - assert.equal(after.tool, null); -}); - -test("reduceActivity: tool.end without a matching start is ignored, not a crash", () => { - const busy = reduceActivity(null, { type: "session.status", status: "busy" }, 0); - assert.equal(busy.tool, null); - const after = reduceActivity(busy, { type: "tool.end", tool_call: { call_id: "never-started" }, output: [] }, 10); - assert.equal(after.phase, "between"); // unchanged from prev, not incorrectly flipped - assert.equal(after.tool, null); - assert.equal(after.lastEventAt, 10); // still counts as activity -}); - -test("reduceActivity: session.status idle closes the turn and clears the current tool", () => { - const busy = reduceActivity(null, { type: "session.status", status: "busy" }, 0); - const withTool = reduceActivity(busy, { type: "tool.start", tool_call: { call_id: "c1", name: "Bash", arguments: {} } }, 10); - const idle = reduceActivity(withTool, { type: "session.status", status: "idle" }, 20); - assert.equal(idle.phase, "idle"); - assert.equal(idle.tool, null); - assert.equal(idle.turn, null); -}); - -test("reduceActivity: turn.end records lastOutcome without altering phase/turn", () => { - const busy = reduceActivity(null, { type: "session.status", status: "busy" }, 0); - const done = reduceActivity(busy, { type: "turn.end", outcome: "completed" }, 30); - assert.equal(done.lastOutcome, "completed"); - assert.equal(done.phase, "between"); -}); - -test("reduceActivity: mid-turn seed (seedActivity) then live events accumulate correctly", () => { - // A monitor connecting mid-turn only has poll data: state busy, no start - // time. seedActivity opens a turn with a null startedAt (never fabricated), - // and subsequent live events must not invent one either. - const seeded = seedActivity({ id: "s1", state: "busy", last_activity_at: new Date(500).toISOString(), last_turn: null }); - assert.equal(seeded.phase, "between"); - assert.deepEqual(reify(seeded.turn), { startedAt: null, toolCalls: 0 }); - assert.equal(seeded.lastEventAt, 500); - - const withTool = reduceActivity(seeded, { - type: "tool.start", tool_call: { call_id: "c1", name: "Bash", arguments: { command: "ls" } }, - }, 900); - assert.equal(withTool.phase, "tool"); - assert.equal(withTool.turn.startedAt, null); // still unknown — never fabricated - assert.equal(withTool.turn.toolCalls, 1); - assert.deepEqual(reify(withTool.tool), { name: "Bash", argsSummary: "ls", sinceAt: 900 }); -}); - -test("seedActivity: idle session seeds an idle activity carrying the last outcome", () => { - const seeded = seedActivity({ - id: "s1", state: "idle", - last_activity_at: new Date(12345).toISOString(), - last_turn: { outcome: "completed" }, - }); - assert.equal(seeded.phase, "idle"); - assert.equal(seeded.turn, null); - assert.equal(seeded.tool, null); - assert.equal(seeded.lastEventAt, 12345); - assert.equal(seeded.lastOutcome, "completed"); -}); - -test("seedActivity: goal-running state seeds a running (between) activity", () => { - const seeded = seedActivity({ id: "s1", state: "goal-running", last_activity_at: new Date(0).toISOString() }); - assert.equal(seeded.phase, "between"); - assert.deepEqual(reify(seeded.turn), { startedAt: null, toolCalls: 0 }); -}); - -test("seedActivity: missing/invalid last_activity_at never fabricates a lastEventAt", () => { - assert.equal(seedActivity({ state: "idle" }).lastEventAt, null); - assert.equal(seedActivity({ state: "idle", last_activity_at: "not-a-date" }).lastEventAt, null); - assert.equal(seedActivity(null).phase, "idle"); -}); - -/* ---------- boardModel ---------- */ - -function activityMap(entries) { - return new Map(Object.entries(entries)); -} - -test("boardModel: queue count is suppressed at 0 and shown when > 0", () => { - const sessions = [ - { id: "a", state: "idle", queued: 0, last_activity_at: new Date(0).toISOString() }, - { id: "b", state: "idle", queued: 3, last_activity_at: new Date(0).toISOString() }, - ]; - const rows = boardModel(sessions, activityMap({}), 0); - const byId = Object.fromEntries(rows.map(r => [r.id, r])); - assert.equal(byId.a.extra, null); - assert.equal(byId.b.extra, "3 queued"); -}); - -test("boardModel: sorts active sessions before idle, and is stable within equal recency", () => { - const now = 100_000; - const sessions = [ - { id: "idle-1", state: "idle", queued: 0, last_activity_at: new Date(now - 1000).toISOString() }, - { id: "live-1", state: "busy", queued: 0, last_activity_at: new Date(now).toISOString() }, - { id: "idle-2", state: "idle", queued: 0, last_activity_at: new Date(now - 500).toISOString() }, - { id: "live-2", state: "busy", queued: 0, last_activity_at: new Date(now).toISOString() }, - ]; - const acts = activityMap({ - "live-1": reduceActivity(null, { type: "text.delta", text: "x" }, now), - "live-2": reduceActivity(null, { type: "text.delta", text: "x" }, now), - "idle-1": seedActivity(sessions[0]), - "idle-2": seedActivity(sessions[2]), - }); - const rows = boardModel(sessions, acts, now); - const order = rows.map(r => r.id); - assert.deepEqual(reify(order), ["live-1", "live-2", "idle-2", "idle-1"]); -}); - -test("boardModel: detail precedence — an active tool wins over stall phase and idle outcome", () => { - const now = 100_000; - const session = { id: "s1", state: "busy", queued: 0, goal: null, last_activity_at: new Date(now).toISOString() }; - let a = reduceActivity(null, { type: "session.status", status: "busy" }, now - 70_000); - a = reduceActivity(a, { type: "tool.start", tool_call: { call_id: "c1", name: "Bash", arguments: { command: "sync_dir" } } }, now - 70_000); - const rows = boardModel([session], activityMap({ s1: a }), now); // 70s silent -> stalled - assert.equal(rows[0].cssClass, "bad"); - assert.equal(rows[0].detail, "Bash · sync_dir"); -}); - -test("boardModel: idle rows show the last turn outcome, or 'worker parked' when the goal is paused on worker failure", () => { - const now = 100_000; - const completed = { id: "s1", state: "idle", queued: 0, last_turn: { outcome: "completed" }, last_activity_at: new Date(now).toISOString() }; - const parked = { - id: "s2", state: "idle", queued: 0, - goal: { condition: "ship it", active: true, paused: true, pause_reason: "worker_failure" }, - last_activity_at: new Date(now).toISOString(), - }; - const rows = boardModel([completed, parked], activityMap({ s1: seedActivity(completed), s2: seedActivity(parked) }), now); - const byId = Object.fromEntries(rows.map(r => [r.id, r])); - assert.equal(byId.s1.detail, "completed"); - assert.equal(byId.s1.detailCritical, false); - assert.equal(byId.s2.detail, "worker parked"); - assert.equal(byId.s2.detailCritical, true); -}); - -/* ---------- boardModel: the "empty" detail cell (RED-FIRST — a zero-message - idle session used to render a blank detail cell, indistinguishable from a - rendering failure; reported via live testing: an operator clicked one - expecting content). Precedence, lowest priority — verified: any outcome, - or a paused goal, still wins; any messages > 0, or a missing/unpolled - messages field, suppresses it. ---------- */ - -test("boardModel: an idle, never-prompted session (messages: 0, no outcome, no goal) shows a dim 'empty' detail cell, not blank", () => { - const now = 100_000; - const fresh = { id: "s1", state: "idle", queued: 0, messages: 0, last_activity_at: new Date(now).toISOString() }; - const row = boardModel([fresh], activityMap({ s1: seedActivity(fresh) }), now)[0]; - assert.equal(row.detail, "empty"); - assert.equal(row.detailCritical, false); -}); - -test("boardModel: 'empty' is suppressed the instant messages > 0", () => { - const now = 100_000; - const used = { id: "s1", state: "idle", queued: 0, messages: 1, last_activity_at: new Date(now).toISOString() }; - const row = boardModel([used], activityMap({ s1: seedActivity(used) }), now)[0]; - assert.equal(row.detail, null); -}); - -test("boardModel: 'empty' never overrides a real outcome or a paused-goal presentation, even when messages is also 0", () => { - const now = 100_000; - const completedButZero = { id: "s1", state: "idle", queued: 0, messages: 0, last_turn: { outcome: "completed" }, last_activity_at: new Date(now).toISOString() }; - const parkedButZero = { - id: "s2", state: "idle", queued: 0, messages: 0, - goal: { condition: "ship it", active: true, paused: true, pause_reason: "worker_failure" }, - last_activity_at: new Date(now).toISOString(), - }; - const rows = boardModel([completedButZero, parkedButZero], activityMap({ s1: seedActivity(completedButZero), s2: seedActivity(parkedButZero) }), now); - const byId = Object.fromEntries(rows.map(r => [r.id, r])); - assert.equal(byId.s1.detail, "completed"); - assert.equal(byId.s2.detail, "worker parked"); -}); - -test("boardModel: a session this page has only seen via a live stub (no `messages` field yet, before the next poll) does not show 'empty'", () => { - const now = 100_000; - const unpolled = { id: "s1", state: "idle", queued: 0, last_activity_at: new Date(now).toISOString() }; // no `messages` field at all - const row = boardModel([unpolled], activityMap({ s1: seedActivity(unpolled) }), now)[0]; - assert.equal(row.detail, null, "an undefined messages count must not be guessed as zero"); -}); - -/* ---------- detail live-chip: fresh-idle phase + the hidden-attribute CSS - bug that actually made it visible (BUG 1) ---------- - - index.html's updateDetailHeader (outside the TESTABLE region — it touches - the DOM) derives the chip from the exact same boardModel call the board's - own syncBoard makes: `boardModel([sessionSummary], state.activities, - nowMs)[0]`, then sets `chip.hidden = row.cssClass === "idle"`. The two - tests below cover the two, independently-necessary halves of that being - correct end to end — the pure model computation (testable in the vm - sandbox) AND the CSS that has to actually respect `chip.hidden` for the - model's "idle" answer to be visible (not testable in the vm sandbox — - there is no DOM/CSS engine there — so this asserts against the raw - stylesheet text instead; see the comment below for why a naive jsdom - getComputedStyle check would not have caught this). */ - -test("detail chip: a freshly created, zero-event session resolves through boardModel (the SAME call updateDetailHeader makes) to cssClass 'idle', matching the board row", () => { - // Mirrors updateDetailHeader's own fallback exactly: a session id not yet - // seen in a GET /session poll response, so it falls back to a minimal - // { id, state: "idle", queued: 0 } summary — and no activity has been - // folded for it yet (activities is empty), so boardModel's activityFor - // falls through to seedActivity(session) — the "zero messages, never - // polled, never streamed to" state a just-opened detail view starts in. - const now = 1_000_000; - const sessionSummary = { id: "s-fresh", state: "idle", queued: 0 }; - const row = boardModel([sessionSummary], activityMap({}), now)[0]; - assert.equal(row.cssClass, "idle", "a fresh, never-active session must resolve idle, not live/quiet/bad"); - assert.equal(row.phase, "idle"); -}); - -test("detail chip: the .livechip CSS rule must not defeat the `hidden` attribute", () => { - // ROOT CAUSE of BUG 1 (verified against a real Chrome tab, not just read): - // the browser's UA stylesheet hides a [hidden] element via a PLAIN (non- - // !important) `[hidden] { display: none }` rule. CSS cascade origin - // outranks specificity: ANY author-stylesheet rule that sets `display` on - // the same element — regardless of that rule's own selector's specificity - // — beats a user-agent-origin rule. index.html used to declare - // `.detail-head .livechip { display: inline-flex; ... }` unconditionally, - // which defeated `chip.hidden = true` outright: in a real browser the - // element stayed fully visible (getComputedStyle().display === - // "inline-flex", non-null offsetParent) with `hidden` set, showing - // whatever text (or the markup's original hardcoded "tool running" - // default) was last assigned — exactly the reported symptom on a freshly - // opened, idle session's detail view. jsdom's own getComputedStyle does - // NOT reproduce this (it special-cases `hidden` rather than emulating the - // real cascade), which is exactly why this had to be found by live - // browser testing and why this regression check is a text/regex - // assertion against the shipped CSS, not a DOM-rendering one. - const styleMatch = html.match(/