fix(addie): enforce persisted outcomes and clarify recovery telemetry

docs/runbooks/addie-gemini-direct.md

@@ -237,8 +237,29 @@ checks remain synchronous and fail closed. SSE clients receive a content-free
 
 `call_adcp_task` failures now carry structured status, safe code/category,
 retryability, operation, attempt count, and same-turn recovery state through
-tool checkpoints. `recovered_tool_errors` and `unrecovered_tool_errors` should
-be compared separately; do not parse Markdown result prefixes. A typed
+tool checkpoints. `recovered_tool_errors` requires later success with the exact
+operation, agent URL, and idempotency key. `unrecovered_tool_errors` retains its
+historical meaning: every error without that exact recovery.
+
+The admin results also split classified unrecovered errors into
+`contained_tool_errors` and `unresolved_tool_errors`. Containment means a
+validation rejection was followed by a successful AdCP read from the same
+literal agent URL and a complete, unflagged generated answer. For example,
+unsupported `request_proposals` followed by successful product discovery can
+be contained without claiming proposals were created. A corrected
+`get_products` call with a changed key can be contained but is not exact
+recovery. Unrelated targets, arbitrary tools, mutation successes, transport or
+authentication failures, and text-only answers do not establish containment.
+This is observed workflow continuation, not proof of fulfillment or answer
+quality; consult delivery outcomes separately. `unresolved_tool_errors` means
+neither form of evidence was observed. Historical/incomplete turns retain
+NULL containment; their errors without exact recovery are reported separately
+as `tool_recovery_unclassified_errors` and excluded from the contained/unresolved
+split. The four categories (exact, contained, unresolved, unclassified) do not
+overlap: a historical turn with two errors and one exact recovery has one
+unclassified error. This uses total errors minus exact recoveries, since rows
+predating recovery telemetry have default-zero recovery counters. No historical
+backfill is inferred from answer prose. Do not parse Markdown result prefixes. A typed
 `call_adcp_get_products` read surface makes the adapter's stable
 idempotency-key and buying-mode contract machine-checkable without acquiring a
 mutation reservation. One automatic retry is allowed only for typed transient

docs/runbooks/addie-reliability-audit.md

@@ -0,0 +1,101 @@
+---
+title: Addie reliability audit follow-up
+description: "Receipt-backed certification and support claims, recovery telemetry, evaluation lifecycle logs, and deployment guidance for Addie."
+"og:title": "AdCP — Addie reliability audit follow-up"
+---
+
+# Addie reliability audit follow-up
+
+This change starts at `0f4512093` on main, after #7649. No production queries,
+writes, or synthetic evaluations were used to implement or validate it.
+
+## Existing fixes
+
+- #7641 added exact AdCP recovery keyed by operation, literal endpoint, and
+  idempotency key. A corrected call with the same identity already recovers;
+  a different operation or key must not be reported as that operation succeeding.
+  The audit did not include request tuples, so it does not establish why the
+  corrected production call failed to match.
+- #7623 reconciles durable certification completion outcomes. It does not stop
+  later model prose from claiming completion after a rejected attempt.
+- #7649 provides durable evaluation ownership, fenced publication, and observer
+  receipts. Owned admissions and coalesced observations previously lacked
+  distinct lifecycle logs.
+- #7650, which landed after #7649, repairs secure organization creation. It does
+  not address Addie's unsupported escalation promises to anonymous users.
+
+## Response truthfulness
+
+The common terminal response pipeline now checks certification outcomes and
+support promises before streaming or non-streaming delivery. Teaching history
+identifies context only; neither user assertions nor prior assistant statements
+are proof of completion. Current successful certification tool receipts or a
+current `get_learner_progress` read provide authoritative evidence. The parser
+accepts anchored application-owned receipt lines, not arbitrary retrieved text.
+Capstone module receipts are emitted only after the module write/reconciliation
+succeeds, independently of the passed attempt.
+
+Recognized completion, mastery, and credential claims are rendered from exact
+recorded outcomes or replaced with a brief unconfirmed-state explanation. Other
+teaching sentences remain available. Credential award and external issuance are
+separate claims. A later exchange alone never makes a rejected completion valid.
+A historical completion can be confirmed by reading persisted progress again.
+Bare identifiers require teaching context; ordinary Q4 or V2 conclusions do not
+count as module claims. Generic sign-in/tool boilerplate cannot establish that
+context; actual conversation or active certification facts can.
+
+This is a deterministic delivery backstop for the covered English outcome
+phrasing, not a semantic proof for every paraphrase or language. Expanding the
+phrase set should include both false-claim and teaching-flow regressions. Tool
+receipt format changes must update these tests and the parser together.
+
+Support claims require an `escalate_to_admin` receipt containing the persisted
+request ID. Notification claims additionally require successful notification.
+Ancillary thread-flag or notification failures preserve the saved request
+receipt. Anonymous sessions cannot create escalation rows, even if they supply
+contact details. Guest registration complaints receive the public
+`support@agenticadvertising.org` route without soliciting unusable contact data.
+
+## Telemetry and evaluation observations
+
+See [Gemini direct](./addie-gemini-direct.md) for exact, contained, unresolved,
+and historically unclassified error definitions. Containment records successful
+same-agent continuation after a validation rejection; it is not proof of task
+fulfillment, original-operation success, or delivery.
+
+The content-free `agent_quality_evaluation` lifecycle event records admission
+(`owned` or `coalesced`, including expired-lease recovery), committed canonical
+or audit-only publication, and completion/failure/lease-loss/unconfirmed outcomes.
+No URLs, prompts, contact details, credential identities, or lease tokens are
+added to these events. Coalescing remains unproven organically until genuine
+production overlap occurs; that observation does not justify synthetic traffic.
+
+## Deferred isolated outliers
+
+The shadow warning is a best-effort stale-recovery catch in
+`jobs/shadow-evaluator.ts`. Cleanup is retried at subsequent scheduled
+invocations. Heartbeat expiry defaults to 20 minutes and is clamped to 15–60
+minutes, beyond the ten-minute provider timeout. Terminal writes and subsequent
+paid dispatches check ownership. Code and tests do not establish a stale worker
+publication defect; changing leases based on one warning would be speculative.
+
+The persistence duration surrounds `threadService.addMessage`; it does not
+measure provider time or browser receipt. Per-thread lock and statement timeouts
+are bounded at five and ten seconds, with five-second pool acquisition. The
+assistant insert and turn finalization share a transaction, and later route
+errors preserve an already-completed turn. Without the outlier's trace, timing,
+and contention evidence, there is no scoped defect to repair here.
+
+## Deployment and rollback
+
+Migration 607 adds one nullable containment count and consistency constraint.
+The constraint is `NOT VALID`: future writes are checked immediately, while
+startup avoids scanning historical rows under the schema lock. Historical rows
+have the new column null and already satisfy the check. Optional validation must
+run in a separate transaction, not inside the startup migration transaction.
+Apply migrations before the new application version (normal startup ordering).
+Historical rows remain unclassified; there is no speculative backfill. Rollback
+can restore the prior application while retaining the additive column. Do not
+drop it during an application rollback. Old writers leave new containment
+values null. Response enforcement and lifecycle logging need no other schema
+change, tool-surface expansion, model-provider change, or protocol changeset.

server/src/addie/claude-client.ts

@@ -5,6 +5,7 @@
  * with tool reference always appended from code.
  */
 
+import { enforceOutcomeClaims, outcomeClaimContext } from './outcome-claims.js';
 import Anthropic from '@anthropic-ai/sdk';
 import { createHash, createHmac } from 'node:crypto';
 import { createLogger } from '../logger.js';
@@ -334,9 +335,9 @@ export const HALLUCINATION_PATTERNS: ReadonlyArray<{ pattern: RegExp; expectedTo
   { pattern: /(?:I'?ve\s+|I\s+)?(?:created|generated|sent)\s+(?:a\s+)?payment\s+link/i, expectedTools: ['create_payment_link'] },
   { pattern: /(?:I'?ve\s+|I\s+)?(?:sent|delivered)\s+(?:a\s+)?(?:DM|direct message|notification)/i, expectedTools: ['send_member_dm', 'resolve_escalation'] },
   { pattern: /(?:I'?ve\s+|I\s+)?added\s+\S+(?:\s+\S+){0,5}\s+to\s+the\s+(?:meeting|call|series)/i, expectedTools: ['add_meeting_attendee'] },
-  // Fake-escalation patterns. `escalate_to_admin` is in the always-available
-  // tool set, so claiming an escalation/notification was made without firing
-  // it is the same class of fabrication as the rest. Real GitHub-issue tools
+  // Fake-escalation monitoring complements the receipt-bound delivery guard.
+  // Escalation availability depends on the authenticated surface. Claiming
+  // escalation without a successful receipt is fabrication. Real GitHub-issue tools
   // count too because Addie sometimes describes filing a ticket as creating
   // an issue.
   //
@@ -498,6 +499,7 @@ function finalizeAssistantText(
   githubIssueRetryReceipts: readonly GithubIssueRetryReceipt[] = [],
   clientRequestId: string | undefined,
   forceTruncation: boolean = false,
+  conversationContext: string = question,
 ): FinalizedAssistantText {
   // This list is supplied only by the same client-request retry path.
   const retryExecutions = rehydratedGithubIssueRetryExecutions(githubIssueRetryReceipts, clientRequestId);
@@ -522,17 +524,21 @@ function finalizeAssistantText(
       'Addie: Replaced unsupported provider prose after failed source lookups',
     );
   }
+  const outcome = enforceOutcomeClaims(evidenceBoundary.text, toolExecutions, conversationContext);
+  if (outcome.reason) {
+    logger.warn({ event: 'addie_unconfirmed_outcome_replaced', reason: outcome.reason }, 'Addie: Replaced unsupported outcome claim');
+  }
   const processed = applyResponsePipelineWithEmptyMonitoring(
     question,
-    evidenceBoundary.text,
+    outcome.text,
     toolExecutions,
   );
   const lengthExceeded = processed.text.length > MAX_OUTPUT_LENGTH;
   const truncated = forceTruncation || lengthExceeded;
   return {
     text: truncated ? formatTruncatedOutput(processed.text) : processed.text,
     emptyReason: processed.reason,
-    localReplacementReason: githubIssueOutcome.reason ?? evidenceBoundary.reason,
+    localReplacementReason: githubIssueOutcome.reason ?? evidenceBoundary.reason ?? outcome.reason,
     lengthExceeded,
   };
 }
@@ -890,6 +896,7 @@ function grantGeminiDirectBoundaryOpportunity(input: Readonly<{
 
 interface TerminalAddieResponseCommon {
   userMessage: string;
+  conversationContext?: string;
   githubIssueCreationRequested: boolean;
   clientRequestId?: string;
   githubIssueRetryReceipts?: readonly GithubIssueRetryReceipt[];
@@ -937,6 +944,7 @@ function buildTerminalAddieResponse(input: TerminalAddieResponseInput): Terminal
     input.githubIssueRetryReceipts,
     input.clientRequestId,
     input.kind === 'provider' && input.disposition === 'truncated',
+    input.conversationContext,
   );
   const terminalExecutions = [
     ...rehydratedGithubIssueRetryExecutions(input.githubIssueRetryReceipts ?? [], input.clientRequestId),
@@ -2054,6 +2062,7 @@ export class AddieClaudeClient {
           kind: 'provider',
           disposition: 'truncated',
           userMessage,
+          conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
           githubIssueCreationRequested,
           clientRequestId: options?.clientRequestId,
           githubIssueRetryReceipts: options?.githubIssueRetryReceipts,
@@ -2107,6 +2116,7 @@ export class AddieClaudeClient {
           kind: 'provider',
           disposition: 'complete',
           userMessage,
+          conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
           githubIssueCreationRequested,
           clientRequestId: options?.clientRequestId,
           githubIssueRetryReceipts: options?.githubIssueRetryReceipts,
@@ -2170,6 +2180,7 @@ export class AddieClaudeClient {
     const terminal = buildTerminalAddieResponse({
       kind: 'max_iterations',
       userMessage,
+      conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
       githubIssueCreationRequested,
       clientRequestId: options?.clientRequestId,
       githubIssueRetryReceipts: options?.githubIssueRetryReceipts,
@@ -2871,6 +2882,7 @@ export class AddieClaudeClient {
             kind: 'provider',
             disposition: 'truncated',
             userMessage,
+            conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
             githubIssueCreationRequested,
             clientRequestId: options?.clientRequestId,
             githubIssueRetryReceipts: options?.githubIssueRetryReceipts,
@@ -2927,6 +2939,7 @@ export class AddieClaudeClient {
             kind: 'provider',
             disposition: 'complete',
             userMessage,
+            conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
             githubIssueCreationRequested,
             clientRequestId: options?.clientRequestId,
             githubIssueRetryReceipts: options?.githubIssueRetryReceipts,
@@ -2987,6 +3000,7 @@ export class AddieClaudeClient {
       const terminal = buildTerminalAddieResponse({
         kind: 'max_iterations',
         userMessage,
+        conversationContext: outcomeClaimContext([userMessage, ...(threadContext ?? []).map(entry => entry.text)], options?.requestContext),
         githubIssueCreationRequested,
         clientRequestId: options?.clientRequestId,
         githubIssueRetryReceipts: options?.githubIssueRetryReceipts,

server/src/addie/config-version.ts

@@ -30,7 +30,7 @@ import { loadRules, loadResponseStyle } from './rules/index.js';
  * Format: YYYY.MM.N where N is incremented for multiple changes in a month
  * Example: 2025.01.1, 2025.01.2, 2025.02.1
  */
-export const CODE_VERSION = '2026.09.66';
+export const CODE_VERSION = '2026.09.67';
 
 // Types
 export interface ConfigVersion {

server/src/addie/gemini-direct-experiment.ts

@@ -9,6 +9,7 @@ import type { WebChatModelPreference } from './web-chat-model-selection.js';
 import { responseClient } from './response-client.js';
 import { getResponseProviderPolicy, responseProviderModel } from './response-provider-policy.js';
 import { anonymousSessionSubjectHmac } from '../routes/helpers/anonymous-session-capability.js';
+import { countContainedToolErrors } from './model-providers/tool-orchestration.js';
 
 const logger = createLogger('addie-gemini-direct');
 export const GEMINI_DIRECT_EXPERIMENT = 'gemini-3.7-direct-v2';
@@ -192,6 +193,9 @@ class ExperimentTurn {
     const totalToolErrors = this.fallbackToolErrors
       + (response?.tool_executions.filter(tool => tool.is_error).length ?? 0);
     const unrecoveredToolErrors = Math.max(0, totalToolErrors - recoveredToolErrors);
+    const containedToolErrors = countContainedToolErrors(response?.tool_executions ?? [], Boolean(
+      response?.text.trim() && !failed && !response.flagged && !response.output_truncation,
+    ));
     logger.info({
       event: 'addie_response_stage',
       stage: 'provider_and_tools',
@@ -216,7 +220,8 @@ class ExperimentTurn {
         member_context_ms = $24, workos_context_ms = $25,
         experiment_routing_ms = $26, pre_provider_ms = $27,
         provider_ms = $28, tool_ms = $29,
-        recovered_tool_errors = $30, unrecovered_tool_errors = $31
+        recovered_tool_errors = $30, unrecovered_tool_errors = $31,
+        contained_tool_errors = $32
         WHERE id = $1`, [
         this.id, this.firstVisibleMs, Date.now() - this.startedAt, this.routerMs,
         this.providerCalls, costMicros, this.usageComplete, this.fallbackReason,
@@ -240,6 +245,7 @@ class ExperimentTurn {
         response?.timing?.total_tool_execution_ms ?? null,
         recoveredToolErrors,
         unrecoveredToolErrors,
+        containedToolErrors,
       ]);
     } catch (error) {
       logger.error({ error, turnId: this.id }, 'Failed to save Gemini Direct outcome');
@@ -457,6 +463,10 @@ async function getResults(experimentId: string) {
     SUM(tool_errors)::int AS tool_errors,
     SUM(recovered_tool_errors)::int AS recovered_tool_errors,
     SUM(unrecovered_tool_errors)::int AS unrecovered_tool_errors,
+    SUM(contained_tool_errors)::int AS contained_tool_errors,
+    SUM(unrecovered_tool_errors - contained_tool_errors)::int AS unresolved_tool_errors,
+    COUNT(*) FILTER (WHERE contained_tool_errors IS NOT NULL)::int AS tool_recovery_classified_turns,
+    SUM(tool_errors - recovered_tool_errors) FILTER (WHERE contained_tool_errors IS NULL)::int AS tool_recovery_unclassified_errors,
     COUNT(m.rating)::int AS rated_turns, ROUND(AVG(m.rating), 2) AS mean_rating,
     COUNT(*) FILTER (WHERE m.outcome = 'resolved')::int AS resolved_turns,
     CASE WHEN bool_and(usage_complete AND completed_at IS NOT NULL)
@@ -478,6 +488,13 @@ async function getResults(experimentId: string) {
       assignment_version: ANONYMOUS_WEB_ASSIGNMENT_VERSION,
     },
     cohorts: result.rows,
+    tool_recovery_definitions: {
+      recovered_tool_errors: 'Later success with the exact operation, agent URL, and idempotency key.',
+      unrecovered_tool_errors: 'Errors without exact recovery, including contained errors; retained for historical comparison.',
+      contained_tool_errors: 'Validation rejection followed by a successful read from the same agent and a complete generated answer. Observed continuation, not original-operation success or fulfillment; delivery is reported separately.',
+      unresolved_tool_errors: 'Classified errors with neither exact recovery nor observed containment. Absence of evidence, not a judgment of answer quality.',
+      tool_recovery_unclassified_errors: 'Errors without exact recovery on historical or incomplete turns without containment classification; excluded from contained and unresolved counts.',
+    },
   };
 }