[kcenon]

Summary

Every ad-sdlc run now records what each execution attempt cost and how the run ended, in versioned records (schemaVersion: 1) for each attempt, stage, and run. Tool invocations are counted from SDK tool_use blocks (deduplicated by block id) and kept apart from turns (num_turns). Failed, retried, timed-out, and cancelled attempts keep the usage they consumed, and usage the SDK did not report is null, never a measured 0. The records are returned on PipelineResult.metrics and in run --format json; with telemetry consent they are appended to .ad-sdlc/scratchpad/metrics/<session-id>.jsonl. The new ad-sdlc telemetry report aggregates the exports, plus an optional review file a person supplies, into a report where every metric states its source, sample, missing count, and denominator.

The vocabulary reuses #951 and #953 instead of inventing parallel names: stage records carry PipelineStageStatus, run records carry PipelineStatus and the PIPELINE_EXIT_CODES status, and the verification and approval summaries are projections of StageVerificationRecord / PipelineVerificationSummary and ApprovalDecisionRecord / PipelineApprovalSummary. Only attempts add timed_out and cancelled, which are not stage statuses.

Related Issue

Closes #956

Parent: #944. #957 depends on this; #955 attaches measured live results through this report.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• CI/CD changes
• Other (please describe): toolCallCount and tokenUsage deprecated with unchanged values; privacy policy 1.1.0; see Compatibility below

Changes Made

SDK usage observation (src/execution/usage.ts, SdkExecutionAdapter.ts, types.ts)

• SdkUsageObserver reduces the SDK message stream into ExecutionMetrics, reported as the new optional StageExecutionResult.metrics on every outcome: usage { inputTokens, outputTokens, cacheReadInputTokens, cacheCreationInputTokens, source }, turnCount, toolUseCount, toolUses (per tool name), modelId, models, sdkVersion (installed package), claudeCodeVersion (system/init).
• Usage source: result (sum of modelUsage, which covers every model call including subagents, else usage), partial (no usable result: the distinct assistant messages seen so far, a lower bound), recovered (resume reused a committed manifest; no SDK call, so a measured 0), none (every field null). A zeroed error result is not treated as a measurement, because the SDK writes zeroed usage on crash results; previously it replaced the partial totals with zeros.
• Tool invocations: distinct tool_use (and server_tool_use, mcp_tool_use) blocks of assistant messages, deduplicated by block id across the per-block streamed messages that share message.id; subagent blocks count; stream_event partials do not; names that are not identifiers count as other.
• Stop finalization: no SDK Stop hook is registered. The SDK Stop hook does not fire for failed or cancelled executions, so usage is finalized at adapter completion; the TODO in hooks.ts is replaced by that explanation.

Records (src/telemetry/deliveryMetrics.ts, src/ad-sdlc-orchestrator/deliveryMetrics.ts, StageScheduler.ts, AdsdlcOrchestratorAgent.ts)

• Strict zod schemas for attempt, stage, and pipeline records; a record with any undefined field is rejected before it is written. Common fields: schemaVersion, recordedAt, sessionId (kept on resume), runId (one per executePipeline call).
• Attempt: stage, agent, retryIndex, attemptId (the ArtifactContext.attemptId, the same identity as the feat(execution): persist structured artifact manifests from stage execution #950 manifest), sdkSessionId, sdkResume, timestamps and duration, outcome (succeeded | failed | timed_out | cancelled), errorType (code or class name, never the message), execution (sdk | recovered | unobserved), usage, turns, tool uses, models, versions, manifest reference.
• The scheduler reports every attempt through the new optional SchedulerHost.onAttempt; classifyAttemptOutcome() maps a StageTimeoutError (also behind a cleanup failure) to timed_out and pipeline cancellation or EXEC-005 to cancelled. The orchestrator pairs the event with the adapter observation of that attempt. Attempt records are written as attempts end, so a killed run keeps them.
• Stage record: PipelineStageStatus, restored (pre-completed on resume), retries, attempt counts per outcome, sums of usage, turns, and tool uses (usage: null without an attempt), errorType, the feat(vnv): define explicit verification coverage and delivery gate policies #951 verification projection (state, check counts, failed required checks) and the feat(orchestrator): make approval policy explicit and reviewable #953 approval projection (outcome, basis, boundaries, effects, decidedBy, reused; no reason, proposed action, or scope).
• Pipeline record: mode, PipelineStatus, exit status, resume { resumed, source: checkpoint | session, priorStatus } (new OrchestratorSession.resumeSource / resumedFromStatus), stage counts, attempt counts and retries, usage sums including failed and cancelled attempts, models, SDK and package versions, feat(vnv): define explicit verification coverage and delivery gate policies #951 and feat(orchestrator): make approval policy explicit and reviewable #953 summary projections. A run that ends with an unexpected error still writes its attempts, stages, and a failed run record with errorType.

Export and consent (src/telemetry/deliveryMetricsExport.ts, Telemetry.ts, src/cli/runCommand.ts, src/cli.ts)

• OrchestratorConfig.deliveryMetrics { sink?, consent? }: records are always built; they are written only to the sink created for the session. JsonlDeliveryMetricsSink appends validated JSON lines, one write at a time; a failed write is logged, counted in metrics.export.failed, and never changes the run.
• ad-sdlc run reads the existing telemetry consent (Telemetry.getDeliveryMetricsConsent()): the JSONL export is enabled only with consent granted under privacy policy 1.1.0 or later. The policy version is bumped to 1.1.0 and lists the local export; consent recorded under 1.0.0 reports outdated, telemetry enable shows the policy again for it, and telemetry status prints whether runs export delivery metrics. No workflow key or environment variable is added.
• The run's text output ends with a Usage: line (attempts per outcome, tokens with unknown for unreported fields, tool uses, turns) and a Metrics: line (export file, or the consent state). RunCommandDependencies.deliveryMetricsConsent replaces the consent lookup in tests.

Report (src/telemetry/deliveryMetricsReport.ts, src/cli/metricsReportCommand.ts)

• New ad-sdlc telemetry report [files...] [--project-dir <dir>] [--session <id>] [--supplied <file>] [--format text|json]. Without files it reads .ad-sdlc/scratchpad/metrics/*.jsonl. Invalid lines are listed with file and line, duplicates are dropped, and it exits 1 without exports or with unreadable input.
• Metrics, each with source (observed / supplied), sample, missing, and denominator (count and description): completed, verified (feat(vnv): define explicit verification coverage and delivery gate policies #951), resumed, and resumed-after-decision run rates; requested and rejected decisions (feat(orchestrator): make approval policy explicit and reviewable #953); totals and means of duration, the four token fields, turns, and tool uses for all attempts, each outcome, and retried attempts, with a partial count for lower bounds; accepted pull requests, human rework minutes, and requirement evidence coverage from the supplied file only (n/a without it, never inferred from CI). No monetary estimate: no price table is built in.
• CLI_COMMANDS gains telemetry report with a values list for --format; docs/cli.md documents it.

Docs

• New docs/guides/delivery-metrics.md: record schema, outcomes, unknown versus zero, tool-use counting, compatibility, export and consent, report and supplied file, PRD-001 KPI mapping, privacy defaults, and how the live lanes (test(import): prove installed CLI delivery and resume through the real SDK #955, test(enhancement): validate document updates and regressions on an existing project #957) read the records.
• docs/telemetry.md (policy 1.1.0, delivery metrics section, getDeliveryMetricsConsent), docs/cli.md, docs/observability.md (pointer), docs/guides/README.md, README.md CLI table, src/execution/README.md, CHANGELOG.md (Added, Changed, Deprecated).
• The second commit only runs prettier over docs/telemetry.md and docs/observability.md, which were never formatted (the pre-commit hook formats staged Markdown), so the third commit's content diff stays reviewable. The fourth commit makes DeliveryMetricsRecorder.finish log and count a record it cannot build instead of throwing, and reads the session in the error path through getSession(), so a metrics failure can never replace a run's error.

Compatibility

• StageExecutionResult.toolCallCount keeps its value (the larger of distinct assistant messages and num_turns) and is @deprecated in favor of metrics.turnCount and metrics.toolUseCount; the CHANGELOG says it counts turns, not tool invocations.
• tokenUsage keeps its shape and main-loop source and is @deprecated in favor of metrics.usage. One documented change: a failed execution whose result carries zeroed usage keeps the partial assistant totals instead of zeros (also in the manifest's execution.tokenUsage). A result without usage no longer fails the execution with a type error.
• Artifact manifests and the stage output JSON are unchanged; PipelineResult.metrics is not saved in session state.
• Internal reads of the deprecated fields (stage output, manifest execution) carry no-deprecated disables with a reason.

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed (offline demo through executeRunCommand and the report command; transcript in the evidence folder)

New tests (54), all offline:

• tests/execution/usage.test.ts (11) on recorded SDK sequences typed against the installed SDKMessage union (tests/execution/fixtures/usageStreams.ts, checked by npm run test:sdk-contract): success (modelUsage over two models), one message with several tool_use blocks plus a re-emitted block (exactly 5), partial failure with a zeroed error result (partial usage kept), a failure result with measured usage, cancellation mid-stream (partial usage, unknown cache fields, no turn count), a result without usage (all null), a measured zero, subagent and non-identifier tool names, ignored stream events, SDK version lookup.
• tests/execution/SdkExecutionAdapter.metrics.test.ts (7): the adapter reports metrics on success, multi-tool, partial failure, max-turns failure, cancellation through a controlled query, no usage, and loader failure; legacy fields keep their values.
• tests/ad-sdlc-orchestrator/deliveryMetrics.test.ts (9), production scheduler and SdkExecutionAdapter with a controlled query and fake timers: a failed first attempt with partial usage next to the successful retry (stage and run sums include it; sink order), a timed-out attempt, an attempt cancelled by disposal, outcome classification, a failing sink, no sink or a sink that cannot be created, an adapter without metrics, a run ended by an unexpected error, a malformed stage result counted instead of thrown.
• tests/integration/delivery-metrics.integration.test.ts (2), scripted SDK through the real adapter, scheduler, approval gate, and verifier: Import run paused at a decision and resumed (both runs in one export, restored stage, resume source and prior status, tool uses from tool_use blocks, report resume rate), a typed rejection; privacy scan of the export and PipelineResult.metrics for a planted API key, GitHub token, requirement marker, tool inputs, the typed rejection reason, and the project path.
• tests/telemetry/deliveryMetricsReport.test.ts (9): the report on a fixture corpus (tests/telemetry/fixtures/delivery-metrics/: four sessions, five runs including a resumed and an unrecorded run, 14 attempts of every outcome, missing usage, one duplicate, a torn line, a rejected free-text record, a supplied review file) against totals computed by hand (table in the test header).
• tests/telemetry/deliveryMetrics.test.ts (8): schema, null versus 0 through JSON, strict rejection of free text and unknown values, consent states (granted, outdated, pending, denied) including the real consent file, JSONL sink.
• tests/cli/deliveryMetrics.test.ts (8): ad-sdlc run exports with granted consent and writes nothing when consent is pending, denied, or outdated; text summary lines; ad-sdlc telemetry report on project exports, one session, given files with a supplied file, and its failure exits; completion values.
• Updated: tests/completion/CompletionGenerator.test.ts for the report subcommand.

Test Results

env -u ANTHROPIC_API_KEY -u CLAUDE_API_KEY npm test -- --reporter=dot --maxWorkers=4
  Test Files  291 passed | 1 skipped (292)
       Tests  7579 passed | 61 skipped (7640)
npm run test:sdk-contract        pass
npm run build                    pass
npm run lint                     0 errors, 437 warnings (develop: 437; none new)
npm run docs:check               symbols: 117 references in 12 documents; inventory passed (36 agents, 39 slots, 36 components)
npm run format:check             pass
npm run test:package             PACKAGE SMOKE PASS
npm audit --audit-level=high     0 vulnerabilities
scripts/ci/validate-traceability.sh --docs-dir docs   pass, 100% (unchanged)
scripts/ci/check-doc-cascade.sh --base origin/develop pass, 0 warnings
python3 scripts/readme_lint.py README.md              pass
git -c core.whitespace=cr-at-eol diff origin/develop...HEAD --check   clean

Plain git diff origin/develop...HEAD --check reports trailing whitespace only for the CR of added lines in src/ad-sdlc-orchestrator/types.ts, which is stored with CRLF in the index on develop (git ls-files --eol: i/crlf), as in #1007 and #1009.

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

• Not validated: no live SDK run and no real Claude Code process. The SDK message shapes come from the installed SDK's types (0.3.301) and its documented streaming behavior; the recorded sequences are synthetic. The live acceptance lane (test(import): prove installed CLI delivery and resume through the real SDK #955) is expected to attach measured results through ad-sdlc telemetry report. No credentials were used.
• Consent decision: the local export follows the existing telemetry consent record rather than a new setting, and requires consent under the policy version that lists it. Telemetry.isEnabled() is unchanged; it still requires config.enabled, which the CLI never sets, so telemetry status keeps printing Enabled: No beside the new Delivery metrics: line. That predates this change and is listed as a follow-up.
• Requirement evidence coverage is supplied-only for now. An observed source would need requirement-level traceability in the run results, which is feat(vnv): wire requirements traceability and consistency into pipeline results #952.
• docs/telemetry.md still documents a workflow telemetry: block that ad-sdlc validate rejects as retired; that predates this change and is left for docs(product): align product promises and onboarding with verified runtime behavior #960.

[github-actions]

✅ Document Traceability: PASS

Metric	Value
Overall Coverage	100%
FR → SF	33/33 (100%)
SF → CMP	31/31 (100%)
Versions	PRD 1.1.0 / SRS 1.1.0 / SDS 1.2.0