feat(roadmap): a trace of every run, kept and shown in the PR comment and in explain #3032

What happened

A stopped Job leaves 40 lines of log with lines like Tool call: shell, program other, refused. The PR comment of the stop on #2175 shows only the last mypy error. A person cannot see what the agent tried, which model call failed, or how many tokens each step used.

What should happen

• Each stage or implement run writes a trace record: ordered events (model call with model name, tokens in and out, duration; tool call with name, arguments digest and result class; check run with command and exit code; retry reason).
• The record is kept under the item's job record (size bounded) and as a Job artifact, and devops roadmap explain <issue> --trace prints it. The PR comment and the stop comment show a summary table.

Acceptance criteria

• A run of the implement job writes the trace with the fields above.
• devops roadmap explain <issue> --trace prints the last run's trace.
• The stop comment shows the step, the attempt count, the tokens and the last failing command.
• The trace holds no secret and no private host name (masking test).

Related: tracker for this set (filed with it), #2908 (issue workflow), #2867 (draft PR evaluator), #3005 (task kind), #3010 to #3015 (plan gates), #2919 (stuck-item metrics), #2953 (Ollama parser).

Proposed design

Problem statement

When a roadmap Job stops, a person sees only the last error in the PR comment. The full sequence of model calls, tool calls, check runs, and retries is lost. A trace of every run is needed so a person can see what the agent tried, which call failed, and how many tokens each step used.

Acceptance criteria

Requirement: implement run writes trace

A run of the implement job SHALL write a trace record with ordered events for each model call, tool call, check run, and retry.

Scenario: implement run produces trace

• WHEN the implement job completes one step
• THEN the job record contains a trace array with at least one event of each type that occurred in that step

Requirement: explain prints trace

devops roadmap explain --trace SHALL print the last run's trace to standard output.

Scenario: explain with trace flag

• WHEN a person runs devops roadmap explain 3032 --trace
• THEN the command prints the trace events of the last run in order

Requirement: stop comment shows summary

The stop comment SHALL show the step name, the attempt count, the total tokens, and the last failing command.

Scenario: stop comment summary table

• WHEN the implement job stops after a failed step
• THEN the stop comment contains a table with the step, attempt count, tokens, and last failing command

Requirement: trace masking

The trace record SHALL contain no secret value and no private host name.

Scenario: masking test

• WHEN a trace event contains a value that matches a secret pattern or a private host name pattern
• THEN the stored trace shows a masked placeholder instead of the original value

Existing Tool Considered

No check has run. No open, maintained tool covers this need because it is tied to the project's job architecture: the Kubernetes Runner pod, the Service lanes, and the issue-body job record. The project builds its own code: a trace recorder, storage in the job record and a Job artifact, the explain --trace display, a summary table renderer, and a masking filter. The written cost is about 5000 tokens to build, about 1000 tokens per fix cycle, and 2 to 3 fix cycles, for an elapsed time of 2 to 3 days.

Key questions

• What format does the trace record use?: A JSON array of event objects. Each object has a type field (model_call, tool_call, check_run, retry) and type-specific fields such as model name, tokens in and out, duration, tool name, arguments digest, result class, command, exit code, and retry reason. (best practice) [sources: CONTEXT.md]
  • Chosen: JSON array of typed events. Standard for structured event logs. Easy to parse, filter by type, and render as a table. No external dependency.
  • Rejected: Flat log lines. Loses structure. Hard to filter by event type or extract specific fields for the summary table.
  • Rejected: OpenTelemetry spans. Designed for distributed tracing. Adds a dependency and a collector for a single-process trace. Overkill for this need.
• Where is the trace stored?: Under the item's job record in the issue body, size bounded. Also as a Kubernetes Job artifact for the full record. [sources: CONTEXT.md]
• What is the size bound for the trace in the job record?: 8192 characters. When the bound is exceeded, the oldest events are dropped and the newest are kept. (best practice) [sources: CONTEXT.md]
  • Chosen: 8192 characters. Fits within GitHub's 65536-character issue body limit with room for the rest of the job record. Holds about 40 to 80 events.
  • Rejected: 4096 characters. Too small for a multi-step run with model calls and tool calls.
  • Rejected: Unbounded. Risks exceeding the issue body limit and breaking the job record.
• How are secrets and private host names masked?: A masking filter runs on every event before storage. It replaces values matching secret patterns (bearer tokens, API keys, passwords) and private host name patterns (internal DNS suffixes) with a [masked] placeholder. (best practice) [sources: CONTEXT.md]
  • Chosen: Regex-based masking filter. Deterministic, testable, no external dependency. Runs in the Runner pod before the event is written.
  • Rejected: External secret scanner. Adds a dependency and a network call. The Runner pod has no network access by design.
  • Rejected: No masking. Violates the acceptance criterion that the trace holds no secret and no private host name.
• What does the stop comment summary table contain?: The step name, the attempt count, the total tokens in and out, and the last failing command with its exit code. [sources: CONTEXT.md]

Fit

Fits in one pull request.

Suspected block

None.

Dependencies

None.