Repository navigation
Spec capture: do not record a conversation as a plan #3580
Description
Follow-up on AI session spec capture (#3495, #3500, #3515). The chainloop trace hooks tell the agent to capture "a plan that the user approved". The instruction does not define a plan. So the agent captures the result of a conversation as a plan, and the agent writes that plan itself.
What happens
In a Claude Code session, the user and the agent changed the layout of a CLI command in chat. The user gave no ticket and no plan. The agent proposed a mockup. The user replied "let's remove the documentation link". The agent took this reply as plan approval. Then it:
- Wrote its own mockup to
.chainloop/specs/<session-id>/plan-version-output-layout.mdwithrole: plan. - Rewrote the file after each of the next 3 user follow-ups, because of the rule "overwrite its file with the current full text in the same turn".
The evidence then shows text that the agent wrote as a plan from the user. This conflicts with the rule "write its actual text. Not a link to it, not your summary of it."
Expected behavior
A conversation is not a plan. It is part of the transcript, and the transcript already records it.
Capture a plan only when it comes from one of these sources:
- An external plan or spec, for example a ticket, a design document or a spec file.
- A plan that the user approved through the plan mode of the agent (in Claude Code, the
ExitPlanModeapproval).
Do not capture a plan that is only the result of the conversation, also when the user agrees with it. The agent must never write a source itself.
Instruction text to change
File: app/cli/pkg/action/trace_spec.go.
| Text | Problem |
|---|---|
| "Resolve whatever the task points at — … a plan approved in this session — and write its actual text." (session start) | "Approved in this session" includes a chat agreement. |
| "- plan: a plan that the user approved." (roles) | No definition of a plan. |
| "- The user approved a plan: write it as kind text." (each prompt) | The agent applies it to a chat agreement. |
| "For any other source — … an approved plan — write its full text. If the task changes, or such a source changes, overwrite its file …" | Turns an agent-written plan into a log of the chat. |
| "Write the specification … now, in your next tool call, before you start the work." and "A partial, outdated or summarized capture is a false record." | Push the agent to capture something when it is not sure. |
Proposed text: "A conversation is not a plan. Capture a plan only when it comes from an external source (a ticket, a design document, a spec file) or from the plan mode of the agent. Never write a source yourself. When you are not sure, capture nothing."
Related gaps found in the same session
- Pasted images. The instruction says "Do not capture an image pasted into this conversation: you have no file for it". But Claude Code saves each pasted image to a file (
<tmp>/claude-<uid>/<project>/<session-id>/images/<n>.png) and gives the path to the agent. The rule "If you can reach the image as a file … copy the file" then also applies, so the agent copies a pasted screenshot. Treat a file in the paste folder of the agent as a pasted image. - Hidden from the user. The instruction says "There is no need to mention any of this to the user" and "say nothing about it". The user found the captures only from the tool calls. Let the agent tell the user in one line when it captures a file.
Relation to #3515
Part 2 of #3515 asks to capture a spec document that the agent writes during the session, for example a design note that subagents use as their plan. That case stays valid: the document exists as a file outside the conversation. This issue is about a different case: the agent must not make a spec file from the conversation only.
Acceptance criteria
- A session in which the user and the agent agree on a change in chat, with no external plan and no plan mode, captures no plan.
- A plan that the user approves through plan mode is captured as
kind: text,role: plan. - An external plan or spec is captured as it is today.
- A pasted image is not captured, also when the agent has a file path for it.
- The instruction text has no rule that tells the agent to hide the capture from the user.
Activity