Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/cli/internal/trace/cursor/announce_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ func TestAnnounceSessionStart(t *testing.T) {
}

// TestAnnouncePromptSubmit pins that Cursor gets no reminder: it documents no
// channel that adds context at prompt submit (D-008 of Spec 003).
// channel that adds context at prompt submit (D-008 of Spec issue-3515).
func TestAnnouncePromptSubmit(t *testing.T) {
out := captureStdout(t, func() {
require.NoError(t, New().AnnouncePromptSubmit("Chainloop spec capture."))
Expand Down
2 changes: 1 addition & 1 deletion app/cli/pkg/action/trace_agent_hook_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ func TestHandleAgentSessionStart(t *testing.T) {
})
}

// TestHandleAgentPromptSubmit covers R-002 of Spec 003: the reminder comes at
// TestHandleAgentPromptSubmit covers R-002 of Spec issue-3515: the reminder comes at
// each user prompt, also in a resumed session whose folder holds files.
func TestHandleAgentPromptSubmit(t *testing.T) {
const sessionID = "abc-123"
Expand Down
2 changes: 1 addition & 1 deletion app/cli/pkg/action/trace_spec_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ func TestSessionSpecReminder(t *testing.T) {
})
}

// TestSessionSpecsAcrossPushes covers R-004 of Spec 003: a file that an
// TestSessionSpecsAcrossPushes covers R-004 of Spec issue-3515: a file that an
// earlier push recorded stays in the evidence of later pushes, also when the
// agent overwrote it and the session went over the limit.
func TestSessionSpecsAcrossPushes(t *testing.T) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ ticket: https://github.com/chainloop-dev/chainloop/issues/3063
prd:
---

# Spec 001: Project and Organization from .chainloop.yml in Attestations
# Spec issue-3063: Project and Organization from .chainloop.yml in Attestations

## Summary

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ ticket: https://github.com/chainloop-dev/chainloop/issues/3495
prd:
---

# Spec 001: Spec capture for AI coding sessions
# Spec issue-3495: Spec capture for AI coding sessions

## Summary
An AI coding session record shows what the agent changed, but not what the user asked for. This spec adds the spec of the session to its evidence. At session start, the trace hook tells the agent to write each source that sets the task into a session folder. At push time, the CLI stores each file as its own material in the attestation. The session material keeps only references to these materials. Reviewers and scoring tools can then compare the change with the request.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ ticket: https://github.com/chainloop-dev/chainloop/issues/3515
prd:
---

# Spec 003: Spec capture during the session
# Spec issue-3515: Spec capture during the session

## Summary
This spec changes the spec capture of [Spec 002](002-session-spec-capture.md). It does not replace it. Today the agent captures the spec only at session start. It also captures the user prompt, which the transcript already holds. With this change, the agent does not capture the user's request prompt, because the transcript holds it. A short reminder at each user turn tells the agent to capture a new spec or image from the user. It also tells the agent to capture an approved plan, and a changed spec again. The limit on spec files for each session goes from 10 to 25.
This spec changes the spec capture of [Spec issue-3495](issue-3495-session-spec-capture.md). It does not replace it. Today the agent captures the spec only at session start. It also captures the user prompt, which the transcript already holds. With this change, the agent does not capture the user's request prompt, because the transcript holds it. A short reminder at each user turn tells the agent to capture a new spec or image from the user. It also tells the agent to capture an approved plan, and a changed spec again. The limit on spec files for each session goes from 10 to 25.

## Problem
- The agent captures the user prompt as a spec of kind `text`. The prompt is the first message of the transcript, so the spec material adds no information. The spec list then shows a duplicate as if it were external context.
Expand All @@ -19,11 +19,11 @@ This spec changes the spec capture of [Spec 002](002-session-spec-capture.md). I
- Goal: the spec materials of a session do not hold the user's request prompt.
- Goal: the evidence holds each spec or image that the user gives after the start, and each plan that the user approves. It holds each changed spec with its content at push time.
- Goal: the reminder is short and narrow, so that it does not overload the session.
- Goal: the agent still decides what a spec is, as Spec 002 decided (D-001 of Spec 002).
- Goal: the agent still decides what a spec is, as Spec issue-3495 decided (D-001 of Spec issue-3495).
- Non-goal: a list of known spec locations, or rules in the CLI that decide if a file is a spec.
- Non-goal: a new attestation when only the specs changed. A push records new specs only with a new AI-assisted commit, as today.
- Non-goal: a spec for a push after the session ends. Spec 002 accepted this loss, and this spec keeps it.
- Non-goal: changes to the materials, the redaction, or the storage that Spec 002 defines.
- Non-goal: a spec for a push after the session ends. Spec issue-3495 accepted this loss, and this spec keeps it.
- Non-goal: changes to the materials, the redaction, or the storage that Spec issue-3495 defines.

## Requirements

Expand All @@ -47,7 +47,7 @@ For a changed spec, the agent MUST overwrite its spec file with the current cont
- Done when: the user approves a plan in the session. The push records that plan as a `text` spec.

### R-004: Higher file limit
The push MUST record at most 25 spec files for each session. When there are more, it MUST record a warning, as Spec 002 does today. The push MUST NOT drop a file that an earlier push of the session recorded, also when the agent overwrote that file later. New files fill the remaining places, oldest first. The order of files on disk changes when the agent overwrites a file, so it MUST NOT decide which files the push keeps.
The push MUST record at most 25 spec files for each session. When there are more, it MUST record a warning, as Spec issue-3495 does today. The push MUST NOT drop a file that an earlier push of the session recorded, also when the agent overwrote that file later. New files fill the remaining places, oldest first. The order of files on disk changes when the agent overwrites a file, so it MUST NOT decide which files the push keeps.
- Done when: a session goes over 25 files after the agent overwrites its starting ticket. The push still records the starting ticket.

## Constraints
Expand All @@ -58,13 +58,13 @@ The push MUST record at most 25 spec files for each session. When there are more
## Proposal
The user does nothing new.

At session start, the trace hook gives the full capture instruction, as in Spec 002. The instruction no longer lists "a written prompt" as a source. It tells the agent that the transcript already holds the conversation, so a prompt is not a spec. Kind `text` stays for a plan that the user approved in the session and for spec text that came from outside the conversation.
At session start, the trace hook gives the full capture instruction, as in Spec issue-3495. The instruction no longer lists "a written prompt" as a source. It tells the agent that the transcript already holds the conversation, so a prompt is not a spec. Kind `text` stays for a plan that the user approved in the session and for spec text that came from outside the conversation.

At each user prompt, the trace hook adds a short reminder to the agent context. The reminder names the session folder and covers four cases. When the user pastes or gives a new spec, copy it. When the user gives an image file or URL, copy it. When the user approves a plan, write it as kind `text`. When a spec changes, copy it again. The reminder says nothing else. A spec that the session writes, for example a design note in a local notes folder, is a spec that changes. Its local path is the source address. The push records the content on disk at push time, so the final version of a plan replaces its drafts.

The reminder replaces the old rule that gave the session-start instruction only while the folder was empty. A resumed session gets the reminder at its first user prompt, so it does not need the full instruction again.

The push keeps the rules of Spec 002, with a limit of 25 files instead of 10. Over the limit, it never drops a file that an earlier push of the session recorded. A push with no new AI-assisted commits still sends no attestation. The next AI-assisted commit records a spec that the agent added after the last push. The session-end hook still deletes the folder.
The push keeps the rules of Spec issue-3495, with a limit of 25 files instead of 10. Over the limit, it never drops a file that an earlier push of the session recorded. A push with no new AI-assisted commits still sends no attestation. The next AI-assisted commit records a spec that the agent added after the last push. The session-end hook still deletes the folder.

Each agent receives the reminder through its own channel:

Expand Down Expand Up @@ -95,22 +95,22 @@ sequenceDiagram
end
User->>CLI: git push
CLI->>Folder: Read up to 25 spec files, earlier recorded files first
CLI->>CLI: Redact, upload, reference (as in Spec 002)
CLI->>CLI: Redact, upload, reference (as in Spec issue-3495)
```

## Decision Record

| ID | Decision | Choice | Why (and what we rejected) | Source |
|----|----------|--------|----------------------------|--------|
| D-001 | Capture of a user prompt | Do not capture the user's request prompt. Capture a spec that the user pastes | The transcript is already part of the session evidence, so the spec material is a duplicate. Rejected: keep it and label it "prompt" in the user interface (more data and UI work for no new information). | drafting |
| D-002 | How the system finds specs after session start | A short reminder at each user prompt, with four cases: a new spec, a new image, an approved plan, a changed spec. The agent decides what a spec is | Simple, and it keeps D-001 of Spec 002. It covers resumed sessions. Four narrow cases keep the reminder short, so it does not overload the session. Rejected: a general rule to capture anything that looks like a spec (too much capture). Rejected: a list of known spec locations that a hook watches (a new configuration, and it misses specs at other paths). Rejected: the hook copies matching files itself (the CLI decides what a spec is, and captures noise). Rejected: a check at push time against the transcript (needs transcript analysis for each agent). Rejected: a reminder after each Markdown write (misses a spec that the agent only reads, and more hook logic). | drafting |
| D-002 | How the system finds specs after session start | A short reminder at each user prompt, with four cases: a new spec, a new image, an approved plan, a changed spec. The agent decides what a spec is | Simple, and it keeps D-001 of Spec issue-3495. It covers resumed sessions. Four narrow cases keep the reminder short, so it does not overload the session. Rejected: a general rule to capture anything that looks like a spec (too much capture). Rejected: a list of known spec locations that a hook watches (a new configuration, and it misses specs at other paths). Rejected: the hook copies matching files itself (the CLI decides what a spec is, and captures noise). Rejected: a check at push time against the transcript (needs transcript analysis for each agent). Rejected: a reminder after each Markdown write (misses a spec that the agent only reads, and more hook logic). | drafting |
| D-003 | A spec added after the last push, with no new AI-assisted commit | Record it with the next AI-assisted commit, as today | An attestation with no new commit is a duplicate of the session. Rejected: push a new attestation when the set of spec digests changed. | drafting |
| D-004 | A push after the session ends | No change: the session-end hook deletes the folder, and that push holds no spec | Accepted in Spec 002. Rejected: keep the folder until the next push, with an age limit (state to track and clean for a rare case). | drafting |
| D-004 | A push after the session ends | No change: the session-end hook deletes the folder, and that push holds no spec | Accepted in Spec issue-3495. Rejected: keep the folder until the next push, with an age limit (state to track and clean for a rare case). | drafting |
| D-005 | Spec file limit | Raise from 10 to 25. Never drop a file that an earlier push recorded. New files fill the remaining places, oldest first | The reminder makes more spec files likely in a long session. The oldest files are the sources the session started from. An overwrite changes the modification time of a file, so an order by modification time can drop an edited start ticket. Rejected: keep the newest files (the start ticket can drop). Rejected: keep the oldest files by modification time only (an overwritten file moves to the end). | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3516#discussion_r4182950724) |
| D-006 | Relation to Spec 002 | This spec changes Spec 002 and does not supersede it | Most of the capture design does not change. | drafting |
| D-006 | Relation to Spec issue-3495 | This spec changes Spec issue-3495 and does not supersede it | Most of the capture design does not change. | drafting |
| D-007 | Control of overcapture from the reminder | Measure the number of spec files for each session after release. Make the reminder narrower if the number grows | The four cases are already narrow. Data from real sessions shows if more limits are necessary. | drafting |
| D-008 | Cursor | The session-start instruction only, until Cursor documents a channel that adds context at prompt submit | No channel to use today. | drafting |
| D-009 | Pasted images | Do not capture a pasted image. Record it as a known limitation | The agent cannot copy the bytes of a pasted image into the folder. A description that the agent writes is not the image, so it adds little value. Rejected: a description of kind `image` (D-010 of Spec 002). This spec changes that decision. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3516#discussion_r4183303763) |
| D-009 | Pasted images | Do not capture a pasted image. Record it as a known limitation | The agent cannot copy the bytes of a pasted image into the folder. A description that the agent writes is not the image, so it adds little value. Rejected: a description of kind `image` (D-010 of Spec issue-3495). This spec changes that decision. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3516#discussion_r4183303763) |

## Open Questions
None.
Expand All @@ -119,7 +119,7 @@ None.

| Risk | Mitigation |
|------|------------|
| The agent ignores the reminder, as it ignored the instruction to add a file when the task changed. | The reminder comes at each turn, near the work, and not only at session start. The capture rate of Spec 002 (R-010) stays measurable. |
| The agent ignores the reminder, as it ignored the instruction to add a file when the task changed. | The reminder comes at each turn, near the work, and not only at session start. The capture rate of Spec issue-3495 (R-010) stays measurable. |
| A spec written and pushed in the same turn is not captured, because the next reminder comes later. | The session-start instruction still says to update the folder when the task changes. The next push with a new commit records it. |
| The reminder adds tokens to each turn. | Keep it to a few lines. The full instruction stays at session start only. |
| Known limitation: the evidence does not hold a pasted image as a spec (D-009). | The transcript still holds the conversation. A later version can take the image bytes from the transcript at push time. |
Loading
Loading