From 9ffb9ae683a25fbeb521935d27bbb2ef0955c627 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 00:35:42 +0200 Subject: [PATCH 1/7] spec: issue-3556 image placeholders in the AI coding session transcript Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/specs/issue-3556-session-image-placeholders.md diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md new file mode 100644 index 000000000..9e18f1900 --- /dev/null +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -0,0 +1,129 @@ +--- +owner: migmartri +ticket: https://github.com/chainloop-dev/chainloop/issues/3556 +prd: +--- + +# Spec issue-3556: Image placeholders in the AI coding session transcript + +## Summary +Today the AI coding session evidence keeps each image of the transcript inline, as base64 text. An image that the agent reads from a file is in the transcript two times. With this change, the CLI removes the image bytes from the transcript at push time and puts a placeholder in their place. The placeholder holds the digest of the image in the CAS. When the image is already a spec material of the same push ([Spec 002](002-session-spec-capture.md)), the placeholder references that material. Otherwise, the CLI uploads the image one time as a separate material. The session evidence gets smaller, the secret redaction does not scan image bytes, and the transcript view can show each image from its placeholder. This change also does the work that D-010 of Spec 002 left for "a later version": it keeps the bytes of pasted images. + +## Problem +- Images make the session evidence too large. In one measured session, the evidence was 13.4 MB, and base64 images were 7.1 MB (53%) of it. The transcript view does not show evidence of this size inline. +- An image that the agent reads with its file tool is in the transcript two times. One copy is in the tool result content, and one is in the tool result metadata. +- Some of these images are also spec materials of the same attestation. The evidence then holds the same image three times. +- The secret redaction scans all transcript text, image bytes included. This is slow. It also gives false matches in base64 data. In one session, the redactor wrote a redaction marker into a JPEG, and the image does not decode now. +- The bytes of a pasted image are only in the transcript. A reader cannot get the image as a file. + +## Goals and Non-Goals +- Goal: the session evidence holds no image bytes, so its size depends on the conversation text only. +- Goal: the secret redaction does not scan image bytes, so it is faster and does not break images. +- Goal: each image keeps its position in the transcript as a placeholder. A transcript view can link the placeholder to the image in the CAS. +- Goal: the evidence holds each image one time. +- Non-goal: the change to the transcript view that shows the images. It is in a different repository. This spec defines only what the view gets. +- Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline images. +- Non-goal: other duplicate data in the transcript, for example repeated attachment blocks in subagent streams. +- Non-goal: image redaction, for example blurring a secret in a screenshot. The CLI stores images as they are, as in Spec 002. + +## Requirements + +### R-001: No image bytes in the transcript +The CLI MUST replace each base64 image in the session transcript with a placeholder before it records the session material. This applies to images that the agent reads from a file and to images that the user pastes. +- Done when: the agent reads one screenshot, and the user pastes one image. The push records a session material with no base64 image data. + +### R-002: Placeholder content +A placeholder MUST stay at the position of the image in the transcript, with the same block type. It MUST hold the digest of the image in the CAS, the media type, and the origin (`file` or `pasted`). It SHOULD hold the original size of the image. A transcript view MUST be able to tell a placeholder from an inline image without other data. + +### R-003: Reuse of spec materials +The agent can read an image from a file that is still on disk at push time. Then the CLI MUST compute the digest of the file. When the digest is equal to the digest of a spec material of the same push, the placeholder MUST reference that material. The CLI MUST NOT upload a second copy. +- Done when: a screenshot that the agent read and also copied into the spec folder gives one material, and the placeholder holds its digest. + +### R-004: Upload of other images +For each other image, the CLI MUST upload the image as an evidence material of the same attestation, and the placeholder MUST hold its digest. The material MUST be linked to the session by an annotation. An image that is in the transcript more than one time MUST give one material. + +### R-005: No secret scan of image bytes +The secret redaction MUST NOT scan image bytes. The CLI MUST remove the bytes before the redaction runs. An uploaded image MUST be stored byte for byte. +- Done when: a pasted image whose base64 text matches a secret pattern decodes after the push. + +### R-006: Failures do not block the push +When the CLI cannot upload an image, the placeholder MUST hold the reason and no digest, and the push MUST continue. The CLI MUST NOT put the bytes back into the transcript. + +### R-007: All agents +The CLI SHOULD apply this to each supported agent that keeps images in its transcript. An agent that keeps no images needs no change. + +## Constraints +- The repository is public. The placeholder format is visible to all users and to external transcript viewers. +- The change is additive. The session schema does not type transcript entries, so the placeholder needs no schema change. A consumer that does not know the placeholder MUST still read the session. +- The transcript view must keep support for evidence that an earlier CLI pushed with inline images. +- The CAS backend limits the size of each material. An image over the limit follows R-006. +- The push runs in a git hook. The extra work must stay small: one digest for each read image, and one upload for each new image. + +## Proposal +The user does nothing new. At push time, after the CLI parses the session and stores the spec materials, it walks the transcript and finds each base64 image. For each image, it decides on a digest: + +1. **Read image that is a spec.** The tool call that read the image keeps the file path. The CLI computes the digest of the file at that path. The agent's file tool can resize the image, so the CLI cannot use the transcript bytes for this match. When the digest is equal to a spec material of this push, the CLI uses it. +2. **Other read image, or pasted image.** The CLI decodes the transcript bytes and uploads them as an evidence material. It keeps a map from digest to material, so a second copy of the same image gives no second upload. + +The CLI then replaces the image data with a placeholder. The image block keeps its type and its position. Its data field holds a Chainloop reference in place of the base64 text: + +| Field | Value | +|-------|-------| +| type | A reference marker, for example `chainloop_material` | +| digest | `sha256:...` of the image in the CAS, or empty when the upload failed | +| media_type | The media type of the image, for example `image/png` | +| origin | `file` or `pasted` | +| size | The original size in bytes | +| error | The reason, only when there is no digest | + +The second copy of a read image, in the tool result metadata, gets the same reference. After this step, the CLI records the session material. The secret redaction then scans only text. + +A transcript view reads a placeholder, finds the material with that digest in the same attestation, and shows the image from the CAS. With no digest, it shows the reason. For evidence from an earlier CLI, it shows the inline image as today. + +```mermaid +flowchart TD + A[Push] --> B[Parse session transcript] + B --> C[Store spec materials] + C --> D{For each base64 image} + D -->|read from file, file on disk| E{File digest equal to a spec material?} + E -->|yes| F[Use spec material digest] + E -->|no| G[Upload image as evidence material] + D -->|pasted, or file gone| G + G -->|ok| H[Use new digest] + G -->|failed| I[Placeholder with reason, no digest] + F --> J[Replace bytes with placeholder] + H --> J + I --> J + J --> K[Record session material, redact text] + K --> L[Push attestation] +``` + +## Decision Record + +| ID | Decision | Choice | Why (and what we rejected) | Source | +|----|----------|--------|----------------------------|--------| +| D-001 | What to do with image bytes | Move them to the CAS, and keep a placeholder with the digest in the transcript | The transcript gets small and keeps the image position. The image stays in the evidence. Rejected: drop the images (the evidence loses what the agent saw). Rejected: remove only the second copy of read images (the transcript is still large, and the redaction still scans the bytes). | drafting | +| D-002 | Where the change runs | In the CLI at push time, before the CLI records the session material | Only the CLI can read the image file on disk for the spec match. The redaction runs when the CLI records the material, so the bytes never reach it. Rejected: in the control plane after the upload (the bytes are already uploaded and scanned, and the file is not on disk). | drafting | +| D-003 | Shape of the placeholder | Replace the image data in place, and keep the block type | A transcript view finds each image where it was, with no join to other data. Rejected: a separate list of images in the session material (a schema change, and the view must join the list to the transcript). | drafting | +| D-004 | Spec match | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript bytes (the file tool can resize the image, so the digest is different). | drafting | + +## Open Questions +- [ ] **Which bytes does the CLI upload for a read image that is not a spec?** Proposed: the transcript bytes. The agent saw these bytes. They are always present, also when the file is gone. The original file can have a higher resolution. But the file can change after the read. +- [ ] **Is there a limit on the number of uploaded images for each push?** Proposed: yes, 50. Over the limit, the placeholder holds the reason and no digest. A long session with many screenshots must not add hundreds of materials to one attestation. +- [ ] **Do uploaded transcript images also become spec entries, with kind `image` and role `reference`?** Proposed: no. They are not specs that the agent chose (D-009 of [Spec 003](003-spec-capture-during-session.md)). They get their own material annotations, so a tool can list them apart from the specs. +- [ ] **Does this apply to other base64 media too, for example a PDF?** Proposed: yes, to all base64 media blocks. The size and redaction problems are the same. + +## Milestones +1. **Claude Code.** The CLI replaces images with placeholders and uploads them. The evidence gets smaller, and the redaction skips image bytes. +2. **Transcript view.** The view shows each image from its placeholder. This is in a different repository. +3. **Other agents.** The same change for each other agent that keeps images in its transcript (R-007). + +## Risks + +| Risk | Mitigation | +|------|------------| +| An attestation gets many new materials, and the materials view gets long. | Each image material has an annotation that links it to the session, so a view can group or hide them. The limit in the open questions caps the number. | +| The image file changed after the read, so the spec match fails. | The CLI uploads the transcript bytes. The evidence holds one more copy, but it is correct. | +| A very large transcript line, with a large image, stops the session parse before the CLI can remove the image. | The parse limit is separate from this change. The CLI can raise it or skip the image data while it reads the line. | +| A transcript view does not know the placeholder and shows raw data. | The placeholder keeps the block type and holds a clear marker, so a view that ignores it shows nothing wrong. Milestone 2 adds the support. | +| An image holds a secret, for example a screenshot of a terminal. | The CLI did not redact images before this change either. It stores images as they are (Spec 002, non-goal). | From ab84fc01755f853fff3a951052c1b2444ff70fdc Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 00:49:18 +0200 Subject: [PATCH 2/7] spec: issue-3556 cover all attachments, attachment-driven replacement Generalize from images to any attachment: never keep inline content that is also an attachment. Start from the attachments, replace their copies in the transcript, then redact. Remove only confirmed attachments, and make the placeholder extensible to other content kinds. Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 148 ++++++++++-------- 1 file changed, 83 insertions(+), 65 deletions(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index 9e18f1900..03fc0122a 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -4,126 +4,144 @@ ticket: https://github.com/chainloop-dev/chainloop/issues/3556 prd: --- -# Spec issue-3556: Image placeholders in the AI coding session transcript +# Spec issue-3556: Attachment placeholders in the AI coding session transcript ## Summary -Today the AI coding session evidence keeps each image of the transcript inline, as base64 text. An image that the agent reads from a file is in the transcript two times. With this change, the CLI removes the image bytes from the transcript at push time and puts a placeholder in their place. The placeholder holds the digest of the image in the CAS. When the image is already a spec material of the same push ([Spec 002](002-session-spec-capture.md)), the placeholder references that material. Otherwise, the CLI uploads the image one time as a separate material. The session evidence gets smaller, the secret redaction does not scan image bytes, and the transcript view can show each image from its placeholder. This change also does the work that D-010 of Spec 002 left for "a later version": it keeps the bytes of pasted images. +Today the AI coding session evidence keeps the content of each file that the agent reads inline in the transcript. A file can also be an attachment of the same attestation, for example a spec material ([Spec 002](002-session-spec-capture.md)). Then the evidence holds the same content two or three times. Images make this worse: they are large base64 text, and the secret redaction scans them. With this change, the CLI never keeps inline content that is also an attachment. At push time, it replaces that content with a placeholder that holds the digest of the attachment. This applies to all file types. The CLI also moves other base64 media, for example pasted images, to new attachments when it can. The session evidence gets smaller, the secret redaction does not scan media bytes, and the transcript view can show each attachment from its placeholder. This change also does the work that D-010 of Spec 002 left for "a later version": it keeps the bytes of pasted images. ## Problem -- Images make the session evidence too large. In one measured session, the evidence was 13.4 MB, and base64 images were 7.1 MB (53%) of it. The transcript view does not show evidence of this size inline. -- An image that the agent reads with its file tool is in the transcript two times. One copy is in the tool result content, and one is in the tool result metadata. -- Some of these images are also spec materials of the same attestation. The evidence then holds the same image three times. -- The secret redaction scans all transcript text, image bytes included. This is slow. It also gives false matches in base64 data. In one session, the redactor wrote a redaction marker into a JPEG, and the image does not decode now. -- The bytes of a pasted image are only in the transcript. A reader cannot get the image as a file. +- Inline media makes the session evidence too large. In one measured session, the evidence was 13.4 MB, and base64 images were 7.1 MB (53%) of it. The transcript view does not show evidence of this size inline. +- A file that the agent reads with its file tool is in the transcript two times. One copy is in the tool result content, and one is in the tool result metadata. +- Some of these files are also spec materials of the same attestation. The evidence then holds the same content three times. +- The secret redaction scans all transcript text, base64 media included. This is slow. It also gives false matches in base64 data. In one session, the redactor wrote a redaction marker into a JPEG, and the image does not decode now. +- The bytes of a pasted image are only in the transcript. A reader cannot download the image as a file. ## Goals and Non-Goals -- Goal: the session evidence holds no image bytes, so its size depends on the conversation text only. -- Goal: the secret redaction does not scan image bytes, so it is faster and does not break images. -- Goal: each image keeps its position in the transcript as a placeholder. A transcript view can link the placeholder to the image in the CAS. -- Goal: the evidence holds each image one time. -- Non-goal: the change to the transcript view that shows the images. It is in a different repository. This spec defines only what the view gets. -- Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline images. +- Goal: the evidence never holds the same content both inline in the transcript and as an attachment, for any file type. +- Goal: the evidence holds as little base64 media inline as possible. Most media moves to attachments. +- Goal: the secret redaction does not scan base64 media, so it is faster and does not break media. +- Goal: each replaced block keeps its position in the transcript as a placeholder. A transcript view can link the placeholder to the attachment in the CAS. +- Non-goal: the change to the transcript view that shows the attachments. It is in a different repository. This spec defines only what the view gets. +- Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline content. +- Non-goal: moving text files that are not attachments out of the transcript. They stay inline, so reviewers and policies can read them. A later spec can move large text files out, with redaction of the new attachment. - Non-goal: other duplicate data in the transcript, for example repeated attachment blocks in subagent streams. -- Non-goal: image redaction, for example blurring a secret in a screenshot. The CLI stores images as they are, as in Spec 002. +- Non-goal: image redaction, for example blurring a secret in a screenshot. The CLI stores media as it is, as in Spec 002. ## Requirements -### R-001: No image bytes in the transcript -The CLI MUST replace each base64 image in the session transcript with a placeholder before it records the session material. This applies to images that the agent reads from a file and to images that the user pastes. -- Done when: the agent reads one screenshot, and the user pastes one image. The push records a session material with no base64 image data. +### R-001: No inline copy of an attachment +The CLI MUST NOT keep inline transcript content that is also an attachment of the same attestation. It MUST replace each copy of that content with a placeholder before it records the session material. This applies to all file types: images, documents, and text. +- Done when: the agent reads a JSON file and a screenshot that are also in the spec folder. The session material holds a placeholder for each copy of both files, and no inline content of them. ### R-002: Placeholder content -A placeholder MUST stay at the position of the image in the transcript, with the same block type. It MUST hold the digest of the image in the CAS, the media type, and the origin (`file` or `pasted`). It SHOULD hold the original size of the image. A transcript view MUST be able to tell a placeholder from an inline image without other data. +A placeholder MUST stay at the position of the replaced block in the transcript, with the same block type. It MUST hold the digest of the attachment in the CAS, the media type, and the origin (`file` or `pasted`). It SHOULD hold the original size. A transcript view MUST be able to tell a placeholder from inline content without other data. -### R-003: Reuse of spec materials -The agent can read an image from a file that is still on disk at push time. Then the CLI MUST compute the digest of the file. When the digest is equal to the digest of a spec material of the same push, the placeholder MUST reference that material. The CLI MUST NOT upload a second copy. +### R-003: Match with spec materials +When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of a spec material of the same push, the placeholder MUST reference that material. The CLI MUST NOT upload a second copy. A partial read of a file, for example a range of lines, is not a copy of the attachment and stays inline. - Done when: a screenshot that the agent read and also copied into the spec folder gives one material, and the placeholder holds its digest. -### R-004: Upload of other images -For each other image, the CLI MUST upload the image as an evidence material of the same attestation, and the placeholder MUST hold its digest. The material MUST be linked to the session by an annotation. An image that is in the transcript more than one time MUST give one material. +### R-004: Upload of other base64 media +Some base64 media blocks are not specs, for example a pasted image. For each of these blocks, the CLI SHOULD upload the bytes as an evidence material of the same attestation. Then the placeholder MUST hold its digest. The material MUST be linked to the session by an annotation. Content that is in the transcript more than one time MUST give one material. -### R-005: No secret scan of image bytes -The secret redaction MUST NOT scan image bytes. The CLI MUST remove the bytes before the redaction runs. An uploaded image MUST be stored byte for byte. +### R-005: No secret scan of base64 media +The secret redaction MUST NOT scan base64 media data, also when the data stays inline. An uploaded media file MUST be stored byte for byte. - Done when: a pasted image whose base64 text matches a secret pattern decodes after the push. ### R-006: Failures do not block the push -When the CLI cannot upload an image, the placeholder MUST hold the reason and no digest, and the push MUST continue. The CLI MUST NOT put the bytes back into the transcript. +When the CLI cannot upload a media block, the block MUST stay inline, and the push MUST continue. This does not break R-001, because the content is not an attachment. ### R-007: All agents -The CLI SHOULD apply this to each supported agent that keeps images in its transcript. An agent that keeps no images needs no change. +The CLI SHOULD apply this to each supported agent that keeps file content or media in its transcript. + +### R-008: Extensible placeholder +The placeholder format MUST NOT be specific to images or files. It MUST hold a content kind and a format version. A later version MUST be able to add a content kind with no change to the existing kinds. An example is a code snippet that is a range of an attachment. A transcript view MUST show a general placeholder for a kind that it does not know, and MUST ignore fields that it does not know. + +### R-009: Remove only known attachments +The CLI MUST NOT remove data from the transcript unless the data is with certainty an attachment of the same attestation. The attachment MUST exist in the CAS before the CLI replaces the block. The digest MUST match exactly. When the CLI is not sure, for example the file is gone, the file changed, or the upload failed, the data MUST stay inline. +- Done when: a screenshot that the agent read and then changed on disk stays inline, or goes up as a new material. It never becomes a placeholder for the spec material. ## Constraints - The repository is public. The placeholder format is visible to all users and to external transcript viewers. - The change is additive. The session schema does not type transcript entries, so the placeholder needs no schema change. A consumer that does not know the placeholder MUST still read the session. -- The transcript view must keep support for evidence that an earlier CLI pushed with inline images. -- The CAS backend limits the size of each material. An image over the limit follows R-006. -- The push runs in a git hook. The extra work must stay small: one digest for each read image, and one upload for each new image. +- The transcript view must keep support for evidence that an earlier CLI pushed with inline content. +- The CAS backend limits the size of each material. A media block over the limit follows R-006. +- Policies read the session evidence. Text that is not an attachment stays inline, so these policies keep working. +- The push runs in a git hook. The extra work must stay small: one digest for each full file read, and one upload for each new media block. ## Proposal -The user does nothing new. At push time, after the CLI parses the session and stores the spec materials, it walks the transcript and finds each base64 image. For each image, it decides on a digest: +The user does nothing new. The attachments drive the change. At push time, after the CLI parses the session, it works in three steps: -1. **Read image that is a spec.** The tool call that read the image keeps the file path. The CLI computes the digest of the file at that path. The agent's file tool can resize the image, so the CLI cannot use the transcript bytes for this match. When the digest is equal to a spec material of this push, the CLI uses it. -2. **Other read image, or pasted image.** The CLI decodes the transcript bytes and uploads them as an evidence material. It keeps a map from digest to material, so a second copy of the same image gives no second upload. +1. **Collect the attachments.** The spec materials of this push are attachments ([Spec 002](002-session-spec-capture.md)). The CLI also extracts each base64 media block that is not a spec, for example a pasted image, and uploads it as an evidence material. It keeps one material for each digest, so a second copy of the same content gives no second upload. When an upload fails, that block is not an attachment and stays inline. +2. **Find each attachment in the transcript, and replace it.** For each attachment, the CLI finds the transcript blocks that hold its content. It replaces each block with a placeholder. For a file that the agent read, the tool call keeps the file path. The CLI computes the digest of the file at that path. It compares this digest with the source digest of the attachment. The agent's file tool can change the content, for example it resizes images and adds line numbers to text. So the CLI cannot compare the transcript copy. For an extracted media block, the CLI already knows the block. Text that is not an attachment stays inline, as today. +3. **Redact.** The CLI records the session material, and the secret redaction scans it. The attachment copies are gone, so the redaction scans less text. It also skips base64 media data that stays inline. -The CLI then replaces the image data with a placeholder. The image block keeps its type and its position. Its data field holds a Chainloop reference in place of the base64 text: +Each placeholder keeps the type and the position of the replaced block. Its data holds a Chainloop reference in place of the content: | Field | Value | |-------|-------| | type | A reference marker, for example `chainloop_material` | -| digest | `sha256:...` of the image in the CAS, or empty when the upload failed | -| media_type | The media type of the image, for example `image/png` | +| version | The placeholder format version | +| kind | The content kind: `media` or `file` in this version. A later version can add kinds, for example `snippet` | +| digest | `sha256:...` of the attachment in the CAS | +| media_type | For example `image/png` or `application/json` | | origin | `file` or `pasted` | | size | The original size in bytes | -| error | The reason, only when there is no digest | -The second copy of a read image, in the tool result metadata, gets the same reference. After this step, the CLI records the session material. The secret redaction then scans only text. +The second copy of a file read, in the tool result metadata, gets the same reference. + +For each content kind, the CLI has one finder. A finder takes an attachment and returns the transcript blocks that hold its content. A new kind, for example a code snippet from a file, adds a finder. When necessary, it also adds its own fields in the placeholder, such as a line range. The other kinds and the transcript view do not change (R-008). -A transcript view reads a placeholder, finds the material with that digest in the same attestation, and shows the image from the CAS. With no digest, it shows the reason. For evidence from an earlier CLI, it shows the inline image as today. +A transcript view reads a placeholder, finds the material with that digest in the same attestation, and shows it from the CAS. For evidence from an earlier CLI, it shows the inline content as today. ```mermaid flowchart TD A[Push] --> B[Parse session transcript] B --> C[Store spec materials] - C --> D{For each base64 image} - D -->|read from file, file on disk| E{File digest equal to a spec material?} - E -->|yes| F[Use spec material digest] - E -->|no| G[Upload image as evidence material] - D -->|pasted, or file gone| G - G -->|ok| H[Use new digest] - G -->|failed| I[Placeholder with reason, no digest] - F --> J[Replace bytes with placeholder] - H --> J - I --> J - J --> K[Record session material, redact text] - K --> L[Push attestation] + B --> D[Extract other base64 media] + D --> E{Upload ok?} + E -->|no| K[Block stays inline] + E -->|yes| F[New evidence material] + C --> G[Attachments] + F --> G + G --> H[For each attachment, the finder of its kind finds its blocks in the transcript] + H --> J[Replace each block with a placeholder] + J --> L[Record session material, then redact text and skip media] + K --> L + L --> M[Push attestation] ``` ## Decision Record | ID | Decision | Choice | Why (and what we rejected) | Source | |----|----------|--------|----------------------------|--------| -| D-001 | What to do with image bytes | Move them to the CAS, and keep a placeholder with the digest in the transcript | The transcript gets small and keeps the image position. The image stays in the evidence. Rejected: drop the images (the evidence loses what the agent saw). Rejected: remove only the second copy of read images (the transcript is still large, and the redaction still scans the bytes). | drafting | -| D-002 | Where the change runs | In the CLI at push time, before the CLI records the session material | Only the CLI can read the image file on disk for the spec match. The redaction runs when the CLI records the material, so the bytes never reach it. Rejected: in the control plane after the upload (the bytes are already uploaded and scanned, and the file is not on disk). | drafting | -| D-003 | Shape of the placeholder | Replace the image data in place, and keep the block type | A transcript view finds each image where it was, with no join to other data. Rejected: a separate list of images in the session material (a schema change, and the view must join the list to the transcript). | drafting | -| D-004 | Spec match | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript bytes (the file tool can resize the image, so the digest is different). | drafting | +| D-001 | What to do with media bytes | Move them to the CAS, and keep a placeholder with the digest in the transcript | The transcript gets small and keeps the position of each block. The media stays in the evidence. Rejected: drop the media (the evidence loses what the agent saw). Rejected: remove only the second copy of read files (the transcript is still large, and the redaction still scans the bytes). | drafting | +| D-002 | Where the change runs | In the CLI at push time, before the CLI records the session material | Only the CLI can read the file on disk for the match. The redaction runs when the CLI records the material, so it never sees the replaced bytes. Rejected: in the control plane after the upload (the bytes are already uploaded and scanned, and the file is not on disk). | drafting | +| D-003 | Shape of the placeholder | Replace the content in place, and keep the block type | A transcript view finds each block where it was, with no join to other data. Rejected: a separate list of attachments in the session material (a schema change, and the view must join the list to the transcript). | drafting | +| D-004 | Match with spec materials | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript copy (the file tool changes the content, so the digest is different). | drafting | +| D-005 | Main rule | No inline copy of an attachment, for all file types. Moving other media out is a SHOULD | A full image cleanup is not always possible, for example when an upload fails. A duplicate is never necessary. Rejected: images only (a JSON or PDF spec file has the same duplicate). Rejected: no media bytes inline as a MUST (a failed upload would lose the media). | owner review | +| D-006 | Failed upload | The media block stays inline | The content is not an attachment, so the inline copy is the only copy. R-005 keeps the redaction from breaking it. Rejected: a placeholder with no digest (the evidence loses the media). | owner review | +| D-007 | Text that is not an attachment | Stays inline | Reviewers read it in the transcript, and policies check it. Rejected: move all read text files to redacted attachments (policies lose the content, and the view must download each file). A later spec can move large text files. | owner review | +| D-008 | Extensibility | A placeholder with a content kind and a version, and one finder for each kind | Other inputs, for example code snippets, can use the same mechanism later. Rejected: an image-only placeholder (each new kind would need a new format and a new view). | owner review | +| D-009 | Direction and order | Start from the attachments: for each one, find its copies in the transcript and replace them. Then redact | The set of attachments is small and known. The rule of R-001 is about attachments, so the code follows the rule directly. The redaction runs last, on less text. Rejected: walk each transcript block and decide what it is (each block type needs a rule, also blocks that are never attachments). | owner review | +| D-010 | Certainty before removal | Replace a block only after the attachment is in the CAS and its digest matches exactly | The transcript is evidence. A wrong placeholder loses content with no way back. Rejected: a best-effort match, for example by size or by file name (a wrong match removes content that is not stored anywhere). | owner review | ## Open Questions -- [ ] **Which bytes does the CLI upload for a read image that is not a spec?** Proposed: the transcript bytes. The agent saw these bytes. They are always present, also when the file is gone. The original file can have a higher resolution. But the file can change after the read. -- [ ] **Is there a limit on the number of uploaded images for each push?** Proposed: yes, 50. Over the limit, the placeholder holds the reason and no digest. A long session with many screenshots must not add hundreds of materials to one attestation. -- [ ] **Do uploaded transcript images also become spec entries, with kind `image` and role `reference`?** Proposed: no. They are not specs that the agent chose (D-009 of [Spec 003](003-spec-capture-during-session.md)). They get their own material annotations, so a tool can list them apart from the specs. -- [ ] **Does this apply to other base64 media too, for example a PDF?** Proposed: yes, to all base64 media blocks. The size and redaction problems are the same. +- [ ] **Which bytes does the CLI upload for read media that is not a spec?** Proposed: the transcript bytes. The agent saw these bytes. They are always present, also when the file is gone. The original file can have a higher resolution. But the file can change after the read. +- [ ] **Is there a limit on the number of uploaded media files for each push?** Proposed: yes, 50. Over the limit, the block stays inline (R-006). A long session with many screenshots must not add hundreds of materials to one attestation. +- [ ] **Do uploaded transcript media also become spec entries, with kind `image` and role `reference`?** Proposed: no. They are not specs that the agent chose (D-009 of [Spec 003](003-spec-capture-during-session.md)). They get their own material annotations, so a tool can list them apart from the specs. ## Milestones -1. **Claude Code.** The CLI replaces images with placeholders and uploads them. The evidence gets smaller, and the redaction skips image bytes. -2. **Transcript view.** The view shows each image from its placeholder. This is in a different repository. -3. **Other agents.** The same change for each other agent that keeps images in its transcript (R-007). +1. **Claude Code.** The CLI replaces attachment copies and media with placeholders, and uploads other media. The evidence gets smaller, and the redaction skips media. +2. **Transcript view.** The view shows each attachment from its placeholder. This is in a different repository. +3. **Other agents.** The same change for each other agent that keeps file content or media in its transcript (R-007). ## Risks | Risk | Mitigation | |------|------------| -| An attestation gets many new materials, and the materials view gets long. | Each image material has an annotation that links it to the session, so a view can group or hide them. The limit in the open questions caps the number. | -| The image file changed after the read, so the spec match fails. | The CLI uploads the transcript bytes. The evidence holds one more copy, but it is correct. | -| A very large transcript line, with a large image, stops the session parse before the CLI can remove the image. | The parse limit is separate from this change. The CLI can raise it or skip the image data while it reads the line. | +| An attestation gets many new materials, and the materials view gets long. | Each new material has an annotation that links it to the session, so a view can group or hide them. The limit in the open questions caps the number. | +| The file changed after the read, so the match with the spec fails. | Media goes up as a new material, and text stays inline. The evidence holds one more copy, but it is correct. | +| A policy reads a spec file from the transcript, and now finds a placeholder. | The spec material holds the same content. The policy can read the material by its digest. Text that is not an attachment stays inline (D-007). | +| A very large transcript line, with a large image, stops the session parse before the CLI can replace the image. | The parse limit is separate from this change. The CLI can raise it or skip the media data while it reads the line. | | A transcript view does not know the placeholder and shows raw data. | The placeholder keeps the block type and holds a clear marker, so a view that ignores it shows nothing wrong. Milestone 2 adds the support. | -| An image holds a secret, for example a screenshot of a terminal. | The CLI did not redact images before this change either. It stores images as they are (Spec 002, non-goal). | +| An image holds a secret, for example a screenshot of a terminal. | The CLI did not redact images before this change either. It stores media as it is (Spec 002, non-goal). | From 1d1f0c3b974f58fff30d6e52044ad4e04011fb55 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 01:19:50 +0200 Subject: [PATCH 3/7] spec: issue-3556 limit replacement to spec sources, capture pasted images, minimal pointer Replace only transcript copies of sources in the session spec list. Capture pasted images as spec sources. The placeholder is a chainloop.replaced pointer with the sibling material digest. Address review on the role of new materials. Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 215 +++++++++++------- 1 file changed, 138 insertions(+), 77 deletions(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index 03fc0122a..2dd4b9ab7 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -7,104 +7,163 @@ prd: # Spec issue-3556: Attachment placeholders in the AI coding session transcript ## Summary -Today the AI coding session evidence keeps the content of each file that the agent reads inline in the transcript. A file can also be an attachment of the same attestation, for example a spec material ([Spec 002](002-session-spec-capture.md)). Then the evidence holds the same content two or three times. Images make this worse: they are large base64 text, and the secret redaction scans them. With this change, the CLI never keeps inline content that is also an attachment. At push time, it replaces that content with a placeholder that holds the digest of the attachment. This applies to all file types. The CLI also moves other base64 media, for example pasted images, to new attachments when it can. The session evidence gets smaller, the secret redaction does not scan media bytes, and the transcript view can show each attachment from its placeholder. This change also does the work that D-010 of Spec 002 left for "a later version": it keeps the bytes of pasted images. +The session material lists its sources in a spec list: each spec, skill, and image that the session captured ([Spec 002](002-session-spec-capture.md)). Each entry holds the digest of a sibling material in the same attestation. Today the transcript also holds the content of many of these sources inline. A screenshot that the agent read and captured is in the evidence three times: two base64 copies in the transcript and one material. With this change, the CLI replaces each transcript copy of a source in the spec list with a small pointer to its sibling material. The CLI also captures images that the user pastes as spec sources, so their bytes become files and their transcript copies become pointers too. Content that is not in the spec list stays inline. The secret redaction skips base64 media. The session evidence gets smaller, the redaction is faster and no longer breaks images, and the transcript view can show each source from its pointer. ## Problem - Inline media makes the session evidence too large. In one measured session, the evidence was 13.4 MB, and base64 images were 7.1 MB (53%) of it. The transcript view does not show evidence of this size inline. - A file that the agent reads with its file tool is in the transcript two times. One copy is in the tool result content, and one is in the tool result metadata. -- Some of these files are also spec materials of the same attestation. The evidence then holds the same content three times. +- When the agent also captures that file as a spec, the evidence holds the same content three times. - The secret redaction scans all transcript text, base64 media included. This is slow. It also gives false matches in base64 data. In one session, the redactor wrote a redaction marker into a JPEG, and the image does not decode now. -- The bytes of a pasted image are only in the transcript. A reader cannot download the image as a file. +- The bytes of a pasted image are only in the transcript. A reader cannot download the image as a file, and the spec list does not show it. D-010 of Spec 002 left this for "a later version". ## Goals and Non-Goals -- Goal: the evidence never holds the same content both inline in the transcript and as an attachment, for any file type. -- Goal: the evidence holds as little base64 media inline as possible. Most media moves to attachments. +- Goal: the evidence never holds a source of the spec list both inline in the transcript and as a material, for any file type. +- Goal: each image that the user pastes becomes a spec source, with its own material. - Goal: the secret redaction does not scan base64 media, so it is faster and does not break media. -- Goal: each replaced block keeps its position in the transcript as a placeholder. A transcript view can link the placeholder to the attachment in the CAS. -- Non-goal: the change to the transcript view that shows the attachments. It is in a different repository. This spec defines only what the view gets. +- Goal: each replaced block keeps its position in the transcript as a pointer. A transcript view can link the pointer to the sibling material. +- Goal: the mechanism can take other content kinds later, for example code snippets. +- Non-goal: content that is not in the spec list. For example, a screenshot that the agent read but did not capture stays inline. The agent decides what to capture (Spec 002 D-001). +- Non-goal: the change to the transcript view that shows the pointers. It is in a different repository. This spec defines only what the view gets. - Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline content. -- Non-goal: moving text files that are not attachments out of the transcript. They stay inline, so reviewers and policies can read them. A later spec can move large text files out, with redaction of the new attachment. -- Non-goal: other duplicate data in the transcript, for example repeated attachment blocks in subagent streams. +- Non-goal: other duplicate data in the transcript, for example the rendered copies of agent attachment entries. - Non-goal: image redaction, for example blurring a secret in a screenshot. The CLI stores media as it is, as in Spec 002. ## Requirements -### R-001: No inline copy of an attachment -The CLI MUST NOT keep inline transcript content that is also an attachment of the same attestation. It MUST replace each copy of that content with a placeholder before it records the session material. This applies to all file types: images, documents, and text. -- Done when: the agent reads a JSON file and a screenshot that are also in the spec folder. The session material holds a placeholder for each copy of both files, and no inline content of them. +### R-001: No inline copy of a spec source +The CLI MUST NOT keep inline transcript content that is also a source in the spec list of the session. It MUST replace each copy of that content with a pointer before it records the session material. This applies to all file types: images, documents, and text. +- Done when: the agent reads a screenshot and a JSON file, and captures both as specs. The session material holds a pointer for each copy of both files, and no inline content of them. -### R-002: Placeholder content -A placeholder MUST stay at the position of the replaced block in the transcript, with the same block type. It MUST hold the digest of the attachment in the CAS, the media type, and the origin (`file` or `pasted`). It SHOULD hold the original size. A transcript view MUST be able to tell a placeholder from inline content without other data. +### R-002: Pointer content +A pointer MUST stay at the position of the replaced block in the transcript, with the same block type. It MUST hold only a `chainloop.replaced` marker and the digest of the sibling material. That digest MUST also be in the spec list of the session. The pointer MUST NOT repeat data that the material or the spec list already holds, for example the name, the size, or the role. A reader MUST be able to see from the marker alone that Chainloop replaced the content. -### R-003: Match with spec materials -When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of a spec material of the same push, the placeholder MUST reference that material. The CLI MUST NOT upload a second copy. A partial read of a file, for example a range of lines, is not a copy of the attachment and stays inline. -- Done when: a screenshot that the agent read and also copied into the spec folder gives one material, and the placeholder holds its digest. +### R-003: Match with spec sources +When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of an entry in the spec list, the CLI MUST replace the copies of that read. A partial read of a file, for example a range of lines, is not a copy of the source and stays inline. +- Done when: the agent reads a screenshot and copies it into the spec folder. The push gives one material, and both transcript copies hold its digest. -### R-004: Upload of other base64 media -Some base64 media blocks are not specs, for example a pasted image. For each of these blocks, the CLI SHOULD upload the bytes as an evidence material of the same attestation. Then the placeholder MUST hold its digest. The material MUST be linked to the session by an annotation. Content that is in the transcript more than one time MUST give one material. +### ~~R-004: Upload of other base64 media~~ +Dropped. See D-013 and R-010. ### R-005: No secret scan of base64 media -The secret redaction MUST NOT scan base64 media data, also when the data stays inline. An uploaded media file MUST be stored byte for byte. +The secret redaction MUST NOT scan base64 media data, also when the data stays inline. A media material MUST be stored byte for byte. - Done when: a pasted image whose base64 text matches a secret pattern decodes after the push. ### R-006: Failures do not block the push -When the CLI cannot upload a media block, the block MUST stay inline, and the push MUST continue. This does not break R-001, because the content is not an attachment. +When the CLI cannot store a pasted image, the image MUST stay inline, and the push MUST continue. ### R-007: All agents -The CLI SHOULD apply this to each supported agent that keeps file content or media in its transcript. +The CLI SHOULD apply this to each supported agent that keeps file content or pasted images in its transcript. -### R-008: Extensible placeholder -The placeholder format MUST NOT be specific to images or files. It MUST hold a content kind and a format version. A later version MUST be able to add a content kind with no change to the existing kinds. An example is a code snippet that is a range of an attachment. A transcript view MUST show a general placeholder for a kind that it does not know, and MUST ignore fields that it does not know. +### R-008: Extensible pointer +The pointer format MUST NOT be specific to images or files. A later version MUST be able to add a content kind with optional fields, with no change to the existing pointers. An example is a code snippet that is a range of lines of a source. A transcript view MUST ignore fields that it does not know, and MUST still link the pointer to its material. -### R-009: Remove only known attachments -The CLI MUST NOT remove data from the transcript unless the data is with certainty an attachment of the same attestation. The attachment MUST exist in the CAS before the CLI replaces the block. The digest MUST match exactly. When the CLI is not sure, for example the file is gone, the file changed, or the upload failed, the data MUST stay inline. -- Done when: a screenshot that the agent read and then changed on disk stays inline, or goes up as a new material. It never becomes a placeholder for the spec material. +### R-009: Remove only known sources +The CLI MUST NOT remove data from the transcript unless the data is with certainty a source in the spec list. The material MUST exist in the CAS before the CLI replaces the block. The digest MUST match exactly. When the CLI is not sure, for example the file is gone, the file changed, or the upload failed, the data MUST stay inline. +- Done when: a screenshot that the agent read and then changed on disk stays inline. + +### R-010: Pasted images are spec sources +The user can paste an image in a prompt or in a message during a turn. For each pasted image, the CLI MUST store the image as a spec material. It MUST add an entry with the kind `image` to the spec list. When the spec list already has an entry with the same digest, the CLI MUST use that entry and MUST NOT store a second copy. The same image pasted two times MUST give one material. +- Done when: the user pastes the same image two times. The session has one image material and one spec entry for it, and both transcript copies hold its digest. ## Constraints -- The repository is public. The placeholder format is visible to all users and to external transcript viewers. -- The change is additive. The session schema does not type transcript entries, so the placeholder needs no schema change. A consumer that does not know the placeholder MUST still read the session. +- The repository is public. The pointer format is visible to all users and to external transcript viewers. +- The change is additive. The session schema does not type transcript entries, and the kind and the role of a spec entry are open values. So the change needs no schema change. A consumer that does not know the pointer MUST still read the session. - The transcript view must keep support for evidence that an earlier CLI pushed with inline content. -- The CAS backend limits the size of each material. A media block over the limit follows R-006. -- Policies read the session evidence. Text that is not an attachment stays inline, so these policies keep working. -- The push runs in a git hook. The extra work must stay small: one digest for each full file read, and one upload for each new media block. +- The CAS backend limits the size of each material. A pasted image over the limit follows R-006. +- Policies read the session evidence. Text that is not a spec source stays inline, so these policies keep working. +- The push runs in a git hook. The extra work must stay small: one digest for each full file read, and one upload for each new pasted image. ## Proposal -The user does nothing new. The attachments drive the change. At push time, after the CLI parses the session, it works in three steps: +The user does nothing new. Each push rebuilds the session evidence from the raw transcript, so the result does not depend on earlier pushes. For each session, the CLI works in these steps: + +1. **Capture specs, as today.** The CLI reads the spec folder into the spec list and stores each file as a sibling material. For each entry it also keeps the source digest: the digest of the file before redaction. For an image, the source digest and the material digest are the same. +2. **Capture pasted images.** The CLI finds each pasted image in the transcript of the main agent and of each subagent. A pasted image can be in a user prompt or in a message that the user sends during a turn. It decodes the bytes and computes the digest. When the spec list has that digest, the CLI uses that entry. Otherwise it stores the image as a spec material and adds an entry. When the upload fails, the image stays inline. +3. **Find and replace.** For each content kind, the CLI has one finder. A finder returns the transcript blocks that hold a source of the spec list. + - The pasted image finder returns the blocks of step 2. + - The file read finder looks at each full read of a file. The tool call keeps the file path. The CLI computes the digest of the file at that path, one time for each path. When the digest is equal to a source digest in the spec list, the finder returns both copies of the read. The agent's file tool can change the content, for example it resizes images and adds line numbers to text. So the CLI does not compare the transcript copy. + + The CLI replaces each returned block with a pointer. +4. **Redact.** The CLI records the session material, and the secret redaction scans it. The source copies are gone, so the redaction scans less text. It skips base64 media that stays inline. + +A pasted image has no file name, so its material gets a number, for example `spec-b5bf8b-pasted-1`. It has the same annotations as the other spec materials. Its bytes were already in the attestation, as base64 in the session material. They now move to a material of their own, which is about 25% smaller than the base64 text. A reader can download it as a file. + +### Pointer format +A pointer holds two fields: + +```json +{ "type": "chainloop.replaced", "digest": "sha256:99c0bca3..." } +``` + +- `type`: always `chainloop.replaced`. It tells a person, a policy, or a view that Chainloop removed the content here. +- `digest`: the digest of the sibling material. The spec list has an entry with the same digest. + +The material and the spec list hold the name, the size, the media type, and the role. The pointer does not repeat them. The position tells the origin. A tool result is a file that the agent read. A user message is a pasted image. + +The CLI puts the pointer where the content was. A content block keeps its type, and only its data changes. When the transcript holds the content in a metadata field, the CLI removes that field and adds a `chainloop.replaced` field next to it. A consumer that reads the old field does not get an object where it expects text. -1. **Collect the attachments.** The spec materials of this push are attachments ([Spec 002](002-session-spec-capture.md)). The CLI also extracts each base64 media block that is not a spec, for example a pasted image, and uploads it as an evidence material. It keeps one material for each digest, so a second copy of the same content gives no second upload. When an upload fails, that block is not an attachment and stays inline. -2. **Find each attachment in the transcript, and replace it.** For each attachment, the CLI finds the transcript blocks that hold its content. It replaces each block with a placeholder. For a file that the agent read, the tool call keeps the file path. The CLI computes the digest of the file at that path. It compares this digest with the source digest of the attachment. The agent's file tool can change the content, for example it resizes images and adds line numbers to text. So the CLI cannot compare the transcript copy. For an extracted media block, the CLI already knows the block. Text that is not an attachment stays inline, as today. -3. **Redact.** The CLI records the session material, and the secret redaction scans it. The attachment copies are gone, so the redaction scans less text. It also skips base64 media data that stays inline. +**A screenshot that the agent read and captured.** Before, the tool result holds the image two times: -Each placeholder keeps the type and the position of the replaced block. Its data holds a Chainloop reference in place of the content: +```json +{ "type": "user", + "message": { "role": "user", "content": [ + { "type": "tool_result", "tool_use_id": "toolu_01A", "content": [ + { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "iVBORw0KGgo..." } } + ] } ] }, + "toolUseResult": { "type": "image", + "file": { "base64": "iVBORw0KGgo...", "type": "image/png", "originalSize": 316801 } } } +``` + +After, the image block keeps the type `image`, and its source is the pointer: + +```json +{ "type": "user", + "message": { "role": "user", "content": [ + { "type": "tool_result", "tool_use_id": "toolu_01A", "content": [ + { "type": "image", "source": { "type": "chainloop.replaced", "digest": "sha256:99c0bca3..." } } + ] } ] }, + "toolUseResult": { "type": "image", + "file": { "type": "image/png", "originalSize": 316801, + "chainloop.replaced": { "digest": "sha256:99c0bca3..." } } } } +``` + +**A pasted image.** The text of the prompt stays: + +```json +{ "type": "user", "message": { "role": "user", "content": [ + { "type": "text", "text": "The button is in the wrong place, see this" }, + { "type": "image", "source": { "type": "chainloop.replaced", "digest": "sha256:9b07..." } } +] } } +``` + +The spec list gets an entry for it: -| Field | Value | -|-------|-------| -| type | A reference marker, for example `chainloop_material` | -| version | The placeholder format version | -| kind | The content kind: `media` or `file` in this version. A later version can add kinds, for example `snippet` | -| digest | `sha256:...` of the attachment in the CAS | -| media_type | For example `image/png` or `application/json` | -| origin | `file` or `pasted` | -| size | The original size in bytes | +```json +{ "kind": "image", "digest": "sha256:9b07...", "captured_at": "2026-10-07T21:30:11Z" } +``` -The second copy of a file read, in the tool result metadata, gets the same reference. +**A JSON file that the agent read and captured.** A text tool result has no block with a source. So its content becomes one pointer block: + +```json +"content": [ { "type": "chainloop.replaced", "digest": "sha256:c2a1..." } ] +``` -For each content kind, the CLI has one finder. A finder takes an attachment and returns the transcript blocks that hold its content. A new kind, for example a code snippet from a file, adds a finder. When necessary, it also adds its own fields in the placeholder, such as a line range. The other kinds and the transcript view do not change (R-008). +A later content kind adds a finder and optional fields to the pointer, for example a line range for a code snippet. The examples use the transcript format of Claude Code. Each other agent puts the same pointer in its own block shape (R-007). -A transcript view reads a placeholder, finds the material with that digest in the same attestation, and shows it from the CAS. For evidence from an earlier CLI, it shows the inline content as today. +A transcript view reads a pointer, finds the entry and the material with that digest, and shows the content from the CAS. For evidence from an earlier CLI, it shows the inline content as today. ```mermaid flowchart TD A[Push] --> B[Parse session transcript] - B --> C[Store spec materials] - B --> D[Extract other base64 media] - D --> E{Upload ok?} - E -->|no| K[Block stays inline] - E -->|yes| F[New evidence material] - C --> G[Attachments] - F --> G - G --> H[For each attachment, the finder of its kind finds its blocks in the transcript] - H --> J[Replace each block with a placeholder] + B --> C[Capture spec folder into the spec list] + B --> D[Find pasted images] + D --> E{Digest in the spec list?} + E -->|yes| G[Use the entry] + E -->|no| F{Store as spec material} + F -->|ok| G + F -->|failed| K[Image stays inline] + C --> H[Finders: pasted images and full file reads with a matching digest] + G --> H + H --> J[Replace each copy with a pointer] J --> L[Record session material, then redact text and skip media] K --> L L --> M[Push attestation] @@ -114,34 +173,36 @@ flowchart TD | ID | Decision | Choice | Why (and what we rejected) | Source | |----|----------|--------|----------------------------|--------| -| D-001 | What to do with media bytes | Move them to the CAS, and keep a placeholder with the digest in the transcript | The transcript gets small and keeps the position of each block. The media stays in the evidence. Rejected: drop the media (the evidence loses what the agent saw). Rejected: remove only the second copy of read files (the transcript is still large, and the redaction still scans the bytes). | drafting | +| D-001 | What to do with media bytes | Keep them in a material, and keep a pointer with the digest in the transcript | The transcript gets small and keeps the position of each block. The media stays in the evidence. Rejected: drop the media (the evidence loses what the agent saw). Rejected: remove only the second copy of read files (the transcript is still large, and the redaction still scans the bytes). | drafting | | D-002 | Where the change runs | In the CLI at push time, before the CLI records the session material | Only the CLI can read the file on disk for the match. The redaction runs when the CLI records the material, so it never sees the replaced bytes. Rejected: in the control plane after the upload (the bytes are already uploaded and scanned, and the file is not on disk). | drafting | -| D-003 | Shape of the placeholder | Replace the content in place, and keep the block type | A transcript view finds each block where it was, with no join to other data. Rejected: a separate list of attachments in the session material (a schema change, and the view must join the list to the transcript). | drafting | -| D-004 | Match with spec materials | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript copy (the file tool changes the content, so the digest is different). | drafting | -| D-005 | Main rule | No inline copy of an attachment, for all file types. Moving other media out is a SHOULD | A full image cleanup is not always possible, for example when an upload fails. A duplicate is never necessary. Rejected: images only (a JSON or PDF spec file has the same duplicate). Rejected: no media bytes inline as a MUST (a failed upload would lose the media). | owner review | -| D-006 | Failed upload | The media block stays inline | The content is not an attachment, so the inline copy is the only copy. R-005 keeps the redaction from breaking it. Rejected: a placeholder with no digest (the evidence loses the media). | owner review | -| D-007 | Text that is not an attachment | Stays inline | Reviewers read it in the transcript, and policies check it. Rejected: move all read text files to redacted attachments (policies lose the content, and the view must download each file). A later spec can move large text files. | owner review | -| D-008 | Extensibility | A placeholder with a content kind and a version, and one finder for each kind | Other inputs, for example code snippets, can use the same mechanism later. Rejected: an image-only placeholder (each new kind would need a new format and a new view). | owner review | -| D-009 | Direction and order | Start from the attachments: for each one, find its copies in the transcript and replace them. Then redact | The set of attachments is small and known. The rule of R-001 is about attachments, so the code follows the rule directly. The redaction runs last, on less text. Rejected: walk each transcript block and decide what it is (each block type needs a rule, also blocks that are never attachments). | owner review | -| D-010 | Certainty before removal | Replace a block only after the attachment is in the CAS and its digest matches exactly | The transcript is evidence. A wrong placeholder loses content with no way back. Rejected: a best-effort match, for example by size or by file name (a wrong match removes content that is not stored anywhere). | owner review | +| D-003 | Shape of the pointer | Replace the content in place, and keep the block type | A transcript view finds each block where it was, with no join to other data. Rejected: a separate list of replacements in the session material (the view must join the list to the transcript). | drafting | +| D-004 | Match with spec sources | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript copy (the file tool changes the content, so the digest is different). | drafting | +| D-005 | ~~Main rule: no inline copy of an attachment, and move other media out as a SHOULD~~ | Replaced by D-013 | | owner review | +| D-006 | Failed upload | The pasted image stays inline | The inline copy is then the only copy. R-005 keeps the redaction from breaking it. Rejected: a pointer with no digest (the evidence loses the image). | owner review | +| D-007 | Text that is not a spec source | Stays inline | Reviewers read it in the transcript, and policies check it. Rejected: move all read text files to redacted materials (policies lose the content, and the view must download each file). | owner review | +| D-008 | Extensibility | One finder for each content kind. A later kind adds optional fields to the pointer | Other inputs, for example code snippets, can use the same mechanism later. Rejected: an image-only pointer (each new kind would need a new format and a new view). Rejected: a version and a kind field in each pointer (optional fields are enough, and the material holds the kind). | owner review | +| D-009 | Direction and order | Start from the spec list: find the copies of its sources in the transcript and replace them. Then redact | The spec list is small and known. The rule of R-001 is about these sources, so the code follows the rule directly. The redaction runs last, on less text. Rejected: walk each transcript block and decide what it is (each block type needs a rule, also blocks that are never sources). | owner review | +| D-010 | Certainty before removal | Replace a block only after the material is in the CAS and its digest matches exactly | The transcript is evidence. A wrong pointer loses content with no way back. Rejected: a best-effort match, for example by size or by file name (a wrong match removes content that is not stored anywhere). | owner review | +| D-011 | Pasted images | The CLI captures each pasted image as a spec source | The user gave the image to the session, as the user gives a ticket. Only the CLI can reach the bytes, because the agent sees only the image (Spec 003 D-009). This does the work that Spec 002 D-010 left for a later version. Rejected: leave pasted images inline (they stay large and cannot be downloaded). Rejected: new attachment materials outside the spec list (a second list for a view to join). | owner review | +| D-012 | Content of the pointer | Only the `chainloop.replaced` marker and the digest of the sibling material | The material and its entry in the spec list already hold the name, the size, the kind, and the role. The position of the block tells the origin. The marker name tells any reader that Chainloop replaced the content. Rejected: a full reference with the name, size, media type, origin, kind, and version (it repeats the material). Rejected: the material name as the pointer (a label, while the digest is unique and lets a reader check the content). | owner review | +| D-013 | Scope of the replacement | Only sources in the spec list, including pasted images (D-011) | Each pointer then resolves in the session itself, and the CLI never decides on its own that content is evidence. Rejected: also upload each other base64 media block. An example is a screenshot that the agent read but did not capture. This turns each image that the agent looked at into a material, and the agent did not choose it. | owner review | ## Open Questions -- [ ] **Which bytes does the CLI upload for read media that is not a spec?** Proposed: the transcript bytes. The agent saw these bytes. They are always present, also when the file is gone. The original file can have a higher resolution. But the file can change after the read. -- [ ] **Is there a limit on the number of uploaded media files for each push?** Proposed: yes, 50. Over the limit, the block stays inline (R-006). A long session with many screenshots must not add hundreds of materials to one attestation. -- [ ] **Do uploaded transcript media also become spec entries, with kind `image` and role `reference`?** Proposed: no. They are not specs that the agent chose (D-009 of [Spec 003](003-spec-capture-during-session.md)). They get their own material annotations, so a tool can list them apart from the specs. +- [ ] **Do pasted images get a role?** Proposed: `reference`. Spec 004 defines it as "supporting material, for example a screenshot". The CLI knows that the user gave the image as support. With no role, a view that lists specs shows pasted screenshots next to the ticket. This extends Spec 004 R-005, which forbids a role guess for sources that the agent writes. A new role `attachment` is a conflict, because Claude Code transcripts already have entries of the type `attachment`. +- [ ] **Do pasted images count toward the limit of 25 spec entries for each session (Spec 003)?** Proposed: no. They get a separate limit of 25. A session with many pastes must not push out the specs that the agent chose. ## Milestones -1. **Claude Code.** The CLI replaces attachment copies and media with placeholders, and uploads other media. The evidence gets smaller, and the redaction skips media. -2. **Transcript view.** The view shows each attachment from its placeholder. This is in a different repository. -3. **Other agents.** The same change for each other agent that keeps file content or media in its transcript (R-007). +1. **Claude Code.** The CLI captures pasted images, replaces the copies of spec sources with pointers, and the redaction skips media. +2. **Transcript view.** The view shows each source from its pointer. This is in a different repository. +3. **Other agents.** The same change for each other agent that keeps file content or pasted images in its transcript (R-007). ## Risks | Risk | Mitigation | |------|------------| -| An attestation gets many new materials, and the materials view gets long. | Each new material has an annotation that links it to the session, so a view can group or hide them. The limit in the open questions caps the number. | -| The file changed after the read, so the match with the spec fails. | Media goes up as a new material, and text stays inline. The evidence holds one more copy, but it is correct. | -| A policy reads a spec file from the transcript, and now finds a placeholder. | The spec material holds the same content. The policy can read the material by its digest. Text that is not an attachment stays inline (D-007). | +| The file changed after the read, so the match with the spec fails. | The read stays inline (R-009). The evidence holds one more copy, but it is correct. | +| The agent reads a screenshot but does not capture it, so it stays inline and the evidence stays large. | The capture instruction asks the agent to capture images that the user gives (Spec 003). The share of read images with no spec entry is measurable from the evidence. | +| A policy reads a spec file from the transcript, and now finds a pointer. | The spec material holds the same content. The policy can read the material by its digest. Text that is not a spec source stays inline (D-007). | | A very large transcript line, with a large image, stops the session parse before the CLI can replace the image. | The parse limit is separate from this change. The CLI can raise it or skip the media data while it reads the line. | -| A transcript view does not know the placeholder and shows raw data. | The placeholder keeps the block type and holds a clear marker, so a view that ignores it shows nothing wrong. Milestone 2 adds the support. | +| A transcript view does not know the pointer. | The pointer keeps the block type and holds a clear marker, so the view shows an image with no data, not wrong data. Milestone 2 adds the support. | | An image holds a secret, for example a screenshot of a terminal. | The CLI did not redact images before this change either. It stores media as it is (Spec 002, non-goal). | From a943eb6de9e339b78d53f9a3d1d341a84bf17b72 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 01:21:45 +0200 Subject: [PATCH 4/7] spec: issue-3556 fix inconsistencies in scope and pointer format Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index 2dd4b9ab7..3c0225c9b 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -4,7 +4,7 @@ ticket: https://github.com/chainloop-dev/chainloop/issues/3556 prd: --- -# Spec issue-3556: Attachment placeholders in the AI coding session transcript +# Spec issue-3556: Pointers to spec sources in the AI coding session transcript ## Summary The session material lists its sources in a spec list: each spec, skill, and image that the session captured ([Spec 002](002-session-spec-capture.md)). Each entry holds the digest of a sibling material in the same attestation. Today the transcript also holds the content of many of these sources inline. A screenshot that the agent read and captured is in the evidence three times: two base64 copies in the transcript and one material. With this change, the CLI replaces each transcript copy of a source in the spec list with a small pointer to its sibling material. The CLI also captures images that the user pastes as spec sources, so their bytes become files and their transcript copies become pointers too. Content that is not in the spec list stays inline. The secret redaction skips base64 media. The session evidence gets smaller, the redaction is faster and no longer breaks images, and the transcript view can show each source from its pointer. @@ -17,12 +17,13 @@ The session material lists its sources in a spec list: each spec, skill, and ima - The bytes of a pasted image are only in the transcript. A reader cannot download the image as a file, and the spec list does not show it. D-010 of Spec 002 left this for "a later version". ## Goals and Non-Goals -- Goal: the evidence never holds a source of the spec list both inline in the transcript and as a material, for any file type. +- Goal: a source of the spec list is never both a material and a full file read in the transcript. The same applies to a pasted image. This is true for all file types. - Goal: each image that the user pastes becomes a spec source, with its own material. - Goal: the secret redaction does not scan base64 media, so it is faster and does not break media. - Goal: each replaced block keeps its position in the transcript as a pointer. A transcript view can link the pointer to the sibling material. - Goal: the mechanism can take other content kinds later, for example code snippets. - Non-goal: content that is not in the spec list. For example, a screenshot that the agent read but did not capture stays inline. The agent decides what to capture (Spec 002 D-001). +- Non-goal: other copies of a spec source in the transcript. Examples are the text of the file write that created a spec file, and the text of a skill that the agent loaded. A later finder can replace them (R-008). - Non-goal: the change to the transcript view that shows the pointers. It is in a different repository. This spec defines only what the view gets. - Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline content. - Non-goal: other duplicate data in the transcript, for example the rendered copies of agent attachment entries. @@ -31,14 +32,14 @@ The session material lists its sources in a spec list: each spec, skill, and ima ## Requirements ### R-001: No inline copy of a spec source -The CLI MUST NOT keep inline transcript content that is also a source in the spec list of the session. It MUST replace each copy of that content with a pointer before it records the session material. This applies to all file types: images, documents, and text. +A full file read or a pasted image in the transcript can be a source in the spec list of the session. Then the CLI MUST replace each copy of it with a pointer. It MUST do this before it records the session material. This applies to all file types: images, documents, and text. - Done when: the agent reads a screenshot and a JSON file, and captures both as specs. The session material holds a pointer for each copy of both files, and no inline content of them. ### R-002: Pointer content -A pointer MUST stay at the position of the replaced block in the transcript, with the same block type. It MUST hold only a `chainloop.replaced` marker and the digest of the sibling material. That digest MUST also be in the spec list of the session. The pointer MUST NOT repeat data that the material or the spec list already holds, for example the name, the size, or the role. A reader MUST be able to see from the marker alone that Chainloop replaced the content. +A pointer MUST stay at the position of the replaced content in the transcript. An image block MUST keep its type. In this version, a pointer MUST hold only a `chainloop.replaced` marker and the digest of the sibling material. That digest MUST also be in the spec list of the session. The pointer MUST NOT repeat data that the material or the spec list already holds, for example the name, the size, or the role. A reader MUST be able to see from the marker alone that Chainloop replaced the content. ### R-003: Match with spec sources -When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of an entry in the spec list, the CLI MUST replace the copies of that read. A partial read of a file, for example a range of lines, is not a copy of the source and stays inline. +When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of an entry in the spec list, the CLI MUST replace the copies of that read. The pointer holds the digest of the material. For a text file, the material is the redacted copy, so the two digests differ. A partial read of a file, for example a range of lines, is not a copy of the source and stays inline. A text spec that the agent wrote with a header is not the file that it read. Its digest differs, and the read stays inline. - Done when: the agent reads a screenshot and copies it into the spec folder. The push gives one material, and both transcript copies hold its digest. ### ~~R-004: Upload of other base64 media~~ @@ -97,9 +98,9 @@ A pointer holds two fields: - `type`: always `chainloop.replaced`. It tells a person, a policy, or a view that Chainloop removed the content here. - `digest`: the digest of the sibling material. The spec list has an entry with the same digest. -The material and the spec list hold the name, the size, the media type, and the role. The pointer does not repeat them. The position tells the origin. A tool result is a file that the agent read. A user message is a pasted image. +The material and the spec list hold the name, the size, the media type, and the role. The pointer does not repeat them. The position tells the origin. A tool result is a file that the agent read. A user prompt, or a message that the user sent during a turn, is a pasted image. -The CLI puts the pointer where the content was. A content block keeps its type, and only its data changes. When the transcript holds the content in a metadata field, the CLI removes that field and adds a `chainloop.replaced` field next to it. A consumer that reads the old field does not get an object where it expects text. +The CLI puts the pointer where the content was. An image block keeps its type, and only its source changes. Text content has no block with a source, so the text becomes one pointer block. The transcript can also hold the content in a metadata field. Then the CLI removes that field, and adds a `chainloop.replaced` field with the same pointer next to it. A consumer that reads the old field does not get an object where it expects text. **A screenshot that the agent read and captured.** Before, the tool result holds the image two times: @@ -123,7 +124,7 @@ After, the image block keeps the type `image`, and its source is the pointer: ] } ] }, "toolUseResult": { "type": "image", "file": { "type": "image/png", "originalSize": 316801, - "chainloop.replaced": { "digest": "sha256:99c0bca3..." } } } } + "chainloop.replaced": { "type": "chainloop.replaced", "digest": "sha256:99c0bca3..." } } } } ``` **A pasted image.** The text of the prompt stays: @@ -141,7 +142,7 @@ The spec list gets an entry for it: { "kind": "image", "digest": "sha256:9b07...", "captured_at": "2026-10-07T21:30:11Z" } ``` -**A JSON file that the agent read and captured.** A text tool result has no block with a source. So its content becomes one pointer block: +**A JSON file that the agent read and copied into the spec folder with no change.** Its content becomes one pointer block. The metadata copy of the text gets the same pointer: ```json "content": [ { "type": "chainloop.replaced", "digest": "sha256:c2a1..." } ] @@ -201,6 +202,7 @@ flowchart TD | Risk | Mitigation | |------|------------| | The file changed after the read, so the match with the spec fails. | The read stays inline (R-009). The evidence holds one more copy, but it is correct. | +| The file changed after the read, and the agent then copied the new version into the spec folder. The digest matches, but the agent saw the old version. | When the tool result records the original size, the CLI also compares it with the size of the file. A different size keeps the read inline. | | The agent reads a screenshot but does not capture it, so it stays inline and the evidence stays large. | The capture instruction asks the agent to capture images that the user gives (Spec 003). The share of read images with no spec entry is measurable from the evidence. | | A policy reads a spec file from the transcript, and now finds a pointer. | The spec material holds the same content. The policy can read the material by its digest. Text that is not a spec source stays inline (D-007). | | A very large transcript line, with a large image, stops the session parse before the CLI can replace the image. | The parse limit is separate from this change. The CLI can raise it or skip the media data while it reads the line. | From 8ad55b68994fc3715b466bc7493af5d4ec9b7149 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 01:25:24 +0200 Subject: [PATCH 5/7] spec: issue-3556 replace every exact copy of a spec source, best effort with match report Cover text sources such as tickets through a file write finder. Move pasted images to a later spec. Make the replacement best effort with a debug match report. Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 150 +++++++++--------- 1 file changed, 77 insertions(+), 73 deletions(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index 3c0225c9b..922566ff9 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -7,23 +7,22 @@ prd: # Spec issue-3556: Pointers to spec sources in the AI coding session transcript ## Summary -The session material lists its sources in a spec list: each spec, skill, and image that the session captured ([Spec 002](002-session-spec-capture.md)). Each entry holds the digest of a sibling material in the same attestation. Today the transcript also holds the content of many of these sources inline. A screenshot that the agent read and captured is in the evidence three times: two base64 copies in the transcript and one material. With this change, the CLI replaces each transcript copy of a source in the spec list with a small pointer to its sibling material. The CLI also captures images that the user pastes as spec sources, so their bytes become files and their transcript copies become pointers too. Content that is not in the spec list stays inline. The secret redaction skips base64 media. The session evidence gets smaller, the redaction is faster and no longer breaks images, and the transcript view can show each source from its pointer. +The session material lists its sources in a spec list: each ticket, document, plan, image, and skill that the session captured ([Spec 002](002-session-spec-capture.md)). Each entry holds the digest of a sibling material in the same attestation. Today the transcript also holds copies of these sources. A screenshot that the agent read and captured is in the evidence three times: two base64 copies in the transcript and one material. A ticket that the agent wrote into the spec folder is in the transcript two times, and in one material. With this change, the CLI replaces each exact copy of a source in the spec list with a small pointer to its sibling material. Content that is not with certainty a copy of a source stays inline. The secret redaction skips base64 media. The session evidence gets smaller, the redaction is faster and no longer breaks images, and the transcript view can show each source from its pointer. ## Problem - Inline media makes the session evidence too large. In one measured session, the evidence was 13.4 MB, and base64 images were 7.1 MB (53%) of it. The transcript view does not show evidence of this size inline. - A file that the agent reads with its file tool is in the transcript two times. One copy is in the tool result content, and one is in the tool result metadata. -- When the agent also captures that file as a spec, the evidence holds the same content three times. +- A spec file that the agent writes is in the transcript two times: in the tool input, and in the tool result metadata. +- The spec material holds the same content again. So the evidence holds each captured source up to three times. - The secret redaction scans all transcript text, base64 media included. This is slow. It also gives false matches in base64 data. In one session, the redactor wrote a redaction marker into a JPEG, and the image does not decode now. -- The bytes of a pasted image are only in the transcript. A reader cannot download the image as a file, and the spec list does not show it. D-010 of Spec 002 left this for "a later version". ## Goals and Non-Goals -- Goal: a source of the spec list is never both a material and a full file read in the transcript. The same applies to a pasted image. This is true for all file types. -- Goal: each image that the user pastes becomes a spec source, with its own material. +- Goal: the transcript holds no exact copy of a source in the spec list. This applies to all file types: images, tickets, documents, plans, and other text. - Goal: the secret redaction does not scan base64 media, so it is faster and does not break media. -- Goal: each replaced block keeps its position in the transcript as a pointer. A transcript view can link the pointer to the sibling material. +- Goal: each replaced copy keeps its position in the transcript as a pointer. A transcript view can link the pointer to the sibling material. - Goal: the mechanism can take other content kinds later, for example code snippets. -- Non-goal: content that is not in the spec list. For example, a screenshot that the agent read but did not capture stays inline. The agent decides what to capture (Spec 002 D-001). -- Non-goal: other copies of a spec source in the transcript. Examples are the text of the file write that created a spec file, and the text of a skill that the agent loaded. A later finder can replace them (R-008). +- Non-goal: pasted images. Their bytes are only in the transcript, and they are not in the spec list. A later spec can capture them as spec sources, and this mechanism then replaces them with no change (D-014). +- Non-goal: content that is not an exact copy of a source. Examples are a screenshot that the agent read but did not capture, and the tracker output from which the agent wrote a ticket file. This content stays inline. - Non-goal: the change to the transcript view that shows the pointers. It is in a different repository. This spec defines only what the view gets. - Non-goal: a change to evidence that an earlier CLI pushed. That evidence keeps its inline content. - Non-goal: other duplicate data in the transcript, for example the rendered copies of agent attachment entries. @@ -31,62 +30,67 @@ The session material lists its sources in a spec list: each spec, skill, and ima ## Requirements -### R-001: No inline copy of a spec source -A full file read or a pasted image in the transcript can be a source in the spec list of the session. Then the CLI MUST replace each copy of it with a pointer. It MUST do this before it records the session material. This applies to all file types: images, documents, and text. -- Done when: the agent reads a screenshot and a JSON file, and captures both as specs. The session material holds a pointer for each copy of both files, and no inline content of them. +### R-001: No copy of a spec source +When a copy in the transcript holds exactly the content of a source in the spec list, the CLI MUST replace it with a pointer. It MUST do this before it records the session material. This applies to all file types. In this version, a copy is a full file read (R-003) or a file write (R-011). +- Done when: the agent reads a screenshot and copies it into the spec folder, and writes a ticket file into the spec folder. The session material holds a pointer for each copy of both, and no inline content of them. ### R-002: Pointer content A pointer MUST stay at the position of the replaced content in the transcript. An image block MUST keep its type. In this version, a pointer MUST hold only a `chainloop.replaced` marker and the digest of the sibling material. That digest MUST also be in the spec list of the session. The pointer MUST NOT repeat data that the material or the spec list already holds, for example the name, the size, or the role. A reader MUST be able to see from the marker alone that Chainloop replaced the content. -### R-003: Match with spec sources -When the agent read a full file that is still on disk at push time, the CLI MUST compute the digest of the file. When the digest is equal to the source digest of an entry in the spec list, the CLI MUST replace the copies of that read. The pointer holds the digest of the material. For a text file, the material is the redacted copy, so the two digests differ. A partial read of a file, for example a range of lines, is not a copy of the source and stays inline. A text spec that the agent wrote with a header is not the file that it read. Its digest differs, and the read stays inline. +### R-003: Match of a file read +The agent can read a full file that is still on disk at push time. Then the CLI MUST compute the digest of the file. When the digest is equal to the source digest of an entry in the spec list, the CLI MUST replace the copies of that read. A partial read, for example a range of lines, stays inline. - Done when: the agent reads a screenshot and copies it into the spec folder. The push gives one material, and both transcript copies hold its digest. ### ~~R-004: Upload of other base64 media~~ -Dropped. See D-013 and R-010. +Dropped. See D-013. ### R-005: No secret scan of base64 media -The secret redaction MUST NOT scan base64 media data, also when the data stays inline. A media material MUST be stored byte for byte. +The secret redaction MUST NOT scan base64 media data that stays inline. A media material MUST be stored byte for byte. - Done when: a pasted image whose base64 text matches a secret pattern decodes after the push. ### R-006: Failures do not block the push -When the CLI cannot store a pasted image, the image MUST stay inline, and the push MUST continue. +When the CLI cannot complete a replacement, the content MUST stay inline, and the push MUST continue. ### R-007: All agents -The CLI SHOULD apply this to each supported agent that keeps file content or pasted images in its transcript. +The CLI SHOULD apply this to each supported agent that keeps file reads or file writes in its transcript. ### R-008: Extensible pointer The pointer format MUST NOT be specific to images or files. A later version MUST be able to add a content kind with optional fields, with no change to the existing pointers. An example is a code snippet that is a range of lines of a source. A transcript view MUST ignore fields that it does not know, and MUST still link the pointer to its material. ### R-009: Remove only known sources -The CLI MUST NOT remove data from the transcript unless the data is with certainty a source in the spec list. The material MUST exist in the CAS before the CLI replaces the block. The digest MUST match exactly. When the CLI is not sure, for example the file is gone, the file changed, or the upload failed, the data MUST stay inline. +The CLI MUST NOT remove data from the transcript unless the data is with certainty a copy of a source in the spec list. The material MUST exist in the CAS before the CLI replaces the copy. The digest MUST match exactly. When the CLI is not sure, for example the file is gone or the file changed, the data MUST stay inline. - Done when: a screenshot that the agent read and then changed on disk stays inline. -### R-010: Pasted images are spec sources -The user can paste an image in a prompt or in a message during a turn. For each pasted image, the CLI MUST store the image as a spec material. It MUST add an entry with the kind `image` to the spec list. When the spec list already has an entry with the same digest, the CLI MUST use that entry and MUST NOT store a second copy. The same image pasted two times MUST give one material. -- Done when: the user pastes the same image two times. The session has one image material and one spec entry for it, and both transcript copies hold its digest. +### ~~R-010: Pasted images are spec sources~~ +Dropped. See D-014. + +### R-011: Match of a file write +The agent can write a full file. The CLI MUST compute the digest of the written content. When it is equal to the source digest of an entry in the spec list, the CLI MUST replace each copy of that content in the write. A write with other content, for example an earlier version of a spec file, stays inline. A partial edit of a file stays inline. +- Done when: the agent writes a ticket file into the spec folder two times, and the second version is the final file. The second write holds pointers, and the first write stays inline. + +### R-012: Best effort, with a match report +The replacement MUST be best effort. The CLI tries each finder on each copy, replaces the exact matches, and keeps all other copies inline with no error. In debug output, the CLI MUST report the result for each session and each finder. The report gives the number of copies that it replaced, and the number that it did not replace, with the reason. Examples of reasons are "no spec entry with this digest", "file gone", "file changed", and "partial read". The report SHOULD give the number of bytes that the replacement removed. +- Done when: a session has 12 image reads, and 8 of the images are specs. A push in debug mode reports 16 replaced copies. It reports 8 copies that it did not replace, with the reason "no spec entry with this digest". ## Constraints - The repository is public. The pointer format is visible to all users and to external transcript viewers. -- The change is additive. The session schema does not type transcript entries, and the kind and the role of a spec entry are open values. So the change needs no schema change. A consumer that does not know the pointer MUST still read the session. +- The change is additive. The session schema does not type transcript entries, so the change needs no schema change. A consumer that does not know the pointer MUST still read the session. - The transcript view must keep support for evidence that an earlier CLI pushed with inline content. -- The CAS backend limits the size of each material. A pasted image over the limit follows R-006. -- Policies read the session evidence. Text that is not a spec source stays inline, so these policies keep working. -- The push runs in a git hook. The extra work must stay small: one digest for each full file read, and one upload for each new pasted image. +- Policies read the session evidence. Content that is not a copy of a source stays inline, so these policies keep working. +- The push runs in a git hook. The extra work must stay small: one digest for each full file read and for each file write. ## Proposal -The user does nothing new. Each push rebuilds the session evidence from the raw transcript, so the result does not depend on earlier pushes. For each session, the CLI works in these steps: +The user does nothing new. Each push rebuilds the session evidence from the raw transcript and the spec folder, so the result does not depend on earlier pushes. For each session, the CLI works in three steps: -1. **Capture specs, as today.** The CLI reads the spec folder into the spec list and stores each file as a sibling material. For each entry it also keeps the source digest: the digest of the file before redaction. For an image, the source digest and the material digest are the same. -2. **Capture pasted images.** The CLI finds each pasted image in the transcript of the main agent and of each subagent. A pasted image can be in a user prompt or in a message that the user sends during a turn. It decodes the bytes and computes the digest. When the spec list has that digest, the CLI uses that entry. Otherwise it stores the image as a spec material and adds an entry. When the upload fails, the image stays inline. -3. **Find and replace.** For each content kind, the CLI has one finder. A finder returns the transcript blocks that hold a source of the spec list. - - The pasted image finder returns the blocks of step 2. - - The file read finder looks at each full read of a file. The tool call keeps the file path. The CLI computes the digest of the file at that path, one time for each path. When the digest is equal to a source digest in the spec list, the finder returns both copies of the read. The agent's file tool can change the content, for example it resizes images and adds line numbers to text. So the CLI does not compare the transcript copy. +1. **Capture specs, as today.** The CLI reads the spec folder into the spec list and stores each file as a sibling material. For each entry it also keeps the source digest: the digest of the file before redaction. For an image, the source digest and the material digest are the same. For a text file, the material is the redacted copy, so the two digests differ. +2. **Find and replace.** For each kind of copy, the CLI has one finder. A finder returns the transcript copies whose digest is equal to a source digest in the spec list. + - The file read finder looks at each full read of a file. The tool call keeps the file path. The CLI computes the digest of the file at that path, one time for each path. The agent's file tool can change the content, for example it resizes images and adds line numbers to text. So the CLI does not compare the transcript copy. + - The file write finder looks at each full write of a file. The tool call holds the written content, so the CLI computes the digest of that content. - The CLI replaces each returned block with a pointer. -4. **Redact.** The CLI records the session material, and the secret redaction scans it. The source copies are gone, so the redaction scans less text. It skips base64 media that stays inline. + The CLI replaces each returned copy with a pointer to the material of that entry. +3. **Redact.** The CLI records the session material, and the secret redaction scans it. The source copies are gone, so the redaction scans less text. It skips base64 media that stays inline. -A pasted image has no file name, so its material gets a number, for example `spec-b5bf8b-pasted-1`. It has the same annotations as the other spec materials. Its bytes were already in the attestation, as base64 in the session material. They now move to a material of their own, which is about 25% smaller than the base64 text. A reader can download it as a file. +For a ticket, the agent first reads it from the tracker, and then writes a spec file from it. The tracker output is in a different format, so it is not a copy and stays inline. It shows where the ticket came from. The write of the spec file is a copy, and becomes a pointer. ### Pointer format A pointer holds two fields: @@ -98,9 +102,12 @@ A pointer holds two fields: - `type`: always `chainloop.replaced`. It tells a person, a policy, or a view that Chainloop removed the content here. - `digest`: the digest of the sibling material. The spec list has an entry with the same digest. -The material and the spec list hold the name, the size, the media type, and the role. The pointer does not repeat them. The position tells the origin. A tool result is a file that the agent read. A user prompt, or a message that the user sent during a turn, is a pasted image. +The material and the spec list hold the name, the size, the media type, and the role. The pointer does not repeat them. The position tells the origin: a file read or a file write. -The CLI puts the pointer where the content was. An image block keeps its type, and only its source changes. Text content has no block with a source, so the text becomes one pointer block. The transcript can also hold the content in a metadata field. Then the CLI removes that field, and adds a `chainloop.replaced` field with the same pointer next to it. A consumer that reads the old field does not get an object where it expects text. +The CLI puts the pointer where the content was: +- An image block keeps its type, and the pointer becomes its source. +- Text content in a tool result becomes one pointer block. +- The transcript can also hold the content in a text field. Examples are the content of a write and the metadata of a tool result. Then the CLI removes that field, and adds a `chainloop.replaced` field with the same pointer next to it. A consumer that reads the old field does not get an object where it expects text. **A screenshot that the agent read and captured.** Before, the tool result holds the image two times: @@ -127,19 +134,17 @@ After, the image block keeps the type `image`, and its source is the pointer: "chainloop.replaced": { "type": "chainloop.replaced", "digest": "sha256:99c0bca3..." } } } } ``` -**A pasted image.** The text of the prompt stays: +**A ticket that the agent wrote into the spec folder.** Before, the write holds the text in the tool input. The tool result metadata holds it again. After, both fields are pointers. The pointer holds the digest of the redacted material: ```json -{ "type": "user", "message": { "role": "user", "content": [ - { "type": "text", "text": "The button is in the wrong place, see this" }, - { "type": "image", "source": { "type": "chainloop.replaced", "digest": "sha256:9b07..." } } -] } } +{ "type": "tool_use", "name": "Write", + "input": { "file_path": ".chainloop/specs//ticket-eng-1234.md", + "chainloop.replaced": { "type": "chainloop.replaced", "digest": "sha256:19b1..." } } } ``` -The spec list gets an entry for it: - ```json -{ "kind": "image", "digest": "sha256:9b07...", "captured_at": "2026-10-07T21:30:11Z" } +"toolUseResult": { "type": "create", "filePath": ".chainloop/specs//ticket-eng-1234.md", + "chainloop.replaced": { "type": "chainloop.replaced", "digest": "sha256:19b1..." } } ``` **A JSON file that the agent read and copied into the spec folder with no change.** Its content becomes one pointer block. The metadata copy of the text gets the same pointer: @@ -155,47 +160,45 @@ A transcript view reads a pointer, finds the entry and the material with that di ```mermaid flowchart TD A[Push] --> B[Parse session transcript] - B --> C[Capture spec folder into the spec list] - B --> D[Find pasted images] - D --> E{Digest in the spec list?} - E -->|yes| G[Use the entry] - E -->|no| F{Store as spec material} - F -->|ok| G - F -->|failed| K[Image stays inline] - C --> H[Finders: pasted images and full file reads with a matching digest] - G --> H - H --> J[Replace each copy with a pointer] - J --> L[Record session material, then redact text and skip media] - K --> L - L --> M[Push attestation] + B --> C[Capture spec folder into the spec list, with source digests] + C --> D[File read finder: digest of the file at the read path] + C --> E[File write finder: digest of the written content] + D --> F{Digest equal to a source digest?} + E --> F + F -->|yes| G[Replace the copy with a pointer] + F -->|no| H[Copy stays inline] + G --> I[Record session material, then redact text and skip media] + H --> I + I --> J[Push attestation] ``` ## Decision Record | ID | Decision | Choice | Why (and what we rejected) | Source | |----|----------|--------|----------------------------|--------| -| D-001 | What to do with media bytes | Keep them in a material, and keep a pointer with the digest in the transcript | The transcript gets small and keeps the position of each block. The media stays in the evidence. Rejected: drop the media (the evidence loses what the agent saw). Rejected: remove only the second copy of read files (the transcript is still large, and the redaction still scans the bytes). | drafting | +| D-001 | What to do with copies of a source | Keep the content in the material, and keep a pointer with the digest in the transcript | The transcript gets small and keeps the position of each copy. The content stays in the evidence. Rejected: drop the copies (a reader loses the position). Rejected: remove only the second copy of read files (the transcript is still large, and the redaction still scans the bytes). | drafting | | D-002 | Where the change runs | In the CLI at push time, before the CLI records the session material | Only the CLI can read the file on disk for the match. The redaction runs when the CLI records the material, so it never sees the replaced bytes. Rejected: in the control plane after the upload (the bytes are already uploaded and scanned, and the file is not on disk). | drafting | -| D-003 | Shape of the pointer | Replace the content in place, and keep the block type | A transcript view finds each block where it was, with no join to other data. Rejected: a separate list of replacements in the session material (the view must join the list to the transcript). | drafting | -| D-004 | Match with spec sources | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript copy (the file tool changes the content, so the digest is different). | drafting | +| D-003 | Shape of the pointer | Replace the content in place, and keep the block type of an image | A transcript view finds each copy where it was, with no join to other data. Rejected: a separate list of replacements in the session material (the view must join the list to the transcript). | drafting | +| D-004 | Match of a file read | Digest of the file at the path of the read tool call | The match is exact. Rejected: match by size (a guess). Rejected: digest of the transcript copy (the file tool changes the content, so the digest is different). | drafting | | D-005 | ~~Main rule: no inline copy of an attachment, and move other media out as a SHOULD~~ | Replaced by D-013 | | owner review | -| D-006 | Failed upload | The pasted image stays inline | The inline copy is then the only copy. R-005 keeps the redaction from breaking it. Rejected: a pointer with no digest (the evidence loses the image). | owner review | -| D-007 | Text that is not a spec source | Stays inline | Reviewers read it in the transcript, and policies check it. Rejected: move all read text files to redacted materials (policies lose the content, and the view must download each file). | owner review | -| D-008 | Extensibility | One finder for each content kind. A later kind adds optional fields to the pointer | Other inputs, for example code snippets, can use the same mechanism later. Rejected: an image-only pointer (each new kind would need a new format and a new view). Rejected: a version and a kind field in each pointer (optional fields are enough, and the material holds the kind). | owner review | +| D-006 | ~~Failed upload of a pasted image~~ | Dropped with D-014 | | owner review | +| D-007 | Text that is not a copy of a source | Stays inline | Reviewers read it in the transcript, and policies check it. Rejected: move all read text files to redacted materials (policies lose the content, and the view must download each file). | owner review | +| D-008 | Extensibility | One finder for each kind of copy. A later kind adds optional fields to the pointer | Other inputs, for example code snippets or pasted images, can use the same mechanism later. Rejected: an image-only pointer (each new kind would need a new format and a new view). Rejected: a version and a kind field in each pointer (optional fields are enough, and the material holds the kind). | owner review | | D-009 | Direction and order | Start from the spec list: find the copies of its sources in the transcript and replace them. Then redact | The spec list is small and known. The rule of R-001 is about these sources, so the code follows the rule directly. The redaction runs last, on less text. Rejected: walk each transcript block and decide what it is (each block type needs a rule, also blocks that are never sources). | owner review | -| D-010 | Certainty before removal | Replace a block only after the material is in the CAS and its digest matches exactly | The transcript is evidence. A wrong pointer loses content with no way back. Rejected: a best-effort match, for example by size or by file name (a wrong match removes content that is not stored anywhere). | owner review | -| D-011 | Pasted images | The CLI captures each pasted image as a spec source | The user gave the image to the session, as the user gives a ticket. Only the CLI can reach the bytes, because the agent sees only the image (Spec 003 D-009). This does the work that Spec 002 D-010 left for a later version. Rejected: leave pasted images inline (they stay large and cannot be downloaded). Rejected: new attachment materials outside the spec list (a second list for a view to join). | owner review | -| D-012 | Content of the pointer | Only the `chainloop.replaced` marker and the digest of the sibling material | The material and its entry in the spec list already hold the name, the size, the kind, and the role. The position of the block tells the origin. The marker name tells any reader that Chainloop replaced the content. Rejected: a full reference with the name, size, media type, origin, kind, and version (it repeats the material). Rejected: the material name as the pointer (a label, while the digest is unique and lets a reader check the content). | owner review | -| D-013 | Scope of the replacement | Only sources in the spec list, including pasted images (D-011) | Each pointer then resolves in the session itself, and the CLI never decides on its own that content is evidence. Rejected: also upload each other base64 media block. An example is a screenshot that the agent read but did not capture. This turns each image that the agent looked at into a material, and the agent did not choose it. | owner review | +| D-010 | Certainty before removal | Replace a copy only when the material is in the CAS and its digest matches exactly | The transcript is evidence. A wrong pointer loses content with no way back. Rejected: a best-effort match, for example by size or by file name (a wrong match removes content that is not stored anywhere). | owner review | +| D-011 | ~~Pasted images become spec sources~~ | Dropped with D-014 | | owner review | +| D-012 | Content of the pointer | Only the `chainloop.replaced` marker and the digest of the sibling material | The material and its entry in the spec list already hold the name, the size, the kind, and the role. The position of the copy tells the origin. The marker name tells any reader that Chainloop replaced the content. Rejected: a full reference with the name, size, media type, origin, kind, and version (it repeats the material). Rejected: the material name as the pointer (a label, while the digest is unique and lets a reader check the content). | owner review | +| D-013 | Scope of the replacement | Each exact copy of a source in the spec list, for all file types, from file reads and file writes | One rule covers images and text sources, for example tickets and plans. Each pointer resolves in the session itself, and the CLI never decides on its own that content is evidence. Rejected: images only (the write copies of tickets stay, and the redaction still scans them). Rejected: also upload each other base64 media block. An example is a screenshot that the agent read but did not capture, and the agent did not choose it. | owner review | +| D-015 | How hard the CLI tries | Best effort coverage, with an exact match for each replacement, and a match report in debug output | A copy that does not match costs only size, so the CLI must not fail or warn for it. A wrong match loses content, so each replacement needs an exact digest (D-010). The debug report shows how well the matching works in real sessions, and which finder to improve. Rejected: fail or warn on each copy that does not match (noise, because most transcript content is not a source). Rejected: no report (nobody can see the share of matches). | owner review | +| D-014 | Pasted images | Out of scope. A later spec | The bytes of a pasted image are only in the transcript. Capturing them adds new materials and a role decision. This spec keeps to sources that the session already captured. Rejected: capture them in this spec (larger scope, and open questions on the role and the limit). | owner review | ## Open Questions -- [ ] **Do pasted images get a role?** Proposed: `reference`. Spec 004 defines it as "supporting material, for example a screenshot". The CLI knows that the user gave the image as support. With no role, a view that lists specs shows pasted screenshots next to the ticket. This extends Spec 004 R-005, which forbids a role guess for sources that the agent writes. A new role `attachment` is a conflict, because Claude Code transcripts already have entries of the type `attachment`. -- [ ] **Do pasted images count toward the limit of 25 spec entries for each session (Spec 003)?** Proposed: no. They get a separate limit of 25. A session with many pastes must not push out the specs that the agent chose. +None. ## Milestones -1. **Claude Code.** The CLI captures pasted images, replaces the copies of spec sources with pointers, and the redaction skips media. +1. **Claude Code.** The CLI replaces the copies of spec sources from file reads and file writes with pointers, and the redaction skips media. 2. **Transcript view.** The view shows each source from its pointer. This is in a different repository. -3. **Other agents.** The same change for each other agent that keeps file content or pasted images in its transcript (R-007). +3. **Other agents.** The same change for each other agent that keeps file reads or file writes in its transcript (R-007). ## Risks @@ -204,7 +207,8 @@ flowchart TD | The file changed after the read, so the match with the spec fails. | The read stays inline (R-009). The evidence holds one more copy, but it is correct. | | The file changed after the read, and the agent then copied the new version into the spec folder. The digest matches, but the agent saw the old version. | When the tool result records the original size, the CLI also compares it with the size of the file. A different size keeps the read inline. | | The agent reads a screenshot but does not capture it, so it stays inline and the evidence stays large. | The capture instruction asks the agent to capture images that the user gives (Spec 003). The share of read images with no spec entry is measurable from the evidence. | -| A policy reads a spec file from the transcript, and now finds a pointer. | The spec material holds the same content. The policy can read the material by its digest. Text that is not a spec source stays inline (D-007). | +| Pasted images stay inline and keep the evidence large. | R-005 stops the redaction from breaking them. A later spec captures them (D-014). | +| A policy reads a spec file from the transcript, and now finds a pointer. | The spec material holds the same content. The policy can read the material by its digest. Content that is not a copy of a source stays inline (D-007). | | A very large transcript line, with a large image, stops the session parse before the CLI can replace the image. | The parse limit is separate from this change. The CLI can raise it or skip the media data while it reads the line. | -| A transcript view does not know the pointer. | The pointer keeps the block type and holds a clear marker, so the view shows an image with no data, not wrong data. Milestone 2 adds the support. | +| A transcript view does not know the pointer. | The pointer keeps the block type of an image and holds a clear marker. So the view shows an image with no data, not wrong data. Milestone 2 adds the support. | | An image holds a secret, for example a screenshot of a terminal. | The CLI did not redact images before this change either. It stores media as it is (Spec 002, non-goal). | From 6908229339ff23c25fe43eed66cdb3bd2b7eb4f6 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 01:26:01 +0200 Subject: [PATCH 6/7] spec: issue-3556 wording Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- docs/specs/issue-3556-session-image-placeholders.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index 922566ff9..c7bf9f726 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -65,7 +65,7 @@ The CLI MUST NOT remove data from the transcript unless the data is with certain Dropped. See D-014. ### R-011: Match of a file write -The agent can write a full file. The CLI MUST compute the digest of the written content. When it is equal to the source digest of an entry in the spec list, the CLI MUST replace each copy of that content in the write. A write with other content, for example an earlier version of a spec file, stays inline. A partial edit of a file stays inline. +The agent can write a full file. The CLI MUST compute the digest of the written content. When it is equal to the source digest of a spec entry, the CLI MUST replace each copy of that content in the write. A write with other content, for example an earlier version of a spec file, stays inline. A partial edit of a file stays inline. - Done when: the agent writes a ticket file into the spec folder two times, and the second version is the final file. The second write holds pointers, and the first write stays inline. ### R-012: Best effort, with a match report From e3d49982161360d984e174897aaae234eb2a9c83 Mon Sep 17 00:00:00 2001 From: Miguel Martinez Trivino Date: Thu, 8 Oct 2026 01:27:11 +0200 Subject: [PATCH 7/7] spec: issue-3556 merge overlapping requirements Refs #3556 Assisted-by: Claude Code Signed-off-by: Miguel Martinez Trivino Chainloop-Trace-Sessions: 383701ed-3811-43b8-a02a-299e49a9a604 --- .../issue-3556-session-image-placeholders.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/specs/issue-3556-session-image-placeholders.md b/docs/specs/issue-3556-session-image-placeholders.md index c7bf9f726..513301764 100644 --- a/docs/specs/issue-3556-session-image-placeholders.md +++ b/docs/specs/issue-3556-session-image-placeholders.md @@ -48,18 +48,17 @@ Dropped. See D-013. The secret redaction MUST NOT scan base64 media data that stays inline. A media material MUST be stored byte for byte. - Done when: a pasted image whose base64 text matches a secret pattern decodes after the push. -### R-006: Failures do not block the push -When the CLI cannot complete a replacement, the content MUST stay inline, and the push MUST continue. +### ~~R-006: Failures do not block the push~~ +Merged into R-012. See D-016. -### R-007: All agents -The CLI SHOULD apply this to each supported agent that keeps file reads or file writes in its transcript. +### ~~R-007: All agents~~ +Moved to Milestone 3. See D-016. ### R-008: Extensible pointer The pointer format MUST NOT be specific to images or files. A later version MUST be able to add a content kind with optional fields, with no change to the existing pointers. An example is a code snippet that is a range of lines of a source. A transcript view MUST ignore fields that it does not know, and MUST still link the pointer to its material. -### R-009: Remove only known sources -The CLI MUST NOT remove data from the transcript unless the data is with certainty a copy of a source in the spec list. The material MUST exist in the CAS before the CLI replaces the copy. The digest MUST match exactly. When the CLI is not sure, for example the file is gone or the file changed, the data MUST stay inline. -- Done when: a screenshot that the agent read and then changed on disk stays inline. +### ~~R-009: Remove only known sources~~ +Merged into R-012. See D-016. ### ~~R-010: Pasted images are spec sources~~ Dropped. See D-014. @@ -68,8 +67,8 @@ Dropped. See D-014. The agent can write a full file. The CLI MUST compute the digest of the written content. When it is equal to the source digest of a spec entry, the CLI MUST replace each copy of that content in the write. A write with other content, for example an earlier version of a spec file, stays inline. A partial edit of a file stays inline. - Done when: the agent writes a ticket file into the spec folder two times, and the second version is the final file. The second write holds pointers, and the first write stays inline. -### R-012: Best effort, with a match report -The replacement MUST be best effort. The CLI tries each finder on each copy, replaces the exact matches, and keeps all other copies inline with no error. In debug output, the CLI MUST report the result for each session and each finder. The report gives the number of copies that it replaced, and the number that it did not replace, with the reason. Examples of reasons are "no spec entry with this digest", "file gone", "file changed", and "partial read". The report SHOULD give the number of bytes that the replacement removed. +### R-012: Exact matches only, best effort, with a match report +The CLI MUST NOT remove data from the transcript unless the data is with certainty a copy of a spec source. The digest MUST match exactly. The replacement MUST be best effort. The CLI tries each finder on each copy, and replaces only the exact matches. All other copies stay inline, for example when the file is gone or changed, or when a step fails. These cases MUST NOT give an error or stop the push. In debug output, the CLI MUST report the result for each session and each finder. The report gives the number of copies that it replaced, and the number that it did not replace, with the reason. Examples of reasons are "no spec entry with this digest", "file gone", "file changed", and "partial read". The report SHOULD give the number of bytes that the replacement removed. - Done when: a session has 12 image reads, and 8 of the images are specs. A push in debug mode reports 16 replaced copies. It reports 8 copies that it did not replace, with the reason "no spec entry with this digest". ## Constraints @@ -153,7 +152,7 @@ After, the image block keeps the type `image`, and its source is the pointer: "content": [ { "type": "chainloop.replaced", "digest": "sha256:c2a1..." } ] ``` -A later content kind adds a finder and optional fields to the pointer, for example a line range for a code snippet. The examples use the transcript format of Claude Code. Each other agent puts the same pointer in its own block shape (R-007). +A later content kind adds a finder and optional fields to the pointer, for example a line range for a code snippet. The examples use the transcript format of Claude Code. Each other agent puts the same pointer in its own block shape (Milestone 3). A transcript view reads a pointer, finds the entry and the material with that digest, and shows the content from the CAS. For evidence from an earlier CLI, it shows the inline content as today. @@ -190,6 +189,7 @@ flowchart TD | D-012 | Content of the pointer | Only the `chainloop.replaced` marker and the digest of the sibling material | The material and its entry in the spec list already hold the name, the size, the kind, and the role. The position of the copy tells the origin. The marker name tells any reader that Chainloop replaced the content. Rejected: a full reference with the name, size, media type, origin, kind, and version (it repeats the material). Rejected: the material name as the pointer (a label, while the digest is unique and lets a reader check the content). | owner review | | D-013 | Scope of the replacement | Each exact copy of a source in the spec list, for all file types, from file reads and file writes | One rule covers images and text sources, for example tickets and plans. Each pointer resolves in the session itself, and the CLI never decides on its own that content is evidence. Rejected: images only (the write copies of tickets stay, and the redaction still scans them). Rejected: also upload each other base64 media block. An example is a screenshot that the agent read but did not capture, and the agent did not choose it. | owner review | | D-015 | How hard the CLI tries | Best effort coverage, with an exact match for each replacement, and a match report in debug output | A copy that does not match costs only size, so the CLI must not fail or warn for it. A wrong match loses content, so each replacement needs an exact digest (D-010). The debug report shows how well the matching works in real sessions, and which finder to improve. Rejected: fail or warn on each copy that does not match (noise, because most transcript content is not a source). Rejected: no report (nobody can see the share of matches). | owner review | +| D-016 | Fewer requirements | Merge R-006 and R-009 into R-012, and move R-007 to a milestone | R-006, R-009, and R-012 all said "replace only exact matches, keep the rest inline, never fail". One requirement says it once. Support for other agents is a delivery step, not a behavior a reviewer can check. | owner review | | D-014 | Pasted images | Out of scope. A later spec | The bytes of a pasted image are only in the transcript. Capturing them adds new materials and a role decision. This spec keeps to sources that the session already captured. Rejected: capture them in this spec (larger scope, and open questions on the role and the limit). | owner review | ## Open Questions @@ -198,13 +198,13 @@ None. ## Milestones 1. **Claude Code.** The CLI replaces the copies of spec sources from file reads and file writes with pointers, and the redaction skips media. 2. **Transcript view.** The view shows each source from its pointer. This is in a different repository. -3. **Other agents.** The same change for each other agent that keeps file reads or file writes in its transcript (R-007). +3. **Other agents.** The same change for each other agent that keeps file reads or file writes in its transcript. ## Risks | Risk | Mitigation | |------|------------| -| The file changed after the read, so the match with the spec fails. | The read stays inline (R-009). The evidence holds one more copy, but it is correct. | +| The file changed after the read, so the match with the spec fails. | The read stays inline (R-012). The evidence holds one more copy, but it is correct. | | The file changed after the read, and the agent then copied the new version into the spec folder. The digest matches, but the agent saw the old version. | When the tool result records the original size, the CLI also compares it with the size of the file. A different size keeps the read inline. | | The agent reads a screenshot but does not capture it, so it stays inline and the evidence stays large. | The capture instruction asks the agent to capture images that the user gives (Spec 003). The share of read images with no spec entry is measurable from the evidence. | | Pasted images stay inline and keep the evidence large. | R-005 stops the redaction from breaking them. A later spec captures them (D-014). |