From fe8c2e87888a2b5112d2a1c48b8f79c853f3af6b Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 7 Oct 2026 10:46:39 +0200 Subject: [PATCH 1/4] spec: 006 Skills used in a Chainloop Trace session Refs #3540 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 62aaeaaa-033c-4fb3-abac-9ee026634f60 --- docs/specs/006-trace-skills-used.md | 123 ++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 docs/specs/006-trace-skills-used.md diff --git a/docs/specs/006-trace-skills-used.md b/docs/specs/006-trace-skills-used.md new file mode 100644 index 000000000..145fa7135 --- /dev/null +++ b/docs/specs/006-trace-skills-used.md @@ -0,0 +1,123 @@ +--- +owner: jiparis +ticket: https://github.com/chainloop-dev/chainloop/issues/3540 +prd: +--- + +# Spec 006: Skills used in a Chainloop Trace session + +## Summary + +Chainloop Trace will record the skills that an agent used in a session. The AI coding session evidence gets a new optional list, `skills_used`. Each entry names one skill and gives the number of uses. It also tells who started the skill (the model or the user) and where the skill came from (project, user, or plugin). It also gives the number of uses that came from subagents. The first version supports Claude Code and OpenCode. The change adds a field to the existing evidence schema and changes no existing field. + +## Problem + +- The evidence shows which skills are available to an agent, but not which skills the agent used. The `ai-agent-config` material lists the skill files in the repository. That list does not show use. +- The tool usage summary counts calls per tool name. All Claude Code skill calls go into one `Skill` entry, so the summary does not show which skills ran. +- The skill names are in the raw transcript only. To answer "did this session use skill X?", a policy or the UI must scan every transcript entry. It must also know the transcript format of each agent. +- A user can start a skill with a slash command (for example `/graphify`). That skill use is not a tool call, so a scan of tool calls alone does not find it. + +## Goals and Non-Goals + +- Goal: A policy or the UI can find the skills that a session used with one lookup in the session evidence. +- Goal: The record separates skills that the model started from skills that the user started. +- Goal: The record includes skills that subagents used. +- Non-goal: Record which version of a skill ran (for example a digest of its `SKILL.md`). The transcript does not hold the skill file content. The `ai-agent-config` material already records digests of repository skills. +- Non-goal: Support Cursor and Pi in the first version. Their transcript formats for skill use are not confirmed yet. A later change can add them without a schema change. +- Non-goal: Record the local path of a skill file in the new field. +- Non-goal: Change the existing tool usage summary, the raw transcript, or the `ai-agent-config` material. + +## Requirements + +### R-001: Skills used list + +The AI coding session evidence MUST include a `skills_used` list when the session used one or more skills. Each skill MUST appear one time only in the list. The list MUST be sorted by name. A session that used no skill MUST leave the field out. + +- Done when: sessions with no skill, one skill, and several skills each produce the expected evidence. Two runs over the same transcript produce the same output. + +### R-002: Counts and who started the skill + +Each entry MUST hold the skill name and the total number of uses. It MUST also hold the number of uses that the model started and the number that the user started. The two start counts MUST add up to the total. A model start is a skill tool call that loaded a skill. A failed call MUST NOT count. A user start is a slash command that loads a skill. One skill use MUST count one time only, also when the transcript records it in more than one entry. + +- Done when: fixtures with a model-started skill, a user-started skill, and the same skill started both ways produce correct counts. + +### R-003: User-started skills only + +A slash command that does not load a skill MUST NOT go into the list. Examples are built-in agent commands such as `/clear` or `/model`, and custom commands that are not skills. + +- Done when: a transcript with built-in commands, custom commands, and skill commands lists only the skill commands. + +### R-004: Subagent skills + +Skills that subagents used MUST count in the session total of the same entry. Each entry MUST also hold the number of uses that came from subagents. + +- Done when: the main agent and a subagent use the same skill in one session. The evidence holds one entry with the correct total and subagent count. + +### R-005: Skill source + +Each entry MUST hold a source. The values are `project` (a skill from the repository), `user` (a skill from the user's agent configuration), `plugin` (a skill from an installed plugin), and `unknown`. The source MUST come from data in the transcript. Examples are a plugin prefix in the skill name and the location of the skill file. The evidence MUST NOT hold the skill file path in this field. When the data is not sufficient, the source MUST be `unknown`. + +- Done when: fixtures for each source value, plus a transcript with no location data, produce the expected source. + +### R-006: Provider support + +Claude Code and OpenCode MUST fill the list. Cursor and Pi MUST leave the field out until a later change adds support. A missing field MUST mean "not recorded" and MUST NOT mean "no skills used" for those providers. + +- Done when: Claude Code and OpenCode fixtures produce the list, and Cursor evidence does not change. + +### R-007: Compatibility and failure safety + +The new field MUST be optional and additive. The schema version MUST stay the same, and existing evidence MUST stay valid. The tool usage summary MUST continue to count skill tool calls as it does today. When the parser cannot read an entry, it MUST skip the entry and add a warning to the evidence. A bad entry MUST NOT stop evidence creation or block the Git push. + +- Done when: old evidence and new evidence both pass schema validation. A malformed skill entry produces a warning and the rest of the evidence. + +## Constraints + +- The skill data comes only from the session transcript that Trace already collects. The change adds no new hook, network call, or file read outside the transcript. +- The evidence must not hold local file paths in the new field. The raw transcript keeps its current content. +- Output ordering must be deterministic, so the same transcript gives the same evidence digest. + +## Proposal + +Each provider parser already reads the full transcript to count tools, tokens, and messages. The parser will also collect skill uses in the same pass and add them to the evidence as `skills_used`. + +```yaml +skills_used: + - name: superpowers:brainstorming + source: plugin + invocation_count: 3 + by_model: 2 + by_user: 1 + in_subagents: 1 +``` + +**Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. The second message tells a skill command apart from a built-in or custom command. Claude Code also writes that meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The meta message gives the skill location, which sets the source. Subagent transcripts go through the same steps, and their counts merge into the session entry. + +**OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. The tool result holds the skill directory, which sets the source. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. + +**Source.** A name with a plugin prefix (`plugin:skill`) or a location in the agent's plugin directory gives `plugin`. A location in the repository gives `project`. A location in the user's agent configuration directory gives `user`. All other cases give `unknown`. + +## Decision Record + +| ID | Decision | Choice | Why (and what we rejected) | Source | +|----|----------|--------|----------------------------|--------| +| D-001 | Where to record skill use | A new optional `skills_used` list in the AI coding session evidence | One lookup answers "was skill X used". Rejected: a per-skill breakdown inside the tool usage summary, which mixes two concepts. Rejected: no change, which leaves every consumer to scan the raw transcript. | drafting | +| D-002 | First-version providers | Claude Code and OpenCode | Both transcripts hold the skill name in a tool input. Rejected: all four providers, because the Cursor and Pi formats are not confirmed. Rejected: Claude Code only, because OpenCode support costs little. | drafting | +| D-003 | User-started skills | Count them, and record who started each use | A user who types a slash command uses the skill as much as a model that calls a tool. Rejected: tool calls only, which misses user starts. Rejected: one count with no split, which hides who chose the skill. | drafting | +| D-004 | Entry shape | One entry per skill, with total, model, user, and subagent counts | A policy finds a skill with one lookup. Rejected: one entry per combination of skill, starter, and agent, which lists one skill several times. | drafting | +| D-005 | Subagent skills | Merge into the session entry and add a subagent count | This matches how the evidence merges subagent token usage. Rejected: a skills list on each subagent, which adds detail no consumer asked for. Rejected: main agent only, which misses skills. | drafting | +| D-006 | Skill version | Out of scope | The transcript does not hold the skill file content. The `ai-agent-config` material records digests of repository skills. | drafting | +| D-007 | Skill source | `project`, `user`, `plugin`, or `unknown`, with no path | The source tells a reviewer if the skill came from the repository. A path exposes local user data. | drafting | +| D-008 | Failed skill calls | Do not count them | A failed call (for example, an unknown skill name) loads no skill. This matches the OpenCode parser, which counts only completed tool calls. Rejected: count every call. | drafting | + +## Open Questions + +None. + +## Risks + +| Risk | Mitigation | +|------|------------| +| An agent changes how it records skill use. | Golden fixtures for each format. An unknown format gives no entry and a warning, never a failed push. | +| A user-started skill and its meta message count two times. | Count starts, not meta messages. A fixture covers both start types for the same skill. | +| A consumer reads a missing field as "no skills used" for Cursor or Pi. | Document that a missing field means "not recorded" for providers without support. | From 5a1fa3831559f93fb2ef76aca507079cc04f7c90 Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 7 Oct 2026 12:02:47 +0200 Subject: [PATCH 2/4] spec: 006 record used skills as spec entries of kind skill Refs #3540 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 62aaeaaa-033c-4fb3-abac-9ee026634f60 --- docs/specs/006-trace-skills-used.md | 138 ++++++++++++++++++---------- 1 file changed, 87 insertions(+), 51 deletions(-) diff --git a/docs/specs/006-trace-skills-used.md b/docs/specs/006-trace-skills-used.md index 145fa7135..ae7d7f1ff 100644 --- a/docs/specs/006-trace-skills-used.md +++ b/docs/specs/006-trace-skills-used.md @@ -8,92 +8,122 @@ prd: ## Summary -Chainloop Trace will record the skills that an agent used in a session. The AI coding session evidence gets a new optional list, `skills_used`. Each entry names one skill and gives the number of uses. It also tells who started the skill (the model or the user) and where the skill came from (project, user, or plugin). It also gives the number of uses that came from subagents. The first version supports Claude Code and OpenCode. The change adds a field to the existing evidence schema and changes no existing field. +Chainloop Trace will record the skills that an agent used in a session, with the text of each skill. Each used skill becomes a spec entry of the new kind `skill`. The skill text that the agent loaded goes into the attestation as an EVIDENCE material, the same as other spec sources. Spec entries get a new optional `metadata` object. Its fields depend on the kind. For a skill, it holds the skill source and the use counts. + +This spec changes the spec capture of [Spec 002](002-session-spec-capture.md) and [Spec 004](004-spec-source-role.md). It does not replace them. It departs from D-001 of Spec 002 for skills only. The CLI makes the skill entries from the transcript. The agent does not write them. The first version supports Claude Code and OpenCode. ## Problem - The evidence shows which skills are available to an agent, but not which skills the agent used. The `ai-agent-config` material lists the skill files in the repository. That list does not show use. - The tool usage summary counts calls per tool name. All Claude Code skill calls go into one `Skill` entry, so the summary does not show which skills ran. -- The skill names are in the raw transcript only. To answer "did this session use skill X?", a policy or the UI must scan every transcript entry. It must also know the transcript format of each agent. +- The skill names and texts are in the raw transcript only. To answer "did this session use skill X?", a policy or the UI must scan every transcript entry. It must also know the transcript format of each agent. +- A skill sets how the agent works, as a spec sets what the agent builds. A reviewer cannot see the skill text next to the other sources of the session. - A user can start a skill with a slash command (for example `/graphify`). That skill use is not a tool call, so a scan of tool calls alone does not find it. ## Goals and Non-Goals -- Goal: A policy or the UI can find the skills that a session used with one lookup in the session evidence. -- Goal: The record separates skills that the model started from skills that the user started. -- Goal: The record includes skills that subagents used. -- Non-goal: Record which version of a skill ran (for example a digest of its `SKILL.md`). The transcript does not hold the skill file content. The `ai-agent-config` material already records digests of repository skills. -- Non-goal: Support Cursor and Pi in the first version. Their transcript formats for skill use are not confirmed yet. A later change can add them without a schema change. -- Non-goal: Record the local path of a skill file in the new field. -- Non-goal: Change the existing tool usage summary, the raw transcript, or the `ai-agent-config` material. +- Goal: A policy or the UI can find the skills that a session used in the spec entries of the session material. +- Goal: The attestation holds the skill text that the agent loaded, as a material that a reviewer can download and verify. +- Goal: The record separates skills that the model started from skills that the user started, and includes skills that subagents used. +- Goal: The `metadata` object can hold kind-specific data for other kinds later, with no new top-level field. +- Non-goal: Store the `SKILL.md` file byte for byte. The material holds the text that the agent loaded. +- Non-goal: Read skill files from disk. All skill data comes from the transcript. +- Non-goal: Support Cursor and Pi in the first version. Their transcript formats for skill use are not known yet. A later change can add them without a schema change. +- Non-goal: Record the local path of a skill file. +- Non-goal: Change how the agent captures other spec sources, the tool usage summary, the raw transcript, or the `ai-agent-config` material. ## Requirements -### R-001: Skills used list +### R-001: One spec entry for each used skill + +The session material MUST hold one spec entry of kind `skill` for each skill that the session used. The title MUST be the skill name. The CLI MUST NOT set a role, a description, or a source address for a skill entry. The capture time MUST be the time of the first use in the transcript. A session that used no skill MUST have no skill entries. + +- Done when: sessions with no skill, one skill, and several skills each produce the expected spec entries. Two runs over the same transcript produce the same output. -The AI coding session evidence MUST include a `skills_used` list when the session used one or more skills. Each skill MUST appear one time only in the list. The list MUST be sorted by name. A session that used no skill MUST leave the field out. +### R-002: Skill text material -- Done when: sessions with no skill, one skill, and several skills each produce the expected evidence. Two runs over the same transcript produce the same output. +Each skill entry MUST point by digest to an EVIDENCE material that holds the skill text that the agent loaded. The CLI MUST take the text from the transcript. It MUST remove the arguments of one call, so that two calls with different arguments give the same text. It MUST use the text of the first use. When a later use gives a different text, the CLI MUST record a warning. The text MUST go through the secret redaction of Spec 002 (R-007) before upload. The material MUST carry the same annotations as other spec materials, with `chainloop.spec.kind` set to `skill` and `chainloop.spec.title` set to the skill name. -### R-002: Counts and who started the skill +- Done when: the attestation holds one EVIDENCE material for each used skill, and its digest downloads the redacted skill text. -Each entry MUST hold the skill name and the total number of uses. It MUST also hold the number of uses that the model started and the number that the user started. The two start counts MUST add up to the total. A model start is a skill tool call that loaded a skill. A failed call MUST NOT count. A user start is a slash command that loads a skill. One skill use MUST count one time only, also when the transcript records it in more than one entry. +### R-003: Metadata object -- Done when: fixtures with a model-started skill, a user-started skill, and the same skill started both ways produce correct counts. +A spec entry MAY hold a `metadata` object. The fields of the object depend on the kind of the entry. This spec defines the fields for the kind `skill` only. Entries of other kinds MUST leave the object out. -### R-003: User-started skills only +### R-004: Use counts and who started the skill -A slash command that does not load a skill MUST NOT go into the list. Examples are built-in agent commands such as `/clear` or `/model`, and custom commands that are not skills. +The metadata of a skill entry MUST hold the total number of uses. It MUST also hold the number of uses that the model started and the number that the user started. The two start counts MUST add up to the total. -- Done when: a transcript with built-in commands, custom commands, and skill commands lists only the skill commands. +- A model start is a skill tool call that loaded a skill. A failed call MUST NOT count. +- A user start is a slash command that loads a skill. A slash command that does not load a skill MUST NOT count. Examples are `/clear`, `/model`, and custom commands that are not skills. +- One skill use MUST count one time only, also when the transcript records it in more than one entry. +- Done when: fixtures with a model start, a user start, both starts for one skill, a failed call, and built-in commands produce correct counts. -### R-004: Subagent skills +### R-005: Subagent skills -Skills that subagents used MUST count in the session total of the same entry. Each entry MUST also hold the number of uses that came from subagents. +Skills that subagents used MUST count in the totals of the same entry. The metadata MUST also hold the number of uses that came from subagents. - Done when: the main agent and a subagent use the same skill in one session. The evidence holds one entry with the correct total and subagent count. -### R-005: Skill source +### R-006: Skill source -Each entry MUST hold a source. The values are `project` (a skill from the repository), `user` (a skill from the user's agent configuration), `plugin` (a skill from an installed plugin), and `unknown`. The source MUST come from data in the transcript. Examples are a plugin prefix in the skill name and the location of the skill file. The evidence MUST NOT hold the skill file path in this field. When the data is not sufficient, the source MUST be `unknown`. +The metadata of a skill entry MUST hold a source. The values are `project` (a skill from the repository), `user` (a skill from the user's agent configuration), `plugin` (a skill from an installed plugin), and `unknown`. The source MUST come from data in the transcript. Examples are a plugin prefix in the skill name and the location of the skill file. The evidence MUST NOT hold the skill file path. When the data is not sufficient, the source MUST be `unknown`. - Done when: fixtures for each source value, plus a transcript with no location data, produce the expected source. -### R-006: Provider support +### R-007: Limit on skill entries -Claude Code and OpenCode MUST fill the list. Cursor and Pi MUST leave the field out until a later change adds support. A missing field MUST mean "not recorded" and MUST NOT mean "no skills used" for those providers. +Skill entries MUST have their own limit of 25 for each session. They MUST NOT count against the limit of the spec sources that the agent writes. When a session used more skills than the limit, the CLI MUST keep the skills that the session used first and record a warning. -- Done when: Claude Code and OpenCode fixtures produce the list, and Cursor evidence does not change. +### R-008: Provider support -### R-007: Compatibility and failure safety +Claude Code and OpenCode MUST produce skill entries. Cursor and Pi MUST produce none until a later change adds support. For those providers, no skill entry MUST mean "not recorded" and MUST NOT mean "no skills used". -The new field MUST be optional and additive. The schema version MUST stay the same, and existing evidence MUST stay valid. The tool usage summary MUST continue to count skill tool calls as it does today. When the parser cannot read an entry, it MUST skip the entry and add a warning to the evidence. A bad entry MUST NOT stop evidence creation or block the Git push. +### R-009: Compatibility and failure safety -- Done when: old evidence and new evidence both pass schema validation. A malformed skill entry produces a warning and the rest of the evidence. +The new kind and the `metadata` object MUST be additive. The schema version MUST stay the same, and existing evidence MUST stay valid. A consumer that does not know the kind `skill` MUST still read the other spec entries. When the parser cannot read a skill use, it MUST skip it and add a warning to the evidence. A bad skill use MUST NOT stop evidence creation or block the Git push. + +- Done when: old evidence and new evidence both pass schema validation. A malformed skill use produces a warning and the rest of the evidence. ## Constraints -- The skill data comes only from the session transcript that Trace already collects. The change adds no new hook, network call, or file read outside the transcript. -- The evidence must not hold local file paths in the new field. The raw transcript keeps its current content. +- All skill data comes from the session transcript that Trace already collects. The change adds no new hook, network call, or file read. +- The repository is public. The evidence format is visible to all users. +- User and plugin skills can hold private text. The secret redaction applies to all skill text before upload. - Output ordering must be deterministic, so the same transcript gives the same evidence digest. ## Proposal -Each provider parser already reads the full transcript to count tools, tokens, and messages. The parser will also collect skill uses in the same pass and add them to the evidence as `skills_used`. +Each provider parser already reads the full transcript to count tools, tokens, and messages. The parser will also collect skill uses in the same pass. At push time, the CLI adds the skill entries to the spec list, after the sources that the agent wrote. It uploads each skill text the same way as a spec file. When the text did not change since an earlier push, it reuses the redacted copy. + +```json +"spec": [ + { "kind": "ticket", "role": "task", "title": "ENG-1234: Add an export button", + "uri": "https://tracker.example.com/issue/ENG-1234", + "digest": "sha256:e4c2...", "captured_at": "2026-10-07T10:12:03Z" }, + { "kind": "skill", "title": "superpowers:brainstorming", + "digest": "sha256:ab12...", "captured_at": "2026-10-07T10:14:51Z", + "metadata": { "source": "plugin", "invocation_count": 3, + "by_model": 2, "by_user": 1, "in_subagents": 1 } } +] +``` -```yaml -skills_used: - - name: superpowers:brainstorming - source: plugin - invocation_count: 3 - by_model: 2 - by_user: 1 - in_subagents: 1 +```json +{ + "name": "superpowers-brainstorming.md", + "annotations": { + "chainloop.material.name": "spec-fd4e67-skill-superpowers-brainstorming", + "chainloop.material.type": "EVIDENCE", + "chainloop.spec.session_id": "fd4e6754-3b26-4f54-9807-13c58465bb35", + "chainloop.spec.kind": "skill", + "chainloop.spec.title": "superpowers:brainstorming" + } +} ``` -**Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. The second message tells a skill command apart from a built-in or custom command. Claude Code also writes that meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The meta message gives the skill location, which sets the source. Subagent transcripts go through the same steps, and their counts merge into the session entry. +**Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. The second message tells a skill command apart from a built-in or custom command. Claude Code also writes that meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The meta message holds the skill text and the skill location. Claude Code removes the frontmatter of the skill file and adds the call arguments at the end. The parser removes the arguments. Subagent transcripts go through the same steps, and their counts merge into the session entry. -**OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. The tool result holds the skill directory, which sets the source. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. +**OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. The tool result holds the skill text and the skill directory. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. **Source.** A name with a plugin prefix (`plugin:skill`) or a location in the agent's plugin directory gives `plugin`. A location in the repository gives `project`. A location in the user's agent configuration directory gives `user`. All other cases give `unknown`. @@ -101,14 +131,18 @@ skills_used: | ID | Decision | Choice | Why (and what we rejected) | Source | |----|----------|--------|----------------------------|--------| -| D-001 | Where to record skill use | A new optional `skills_used` list in the AI coding session evidence | One lookup answers "was skill X used". Rejected: a per-skill breakdown inside the tool usage summary, which mixes two concepts. Rejected: no change, which leaves every consumer to scan the raw transcript. | drafting | -| D-002 | First-version providers | Claude Code and OpenCode | Both transcripts hold the skill name in a tool input. Rejected: all four providers, because the Cursor and Pi formats are not confirmed. Rejected: Claude Code only, because OpenCode support costs little. | drafting | -| D-003 | User-started skills | Count them, and record who started each use | A user who types a slash command uses the skill as much as a model that calls a tool. Rejected: tool calls only, which misses user starts. Rejected: one count with no split, which hides who chose the skill. | drafting | -| D-004 | Entry shape | One entry per skill, with total, model, user, and subagent counts | A policy finds a skill with one lookup. Rejected: one entry per combination of skill, starter, and agent, which lists one skill several times. | drafting | -| D-005 | Subagent skills | Merge into the session entry and add a subagent count | This matches how the evidence merges subagent token usage. Rejected: a skills list on each subagent, which adds detail no consumer asked for. Rejected: main agent only, which misses skills. | drafting | -| D-006 | Skill version | Out of scope | The transcript does not hold the skill file content. The `ai-agent-config` material records digests of repository skills. | drafting | -| D-007 | Skill source | `project`, `user`, `plugin`, or `unknown`, with no path | The source tells a reviewer if the skill came from the repository. A path exposes local user data. | drafting | -| D-008 | Failed skill calls | Do not count them | A failed call (for example, an unknown skill name) loads no skill. This matches the OpenCode parser, which counts only completed tool calls. Rejected: count every call. | drafting | +| D-001 | Where to record skill use | A spec entry of kind `skill` for each used skill | A skill is a source of the session, like a ticket or a plan. Its text gets the same storage, redaction, and annotations. Rejected: a separate `skills_used` list in the session material, which keeps the counts but not the text. | owner | +| D-002 | Who makes skill entries | The CLI, from the transcript | Skill use is a fact in the transcript, and no rule must pick it. This departs from D-001 of Spec 002 for skills only. That decision rejected transcript extraction because the CLI cannot tell which input sets the task. That reason does not apply to skills. | drafting | +| D-003 | Shape of the counts | A generic `metadata` object on the spec entry, with fields that depend on the kind | Other kinds can add data later with no new top-level field. Rejected: a field named `skill` (one field for each kind). Rejected: a separate `skills_used` list linked by digest (two places for one skill). | owner | +| D-004 | Skill text | The text that the agent loaded, taken from the transcript, without the call arguments | It is what the model received, and it needs no file read. It works for user and plugin skills. Rejected: the `SKILL.md` file on disk (it can change after the session, and the CLI would read files outside the repository). Rejected: the disk file with a fallback to the transcript (two origins for one field). | drafting | +| D-005 | Which skills get a material | All used skills, after secret redaction | A reviewer needs the text of every skill that shaped the work. Rejected: project skills only, and project and plugin skills only (both leave out skills that the session used). | drafting | +| D-006 | Limit on skill entries | A separate limit of 25 | Many skills must not push out the task and spec sources. Rejected: one shared limit. | drafting | +| D-007 | Role of a skill entry | No role | The CLI does not guess a role (D-004 of Spec 004). The kind `skill` already tells the purpose. | drafting | +| D-008 | First-version providers | Claude Code and OpenCode | Both transcripts hold the skill name and text. Rejected: all four providers, because the Cursor and Pi formats are not known yet. | drafting | +| D-009 | User-started skills | Count them, and record who started each use | A user who types a slash command uses the skill as much as a model that calls a tool. Rejected: tool calls only. Rejected: one count with no split. | drafting | +| D-010 | Subagent skills | Merge into the session entry and add a subagent count | This matches how the evidence merges subagent token usage. Rejected: a list on each subagent. Rejected: main agent only. | drafting | +| D-011 | Skill source | `project`, `user`, `plugin`, or `unknown`, with no path | The source tells a reviewer if the skill came from the repository. A path exposes local user data. | drafting | +| D-012 | Failed skill calls | Do not count them | A failed call loads no skill. This matches the OpenCode parser, which counts only completed tool calls. | drafting | ## Open Questions @@ -118,6 +152,8 @@ None. | Risk | Mitigation | |------|------------| -| An agent changes how it records skill use. | Golden fixtures for each format. An unknown format gives no entry and a warning, never a failed push. | +| An agent changes how it records skill use or skill text. | Golden fixtures for each format. An unknown format gives no entry and a warning, never a failed push. | | A user-started skill and its meta message count two times. | Count starts, not meta messages. A fixture covers both start types for the same skill. | -| A consumer reads a missing field as "no skills used" for Cursor or Pi. | Document that a missing field means "not recorded" for providers without support. | +| A skill text holds a secret or private data. | The secret redaction of Spec 002 runs on each skill text before upload. | +| A consumer reads no skill entry as "no skills used" for Cursor or Pi. | Document that no entry means "not recorded" for providers without support. | +| A consumer compares the material digest with the digest of the `SKILL.md` file. | Document that the material holds the loaded text, not the file. | From ef559262790a402b437285cd3c21504fe41474f1 Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 7 Oct 2026 12:12:48 +0200 Subject: [PATCH 3/4] spec: issue-3540 capture full skill packages and use issue-based name Refs #3540 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 62aaeaaa-033c-4fb3-abac-9ee026634f60 --- docs/specs/006-trace-skills-used.md | 159 ------------------- docs/specs/issue-3540-trace-skills-used.md | 170 +++++++++++++++++++++ 2 files changed, 170 insertions(+), 159 deletions(-) delete mode 100644 docs/specs/006-trace-skills-used.md create mode 100644 docs/specs/issue-3540-trace-skills-used.md diff --git a/docs/specs/006-trace-skills-used.md b/docs/specs/006-trace-skills-used.md deleted file mode 100644 index ae7d7f1ff..000000000 --- a/docs/specs/006-trace-skills-used.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -owner: jiparis -ticket: https://github.com/chainloop-dev/chainloop/issues/3540 -prd: ---- - -# Spec 006: Skills used in a Chainloop Trace session - -## Summary - -Chainloop Trace will record the skills that an agent used in a session, with the text of each skill. Each used skill becomes a spec entry of the new kind `skill`. The skill text that the agent loaded goes into the attestation as an EVIDENCE material, the same as other spec sources. Spec entries get a new optional `metadata` object. Its fields depend on the kind. For a skill, it holds the skill source and the use counts. - -This spec changes the spec capture of [Spec 002](002-session-spec-capture.md) and [Spec 004](004-spec-source-role.md). It does not replace them. It departs from D-001 of Spec 002 for skills only. The CLI makes the skill entries from the transcript. The agent does not write them. The first version supports Claude Code and OpenCode. - -## Problem - -- The evidence shows which skills are available to an agent, but not which skills the agent used. The `ai-agent-config` material lists the skill files in the repository. That list does not show use. -- The tool usage summary counts calls per tool name. All Claude Code skill calls go into one `Skill` entry, so the summary does not show which skills ran. -- The skill names and texts are in the raw transcript only. To answer "did this session use skill X?", a policy or the UI must scan every transcript entry. It must also know the transcript format of each agent. -- A skill sets how the agent works, as a spec sets what the agent builds. A reviewer cannot see the skill text next to the other sources of the session. -- A user can start a skill with a slash command (for example `/graphify`). That skill use is not a tool call, so a scan of tool calls alone does not find it. - -## Goals and Non-Goals - -- Goal: A policy or the UI can find the skills that a session used in the spec entries of the session material. -- Goal: The attestation holds the skill text that the agent loaded, as a material that a reviewer can download and verify. -- Goal: The record separates skills that the model started from skills that the user started, and includes skills that subagents used. -- Goal: The `metadata` object can hold kind-specific data for other kinds later, with no new top-level field. -- Non-goal: Store the `SKILL.md` file byte for byte. The material holds the text that the agent loaded. -- Non-goal: Read skill files from disk. All skill data comes from the transcript. -- Non-goal: Support Cursor and Pi in the first version. Their transcript formats for skill use are not known yet. A later change can add them without a schema change. -- Non-goal: Record the local path of a skill file. -- Non-goal: Change how the agent captures other spec sources, the tool usage summary, the raw transcript, or the `ai-agent-config` material. - -## Requirements - -### R-001: One spec entry for each used skill - -The session material MUST hold one spec entry of kind `skill` for each skill that the session used. The title MUST be the skill name. The CLI MUST NOT set a role, a description, or a source address for a skill entry. The capture time MUST be the time of the first use in the transcript. A session that used no skill MUST have no skill entries. - -- Done when: sessions with no skill, one skill, and several skills each produce the expected spec entries. Two runs over the same transcript produce the same output. - -### R-002: Skill text material - -Each skill entry MUST point by digest to an EVIDENCE material that holds the skill text that the agent loaded. The CLI MUST take the text from the transcript. It MUST remove the arguments of one call, so that two calls with different arguments give the same text. It MUST use the text of the first use. When a later use gives a different text, the CLI MUST record a warning. The text MUST go through the secret redaction of Spec 002 (R-007) before upload. The material MUST carry the same annotations as other spec materials, with `chainloop.spec.kind` set to `skill` and `chainloop.spec.title` set to the skill name. - -- Done when: the attestation holds one EVIDENCE material for each used skill, and its digest downloads the redacted skill text. - -### R-003: Metadata object - -A spec entry MAY hold a `metadata` object. The fields of the object depend on the kind of the entry. This spec defines the fields for the kind `skill` only. Entries of other kinds MUST leave the object out. - -### R-004: Use counts and who started the skill - -The metadata of a skill entry MUST hold the total number of uses. It MUST also hold the number of uses that the model started and the number that the user started. The two start counts MUST add up to the total. - -- A model start is a skill tool call that loaded a skill. A failed call MUST NOT count. -- A user start is a slash command that loads a skill. A slash command that does not load a skill MUST NOT count. Examples are `/clear`, `/model`, and custom commands that are not skills. -- One skill use MUST count one time only, also when the transcript records it in more than one entry. -- Done when: fixtures with a model start, a user start, both starts for one skill, a failed call, and built-in commands produce correct counts. - -### R-005: Subagent skills - -Skills that subagents used MUST count in the totals of the same entry. The metadata MUST also hold the number of uses that came from subagents. - -- Done when: the main agent and a subagent use the same skill in one session. The evidence holds one entry with the correct total and subagent count. - -### R-006: Skill source - -The metadata of a skill entry MUST hold a source. The values are `project` (a skill from the repository), `user` (a skill from the user's agent configuration), `plugin` (a skill from an installed plugin), and `unknown`. The source MUST come from data in the transcript. Examples are a plugin prefix in the skill name and the location of the skill file. The evidence MUST NOT hold the skill file path. When the data is not sufficient, the source MUST be `unknown`. - -- Done when: fixtures for each source value, plus a transcript with no location data, produce the expected source. - -### R-007: Limit on skill entries - -Skill entries MUST have their own limit of 25 for each session. They MUST NOT count against the limit of the spec sources that the agent writes. When a session used more skills than the limit, the CLI MUST keep the skills that the session used first and record a warning. - -### R-008: Provider support - -Claude Code and OpenCode MUST produce skill entries. Cursor and Pi MUST produce none until a later change adds support. For those providers, no skill entry MUST mean "not recorded" and MUST NOT mean "no skills used". - -### R-009: Compatibility and failure safety - -The new kind and the `metadata` object MUST be additive. The schema version MUST stay the same, and existing evidence MUST stay valid. A consumer that does not know the kind `skill` MUST still read the other spec entries. When the parser cannot read a skill use, it MUST skip it and add a warning to the evidence. A bad skill use MUST NOT stop evidence creation or block the Git push. - -- Done when: old evidence and new evidence both pass schema validation. A malformed skill use produces a warning and the rest of the evidence. - -## Constraints - -- All skill data comes from the session transcript that Trace already collects. The change adds no new hook, network call, or file read. -- The repository is public. The evidence format is visible to all users. -- User and plugin skills can hold private text. The secret redaction applies to all skill text before upload. -- Output ordering must be deterministic, so the same transcript gives the same evidence digest. - -## Proposal - -Each provider parser already reads the full transcript to count tools, tokens, and messages. The parser will also collect skill uses in the same pass. At push time, the CLI adds the skill entries to the spec list, after the sources that the agent wrote. It uploads each skill text the same way as a spec file. When the text did not change since an earlier push, it reuses the redacted copy. - -```json -"spec": [ - { "kind": "ticket", "role": "task", "title": "ENG-1234: Add an export button", - "uri": "https://tracker.example.com/issue/ENG-1234", - "digest": "sha256:e4c2...", "captured_at": "2026-10-07T10:12:03Z" }, - { "kind": "skill", "title": "superpowers:brainstorming", - "digest": "sha256:ab12...", "captured_at": "2026-10-07T10:14:51Z", - "metadata": { "source": "plugin", "invocation_count": 3, - "by_model": 2, "by_user": 1, "in_subagents": 1 } } -] -``` - -```json -{ - "name": "superpowers-brainstorming.md", - "annotations": { - "chainloop.material.name": "spec-fd4e67-skill-superpowers-brainstorming", - "chainloop.material.type": "EVIDENCE", - "chainloop.spec.session_id": "fd4e6754-3b26-4f54-9807-13c58465bb35", - "chainloop.spec.kind": "skill", - "chainloop.spec.title": "superpowers:brainstorming" - } -} -``` - -**Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. The second message tells a skill command apart from a built-in or custom command. Claude Code also writes that meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The meta message holds the skill text and the skill location. Claude Code removes the frontmatter of the skill file and adds the call arguments at the end. The parser removes the arguments. Subagent transcripts go through the same steps, and their counts merge into the session entry. - -**OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. The tool result holds the skill text and the skill directory. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. - -**Source.** A name with a plugin prefix (`plugin:skill`) or a location in the agent's plugin directory gives `plugin`. A location in the repository gives `project`. A location in the user's agent configuration directory gives `user`. All other cases give `unknown`. - -## Decision Record - -| ID | Decision | Choice | Why (and what we rejected) | Source | -|----|----------|--------|----------------------------|--------| -| D-001 | Where to record skill use | A spec entry of kind `skill` for each used skill | A skill is a source of the session, like a ticket or a plan. Its text gets the same storage, redaction, and annotations. Rejected: a separate `skills_used` list in the session material, which keeps the counts but not the text. | owner | -| D-002 | Who makes skill entries | The CLI, from the transcript | Skill use is a fact in the transcript, and no rule must pick it. This departs from D-001 of Spec 002 for skills only. That decision rejected transcript extraction because the CLI cannot tell which input sets the task. That reason does not apply to skills. | drafting | -| D-003 | Shape of the counts | A generic `metadata` object on the spec entry, with fields that depend on the kind | Other kinds can add data later with no new top-level field. Rejected: a field named `skill` (one field for each kind). Rejected: a separate `skills_used` list linked by digest (two places for one skill). | owner | -| D-004 | Skill text | The text that the agent loaded, taken from the transcript, without the call arguments | It is what the model received, and it needs no file read. It works for user and plugin skills. Rejected: the `SKILL.md` file on disk (it can change after the session, and the CLI would read files outside the repository). Rejected: the disk file with a fallback to the transcript (two origins for one field). | drafting | -| D-005 | Which skills get a material | All used skills, after secret redaction | A reviewer needs the text of every skill that shaped the work. Rejected: project skills only, and project and plugin skills only (both leave out skills that the session used). | drafting | -| D-006 | Limit on skill entries | A separate limit of 25 | Many skills must not push out the task and spec sources. Rejected: one shared limit. | drafting | -| D-007 | Role of a skill entry | No role | The CLI does not guess a role (D-004 of Spec 004). The kind `skill` already tells the purpose. | drafting | -| D-008 | First-version providers | Claude Code and OpenCode | Both transcripts hold the skill name and text. Rejected: all four providers, because the Cursor and Pi formats are not known yet. | drafting | -| D-009 | User-started skills | Count them, and record who started each use | A user who types a slash command uses the skill as much as a model that calls a tool. Rejected: tool calls only. Rejected: one count with no split. | drafting | -| D-010 | Subagent skills | Merge into the session entry and add a subagent count | This matches how the evidence merges subagent token usage. Rejected: a list on each subagent. Rejected: main agent only. | drafting | -| D-011 | Skill source | `project`, `user`, `plugin`, or `unknown`, with no path | The source tells a reviewer if the skill came from the repository. A path exposes local user data. | drafting | -| D-012 | Failed skill calls | Do not count them | A failed call loads no skill. This matches the OpenCode parser, which counts only completed tool calls. | drafting | - -## Open Questions - -None. - -## Risks - -| Risk | Mitigation | -|------|------------| -| An agent changes how it records skill use or skill text. | Golden fixtures for each format. An unknown format gives no entry and a warning, never a failed push. | -| A user-started skill and its meta message count two times. | Count starts, not meta messages. A fixture covers both start types for the same skill. | -| A skill text holds a secret or private data. | The secret redaction of Spec 002 runs on each skill text before upload. | -| A consumer reads no skill entry as "no skills used" for Cursor or Pi. | Document that no entry means "not recorded" for providers without support. | -| A consumer compares the material digest with the digest of the `SKILL.md` file. | Document that the material holds the loaded text, not the file. | diff --git a/docs/specs/issue-3540-trace-skills-used.md b/docs/specs/issue-3540-trace-skills-used.md new file mode 100644 index 000000000..a0f5db3a3 --- /dev/null +++ b/docs/specs/issue-3540-trace-skills-used.md @@ -0,0 +1,170 @@ +--- +owner: jiparis +ticket: https://github.com/chainloop-dev/chainloop/issues/3540 +prd: +--- + +# Spec issue-3540: Skills used in a Chainloop Trace session + +## Summary + +Chainloop Trace will record the skills that an agent used in a session, and it will put each full skill into the attestation. Each used skill becomes a spec entry of the new kind `skill`. The CLI copies the skill folder when the session uses the skill. That copy includes `SKILL.md` and any scripts or other files. At push time, the CLI uploads the copy as one archive, as an EVIDENCE material, the same as other spec sources. Spec entries get a new optional `metadata` object. Its fields depend on the kind. For a skill, it holds the skill source and the use counts. + +This spec changes the spec capture of [Spec issue-3495](issue-3495-session-spec-capture.md) and [Spec issue-3531](issue-3531-spec-source-role.md). It does not replace them. It departs from D-001 of Spec issue-3495 for skills only. The CLI makes the skill entries. The agent does not write them. The first version supports Claude Code and OpenCode. + +## Problem + +- The evidence shows which skills are available to an agent, but not which skills the agent used. The `ai-agent-config` material lists the skill files in the repository. That list does not show use. +- The `ai-agent-config` material holds only `SKILL.md` files in the repository. It has no user or organization skills, and no scripts that a skill runs. +- The tool usage summary counts calls per tool name. All Claude Code skill calls go into one `Skill` entry, so the summary does not show which skills ran. +- A skill sets how the agent works, as a spec sets what the agent builds. A reviewer cannot see the skill, or the code that it runs, next to the other sources of the session. +- A user can start a skill with a slash command (for example `/graphify`). That skill use is not a tool call, so a scan of tool calls alone does not find it. + +## Goals and Non-Goals + +- Goal: A policy or the UI can find the skills that a session used in the spec entries of the session material. +- Goal: The attestation holds each used skill in full: its definition and its files, at all skill levels. +- Goal: The attestation does not need the `ai-agent-config` material to show a used skill. +- Goal: The record separates skills that the model started from skills that the user started, and includes skills that subagents used. +- Goal: The `metadata` object can hold kind-specific data for other kinds later, with no new top-level field. +- Non-goal: Record the arguments of each skill call. The raw transcript already holds them. +- Non-goal: Support Cursor and Pi in the first version. Their transcript formats for skill use are not known yet. A later change can add them without a schema change. +- Non-goal: Record the local path of a skill folder. +- Non-goal: Change how the agent captures other spec sources, the tool usage summary, the raw transcript, or the `ai-agent-config` material. + +## Requirements + +### R-001: One spec entry for each used skill + +The session material MUST hold one spec entry of kind `skill` for each skill that the session used. The title MUST be the skill name. The CLI MUST NOT set a role, a description, or a source address for a skill entry. The capture time MUST be the time of the first use in the transcript. A session that used no skill MUST have no skill entries. + +- Done when: sessions with no skill, one skill, and several skills each produce the expected spec entries. Two runs over the same transcript and the same copies produce the same output. + +### R-002: Copy at use time + +The CLI MUST copy the skill folder at the first hook event after the first use of the skill in the session. When no hook event comes before the push, the CLI MUST copy the folder at push time and record a warning. The copy MUST stay in the Trace state until the session ends. When a later use finds different folder content, the CLI MUST keep the first copy and record a warning. + +- Done when: a skill file that changes after the use and before the push does not change the uploaded package. + +### R-003: Skill package material + +Each skill entry MUST point by digest to one EVIDENCE material that holds the skill package. The package MUST be a reproducible archive of the skill folder: sorted entries, fixed timestamps, and no owner data. The same folder content MUST give the same digest in every session. Before it makes the archive, the CLI MUST redact each text file. It MUST use the secret redaction of Spec issue-3495 (R-007). The material MUST carry the same annotations as other spec materials, with `chainloop.spec.kind` set to `skill` and `chainloop.spec.title` set to the skill name. + +- Done when: the attestation holds one archive for each used skill, and the archive holds `SKILL.md` and the other files of the folder, redacted. + +### R-004: Package limits + +The CLI MUST read only a folder that holds a `SKILL.md` file. It MUST NOT follow a link that points out of the skill folder. It MUST leave out version-control folders. When a package is larger than 5 MB, the CLI MUST NOT upload it. It MUST keep the spec entry with no digest and record a warning. + +- Done when: fixtures cover a link out of the folder, a `.git` folder, and a package over 5 MB. Each gives the expected package or warning. + +### R-005: Metadata object + +A spec entry MAY hold a `metadata` object. The fields of the object depend on the kind of the entry. This spec defines the fields for the kind `skill` only. Entries of other kinds MUST leave the object out. + +### R-006: Use counts and who started the skill + +The metadata of a skill entry MUST hold the total number of uses. It MUST also hold the number of uses that the model started and the number that the user started. The two start counts MUST add up to the total. + +- A model start is a skill tool call that loaded a skill. A failed call MUST NOT count. +- A user start is a slash command that loads a skill. A slash command that does not load a skill MUST NOT count. Examples are `/clear`, `/model`, and custom commands that are not skills. +- One skill use MUST count one time only, also when the transcript records it in more than one entry. +- Done when: fixtures cover a model start, a user start, both starts for one skill, a failed call, and built-in commands. Each gives the expected counts. + +### R-007: Subagent skills + +Skills that subagents used MUST count in the totals of the same entry. The metadata MUST also hold the number of uses that came from subagents. + +### R-008: Skill source + +The metadata of a skill entry MUST hold a source, with one of these values: + +- `project`: a skill from the repository. +- `user`: a skill from the user's agent configuration. +- `organization`: a skill that an administrator installs for all users. +- `plugin`: a skill from an installed plugin. +- `unknown`: all other cases. + The evidence MUST NOT hold the skill folder path. When the data is not sufficient, the source MUST be `unknown`. + +- Done when: fixtures for each source value produce the expected source. + +### R-009: Limit on skill entries + +Skill entries MUST have their own limit of 25 for each session. They MUST NOT count against the limit of the spec sources that the agent writes. When a session used more skills than the limit, the CLI MUST keep the skills that the session used first and record a warning. + +### R-010: Providers, compatibility, and failure safety + +Claude Code and OpenCode MUST produce skill entries. Cursor and Pi MUST produce none until a later change adds support. For those providers, no skill entry MUST mean "not recorded". The new kind and the `metadata` object MUST be additive, and the schema version MUST stay the same. A skill that the CLI cannot read or copy MUST give a warning, and it MUST NOT stop evidence creation or block the Git push. + +- Done when: old evidence and new evidence both pass schema validation. A missing skill folder produces a warning and the rest of the evidence. + +## Constraints + +- The repository is public. The evidence format is visible to all users. +- User and organization skills can hold private text. The secret redaction applies to all text files of a package before upload. +- A hook must stay fast. The copy at use time must not delay the agent for a normal skill folder. +- Output ordering and archives must be deterministic, so the same input gives the same digests. + +## Proposal + +The trace hooks already run on agent events. They will also find new skill uses and copy each skill folder into the Trace state. At push time, the provider parser counts the skill uses in the transcript. The CLI then adds the skill entries to the spec list, after the sources that the agent wrote. It uploads each copy as an archive. When the copy did not change since an earlier push, it reuses the redacted archive. + +```json +"spec": [ + { "kind": "ticket", "role": "task", "title": "ENG-1234: Add an export button", + "uri": "https://tracker.example.com/issue/ENG-1234", + "digest": "sha256:e4c2...", "captured_at": "2026-10-07T10:12:03Z" }, + { "kind": "skill", "title": "asd-ste100", + "digest": "sha256:ab12...", "captured_at": "2026-10-07T10:14:51Z", + "metadata": { "source": "user", "invocation_count": 3, + "by_model": 2, "by_user": 1, "in_subagents": 1 } } +] +``` + +```text +asd-ste100.tar.gz +├── SKILL.md +├── references/writing-rules.md +├── examples/before-after.md +└── scripts/ste-lint.py +``` + +**Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. That meta message tells a skill command apart from a built-in or custom command. It gives the skill folder. Claude Code also writes the meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The tool-use hooks will also match the `Skill` tool. Each hook reads the new transcript entries, finds new skill folders, and copies them. Subagent transcripts go through the same steps, and their counts merge into the session entry. + +**OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The tool result holds the skill folder, so the plugin copies the folder after the call. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. + +**Source.** A name with a plugin prefix (`plugin:skill`) or a folder in the agent's plugin directory gives `plugin`. A folder in the repository gives `project`. A folder in the agent's managed configuration gives `organization`. A folder in the user's agent configuration gives `user`. All other cases give `unknown`. + +## Decision Record + +| ID | Decision | Choice | Why (and what we rejected) | Source | +|----|----------|--------|----------------------------|--------| +| D-001 | Where to record skill use | A spec entry of kind `skill` for each used skill | A skill is a source of the session, like a ticket or a plan. It gets the same storage, redaction, and annotations. Rejected: a separate `skills_used` list in the session material. | owner | +| D-002 | Who makes skill entries | The CLI, from the transcript and hook events | Skill use is a fact in the transcript, and no rule must pick it. This departs from D-001 of Spec issue-3495 for skills only. That decision rejected transcript extraction because the CLI cannot tell which input sets the task. That reason does not apply to skills. | drafting | +| D-003 | Shape of the counts | A generic `metadata` object on the spec entry, with fields that depend on the kind | Other kinds can add data later with no new top-level field. Rejected: a field named `skill`. Rejected: a separate list linked by digest. | owner | +| D-004 | Skill content | The full skill folder, not only the text that the agent loaded | A skill can run its own scripts, and a reviewer needs that code. The attestation must not depend on the `ai-agent-config` material. Rejected: the `SKILL.md` text from the transcript (no scripts, and Claude Code removes the frontmatter). | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | +| D-005 | When to copy the folder | At the first hook event after the use, with a fallback at push time | The copy matches what ran, also when the files change before the push. Rejected: read the folder at push time only. | drafting | +| D-006 | Package format | One reproducible archive for each skill | One material and one digest for each skill. The same skill version gives the same digest in all sessions. Rejected: one material for each file, which fills the attestation for a skill with many files. | drafting | +| D-007 | Package limits | Skip a package over 5 MB with a warning, no links out of the folder, no version-control folders | A large or unsafe folder must not block the push or read files outside the skill. Rejected: no size limit. | drafting | +| D-008 | Arguments of each call | Not recorded | Arguments are user text that can hold private data. The raw transcript already holds them. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | +| D-009 | Which skills get a package | All used skills, at all levels, after secret redaction | A reviewer needs every skill that shaped the work. Rejected: project skills only. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | +| D-010 | Limit on skill entries | A separate limit of 25 | Many skills must not push out the task and spec sources. Rejected: one shared limit. | drafting | +| D-011 | Role of a skill entry | No role | The CLI does not guess a role (D-004 of Spec issue-3531). The kind `skill` already tells the purpose. | drafting | +| D-012 | First-version providers | Claude Code and OpenCode | Both agents give the skill name and folder. Rejected: all four providers, because the Cursor and Pi formats are not known yet. | drafting | +| D-013 | Who started the skill, and subagents | Count model and user starts apart, and merge subagent uses into the session entry with their own count | A user who types a slash command uses the skill as much as a model that calls a tool. Subagent token usage merges the same way. | drafting | +| D-014 | Skill source | `project`, `user`, `organization`, `plugin`, or `unknown`, with no path | The source tells a reviewer where the skill came from. A path exposes local user data. | drafting | +| D-015 | Failed skill calls | Do not count them | A failed call loads no skill. This matches the OpenCode parser, which counts only completed tool calls. | drafting | + +## Open Questions + +None. + +## Risks + +| Risk | Mitigation | +|------|------------| +| An agent changes how it records skill use. | Golden fixtures for each format. An unknown format gives no entry and a warning, never a failed push. | +| The skill folder changes between the use and the first hook event. | The window is short. A later difference gives a warning. | +| A skill script or file holds a secret. | The secret redaction runs on each text file. Binary files go into the archive as they are, as for other spec materials. | +| A skill folder is very large. | The 5 MB limit skips the package with a warning. The entry and counts stay. | +| A consumer reads no skill entry as "no skills used" for Cursor or Pi. | Document that no entry means "not recorded" for providers without support. | From a5d888d7114c8c28dce41fe8317072a6d36b3c03 Mon Sep 17 00:00:00 2001 From: "Jose I. Paris" Date: Wed, 7 Oct 2026 13:38:54 +0200 Subject: [PATCH 4/4] spec: issue-3540 upload SKILL.md as the viewable skill definition Refs #3540 Assisted-by: Claude Code Signed-off-by: Jose I. Paris Chainloop-Trace-Sessions: 62aaeaaa-033c-4fb3-abac-9ee026634f60 --- docs/specs/issue-3540-trace-skills-used.md | 37 ++++++++++++++-------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/specs/issue-3540-trace-skills-used.md b/docs/specs/issue-3540-trace-skills-used.md index a0f5db3a3..19802d96b 100644 --- a/docs/specs/issue-3540-trace-skills-used.md +++ b/docs/specs/issue-3540-trace-skills-used.md @@ -8,7 +8,7 @@ prd: ## Summary -Chainloop Trace will record the skills that an agent used in a session, and it will put each full skill into the attestation. Each used skill becomes a spec entry of the new kind `skill`. The CLI copies the skill folder when the session uses the skill. That copy includes `SKILL.md` and any scripts or other files. At push time, the CLI uploads the copy as one archive, as an EVIDENCE material, the same as other spec sources. Spec entries get a new optional `metadata` object. Its fields depend on the kind. For a skill, it holds the skill source and the use counts. +Chainloop Trace will record the skills that an agent used in a session, and it will put each full skill into the attestation. Each used skill becomes a spec entry of the new kind `skill`. The CLI copies the skill folder when the session uses the skill. That copy includes `SKILL.md` and any scripts or other files. At push time, the CLI uploads two EVIDENCE materials, the same as other spec sources. One is the `SKILL.md` file, which a viewer shows. The other is an archive of the full folder, which a viewer offers as the download. Spec entries get a new optional `metadata` object. Its fields depend on the kind. For a skill, it holds the skill source and the use counts. This spec changes the spec capture of [Spec issue-3495](issue-3495-session-spec-capture.md) and [Spec issue-3531](issue-3531-spec-source-role.md). It does not replace them. It departs from D-001 of Spec issue-3495 for skills only. The CLI makes the skill entries. The agent does not write them. The first version supports Claude Code and OpenCode. @@ -48,13 +48,18 @@ The CLI MUST copy the skill folder at the first hook event after the first use o ### R-003: Skill package material -Each skill entry MUST point by digest to one EVIDENCE material that holds the skill package. The package MUST be a reproducible archive of the skill folder: sorted entries, fixed timestamps, and no owner data. The same folder content MUST give the same digest in every session. Before it makes the archive, the CLI MUST redact each text file. It MUST use the secret redaction of Spec issue-3495 (R-007). The material MUST carry the same annotations as other spec materials, with `chainloop.spec.kind` set to `skill` and `chainloop.spec.title` set to the skill name. +Each used skill MUST give two EVIDENCE materials: -- Done when: the attestation holds one archive for each used skill, and the archive holds `SKILL.md` and the other files of the folder, redacted. +- The definition: the `SKILL.md` file of the copy. The `digest` of the skill entry MUST point to it, as for other spec sources, so that a viewer can show it. +- The package: a reproducible archive of the full skill folder, with sorted entries, fixed timestamps, and no owner data. The `package_digest` field of the entry metadata MUST point to it. A viewer offers it as the download of the skill. + +The same folder content MUST give the same digests in every session. Before it makes the materials, the CLI MUST redact each text file with the secret redaction of Spec issue-3495 (R-007). The definition material and the `SKILL.md` file in the package MUST be the same bytes. Both materials MUST carry the same annotations as other spec materials, with `chainloop.spec.kind` set to `skill` and `chainloop.spec.title` set to the skill name. The annotation `chainloop.spec.skill.content` MUST tell them apart, with the value `definition` or `package`. + +- Done when: the attestation holds a definition and a package for each used skill. The package holds `SKILL.md` and the other files of the folder, redacted. ### R-004: Package limits -The CLI MUST read only a folder that holds a `SKILL.md` file. It MUST NOT follow a link that points out of the skill folder. It MUST leave out version-control folders. When a package is larger than 5 MB, the CLI MUST NOT upload it. It MUST keep the spec entry with no digest and record a warning. +The CLI MUST read only a folder that holds a `SKILL.md` file. It MUST NOT follow a link that points out of the skill folder. It MUST leave out version-control folders. When a package is larger than 5 MB, the CLI MUST NOT upload it. It MUST still upload the definition, leave out `package_digest`, and record a warning. - Done when: fixtures cover a link out of the folder, a `.git` folder, and a package over 5 MB. Each gives the expected package or warning. @@ -116,19 +121,24 @@ The trace hooks already run on agent events. They will also find new skill uses "digest": "sha256:e4c2...", "captured_at": "2026-10-07T10:12:03Z" }, { "kind": "skill", "title": "asd-ste100", "digest": "sha256:ab12...", "captured_at": "2026-10-07T10:14:51Z", - "metadata": { "source": "user", "invocation_count": 3, - "by_model": 2, "by_user": 1, "in_subagents": 1 } } + "metadata": { "source": "user", "package_digest": "sha256:cd34...", + "invocation_count": 3, "by_model": 2, "by_user": 1, + "in_subagents": 1 } } ] ``` ```text -asd-ste100.tar.gz -├── SKILL.md -├── references/writing-rules.md -├── examples/before-after.md -└── scripts/ste-lint.py +predicate.materials +├── spec-fd4e67-skill-asd-ste100 EVIDENCE SKILL.md sha256:ab12... definition +└── spec-fd4e67-skill-asd-ste100-pkg EVIDENCE asd-ste100.tar.gz sha256:cd34... package + ├── SKILL.md + ├── references/writing-rules.md + ├── examples/before-after.md + └── scripts/ste-lint.py ``` +A viewer shows the definition with the Markdown view that it uses for other spec sources. Its download link gets the package. + **Claude Code.** A model start is a `Skill` tool call. Its input holds the skill name. A user start is a user message that holds a `` marker, followed by a meta message that starts with `Base directory for this skill:`. That meta message tells a skill command apart from a built-in or custom command. It gives the skill folder. Claude Code also writes the meta message after a `Skill` tool call. The parser counts one use for each start, not for each meta message. The tool-use hooks will also match the `Skill` tool. Each hook reads the new transcript entries, finds new skill folders, and copies them. Subagent transcripts go through the same steps, and their counts merge into the session entry. **OpenCode.** OpenCode loads skills through its native `skill` tool, with one input, `name`. A model start is a completed call to this tool. The tool result holds the skill folder, so the plugin copies the folder after the call. The OpenCode documentation does not describe a slash command that starts a skill. For OpenCode, the user count stays at zero. OpenCode finds project skills in `.opencode/skills`, `.claude/skills`, and `.agents/skills` in the repository. It finds user skills in the same folders in the user's home configuration. @@ -144,8 +154,9 @@ asd-ste100.tar.gz | D-003 | Shape of the counts | A generic `metadata` object on the spec entry, with fields that depend on the kind | Other kinds can add data later with no new top-level field. Rejected: a field named `skill`. Rejected: a separate list linked by digest. | owner | | D-004 | Skill content | The full skill folder, not only the text that the agent loaded | A skill can run its own scripts, and a reviewer needs that code. The attestation must not depend on the `ai-agent-config` material. Rejected: the `SKILL.md` text from the transcript (no scripts, and Claude Code removes the frontmatter). | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | | D-005 | When to copy the folder | At the first hook event after the use, with a fallback at push time | The copy matches what ran, also when the files change before the push. Rejected: read the folder at push time only. | drafting | -| D-006 | Package format | One reproducible archive for each skill | One material and one digest for each skill. The same skill version gives the same digest in all sessions. Rejected: one material for each file, which fills the attestation for a skill with many files. | drafting | -| D-007 | Package limits | Skip a package over 5 MB with a warning, no links out of the folder, no version-control folders | A large or unsafe folder must not block the push or read files outside the skill. Rejected: no size limit. | drafting | +| D-006 | Package format | One reproducible archive for each skill | One package material and one package digest for each skill. The same skill version gives the same digest in all sessions. Rejected: one material for each file, which fills the attestation for a skill with many files. | drafting | +| D-007 | Package limits | Skip a package over 5 MB with a warning, no links out of the folder, no version-control folders | A large or unsafe folder must not block the push or read files outside the skill. The definition still uploads. Rejected: no size limit. | drafting | +| D-016 | How a viewer shows a skill | `SKILL.md` as its own definition material, which the entry digest points to. The archive is the download | A viewer shows the definition with the existing Markdown view and needs no archive support. Rejected: the archive only (a viewer must unpack it to show anything). Rejected: a file list in the metadata (more data in the session material, and a viewer still needs the archive to show a file). | owner | | D-008 | Arguments of each call | Not recorded | Arguments are user text that can hold private data. The raw transcript already holds them. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | | D-009 | Which skills get a package | All used skills, at all levels, after secret redaction | A reviewer needs every skill that shaped the work. Rejected: project skills only. | [PR comment](https://github.com/chainloop-dev/chainloop/pull/3541#discussion_r4205209188) | | D-010 | Limit on skill entries | A separate limit of 25 | Many skills must not push out the task and spec sources. Rejected: one shared limit. | drafting |