feat(repo): skill/skill-catalog arms, cache tokens, MCP config when repo ships one

dokimi/changelog.d/20261020_090000_napetrov_repo-skill-arms.md

@@ -0,0 +1,30 @@
+### Added
+
+- **`repo run` skill arms** — `--arm guidance:none+skill:<path>` installs one
+  skill under `.claude/skills/<name>/`, and `skill-catalog:<dir>` installs
+  every skill in a directory. `result.json` records `skills`
+  (`skills_used`, `skill_invoked`, `skill_selected_correct`).
+  `repo report --json` gives, per row, the mean of the token fields `result.json` already has
+  (`input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`).
+  `repo tasks check --arm` rejects a skill
+  path that does not exist, and a skill whose `SKILL.md` `name` differs from its
+  directory name. The arm is named by the directory (`skill:lint`), so
+  `repo compare` can select it. Resuming a batch with other skill content
+  under the same arm name is refused.
+- **`args_if_present` in harness profiles** — arguments appended only when a
+  file exists in the workspace and the tool mode is `all`.
+
+### Fixed
+
+- **claude-code starts a repo's MCP servers** — the stock profile passed
+  `--strict-mcp-config` without `--mcp-config`, so a repo that ships
+  `.mcp.json` ran with no MCP tools. It now passes `--mcp-config .mcp.json`
+  when that file is present and `--tools all` is used (`--tools none`/`read` do not
+  limit MCP tools, so they start no MCP server).
+  This shifts baselines: a repo that ships `.mcp.json` now starts its MCP
+  servers under `--tools all`, so its results differ from earlier runs of the
+  same repo and arm. Compare only batches run after this change.
+- **A malformed `.mcp.json` is caught up front** — `repo validate` warns (and still succeeds) about one that is not a JSON object (or whose `mcpServers` is not) at a task base (a file that is not UTF-8 text included), and `repo run --tools all` exits before any cell, instead of every cell failing as a task failure when the CLI rejects the file; a `patch:` arm that writes one is a cell error.
+- `check_arms` refuses, before any cell, a base that tracks `.claude` or `.claude/skills` as a symlink.
+- A `guidance:none` arm can install a skill that the base ships under the guidance globs, because the arm drops those files first.
+- A `SKILL.md` `name` that YAML reads as an impossible date is not a crash.

dokimi/docs/benchmarking-and-comparison.md

@@ -274,6 +274,33 @@ python cli.py arms run \
 The `baseline` arm is the control. Every other arm is compared against it. Keep
 the answer model and judge fixed when comparing arms.
 
+### Skill arms in repository mode
+
+`repo run` arms take the same `+` composition. Besides `guidance:none`,
+`guidance:raw` and `guidance:bridge`, an arm can install Claude Code skills into
+the task workspace at `.claude/skills/<dir name>/` before the agent starts:
+
+- `skill:<path>`: one skill directory (holding `SKILL.md`);
+- `skill-catalog:<dir>`: every subdirectory of `<dir>` with a `SKILL.md`, so the
+  agent has to pick the right one.
+
+A relative path is relative to the manifest's `eval_dir`. An arm without a
+guidance part keeps the repository's guidance (`guidance:raw`).
+
+```bash
+dokimi repo run <repo>@<rev> --model <m1> --model <m2> --batch skills \
+  --arm guidance:raw,guidance:raw+skill:skills/lint,guidance:none+skill-catalog:skills
+dokimi repo tasks check <repo>@<rev> --arm skill-catalog:skills   # paths exist, catalog has expected_skill
+```
+
+When a cell installs skills, its `result.json` gets a `skills` object:
+`skills_used` (the skills the agent called with the `Skill` tool, or whose
+`SKILL.md` it read), `skill_invoked`, and `skill_selected_correct` (the task's
+optional `expected_skill` was the first skill it used). `repo report` counts
+both per row. Every cell also records `cache_read_tokens` and
+`cache_write_tokens` with the input and output tokens, and
+`repo report --json` gives their means.
+
 ### Reading the significance tables
 
 Arms are compared **paired per question** against the one shared `baseline`

dokimi/docs/preparing-a-repository.md

@@ -7,9 +7,11 @@ SPDX-License-Identifier: Apache-2.0
 
 > **Status: recommendations.** §1–§7 apply today. The measurement loop in §8 runs
 > today for guidance arms (`guidance:none`, `guidance:raw`, `guidance:bridge`).
-> `guidance:bridge` measures the `CLAUDE.md` import (§7). Arms that add a tool are
-> planned; a changed guidance line can be tested today with two tasks that differ
-> in a `setup_patches` entry (§8).
+> `guidance:bridge` measures the `CLAUDE.md` import (§7). The `skill:` and
+> `skill-catalog:` arms, which install skills, run today too
+> ([skill arms](benchmarking-and-comparison.md#skill-arms-in-repository-mode)). Arms
+> that add any other tool are planned; a changed guidance line can be tested today
+> with two tasks that differ in a `setup_patches` entry (§8).
 
 This page is for repository owners who want coding agents (Claude Code, Codex,
 Copilot and others) to build, test, change and review their code reliably. It
@@ -240,7 +242,7 @@ Treat each item above as a hypothesis for your repository.
    use. Today an arm can remove or keep the guidance (`guidance:none`,
    `guidance:raw`), or inject a file (`guidance:bridge`, which writes a `CLAUDE.md`
    containing `@AGENTS.md` next to each `AGENTS.md` for profiles with a bridge key).
-   Testing an added tool needs planned arms. Today, test one added guidance line
+   Testing an added tool, other than a skill (`skill:`, `skill-catalog:`), needs planned arms. Today, test one added guidance line
    with two task directories that differ only in a `setup_patches` entry (same
    prompt, same graders, distinct ids), or in their `base` commit (see
    [workspace preparation](repo-evaluation-contract.md#workspace-preparation-done-by-the-framework)).
@@ -251,9 +253,12 @@ Treat each item above as a hypothesis for your repository.
    Wilson intervals in `repo report` do not overlap, or raise the cell count. A patch or a second `base` changes only the
    workspace tree: one image serves every task, so a tool or binary (a pinned
    formatter, a navigation server) has to be in the image already. With the
-   `claude-code` profile, which passes `--strict-mcp-config` with no
-   `--mcp-config`, an MCP server added by a patch is not loaded, so this route
-   cannot test an added MCP tool yet.
+   `claude-code` profile, which passes `--strict-mcp-config`, an MCP server is
+   loaded only from the `.mcp.json` at the workspace root, and only with
+   `--tools all` (the profile's `args_if_present` rule: the file must exist in the
+   workspace, and `read` and `none` never pass `--mcp-config`). A patch that adds
+   or edits that file therefore adds the server in `all` mode; `repo run --tools all`
+   refuses a task base whose `.mcp.json` is malformed before any cell starts.
 3. Look at whether the agent used the change, not only at the score. A guidance
    line can move the score without the tool it names being called; tool use tells
    you whether the tool itself mattered. An arm in which the tool was never called

dokimi/docs/repo-evaluation-contract.md

@@ -263,6 +263,7 @@ belongs to the repository's grader and is passed through unread, except `grader`
 | `timeout_s`, `max_turns`, `budget_usd` | no | framework | per-task caps, overriding run defaults; each a finite number > 0 |
 | `host` | no | framework | `{"min_cpus": <int>, "cpu_vendors": [<str>], "cpu_flags": [<str>]}`, every key optional: what the hidden test needs to show the bug (a thread race that needs 3+ CPUs, an ISA path). Vendor and flag names are `/proc/cpuinfo`'s (`GenuineIntel`, `avx512f`); both match in any case, and a vendor other than `GenuineIntel`, `AuthenticAMD`, `HygonGenuine`, `CentaurHauls`, `Shanghai` is refused (`repo validate` too). A probe that read neither vendor nor flags from `/proc/cpuinfo` is a probe failure, not a miss They are checked against a probe container started like the grading one: CPUs are its affinity mask capped by its cgroup CPU quota (`docker --cpus`). On a host that misses them the task is unsupported (§4.3), and the probed facts go to `meta.json` `host` of a cell that runs, or to its `unsupported.json` `host` when it does not |
 | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` |
+| `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. `result.json` `skills` records whether the agent used it and whether it used it first |
 | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` |
 
 `setup_patch` (a single string) is accepted as a synonym for a one-element
@@ -325,6 +326,18 @@ Both write files, as step 1 does: no object of the source's git dir is copied
 into the workspace, so its object store gains only the blobs of the files the
 arm wrote, and step 8 drops the guidance the arm replaced.
 
+A skill arm (`skill:<path>`, `skill-catalog:<dir>`, joined to a guidance part
+with `+`) installs skill directories into `.claude/skills/<dir name>/` after the
+guidance part, so `guidance:none+skill:x` keeps `x`. The arm is named by the
+directory names (`guidance:none+skill:x`, no `/`), so `repo compare` can select it.
+The `name` in a skill's `SKILL.md` has to equal its directory name. A skill is refused before
+any cell runs if it holds a link that copying cannot take (out of the skill,
+dangling, looping, to a directory) or a file that is not regular. Preparation
+also refuses to write through a `.claude` or `.claude/skills` the repository
+ships as a link. A batch records the content of each skill arm it ran
+(`_eval/skills.json`), so resuming it with another path of the same
+directory name, or with edited skill content, is refused.
+
 `meta.json` records the base (`base_rev`, `base_sha`, `base_date`), the arm,
 the SHA of every applied patch, and `start_sha`: the workspace `HEAD` after
 step 7, the commit the agent starts from (§4). `base_sha` names the upstream
@@ -335,7 +348,9 @@ grader diffs the agent's changes from `start_sha`. With the arm come
 files with no counterpart at `<rev>`); for `patch:` an `arm_patch` object (the
 file's absolute `path` and full `sha256`, `added`: the files the patch added
 that neither the guidance globs nor `.mcp.json` match, and `mcp_servers`: the servers
-the patch added to the `.mcp.json` at the root, read before the agent runs).
+the patch added to the `.mcp.json` at the root, read before the agent runs). `args_if_present` lists the profile's
+`args_if_present` files the cell's command line carries (§6): `[]` when the
+workspace has none of them, and always outside the `all` tool mode.
 
 Step 3 removes history only. A fix that landed upstream before a model's
 training cutoff may still be known to the model, so reports split results by
@@ -387,6 +402,17 @@ Code's `modelUsage`), `main` when the harness reported the parent's only, and
 `null` when it reported none. `subagent_calls` counts the parent's and
 subagents' `Agent`/`Task` calls; `null` when the harness wrote no result record.
 
+A cell of an arm that installs skills also has a `skills` object in
+`result.json`: `measured`, `skills_installed` (the installed skill names),
+`skills_used` (the installed skills the agent called with the `Skill` tool or
+whose `SKILL.md` it read, in order of first use), `expected_skill` (the task's,
+or `null`), `skill_invoked` (the expected skill was used; any installed skill
+when the task names none) and `skill_selected_correct` (the expected skill was
+the first one used; `null` when the task names none). When the trace could not
+be read (the harness never started), `measured` is `false` and `skill_invoked`
+and `skill_selected_correct` are `null`: the usage is unknown, not "did not use
+the skill". `repo report` leaves such cells out of its skill counts.
+
 For a `patch:` arm with `added` files or `mcp_servers`, `result.json` has an
 `uptake` object: whether the agent used what the arm added. `files` and
 `mcp_servers` list what it added (sorted); `commands` counts
@@ -637,6 +663,8 @@ guidance:                            # DECLARED; verified by the reach probe, ne
   resolution: ancestors-of-cwd + lazy-on-subtree-access
   imports: "@path"
 bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"}
+args_if_present:                     # argv added after the "{tools}" element when the workspace file exists and the tool mode is `all`
+  .mcp.json: ["--mcp-config", ".mcp.json"]
 ```
 
 - `invoke` and `version_cmd` are argv lists. Each element becomes exactly one
@@ -653,6 +681,18 @@ bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"}
   directory, with `{config_dir}` and `{support_dir}` (the framework's helper
   scripts) as the agent's container sees them. The built-in `claude-code`
   profile uses them for a hook that refuses backgrounded shell commands.
+- `args_if_present` checks that the file exists, not that the harness accepts
+  it. The built-in `claude-code` profile passes `.mcp.json` as `--mcp-config`
+  in the `all` tool mode, and the CLI exits 1 before its first turn on a file
+  that is not valid MCP configuration (`{` for one). So a `.mcp.json` that is
+  not a JSON object, or whose `mcpServers` is not an object, is caught up
+  front: `repo validate` warns about the one at each task base (at
+  the eval tree when the source has no git history) and still succeeds, `repo run --tools all`
+  exits 1 before any cell, and a cell whose arm wrote
+  such a file (a `patch:` arm) is a cell error, not a graded run. The check
+  reads the task base, so a `patch:` arm that repairs the file does not help.
+  Fix or remove the file in the task base; `--tools read` or `none` do not
+  pass it, so `repo run` does not check it there.
 
 Profiles for other harnesses (Codex, Gemini CLI, Copilot CLI, Cursor agent,
 Aider, OpenCode, in-house agents) follow the same shape. Their `guidance`