Repository navigation
feat(repo): skill/skill-catalog arms, cache tokens, MCP config when repo ships one - #137
Conversation
…n results `repo run` arms now compose with `+` like the Q&A --arms: a guidance part (none/raw/bridge, raw by default) plus `skill:<path>` (one skill dir) and `skill-catalog:<dir>` (every skill subdir), installed into the workspace at .claude/skills/<name>/ in prep step 5 and listed with digests in meta.json. Relative paths are under eval_dir; `repo run` and `repo tasks check --arm` refuse missing skills and a catalog without a task's `expected_skill`. A cell with skills records which ones the agent used (Skill tool calls or reads of their SKILL.md) as skills.skill_invoked / skill_selected_correct; `repo report` counts them. Each cell also records cache read/creation input tokens, and the report gives per-row token means. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… repo's .mcp.json servers The stock claude-code profile passes --strict-mcp-config with no --mcp-config, so a repo that ships .mcp.json ran with no MCP tools. Passing --mcp-config unconditionally breaks repos without the file (Claude Code exits on a missing config). args_if_present appends arguments only when the named file exists in the run's workspace. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #137 +/- ##
==========================================
+ Coverage 92.24% 92.28% +0.04%
==========================================
Files 239 240 +1
Lines 29934 30293 +359
Branches 4917 5003 +86
==========================================
+ Hits 27613 27957 +344
- Misses 1633 1642 +9
- Partials 688 694 +6
🚀 New features to boost your workflow:
|
ff1503d to
5c17c68
Compare
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 5c17c68. The arm grammar, the guidance-then-skills ordering and the args_if_present design hold up. Four things are worth fixing before merge, three inline and one below.
What I ran, on a Linux review host at 5c17c68:
make ci-local: 20 passed, 0 failed, 4 skipped (docker-only and org-level checks); changed-line coverage gate passes.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the new code: 56 mutants, 23 of which pass the full suite. The one that matters is inline on
runner.py.
Findings that cannot be anchored inline:
- A skill is matched by its directory name, and nothing checks the
namein itsSKILL.mdfrontmatter.Skill.nameisroot.name(arms.py:100), andskill_usageand theexpected_skillcheck compare against it. The Agent Skills spec requires the frontmatternameto equal the directory name, butSKILL.mdis never parsed, so a skill that breaks this is accepted. If the agent reports the frontmatter name, as inskills/dirname/SKILL.mdwithname: pretty-nameand an event{"kind": "tool", "name": "Skill", "skill": "pretty-name"},skill_usagereturnsskills_used: [],skill_invoked: False,skill_selected_correct: False. The tests'_skillhelper always writesnameequal to the directory name. Suggestion: refuse a skill whose frontmatternamediffers from its directory (or read the frontmatter name), and add a test with differing names.
Checked, no issues
- Schema versioning:
args_if_presentis an optional key added toharness_profile.v1.jsonin place; old profiles still validate, andschema_compatagainst the base passes. - Old results:
report.summarizereads the token fields with.get, so result files without cache fields orskillsstill summarize. - Arm ordering: guidance is applied before skills are installed, so
guidance:none+skill:xkeeps the skill...in a top-levelskill:path is refused afterresolve(). - Housekeeping: the docs page is already in
nav:, the changelog fragment is inchangelog.d/, no generated artifacts are committed, and the tests run offline.
…e a result dir A skill or catalog entry with a symlink to a host path was copied with its target's bytes (copytree dereferences), so the agent could read host files. Links that leave a skill, and catalog entries that leave the catalog, are now refused before any cell runs. Two arms whose strings slug to the same cell name (skill:a/b and skill:a_b) shared one result directory, so the second arm silently got no row. check_arms now refuses such arms up front. Regression tests cover both, and the runner's --mcp-config wiring. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| def check_arms(arms: list[str], content_dir: Path, tasks: list[Any]) -> list[RepoArm]: | ||
| """Parse every arm and find its skills before anything runs; a catalog must hold each task's | ||
| ``expected_skill``. Returns the parsed arms.""" | ||
| from .runner import slug # runner imports this module |
There was a problem hiding this comment.
The collision check works, but from .runner import slug here adds a new import cycle (arms -> runner -> grading -> workspace -> arms), and test_import_cycles_do_not_grow fails on a9b74b5.
Moving slug into arms.py (or a small module both import) would fix it.
There was a problem hiding this comment.
Fixed in f7280b0: slug() moved into arms.py, and runner.py imports it from there, so arms no longer imports runner. test_import_cycles_do_not_grow passes.
| dest = ws / skill.target | ||
| if dest.exists(): | ||
| raise PrepError(f"task {task.id}: the workspace already has {skill.target}") | ||
| dest.parent.mkdir(parents=True, exist_ok=True) |
There was a problem hiding this comment.
If the repo at the task base tracks .claude (or .claude/skills) as a symlink to a host path, this mkdir/copytree writes through it: with .claude -> /tmp/x in the base, --arm skill:skills/lint created /tmp/x/skills/lint/SKILL.md on the host and the cell still passed. Please refuse when ws/.claude or ws/.claude/skills is a symlink, or when dest.resolve() is not inside ws.resolve().
There was a problem hiding this comment.
Fixed in f7280b0: _install_skills refuses when .claude or .claude/skills is a symlink in the workspace, and when the destination would resolve outside the workspace. Regression test test_a_skill_is_not_written_through_a_repository_link (both link positions) fails on the old code.
| if element == TOOLS_ELEMENT: | ||
| out.extend(self.tools[tools]) | ||
| for name in sorted(present & self.args_if_present.keys()): | ||
| out.extend(self.args_if_present[name]) |
There was a problem hiding this comment.
These args are added for every tool mode, so --tools none / --tools read on a repo with .mcp.json now start its MCP servers, and --tools does not limit MCP tools. The test in test_repo_reach.py even pins --tools "" --mcp-config .mcp.json. Should args_if_present apply only to the all mode (or be keyed per tool mode)?
There was a problem hiding this comment.
Done in 4e4e402 (option a): args_if_present is added only in the all tool mode, so --tools none/read no longer start the repo's MCP servers. tests/test_repo_reach.py now pins that none/read get no --mcp-config and all does; the end-to-end MCP test runs with --tools all.
| record: dict[str, Any] = {"task": task.id, "family": task.family, "arm": arm, "model": opts.model, "rep": rep, | ||
| "harness": opts.profile.name, "harness_version": meta["harness_version"], | ||
| "base_date": meta["base_date"], "score_task": bool(task.score), "agent": agent, "grade": grade} | ||
| if meta.get("skills"): # without a trace the agent used none of them |
There was a problem hiding this comment.
When the trace could not be read, this records skill_invoked=False and skill_selected_correct=False, and repo report counts the cell as "did not use the skill". That is "not measured" read as a result.
Please record null for both (or a measured: false flag) here and skip such cells in the report counts.
There was a problem hiding this comment.
Fixed in f7280b0: a cell whose trace cannot be read now records skill_invoked and skill_selected_correct as null with measured: false. The report leaves those cells out of the counts, and the n/… denominator is the measured cells only. Test test_unmeasured_skill_usage_is_left_out_of_the_counts fails on the old code.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at a9b74b5. The three threads from my earlier review are fixed and each has a regression test. make ci-local fails on the import cycle @vlad-perevezentsev already flagged at arms.py:150; I agree, and the test passes on the PR's base. The link check still lets through links that make the copy fail. Two inline points, two more below.
What I ran, on a Linux review host at a9b74b5:
make ci-local: 19 passed, 1 failed, 4 skipped (docker-only and org-level checks). The failure istests/test_import_architecture.py::test_import_cycles_do_not_grow; 5318 tests passed, 1 skipped, and the changed-line coverage gate passes.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the new code: 75 mutants, 30 of which pass the touched test files. The ones that matter are inline.
Findings that cannot be anchored inline:
repo comparecannot select a skill arm, because the selector splits on the last/.parse_selector(compare.py:75, outside this diff, predates the PR) doesrpartition("/"). The PR makes/normal in an arm string. After running the docs' batch (guidance:rawandguidance:raw+skill:skills/lint),--treatment guidance:raw+skill:skills/lintis parsed as batchguidance:raw+skill:skills, armlint, and fails withno finished cell matches the treatment 'guidance:raw+skill:skills/lint'. Suggestion: try the whole string as an arm first, and fall back toBATCH/ARMonly when it matches no cell, or document that skill arms cannot be compared byrepo compare.- Nit:
docs/repo-evaluation-contract.mdis not updated. §3 still says every arm is aguidance:*arm, the §6 profile example omitsargs_if_presentand shows--strict-mcp-configwithout--mcp-config, and the result tables omitskills,expected_skilland the cache-token fields.
Still open from my earlier review: none. The link-out, cell-name collision and runner-wiring threads are fixed.
Checked, no issues
- Cell names:
check_armsbuilds the slug exactly ascell_namedoes (slug(arm.replace(":", "-"))).skill:sandskill:s/are refused, andskill:sandguidance:raw+skill:sare not wrongly refused.check_armsruns on bothrepo runandtasks check --arm. - Links out of a skill: a direct link, a chain
a -> b -> host file, a dangling link to a host path and a catalog entry linking out are all refused. - MCP wiring:
presentis built fromworkspace/<name>for names in the profile's ownargs_if_present. Disablingpresent=or usingrd / nmakes the new runner test fail. - Schema versioning and housekeeping: the optional
args_if_presentkey is added in place,schema_compatpasses, old results still summarize, and no generated artifacts are committed.
- arms: move slug() out of runner so arms no longer imports runner (closes the import cycle test_import_cycles_do_not_grow catches) - arms: refuse skill links that dangle, loop, point at a directory, or are not regular files; they crashed every cell at workspace prep - workspace: refuse to install a skill through a .claude or .claude/skills that the repository ships as a link - runner/report: a cell whose trace could not be read records skill usage as null and is left out of the skill counts - docs: contract §3 covers skill arms; §6 shows args_if_present - tests: link cases, catalog with the expected skill first, guidance:none keeping a skill, tree hash sensitivity, unmeasured cells, write-through Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Skill arms become forms of workspace.Arm (skill:<dir>, skill-catalog:<dir>, joined to a guidance part with +), named by directory so the name holds no '/'. The separate parse_arm grammar of arms.py is gone; guidance:rev: and patch: and pin_arm work with skills. Cache token fields use main's cache_read_tokens/cache_write_tokens next to usage_scope and subagent_calls; skill cells follow main's unsupported.json handling. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
--tools none/read do not limit MCP tools, so a repo's .mcp.json started its servers in arms meant to give the agent no tools. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…tory The agent calls a skill by its frontmatter name, usage matches on the directory name; a mismatch scored skill_invoked False. The merge with main added the check; this adds its regression test and the changelog line. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Skill arms are named by directory (no '/'), so parse_selector needs no whole-string fallback; the test pins that. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Pushed e1dd820 (merge of main in 263da4a, then one commit per point).
|
Arm.skills is a tuple of (kind, path) pairs; the annotation said tuple[str, ...] and mypy strict failed on it. Add a test for the guard, which no test ran: two paths with one directory name in one invocation are refused. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
A skill arm is named by its directories, so resuming a batch with another path of that name, or after editing a host skill, pooled two skills in one row. The batch now records each skill arm's tree sha256 in _eval/skills.json and refuses a mismatch; a new arm is added, the same bytes resume. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
4e4e402 limited the arguments to --tools all and updated the changelog and the contract, but the profile schema still said they apply whenever the file exists. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
An agent that exits before its first turn (for example on an MCP config it rejects) leaves an empty trace that reads without error, so the cell was recorded as measured and counted as not using the skill. A trace with no event and no result line is now unmeasured, as an unreadable one is. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
These mutants passed the suite: skill_invoked as bool(used), the Skill event without the command fallback, the expected_skill name check disabled, the report's selection count for tasks without expected_skill, and the frontmatter quote strip removed. Each now fails a test. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…irectory reads; document a malformed .mcp.json Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Review at 135b273, items not on a thread:
|
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at ecd26bf. The three test gaps, the patch docstring and the stale comment from my last review are closed, and the new tests kill the mutants they were written for. The base check still disagrees with preparation in two places, and one frontmatter value crashes check_arms. Four things are worth fixing before merge: three inline and one below.
What I ran, on a Linux review host at ecd26bf:
make ci-local: 25 passed, 1 failed, 6 skipped. The failure ispip-audit: it reportsgitpython 3.2.0andvirtualenv 21.14.7in the CI tooling lock, which this PR does not touch (mainfixed the virtualenv one in #152; the gitpython advisory is new). 6770 tests passed, 1 skipped.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the code my threads named: 7 mutants, all killed by the new tests.
Findings that cannot be anchored inline:
docs/preparing-a-repository.md:250-252(outside this diff) is now false. It says theclaude-codeprofile "passes--strict-mcp-configwith no--mcp-config, an MCP server added by a patch is not loaded, so this route cannot test an added MCP tool yet". With this PR,--tools allpasses--mcp-config .mcp.jsonwhenever the prepared workspace has the file, andpresentis computed after the arm and patch are applied (runner.py:163), so apatch:arm that adds.mcp.jsonis loaded. A reader would skip a test that now works. Suggestion: replace the sentence with theargs_if_presentrule (allmode only, file must exist in the workspace).
@xaleryb's question on a malformed .mcp.json waits for the author.
Still open from my earlier review: none. All five threads are fixed.
Checked, no issues
- Thread tests: each of the seven mutants from my threads (
rstrip, unfilteredwritten,arm.rev,utf-8, unstripped fence,if meta,len(parts) == 4 and) fails the touched suite now. args_if_present:meta.jsonand the argv use the same rule; the list is[]outside--tools alland when the workspace has no matching file, as the new contract sentence says.- Schema versioning and housekeeping: the optional key was added in place to
harness_profile.v1.json; both docs pages are innav:; the changelog fragment is inchangelog.d/; no generated artifacts are committed.
|
On xaleryb's malformed |
…vers links and emptied dirs repo validate and the repo run pre-flight (and a cell whose arm wrote the file) refuse a .mcp.json the agent's CLI would exit on. check_arms refuses a base that tracks .claude or .claude/skills as a symlink, a guidance:none arm installs a skill whose directory the guidance drop emptied, and a SKILL.md name YAML reads as an impossible date is not a crash. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…helpers from the arm, manifest and workspace checks repo validate reports a malformed .mcp.json as a warning and succeeds; repo run --tools all and a patch arm that writes one still refuse it. check_arms, resolve_skills, check_repo and prepare_workspace are back under the complexity cap. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Follow-up to the .mcp.json threads: 09f903e changes the earlier 36a9448 behavior. |
…der its complexity baseline Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 980e7d8. The three fixes from my last review hold: a guidance:none skill arm now passes both check_arms and preparation, a tracked .claude or .claude/skills symlink is refused up front, and name: 2024-13-45 no longer crashes. The new .mcp.json pre-flight has a hole for non-UTF-8 files and two test gaps. Four things are worth fixing before merge, and there are three nits inline.
What I ran, on a Linux review host at 980e7d8:
make ci-local: 28 passed, 1 failed, 7 skipped. The failure ispip-audit: it reportsgitpython 3.2.0(CVE-2026-114145, fix 3.2.1) in the lock file, which this PR does not touch. 7276 tests passed, 1 skipped; the changed-line coverage gate passes.- Reproductions for the code findings, with the LLM transport mocked.
- Mutation runs over the code added since my last review: the
_only_empty_dirsandcheck_mcp_configsmutants that matter are inline.
Findings that cannot be anchored inline:
docs/preparing-a-repository.md:244-252(outside this diff) is still false, and it now also hides the skill arms. It says theclaude-codeprofile "passes--strict-mcp-configwith no--mcp-config, an MCP server added by a patch is not loaded, so this route cannot test an added MCP tool yet", and its status banner says arms that add a tool are planned. With this PR,--tools allpasses--mcp-config .mcp.jsonwhenever the prepared workspace has the file; the profile's argv withallends in--mcp-config .mcp.json, and withreadit has no--mcp-config. A reader skips a route that works, or never learns thatskill:andskill-catalog:arms exist. Suggestion: replace the sentence with theargs_if_presentrule (allmode only, the file must exist in the workspace) and point to the skill arms inbenchmarking-and-comparison.md.
Checked, no issues
- Thread fixes: for a
guidance:nonearm with a skill under the globs (also nestedreferences/deep/a.md),check_armsandprepare_workspaceagree. Both refuse a non-globhelper.pyin the skill directory,guidance:raw, a plainskill:arm, apatch:arm andguidance:rev:HEADwhere HEAD ships the skill. A base with only.claude/settings.jsonis not refused. - Frontmatter dates:
2024-13-45,2024-02-30and out-of-range timezone forms return no name;2024-01-02,123,yesand[a]come back as strings and hit the name-mismatchArmError. manifest.pyrefactor:_check_eval_pathskeeps the messages, order and conditions of the base..mcp.jsonedge cases: an empty file, a BOM, a duplicatemcpServerskey,mcpServers: null, a 5000-digit integer and 100000 nested[give a message, not a crash;--tools readleaves a malformed base alone;repo validatestaysokwith a warning.- Housekeeping: the optional
args_if_presentkey was added in place toharness_profile.v1.jsonandschema_compatpasses; no new doc page, so nonav:change; no generated artifacts; the changelog fragment is inchangelog.d/; the docs build with--strict.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
I pushed fixes for the three nits from my review, one commit each:
Still open: the four risks from my review (the non-UTF-8 |
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Review-body items from 980e7d8: the non-UTF-8 .mcp.json hole and the two test gaps are fixed in 75e181b (replied on the three threads). docs/preparing-a-repository.md: the status banner now names the skill: and skill-catalog: arms (linking benchmarking-and-comparison.md), and the false "--strict-mcp-config with no --mcp-config" paragraph is replaced by the args_if_present rule (--tools all only, .mcp.json must exist in the workspace, a patch can add it), also 75e181b. The branch is merged with main in 9ef37f6 (conflicts in the contract, report.py, runner.py and two test fixtures; run_cell was split so the growth ratchet stays green). |
…and the skill-dir guard; fix the MCP paragraph Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi repo tasks check` parameters changed:
+ added option `--arm`Updated on every push; the |
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 75e181b. The three threads from my last review are fixed: the non-UTF-8 .mcp.json hole is closed on the pre-flight side, the bases.py pre-flight tests kill the mutants I listed, and _only_empty_dirs is pinned. One risk remains, in the same fix: the runner half of it is not tested. One nit inline.
What I ran, on a Linux review host at 75e181b:
make ci-local: 26 passed, 3 failed, 7 skipped. 7816 tests passed, 1 skipped; changed-line coverage 96% (400 lines). The failures are not the PR's:pip-auditreportsfilelock 3.32.7andsetuptools 84.0.0in the lock files, which this PR does not touch, and bothgitleaksjobs fail because the tool download failed. A gitleaks scan of the PR's commits finds no leaks.- Reproductions for the code findings, with the LLM transport mocked.
- A mutation run over the code added since my last review:
tree_mcp_error, the every-base pre-flight and_only_empty_dirsmutants are killed by the new tests, except the two inline.
Checked, no issues
- Pre-flight and runner agree on the bytes:
\xff{}, UTF-16, an overlong sequence and surrogates givenot valid UTF-8 textfromGitTree,WorkTreeand the runner's strict read; a BOM, a NUL byte, CRLF and an empty file give the same result on all three. - Failure class: a non-UTF-8
.mcp.jsonin the workspace raisesPrepErrorbeforemeta.jsonis written and before any agent starts, so it is a cell error, not a graded task failure. - Docs: the new
preparing-a-repository.mdparagraph matches the profile (readandnonenever pass--mcp-config; a patch that adds.mcp.jsonis loaded inallmode), the#skill-arms-in-repository-modeanchor exists, andmkdocs build --strictpasses. - Housekeeping:
harness_profile.v1.jsononly gained an optional key,schema_compatpasses, no generated artifacts are committed.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 61b3f25, which fix the nit from my review:
61b3f25:dokimi/tests/test_repo_skill_arms.py:181: added aregular_filecase, a regular file at the skill destination, to the parametrized refusal test.
make ci-local at 61b3f25: 26 passed, 3 failed, 7 skipped.
Not approving yet. Still open: the risk from my review.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…reads it The runner's strict read of the prepared workspace's .mcp.json had no test: a mutant that decodes with errors="replace" passed the whole suite. The new test writes the bytes after preparation (the base check would stop a tracked file first) and asserts the cell is refused as not valid UTF-8 text. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The runner half of the non-UTF-8 .mcp.json refusal is now pinned, and nothing else changed in the PR's own files since my last review.
What I ran, on a Linux review host at 94d4323:
make ci-local: 29 passed, 0 failed, 7 skipped (docker-only and org-level checks). 8,275 tests passed, 2 skipped; changed-line coverage 96% (400 lines).- A mutation run over the new test:
read_text(encoding="utf-8", errors="replace")atrunner.py:179fails it, and so does removing theUnicodeDecodeErrorhandling. It passes on the unmutated code.
Checked, no issues
- Runner test:
test_run_refuses_a_non_utf8_mcp_json_in_the_workspace_as_the_runner_reads_itwritesb"\xff{}"afterprepare_workspace, so the base check cannot stop it first. It asserts exit 1, no agent started, and thenot valid UTF-8 textmessage. - Housekeeping: the added lines contain no network or credential access; the
subprocessandenvironhits are in test fixtures (a fake agent script).schema_compatpasses, and no generated artifacts are committed.
Summary
Adds two arm types,
skill:andskill-catalog:, torepo run, and lets the claude-code profile start a repo's MCP servers.repo tasks check --armrejects a skill path that does not exist.--strict-mcp-configwithout--mcp-config, so a repo that ships.mcp.jsonran with no MCP tools. Passing--mcp-configevery time is not a fix either: Claude Code exits withMCP config file not foundwhen the repo has no.mcp.json.Evidence
Before:
--arm skill:...was rejected. A blueprint repo with.mcp.jsonshowed nomcp__*tools.After: 5315 tests passed, 2 skipped. New tests:
tests/test_repo_skill_arms.py: skill arms, check validation, and theskills/ cache fields inresult.json.test_claude_code_profile_passes_mcp_config_only_when_the_workspace_has_one.The change was also used for about 870 real cells across 4 swcatalog blueprint task sets on Haiku 5.5 and Sonnet 5.5. A sample: banking-assistant on Haiku went from 23/25 with no guidance to 25/25 with a skill, and the skill was invoked in 19 of 25 runs.
Merge Danger
Door: two-way. The new profile key and result fields are additive. Existing arms behave as before.
Blast Radius: claude-code-runs. Repos that ship
.mcp.jsonwill now start those MCP servers, which changes the agent's tool set compared with earlier baselines for those repos.🤖 Generated with Claude Code