Add a skill that runs an agent evaluation round of the benchup tools

.agents/skills/benchup-agent-eval/SKILL.md

@@ -0,0 +1,20 @@
+---
+name: benchup-agent-eval
+description: Run a round of the agent evaluation of the benchup CLI and MCP tools (crates/benchup-cli/evals) — one subagent per case, a results table compared with earlier rounds, and findings sorted into tool bugs, guide gaps and core limits. Use when asked to evaluate, test or "run the cases/battery/round" for the benchup CLI, MCP or agent guide, or after changing agent tools, their descriptions or docs/agent-guide.md.
+---
+
+# Agent evaluation round
+
+The cases, rules and earlier results are in `crates/benchup-cli/evals/` (read its README.md first). A round spends many tokens: run it only when the user asked for it, and say how many agents it will start.
+
+1. **Regression first.** `cargo test -p benchup-cli --test evals`. If it fails, stop and report: a reference solution no longer applies, which is a regression or a reference to update.
+2. **Build and prepare.** `cargo build --release -p benchup-cli`, then `python3 crates/benchup-cli/evals/setup.py --round N` (N = the last round in `results/` plus one; `--cases 01,13` for a subset). It prints each case's folder and prompt file.
+3. **Launch one subagent per case, in parallel, in the background.** Give each the content of its prompt file verbatim, with nothing added: no hints, no knowledge of the code, no expectations. The agents must learn the tools only from what the tools say.
+4. **Check the results yourself; do not trust the reports alone.** For each case, run `benchup call <file> check_model` and `benchup render <file> --view iso` (look at the image) on the file the agent left, and compare with the request.
+5. **Write `crates/benchup-cli/evals/results/round-N.md`:**
+   - a table per case: commands (errors), result ✅ / ⚠️ / ❌ checked in step 4, and the same columns from the previous round;
+   - which of each case's `focus` tools the agent actually used, and where it fell back to a workaround;
+   - the findings, sorted into: tool bugs and unclear errors or descriptions (fix in `benchup-tools`); guide gaps or mistakes (`docs/agent-guide.md`); core limits (an issue, with a minimal reproduction); missing features.
+6. **Report to the user** with the table and the proposed fixes, and wait before changing code. After fixing, update reference batches if a tool changed on purpose, and add a reference for any new case an agent solved well.
+
+Keep the cases comparable across rounds: do not edit a case's request or seed; add a new case instead.

.claude/skills

@@ -0,0 +1 @@
+../.agents/skills

.github/workflows/build.yml

@@ -235,6 +235,11 @@ jobs:
       - name: Test native import and export fixtures
         run: bash scripts/ci_cargo.sh test -p benchup-io -p benchup-skp --profile ci-test --locked --target "${{ matrix.target }}"
 
+      # Includes the replay of the agent evaluation's reference solutions
+      # (crates/benchup-cli/tests/evals.rs); the agent rounds themselves are run on demand.
+      - name: Test the agent tools and command line
+        run: bash scripts/ci_cargo.sh test -p benchup-tools -p benchup-cli --profile ci-test --locked --target "${{ matrix.target }}"
+
   wasm:
     name: Build WebAssembly
     needs: validate

crates/benchup-cli/evals/README.md

@@ -0,0 +1,28 @@
+# Agent evaluation of the `benchup` tools
+
+The CLI and MCP tools are meant to be used by coding agents that have never seen them. These cases check that: realistic requests from a woodworker, worded as a user would, each solved by an agent that knows only what the tool tells it (`benchup --help`, `benchup guide`, the tool schemas and its errors).
+
+There are two layers:
+
+1. **Regression, on every change** — `cargo test -p benchup-cli --test evals`. Replays the reference solution of each case (`reference/<case>/*.jsonl`, batches that agents or maintainers got right) with the real binary and checks the result with `check_model`. It needs no agent and takes a few seconds. When a tool changes on purpose, update the reference batch.
+2. **Agent rounds, on demand** — before a release, or after changing tools, descriptions or the [agent guide](../../../docs/agent-guide.md). One agent per case, run in parallel; each one reports what it did, how many commands it needed and where it got stuck. Not in CI: it costs tokens and the results vary from run to run. With Claude Code or Codex, the `benchup-agent-eval` skill in [`.agents/skills`](../../../.agents/skills/benchup-agent-eval/SKILL.md) runs a round (`.claude/skills` links to it); by hand:
+
+   ```sh
+   cargo build --release -p benchup-cli
+   python3 crates/benchup-cli/evals/setup.py --round 5            # or --cases 01,13
+   # give each agent the prompt in /tmp/benchup-eval5/prompts/<case>.md, verbatim
+   ```
+
+## Files
+
+- `cases.json` — the cases: request, seed project, mode (`cli`, or `mcp` through `mcp_call.py`), the tools it is expected to exercise, and the reference solution with what `check_model` must say about it.
+- `prompt.md` — the rules every agent gets, and the report it must return.
+- `setup.py` — prepares a round in `/tmp/benchup-eval<round>/` (binary, one folder per case with its seed, prompts). It never builds.
+- `mcp_call.py` — a minimal MCP client over stdio, for the MCP cases.
+- `fixtures/` — seed projects. `bookcase.bup` is a 1000 × 1800 mm bookcase with two columns of five housed shelves whose ten joints are reversed on purpose (agents should notice and offer `flip_joint`).
+- `reference/` — reference batches, per case.
+- `results/` — one file per round: the table of commands (errors) per case compared with earlier rounds, and the findings, sorted into tool bugs, guide gaps, core limits and missing features.
+
+## Adding a case
+
+Add it to `cases.json` with a request a real user would write, without naming tools. Give it a reference batch when the result can be checked with `check_model`; keep the request achievable with the current core (a case that hits a core limit belongs in an issue). Cases that only make sense with an agent (judgement, visual checks, a mode such as MCP) can go without a reference.