Realign two eval cases with the mislabels #22 corrected

[cmj0121]

Found while checking the suite before spending money on the first make eval run.

Two cases asserted what this repo has since rejected

#20 built 13 cases from the blind-router baseline's misses. #22 then re-examined two of those misses and concluded they were mislabels, replacing the queries in the plugins' trigger-queries.json. The cases were not updated, so they contradicted the sets beside them.

case	asserted	what #22 concluded
agent-ellis/ticket-acceptance	ellis fires, pr-flow silent, on "does this change do what the ticket asked"	pr-flow is right. Its Phase 2 is literally "Compare what the branch does with what the issue asked for", its description says it checks the diff against the issue body, and its own train half already carried the same question. Ellis's Phase 5 verifies against PLAN.md, not a ticket.
agent-smith/agreed-design-run-it	smith fires, on "we agreed the design, now run the implementation"	agent-hale is right. That query names the act of implementing with an approach already settled — hale's description verbatim — and carries none of the scope or coordination that distinguishes smith.

Left alone, the first make eval run would have paid to assert both and failed correctly, for a reason unrelated to routing health.

The fix

Each case is rebuilt around its plugin's replacement query, as a realistic turn — a bare query does not fire a skill, which was measured (Skill called 0x for a bare string, 1x for the same intent in a real turn). Still 13 cases; the directories are renamed because the name is the case name in the report.

• agent-ellis/clean-enough-to-land — finished branch, tests, acceptance criteria in PLAN.md, asking for quality/tests/acceptance together before merge. That combination on finished work is ellis's gate rather than any one of the three alone.
• agent-smith/agreed-design-break-into-units — a settled design split into five units with an ordering between them, asking to break it up and run the iterations. The scope is the differentiator, and it is stated in the prompt rather than implied.

Both dropped their exclusion grader, each for a stated reason

The reasons are written into the grader bodies, which the CLI prints in its report:

• ellis no longer asserts pr-flow silent. That exclusion was measured against the old query, which Bound the six descriptions a blind router measured misfiring #22 reclassified as pr-flow's own ground. Nothing in the new prompt names an issue, a pull request or a reviewer comment, so there is no confusion left to exclude — and asserting one no baseline has measured would be inventing a boundary.
• smith does not assert agent-hale silent. Dispatching hale per unit is what running the iterations consists of, so a grader requiring hale's silence would fail runs that routed correctly. Adopt claude plugin eval behind make eval, with 13 cases for the measured boundaries #20 omitted it for the same reason on the old prompt; the reasoning survives the rewrite.

input_match keeps the repo's no-closing-quote spelling, which a real run settled: it reported Skill called 1x correctly, where the closing-quote variant risks every fires grader reporting 0 and every does-not-fire grader passing trivially.

Versions

Nothing moves. An evals/ change is neither the SKILL.md body nor catalogue copy.

Checks

make test — all four pass. bash scripts/validate-fixtures — all fixtures behaved as expected. make eval deliberately not run here.

🤖 Generated with Claude Code