Record when a skill should fire, and what each runner measures

[cmj0121]

Addresses #24's first and third boxes. The relabelling and the agent-page correction are a separate PR.

Two findings from the first real make eval run, neither written down anywhere

1. When a skill should fire at all

A skill should fire when the request needs something the assistant cannot do unaided — persist state to a file, run a tool, coordinate other agents, emit a machine-readable verdict. Where the assistant can simply comply in the current turn, nothing fires, and that is the right outcome rather than a defect: a trigger-eval positive expecting one is mislabelled, and no description edit will fix it.

The evidence sits inside a single plugin's own results, which is what makes it more than plausible:

lingua case	result	why
restate my question before you answer it	3/3	complying means writing refine_question into a config file
stop refining my questions	0/3	complying means not doing something, in this turn

Same skill, same vocabulary, opposite results. The discriminator is not how near the wording sits to the description — it is whether anything has to happen that the assistant cannot do alone. The same reading covers three of the other four failures: briefing's two cases and agent-hale's the plan is agreed — write it all ask for something the assistant just does.

2. The two runners answer different questions, so a disagreement is not a tie

The blind router shows a model only the 19 name + description pairs and makes it pick one skill or none. Free, offline, all 380 queries — but a forced choice: given nineteen candidates it returns the nearest description even where a real session would invoke nobody, so its "skill A steals from skill B" verdicts overstate. make eval instead observes whether the Skill tool was invoked at all, at $0.142 a run over 13 of the 380.

The measurement that separates them: in all five failing cases the fires grader reported Skill called 0x, and in every case carrying an exclusion grader the rival reported 0x too and the exclusion passed. The blind router had compactor taking both of briefing's queries 3/3 and spec-writer taking the plan is agreed -- write it 3/3. In a real session neither fires at all.

So: a blind-router miss is a hypothesis; a make eval miss is an observation. Do not edit a description on the former alone when a case can be written for the latter. And the blind router's noise floor is about two queries per plugin at three samples — plugins nobody touched moved that much between runs — so a swing smaller than that is not a finding either.

Where it went

Two bullets appended to the "What that means for edits here" list in ## Checks, directly after #20's case-anatomy bullet, which already ends on "A bare query does not fire a skill" — the nearest thing in the file to finding 1. The list now reads as one progression: what the cheap 380 are → what make eval costs and covers → what a case looks like → when a skill should fire at all → which runner's verdict to trust. Neither new bullet restates #20's cost derivation, its 13-of-380 scoping, or the grader syntax.

What this deliberately does not say

#24 currently calls agent-page "the clearest case for a description edit". That claim is wrong — agent-page's Phase 1 says "Page reviews the change, not the codebase", requires a git diff, and stops outright without a __REVIEW_VERDICT__; its nine other positives are all review-of-a-change and its own set already carries fix the slow query as a negative. It is a mislabel. This PR does not argue that one case — finding 1's clause "no description edit will fix it" states the general rule that makes it wrong, and the label itself is fixed in the companion PR.

agent-page is not named in CLAUDE.md, before or after.

Versions

Nothing moves. CLAUDE.md is neither a plugin body nor catalogue copy, and no check reads it — grep -rn "CLAUDE.md" scripts/ Makefile .pre-commit-config.yaml returns nothing.

One note for the next editor of this file

Prettier de-indents any line where a backtick span straddles a line break, breaking the bullet's alignment, and it normalises *emphasis* to _emphasis_. Treat backtick spans as unbreakable tokens when wrapping. Prose here wraps at 95 characters against the file's existing 96 — and awk length miscounts, because em-dashes are three bytes.

Checks

make test — all four pass. bash scripts/validate-fixtures — all 67 fixtures pass. Pre-commit green on the commit.

🤖 Generated with Claude Code