Repository navigation
The blind router and make eval disagree, and the disagreement has a shape #24
Description
Summary
The first real make eval run (13 cases, 3 runs each, $5.92) disagrees with the 3-sample blind
router that this repo has been judging descriptions with. The disagreement is not noise and it
is not random — it has a shape, and that shape means the two runners answer different questions.
Both are useful; neither substitutes for the other, and #14's scoping decisions were made on the
weaker one.
The two methods
- Blind router — a model sees only the 19
name+descriptionpairs and must pick exactly
one skill ornoneper query. Free, offline, 380 queries in three samples. This is the
method in everyhow_to_runand the one behind the 367/380 baseline. make eval— a full Claude child session per run with the plugin loaded, graded on whether
theSkilltool was actually invoked. $0.142 a run, measured.
The finding: every failure is "nothing fired", not "the wrong thing fired"
In all five failing cases, the fires grader reported Skill called 0x — and in each case that
carried an exclusion grader, the rival also reported 0x and the exclusion passed:
agreed-plan-write-it run 1-3 ✗ fires: agent-hale 0x ✓ spec-writer-silent: 0x
result-and-options run 1-3 ✗ fires: briefing 0x ✓ compactor-silent: 0x
stop-restating run 1-3 ✗ fires: lingua 0x ✓ briefing-silent: 0x
The blind router said compactor takes both of briefing's queries 3/3, and spec-writer takes
the plan is agreed -- write it 3/3. In a real session neither fires at all.
That is an artifact of the method, not a fact about the descriptions: the blind router is forced
to choose. Given a query and nineteen candidates it returns the nearest description even when a
real session would invoke nothing. So a blind-router "miss" reads as skill A steals from skill B
when the truth is often no skill fires, and A's description is merely the closest string.
The consequence for #14
Six boundary clauses were written on that reading, four of them to stop a "theft" that a real
session does not perform. They are not wrong — agent-ward, spec-writer and both lingua
negatives are confirmed 3/3 by make eval. But compactor's clause cannot be distinguished
from doing nothing: its case passes compactor-silent on every run, which is equally consistent
with "the clause worked" and with "compactor was never going to fire". The measurement that would
separate them does not exist.
The concrete contradiction: agent-page
why is this endpoint slow in production — an agent-page positive.
| measurement | result |
|---|---|
| blind router, baseline | 0.00 (none ×3) |
| blind router, run 2 | 1.00 |
| blind router, run 3 | 0.33 |
make eval, 3 runs |
0/3, Skill called 0x every time |
The blind router's three readings spanned the entire range with no edit at any point, which is why
#14 classified this as inside the ±2-query noise floor and left agent-page's description
alone. The real runner is unambiguous and repeatable: the skill never fires.
That decision was made on the weaker measurement and should be revisited. agent-page is
"reviews code and infrastructure for operational readiness: observability, reliability,
performance, security posture. Use before a release." A production latency question names none of
that vocabulary and is not "before a release", so there is a real gap here, not a labelling error.
What to do
- Record in
CLAUDE.mdwhat each runner measures and what each cannot see: the blind router
is a forced choice among 19 +none, so its "wrong skill" verdicts overstate;make eval
observes real invocation but costs money and covers 13 of 380. - Re-judge
agent-page's description against the real runner rather than the blind one. - State the rule that follows: a blind-router miss is a hypothesis; a
make evalmiss is an
observation. Do not edit a description on the former alone when a case can be written for
the latter. - Consider whether
compactor's clause (Bound the six descriptions a blind router measured misfiring #22) survives, given nothing can currently show it
does anything.
Not proposed
Converting more queries to cases. At $0.142 a run, all 380 is ~$54 at one sample each and ~$162 at
three; the 13 exist because they are the measured boundaries. The cheap record stays cheap.
Refs #13, #14. Evidence: make eval on dab7767, 8/13 passing, exit 1.
Activity
Merge: record when a skill should fire, and what each runner measures
Merge: relabel the four positives nothing should have fired on
cmj0121 commentedon Sep 16, 2026
Correction first: this issue's
agent-pageclaim is wrongThis issue says
agent-page's failure is "the clearest case for a description edit" and that"there is a real gap here, not a labelling error". Both sentences are wrong, and they are wrong
in the direction this issue itself warns about — they read a
make evalobservation through theblind router's framing instead of reading the skill.
plugins/agent-page/skills/agent-page/SKILL.mdsettles it:git diffto see all changes, then "Page reviews the change, not the codebase."caching, payload sizes.
__REVIEW_VERDICT__reached Page → say so and stop."read, Ellis's findings to build on."
A live latency regression supplies none of the three, so Page stops on turn one. Its nine other
positives are all review-of-a-change; this was the only diagnostic one, and the set already
carried
fix the slow queryas a negative drawing the same line by hand.So
why is this endpoint slow in productionis a mislabel. #27 relabels it and spendsfix the slow queryto pay for it, so the line is now drawn by three measured runs instead of byassertion. No description edit; none was needed.
The lesson is this issue's own, turned on itself: a
make evalmiss is an observation thatnothing fired. It is not an observation of why. Reading the body is still the step that decides
between a description gap and a mislabel, and skipping it is how this issue got it backwards.
The four boxes
CLAUDE.md's"What that means for edits here" list, placed after Adopt claude plugin eval behind make eval, with 13 cases for the measured boundaries #20's case-anatomy bullet, which already
ended on "A bare query does not fire a skill".
agent-pageagainst the real runner — done, and the answer is that the labelwas wrong rather than the description. See above.
make evalmiss is anobservation, plus the ±2-per-plugin noise floor, so a smaller swing is not a finding either.
compactor's clause survives — it stands. Three reasons, in Relabel the four positives nothing should have fired on, and keep the one that should #27: the0x measurement was taken with the clause in place, so reverting trades one unmeasured state
for another at the cost of a four-location catalogue edit; under the criterion now settled it
is no longer an anti-theft patch but a real boundary (standing style is briefing's, one-turn
re-render is compactor's); and "cannot be distinguished from doing nothing" is an absence of
evidence, which this issue's own rule says is not grounds for a description edit.
What came out of it that this issue did not anticipate
The criterion recorded in #26 is stronger than the runner comparison that prompted it, and it came
from
linguarather than from the disagreement: a skill should fire when the request needssomething the assistant cannot do unaided.
restate my question before you answer itpasses 3/3because complying means writing
refine_questionto a config file;stop refining my questionsfires 0/3 because complying means not doing something. Same skill, same vocabulary, opposite
results — so the discriminator is not vocabulary at all.
Closing: all four boxes are answered, one of them by overturning the issue's own premise.