Repository navigation
Five committed cases where no skill fires at all #25
Description
Summary
Five of the 13 committed eval cases fail, and they fail the same way: Skill called 0x. Not the
wrong skill — no skill. Each was built from a blind-router miss, and #14 deliberately left
four of the five unfixed with stated reasons. This issue records what the real runner observed, so
the next pass starts from evidence rather than from the blind router's forced choice (see #24).
make eval on dab7767: 8/13 pass, $5.92, exit 1.
The five
1. agent-hale — the plan is agreed -- write it (0/3)
run 1-3 ✗ fires: agent-hale 0x ✓ spec-writer-silent: 0x
#14 left agent-hale alone, and the reasoning was sound: hale wins the mirror-image query
(we agreed the design, now run the implementation), so its description is behaving as written,
and a clause would be #9's mistake. But hale is "writes clean, maintainable code and its tests
from an agreed plan" — the query says the plan is agreed and asks for exactly that, and still
nothing fires. Whatever is missing, it is not a competing skill.
2. agent-page — why is this endpoint slow in production (0/3)
run 1-3 ✗ fires: agent-page 0x (no exclusion grader)
The blind router scored this 0.00, then 1.00, then 0.33 across three runs with no edit, so #14
classified it as noise and left the description alone. The real runner is repeatable. This is
the one of the five with no prior decision behind it and the clearest case for a description
edit — see #24, which carries the full contradiction.
3-4. briefing — both cases
result-and-options run 1-3 ✗ fires: briefing 0x ✓ compactor-silent: 0x
options-table-recommendation run 1 ✗ fires: briefing 0x ✓ compactor-silent: 0x
run 2 ✗ fires: briefing 0x ✓ compactor-silent: 0x
run 3 ✓ fires: briefing 1x ✓ compactor-silent: 0x
briefing is the hard one and it was excluded from #14 on purpose: at 284 of the 320-character
ceiling a boundary clause does not fit, and a briefing rewrite is precisely what #9 had to
revert outright when its validation half judged the rewrite worse.
The blind router said compactor was taking both. It is not — compactor fires 0x on every
run. So the fix attempted in #22 (a clause on compactor) was aimed at a theft that does not
happen, and briefing's real problem is untouched: a standing style instruction does not read as
a skill invocation at all.
One run in three does fire it, so this is a threshold case rather than a flat gap.
5. lingua — stop refining my questions (0/3)
run 1-3 ✗ fires: lingua 0x ✓ briefing-silent: 0x
Note the asymmetry within the same plugin: restate-before-answering passes 3/3, and both
translation negatives pass 3/3 — #14's lingua clause demonstrably works. But the negation
of the same behaviour ("stop refining my questions") fires nothing. lingua's description says it
restates "the question in correct terminology first"; it says nothing about turning that off,
and refine_question: true is a setting the skill owns.
The pattern worth naming
Four of the five carry an exclusion grader, and all four exclusions passed — the supposed thief
was silent too. The common failure in this marketplace is not misrouting. It is a description
that no realistic phrasing of its own work reaches, which a forced-choice blind router structurally
cannot show, because it must name someone.
Two sub-classes:
- Nothing in the roster owns it —
briefing's standing-style instruction,lingua's
"stop doing that". These are about turning a behaviour on or off rather than asking for work. - The description names the work in vocabulary a user would not use —
agent-page's
"operational readiness … before a release" versus "why is this endpoint slow in production".
Not proposed
Editing briefing. Its ceiling and #9's reverted rewrite are both still true, and this issue is
here to record the observation, not to re-authorise the edit that failed before. If briefing is
to be fixed, it needs its ceiling raised or its existing text rewritten — a decision, not a clause.
Activity
cmj0121 commentedon Sep 16, 2026
Two things this issue got wrong, corrected in #27
Closed by #27, but the reasoning in the issue body should not be read as it stands.
1.
agent-pageis a mislabel, not "the clearest case for a description edit". The bodydecides it: Phase 1 is "Page reviews the change, not the codebase" after a
git diff, Phase 4 isa checklist over that diff, and the phase stops outright when no
__REVIEW_VERDICT__arrived. Alive latency regression supplies none of those, so Page stops on turn one. Nine other positives are
review-of-a-change and the set already carried
fix the slow queryas a negative. No descriptionedit was needed or made.
2.
briefing's failure is not the standing-versus-one-turn distinction this issue proposed.It says these are "about turning a behaviour on or off rather than asking for work". But both case
prompts were already written as standing instructions — "about a dozen of these to get through
this afternoon" and "From here on" over six architecture calls — and they still fired 0x.
The real reason is simpler and sharper:
briefingholds no state.allowed-tools: [Read]alone, no config file, no session hook. Nothing it offers is beyond the session's own reach, so
answering more briefly is compliable whether it covers one answer or twelve. Contrast
lingua,whose
refine_questionis written to a file and reloaded by a SessionStart hook — which is whyrestate my question before you answer itfires 3/3 andstop refining my questionsdoes not.That contrast is the whole basis of the criterion now recorded in
CLAUDE.md(#26), and it isbetter evidence than the sub-classes this issue proposed.
What actually happened to the five
agent-page/why is this endpoint slow in productionfix the slow queryto pay for itagent-hale/the plan is agreed -- write itbriefing×2lingua/stop refining my questionsThe
linguaone is the interesting one, because it was decided against the criterion ratherthan by it. Its body names "stop refining" by name as a mid-session change that writes the config
file, and the SessionStart hook reloads that file — so a stop that is not persisted does not stop
anything past this session. It is recorded in that set's
aboutas a body-versus-descriptionmismatch: the description names the on-state and never the off-state. The failing case is the
observation that would justify a future description edit.
make evalshould now read 12/13, with that one the deliberate remainder. Not yet re-run.Three follow-ups this leaves open
lingua's description does not name the off-state ofrefine_question. Real, unfixed.briefing's description still claims "when the user asks for shorter output", which its ownset now labels negative. Someone has to decide whether the ceiling gets raised or the claim gets
dropped — this issue declines to re-authorise the rewrite Cut router cost where it was real, and prove the rewrite did not break routing #9 reverted.
agent-hale's remaining plain-implementation positives sit on the boundary this relabelestablishes and are unmeasured. Worth a run before anyone touches them, not a blanket
relabel.