Skip to content

Five committed cases where no skill fires at all #25

Closed

Description

@cmj0121

Summary

Five of the 13 committed eval cases fail, and they fail the same way: Skill called 0x. Not the
wrong skill — no skill. Each was built from a blind-router miss, and #14 deliberately left
four of the five unfixed with stated reasons. This issue records what the real runner observed, so
the next pass starts from evidence rather than from the blind router's forced choice (see #24).

make eval on dab7767: 8/13 pass, $5.92, exit 1.

The five

1. agent-hale — the plan is agreed -- write it (0/3)

run 1-3   ✗ fires: agent-hale 0x      ✓ spec-writer-silent: 0x

#14 left agent-hale alone, and the reasoning was sound: hale wins the mirror-image query
(we agreed the design, now run the implementation), so its description is behaving as written,
and a clause would be #9's mistake. But hale is "writes clean, maintainable code and its tests
from an agreed plan"
— the query says the plan is agreed and asks for exactly that, and still
nothing fires. Whatever is missing, it is not a competing skill.

2. agent-page — why is this endpoint slow in production (0/3)

run 1-3   ✗ fires: agent-page 0x      (no exclusion grader)

The blind router scored this 0.00, then 1.00, then 0.33 across three runs with no edit, so #14
classified it as noise and left the description alone. The real runner is repeatable. This is
the one of the five with no prior decision behind it and the clearest case for a description
edit
— see #24, which carries the full contradiction.

3-4. briefing — both cases

result-and-options            run 1-3   ✗ fires: briefing 0x   ✓ compactor-silent: 0x
options-table-recommendation  run 1     ✗ fires: briefing 0x   ✓ compactor-silent: 0x
                              run 2     ✗ fires: briefing 0x   ✓ compactor-silent: 0x
                              run 3     ✓ fires: briefing 1x   ✓ compactor-silent: 0x

briefing is the hard one and it was excluded from #14 on purpose: at 284 of the 320-character
ceiling
a boundary clause does not fit, and a briefing rewrite is precisely what #9 had to
revert outright when its validation half judged the rewrite worse.

The blind router said compactor was taking both. It is not — compactor fires 0x on every
run. So the fix attempted in #22 (a clause on compactor) was aimed at a theft that does not
happen, and briefing's real problem is untouched: a standing style instruction does not read as
a skill invocation at all.

One run in three does fire it, so this is a threshold case rather than a flat gap.

5. lingua — stop refining my questions (0/3)

run 1-3   ✗ fires: lingua 0x          ✓ briefing-silent: 0x

Note the asymmetry within the same plugin: restate-before-answering passes 3/3, and both
translation negatives pass 3/3 — #14's lingua clause demonstrably works. But the negation
of the same behaviour ("stop refining my questions") fires nothing. lingua's description says it
restates "the question in correct terminology first"; it says nothing about turning that off,
and refine_question: true is a setting the skill owns.

The pattern worth naming

Four of the five carry an exclusion grader, and all four exclusions passed — the supposed thief
was silent too. The common failure in this marketplace is not misrouting. It is a description
that no realistic phrasing of its own work reaches
, which a forced-choice blind router structurally
cannot show, because it must name someone.

Two sub-classes:

  • Nothing in the roster owns it — briefing's standing-style instruction, lingua's
    "stop doing that". These are about turning a behaviour on or off rather than asking for work.
  • The description names the work in vocabulary a user would not use — agent-page's
    "operational readiness … before a release" versus "why is this endpoint slow in production".

Not proposed

Editing briefing. Its ceiling and #9's reverted rewrite are both still true, and this issue is
here to record the observation, not to re-authorise the edit that failed before. If briefing is
to be fixed, it needs its ceiling raised or its existing text rewritten — a decision, not a clause.

Refs #13, #14, #24.

Activity

  1. cmj0121 commented on Sep 16, 2026

    @cmj0121
    OwnerAuthor

    Two things this issue got wrong, corrected in #27

    Closed by #27, but the reasoning in the issue body should not be read as it stands.

    1. agent-page is a mislabel, not "the clearest case for a description edit". The body
    decides it: Phase 1 is "Page reviews the change, not the codebase" after a git diff, Phase 4 is
    a checklist over that diff, and the phase stops outright when no __REVIEW_VERDICT__ arrived. A
    live latency regression supplies none of those, so Page stops on turn one. Nine other positives are
    review-of-a-change and the set already carried fix the slow query as a negative. No description
    edit was needed or made.

    2. briefing's failure is not the standing-versus-one-turn distinction this issue proposed.
    It says these are "about turning a behaviour on or off rather than asking for work". But both case
    prompts were already written as standing instructions — "about a dozen of these to get through
    this afternoon"
    and "From here on" over six architecture calls — and they still fired 0x.

    The real reason is simpler and sharper: briefing holds no state. allowed-tools: [Read]
    alone, no config file, no session hook. Nothing it offers is beyond the session's own reach, so
    answering more briefly is compliable whether it covers one answer or twelve. Contrast lingua,
    whose refine_question is written to a file and reloaded by a SessionStart hook — which is why
    restate my question before you answer it fires 3/3 and stop refining my questions does not.

    That contrast is the whole basis of the criterion now recorded in CLAUDE.md (#26), and it is
    better evidence than the sub-classes this issue proposed.

    What actually happened to the five

    case outcome
    agent-page / why is this endpoint slow in production relabelled negative; spent fix the slow query to pay for it
    agent-hale / the plan is agreed -- write it relabelled negative — writing the code a settled plan describes is ordinary compliance
    briefing ×2 relabelled negative; no description edit, still 284/320, #9's revert stands
    lingua / stop refining my questions label stands — left failing on purpose

    The lingua one is the interesting one, because it was decided against the criterion rather
    than by it. Its body names "stop refining" by name as a mid-session change that writes the config
    file, and the SessionStart hook reloads that file — so a stop that is not persisted does not stop
    anything past this session. It is recorded in that set's about as a body-versus-description
    mismatch: the description names the on-state and never the off-state. The failing case is the
    observation that would justify a future description edit.

    make eval should now read 12/13, with that one the deliberate remainder. Not yet re-run.

    Three follow-ups this leaves open

    1. lingua's description does not name the off-state of refine_question. Real, unfixed.
    2. briefing's description still claims "when the user asks for shorter output", which its own
      set now labels negative. Someone has to decide whether the ceiling gets raised or the claim gets
      dropped — this issue declines to re-authorise the rewrite Cut router cost where it was real, and prove the rewrite did not break routing #9 reverted.
    3. agent-hale's remaining plain-implementation positives sit on the boundary this relabel
      establishes and are unmeasured. Worth a run before anyone touches them, not a blanket
      relabel.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions