Relabel the four positives nothing should have fired on, and keep the one that should

[cmj0121]

Closes #25. Applies the criterion #26 recorded, and corrects a claim #24 and #25 both make.

The criterion

A skill should fire when the request needs something the assistant cannot do unaided — persist state to a file, run a tool, coordinate other agents, emit a machine-readable verdict. Where the assistant can simply comply in the current turn, nothing fires, and a positive expecting one is mislabelled.

Four of the five failures are mislabels under it. One is not — and it was decided against the rule, on evidence.

Four relabelled

agent-page — why is this endpoint slow in production → negative

Verified in the body rather than assumed. Phase 1: "Page reviews the change, not the codebase", after a git diff. Phase 4's Performance Review is a checklist over that diff. Phase 4 also: "No __REVIEW_VERDICT__ reached Page → say so and stop." And the failure-handling preamble: "Every phase below assumes its input arrived — a diff to read, Ellis's findings to build on."

A live latency regression supplies none of the three, so Page stops on turn one. Its nine other positives are all review-of-a-change; this was the only diagnostic one, and the set already carried fix the slow query as a negative.

#24 and #25 both call this "the clearest case for a description edit". Both are wrong — they read a repeatable 0/3 as a description gap when the body says Page was never asked.

agent-hale — the plan is agreed -- write it → negative

The case prompt already supplies a fully settled approach and still fires 0x with spec-writer silent. So the failure is not an empty referent: writing the code a settled plan describes is ordinary compliance in the current turn.

briefing — both cases → negative

The decisive finding is not the one #25 guessed. Both prompts were already written as standing instructions — "about a dozen of these to get through this afternoon", "From here on" — so the standing-versus-one-turn distinction is not what fails. briefing holds no state: allowed-tools: [Read] alone, no config file, no session hook. Nothing it offers is beyond the session's own reach, so answering more briefly is compliable whether it covers one answer or twelve.

One that stands: lingua — stop refining my questions

Decided on evidence, and it comes out against a blanket application of the criterion:

• SKILL.md Phase 4 lists "stop refining" by name as a mid-session rule change that updates the config file with Edit.
• refine_question is reloaded by the SessionStart hook every session — so a stop that is not persisted does not stop anything past this one. That is a write, which is exactly what the criterion asks for.

On swapping in lingua off instead, which was suggested: it is the wrong replacement, for two reasons that run opposite to the suggestion. It carries the magic word lingua, so it fires whatever the description says and tests nothing about the description. And the body has it suspend for the session without writing the file — making it the less persistent of the two.

So: not a relabel, which would assert something the body contradicts; and not a replacement, which would delete the evidence — the restate… 3/3 versus stop refining… 0/3 asymmetry is the whole basis of the criterion in #26. The case is left failing on purpose, recorded in about as a body-versus-description mismatch: the description names the on-state and never the off-state. That is the observation which would justify a future description edit.

compactor's #22 clause: stands

#24 asked whether to revert it, since make eval shows compactor firing 0x on every run and the clause therefore cannot be distinguished from doing nothing. Three reasons it stays:

1. The 0x measurement was taken with the clause in place. Reverting is unmeasured — it trades one unmeasured state for another, at the cost of a catalogue-copy edit across four compared locations.
2. Under the criterion now settled, it is no longer an anti-theft patch. A standing output style is briefing's, a one-turn re-render is compactor's; the clause states the real boundary.
3. The blind router and make eval disagree, and the disagreement has a shape #24's own rule bars editing a description on a blind-router reading alone. "Cannot be distinguished from doing nothing" is an absence of evidence, not evidence of absence.

Mechanics

Every relabel that flipped a positive spent an existing negative that asserted the same line by hand, so the line is now drawn by three measured runs instead:

set	spent negative	restored positive
agent-page	fix the slow query	before we ship this change, check its timeout and retry handling
agent-hale	why is this endpoint slow	take unit 3 from the plan, build it test-first, and report what the suite did
briefing (train)	summarise the previous answer	load the project's output style for this planning run
briefing (validation)	condense this table	what output style do the planning agents use in this project

Both new briefing positives ask for the one thing the session cannot supply unaided — what this project's output style actually says — mirroring lingua's existing what response language is set for this project.

All four sets keep 20 queries, 10/10, 6/6 train and 4/4 validation. New positives scanned clean against every magic word with the same \b-anchored regex scripts/validate uses; no new cross-set duplicates.

Still 13 cases — none removed. Four had graders/fires.md replaced by graders/does-not-fire.md, matching the house spelling in lingua/translate-readme. Prompts kept verbatim: each is still a realistic turn, and for briefing the firing-favourable standing framing makes it a stronger negative. The spec-writer-silent and both compactor-silent graders are kept, so each briefing case now asserts the criterion's full prediction — neither skill fires.

No briefing description edit. Still 284/320; #9's revert stands.

Expected movement

make eval should go 8/13 → 12/13, with lingua/stop-restating the one deliberate remaining failure. Not run here.

Three follow-ups this opens, none in scope

1. lingua's description gap is real and unfixed — the off-state of refine_question is unnamed.
2. briefing's description still claims "when the user asks for shorter output", which its own set now labels negative. Someone has to decide whether the ceiling gets raised or the claim gets dropped.
3. agent-hale's remaining plain-implementation positives sit on the boundary this relabel establishes and are unmeasured. Worth a run before anyone touches them — not a blanket relabel.

Versions

Nothing moves at any level. Every changed file is under evals/, and since the compactor clause stands, no description changed either.

Checks

make test — all four pass. bash scripts/validate-fixtures — all fixtures behaved as expected.

🤖 Generated with Claude Code