From 30c785060dd1f53be72fa74c6289bd9190560b32 Mon Sep 17 00:00:00 2001 From: cmj Date: Wed, 16 Sep 2026 15:00:13 +0800 Subject: [PATCH 1/2] fix(evals): relabel the four positives nothing should have fired on `make eval` on dab7767 left five cases at `Skill called 0x`, and in every one carrying an exclusion grader the supposed thief was silent too. On the criterion the maintainer settled -- a skill fires when the request needs what the assistant cannot do unaided -- four of those five are mislabelled, and each case is inverted to assert the silence it already measured. - agent-page `why is this endpoint slow in production`: Page reviews a change, not a running system. Phase 1 wants a `git diff`, Phase 4's performance list is a checklist applied to that diff, and the phase stops when no `__REVIEW_VERDICT__` arrived. This overturns what #24 and #25 both claim, that the 0/3 is a description gap. - agent-hale `the plan is agreed -- write it`: the case states the agreed approach in full and still fires nothing, because writing the code a settled plan describes is ordinary compliance. - briefing, both cases: briefing holds no state -- `allowed-tools` is `Read` alone, no config file, no session hook -- so a standing instruction to answer more briefly is one the session can honour by itself, whether it covers one answer or a dozen. Each set keeps 20 queries at 10/10 and its train/validation sizes. Every relabel spends the negative that had asserted the same line by hand, and the slot restores a positive naming what the skill supplies that the session cannot: a review of a change before shipping, a unit built test-first with the suite reported back, and the project's own output style loaded and read back. compactor's #22 clause stands. It was measured with the clause in place, so removing it trades one unmeasured state for another across four files of catalogue copy, and under the criterion it now states the real boundary rather than patching a theft that does not happen. Refs: #25, #24 Co-Authored-By: Claude Opus 5 (1M context) --- .../agreed-plan-write-it/graders/does-not-fire.md | 12 ++++++++++++ .../evals/agreed-plan-write-it/graders/fires.md | 8 -------- plugins/agent-hale/evals/trigger-queries.json | 14 +++++++------- .../endpoint-slow-in-prod/graders/does-not-fire.md | 14 ++++++++++++++ .../evals/endpoint-slow-in-prod/graders/fires.md | 9 --------- plugins/agent-page/evals/trigger-queries.json | 6 +++--- .../graders/does-not-fire.md | 12 ++++++++++++ .../options-table-recommendation/graders/fires.md | 9 --------- .../result-and-options/graders/does-not-fire.md | 13 +++++++++++++ .../evals/result-and-options/graders/fires.md | 9 --------- plugins/briefing/evals/trigger-queries.json | 12 ++++++------ 11 files changed, 67 insertions(+), 51 deletions(-) create mode 100644 plugins/agent-hale/evals/agreed-plan-write-it/graders/does-not-fire.md delete mode 100644 plugins/agent-hale/evals/agreed-plan-write-it/graders/fires.md create mode 100644 plugins/agent-page/evals/endpoint-slow-in-prod/graders/does-not-fire.md delete mode 100644 plugins/agent-page/evals/endpoint-slow-in-prod/graders/fires.md create mode 100644 plugins/briefing/evals/options-table-recommendation/graders/does-not-fire.md delete mode 100644 plugins/briefing/evals/options-table-recommendation/graders/fires.md create mode 100644 plugins/briefing/evals/result-and-options/graders/does-not-fire.md delete mode 100644 plugins/briefing/evals/result-and-options/graders/fires.md diff --git a/plugins/agent-hale/evals/agreed-plan-write-it/graders/does-not-fire.md b/plugins/agent-hale/evals/agreed-plan-write-it/graders/does-not-fire.md new file mode 100644 index 0000000..c107989 --- /dev/null +++ b/plugins/agent-hale/evals/agreed-plan-write-it/graders/does-not-fire.md @@ -0,0 +1,12 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"agent-hale' +min: 0 +max: 0 +--- + +Relabelled by #25 and #24 after this case fired 0x on three runs. Nothing is left to decide and +the code can simply be written, which is a turn the session can comply with unaided — and where +it can, nothing should fire. Hale's own ground is a unit dispatched from a plan, built +test-first, with the suite result reported back; this turn asks for none of that. diff --git a/plugins/agent-hale/evals/agreed-plan-write-it/graders/fires.md b/plugins/agent-hale/evals/agreed-plan-write-it/graders/fires.md deleted file mode 100644 index 3075fcb..0000000 --- a/plugins/agent-hale/evals/agreed-plan-write-it/graders/fires.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"agent-hale' -min: 1 ---- - -Nothing is left to decide and the work is to produce the code, which is this skill's job. diff --git a/plugins/agent-hale/evals/trigger-queries.json b/plugins/agent-hale/evals/trigger-queries.json index 634bccf..1546159 100644 --- a/plugins/agent-hale/evals/trigger-queries.json +++ b/plugins/agent-hale/evals/trigger-queries.json @@ -1,6 +1,6 @@ { "skill": "agent-hale", - "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.", + "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. One validation label was corrected by hand (issues #25, #24): `the plan is agreed -- write it` is a negative here, not a positive. Its committed case states the agreed approach in full and still fired 0x across three runs, with spec-writer silent too, so nothing was stealing it. On the criterion settled for this marketplace -- a skill fires when the request needs what the assistant cannot do unaided -- writing the code that a settled plan describes is ordinary compliance in the current turn, and correct behaviour is that nothing fires. The compensating positive names what Hale adds beyond typing the code: a unit taken from the plan, built test-first, and a suite result reported back, which is the coordination limb of the criterion. It replaced `why is this endpoint slow`, whose line agent-page's set now draws from measurement rather than assertion. The set's remaining plain-implementation positives sit on the same boundary and are unmeasured; deciding them needs a run, not a blanket relabel.", "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.", "runs_per_query": 3, "trigger_rate_threshold": 0.5, @@ -36,22 +36,22 @@ "split": "train" }, { - "query": "the plan is agreed -- write it", + "query": "fill in the handlers for these three routes", "should_trigger": true, "split": "validation" }, { - "query": "fill in the handlers for these three routes", + "query": "write the unit tests for the cache layer", "should_trigger": true, "split": "validation" }, { - "query": "write the unit tests for the cache layer", + "query": "make this compile against the new api", "should_trigger": true, "split": "validation" }, { - "query": "make this compile against the new api", + "query": "take unit 3 from the plan, build it test-first, and report what the suite did", "should_trigger": true, "split": "validation" }, @@ -86,12 +86,12 @@ "split": "train" }, { - "query": "is this code secure", + "query": "the plan is agreed -- write it", "should_trigger": false, "split": "validation" }, { - "query": "why is this endpoint slow", + "query": "is this code secure", "should_trigger": false, "split": "validation" }, diff --git a/plugins/agent-page/evals/endpoint-slow-in-prod/graders/does-not-fire.md b/plugins/agent-page/evals/endpoint-slow-in-prod/graders/does-not-fire.md new file mode 100644 index 0000000..652b0ad --- /dev/null +++ b/plugins/agent-page/evals/endpoint-slow-in-prod/graders/does-not-fire.md @@ -0,0 +1,14 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"agent-page' +min: 0 +max: 0 +--- + +Relabelled by #25 and #24 after this case fired 0x on three runs. Page reviews a change, not a +running system: Phase 1 wants a `git diff`, Phase 4's performance list is a review checklist +applied to that diff, and the phase stops outright when no `__REVIEW_VERDICT__` arrived. This +turn supplies none of the three, so reading the pool-wait graph is ordinary work and nothing +should load. The baseline fired nothing at all here, so there is still no second skill to rule +out. diff --git a/plugins/agent-page/evals/endpoint-slow-in-prod/graders/fires.md b/plugins/agent-page/evals/endpoint-slow-in-prod/graders/fires.md deleted file mode 100644 index d124d37..0000000 --- a/plugins/agent-page/evals/endpoint-slow-in-prod/graders/fires.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"agent-page' -min: 1 ---- - -A production latency regression with pool-wait evidence is an operational question. -The baseline fired nothing at all here, so there is no second skill to rule out. diff --git a/plugins/agent-page/evals/trigger-queries.json b/plugins/agent-page/evals/trigger-queries.json index ce99c94..26e95d8 100644 --- a/plugins/agent-page/evals/trigger-queries.json +++ b/plugins/agent-page/evals/trigger-queries.json @@ -1,6 +1,6 @@ { "skill": "agent-page", - "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.", + "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. One validation label was corrected by hand (issues #25, #24): `why is this endpoint slow in production` is a negative here, not a positive. `make eval` on `dab7767` ran it three times as a realistic turn and Page fired 0x on every one, and the body says why -- Phase 1 reviews `the change, not the codebase` and needs a `git diff`, Phase 4's performance list is a review checklist applied to that diff, and the phase stops outright when no `__REVIEW_VERDICT__` arrived. A live latency regression supplies none of those, so complying means diagnosing a running system, which is not what this skill does. Its nine other positives are all review-of-a-change; this was the only diagnostic one, and the set already drew the same line with `fix the slow query`. That asserted negative was spent to restore the positive, because the relabelled query now draws the line with three measured runs behind it. This corrects what #24 and #25 both claim -- they read the 0/3 as a description gap.", "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.", "runs_per_query": 3, "trigger_rate_threshold": 0.5, @@ -51,7 +51,7 @@ "split": "validation" }, { - "query": "why is this endpoint slow in production", + "query": "before we ship this change, check its timeout and retry handling", "should_trigger": true, "split": "validation" }, @@ -86,7 +86,7 @@ "split": "train" }, { - "query": "fix the slow query", + "query": "why is this endpoint slow in production", "should_trigger": false, "split": "validation" }, diff --git a/plugins/briefing/evals/options-table-recommendation/graders/does-not-fire.md b/plugins/briefing/evals/options-table-recommendation/graders/does-not-fire.md new file mode 100644 index 0000000..825b598 --- /dev/null +++ b/plugins/briefing/evals/options-table-recommendation/graders/does-not-fire.md @@ -0,0 +1,12 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"briefing' +min: 0 +max: 0 +--- + +Relabelled by #25 and #24 after this case fired 1x on three runs, below the 0.5 threshold. "From +here on" does set a format for six upcoming decisions, but putting options in a table with a +recommendation is as easy to do six times as once, and briefing persists nothing that would carry +the format past this session. The one run that did fire is what a threshold is for. diff --git a/plugins/briefing/evals/options-table-recommendation/graders/fires.md b/plugins/briefing/evals/options-table-recommendation/graders/fires.md deleted file mode 100644 index 2a75bde..0000000 --- a/plugins/briefing/evals/options-table-recommendation/graders/fires.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"briefing' -min: 1 ---- - -"From here on" over six upcoming decisions is a format being set for the rest of the -session, not a complaint about one answer already given. diff --git a/plugins/briefing/evals/result-and-options/graders/does-not-fire.md b/plugins/briefing/evals/result-and-options/graders/does-not-fire.md new file mode 100644 index 0000000..aa62f55 --- /dev/null +++ b/plugins/briefing/evals/result-and-options/graders/does-not-fire.md @@ -0,0 +1,13 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"briefing' +min: 0 +max: 0 +--- + +Relabelled by #25 and #24 after this case fired 0x on three runs. A dozen more to come does make +this a standing instruction rather than a request to redo the last answer — but briefing holds no +state to set. Its `allowed-tools` is `Read` alone, with no config file and no session hook, so +leading with the result for the rest of the afternoon is something the session can just do, and a +standing instruction it can honour unaided is not a skill invocation. diff --git a/plugins/briefing/evals/result-and-options/graders/fires.md b/plugins/briefing/evals/result-and-options/graders/fires.md deleted file mode 100644 index bc250ca..0000000 --- a/plugins/briefing/evals/result-and-options/graders/fires.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: tool_used -tool: Skill -input_match: '"skill"\s*:\s*"briefing' -min: 1 ---- - -A dozen more to come makes this a standing instruction about how answers should arrive, -not a request to redo the last one. diff --git a/plugins/briefing/evals/trigger-queries.json b/plugins/briefing/evals/trigger-queries.json index c8a8bf7..921704f 100644 --- a/plugins/briefing/evals/trigger-queries.json +++ b/plugins/briefing/evals/trigger-queries.json @@ -1,6 +1,6 @@ { "skill": "briefing", - "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.", + "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. Two labels were corrected by hand (issues #25, #24): `too long -- just the result and the options` (train) and `put the options in a table with a recommendation` (validation) are negatives here, not positives. Both committed cases were already written as standing instructions -- a dozen answers to come, `from here on` over six decisions -- and still fired 0x and 1x in three runs, with compactor silent on every run of both. Briefing holds no state: `allowed-tools` is `Read` alone, there is no config file and no session hook, so a request to answer more briefly is compliable in the current turn whether it is meant for one answer or for all of them, and nothing firing is correct. What does need this skill is the one thing the assistant cannot supply unaided -- what this project's output style actually says -- so both compensating positives ask for it to be loaded or read back, mirroring lingua's `what response language is set for this project`. Each replaced the negative that had asserted the same line the relabelled query now draws with runs behind it: `summarise the previous answer` and `condense this table`. The description still claims `when the user asks for shorter output`; that is a description-level mismatch and out of scope here -- #9 reverted a briefing rewrite and #25 does not re-authorise one.", "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.", "runs_per_query": 3, "trigger_rate_threshold": 0.5, @@ -31,7 +31,7 @@ "split": "train" }, { - "query": "too long -- just the result and the options", + "query": "load the project's output style for this planning run", "should_trigger": true, "split": "train" }, @@ -46,17 +46,17 @@ "split": "validation" }, { - "query": "put the options in a table with a recommendation", + "query": "ask me one thing at a time", "should_trigger": true, "split": "validation" }, { - "query": "ask me one thing at a time", + "query": "what output style do the planning agents use in this project", "should_trigger": true, "split": "validation" }, { - "query": "summarise the previous answer", + "query": "too long -- just the result and the options", "should_trigger": false, "split": "train" }, @@ -86,7 +86,7 @@ "split": "train" }, { - "query": "condense this table", + "query": "put the options in a table with a recommendation", "should_trigger": false, "split": "validation" }, From 5b3fc7815ae95bd66270ca126a57fc26b8057bba Mon Sep 17 00:00:00 2001 From: cmj Date: Wed, 16 Sep 2026 15:00:25 +0800 Subject: [PATCH 2/2] docs(evals): keep lingua's stop-refining positive, and record why The fifth failing case is the one the criterion does not decide by itself, so it is decided on the body instead. Phase 4 lists `stop refining` by name as a mid-session rule change that updates the config file with `Edit`, and the SessionStart hook reloads `refine_question` every session -- so a stop that is not persisted does not stop anything past this one. That is a write, which is exactly what the criterion asks for, and the label stands. `lingua off` is not the sharper phrasing to swap in: it carries the magic word, so it fires whatever the description says, and the body has it suspend for the session without writing the file, making it the less persistent of the two. Relabelling would assert something the body contradicts, and replacing the query would delete the evidence -- the asymmetry against `restate my question before you answer it` at 3/3, same skill and same vocabulary, opposite results. So this is recorded as a body-versus-description mismatch: the description names the on-state and never the off-state. No query, label or grader changes, and the case stays failing as the observation that would justify fixing the description. Refs: #25, #24 Co-Authored-By: Claude Opus 5 (1M context) --- plugins/lingua/evals/trigger-queries.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/lingua/evals/trigger-queries.json b/plugins/lingua/evals/trigger-queries.json index dec2eb4..8ee8a97 100644 --- a/plugins/lingua/evals/trigger-queries.json +++ b/plugins/lingua/evals/trigger-queries.json @@ -1,6 +1,6 @@ { "skill": "lingua", - "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.", + "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. `stop refining my questions` (validation) was reviewed under the same criterion (issues #25, #24) and its positive label stands, though the case fires 0x. The criterion asks what complying requires, and here it requires a write: Phase 4 lists `stop refining` by name as a mid-session rule change that updates the config file with `Edit`, and the SessionStart hook reloads `refine_question` every session, so a stop that is not persisted does not stop anything past this one. `lingua off` is not the sharper phrasing to swap in -- it carries the magic word, so it fires whatever the description says, and the body has it suspend for the session without writing the file, making it the less persistent of the two. Relabelling would assert something the body contradicts, and replacing the query would delete the evidence, which is the asymmetry against `restate my question before you answer it` at 3/3: same skill, same vocabulary, opposite results. So this is recorded as a body-versus-description mismatch -- the description names the on-state and never the off-state -- and the failing case is left standing as the observation that would justify fixing it.", "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.", "runs_per_query": 3, "trigger_rate_threshold": 0.5,