Relabel the four positives nothing should have fired on, and keep the one that should

plugins/agent-hale/evals/agreed-plan-write-it/graders/does-not-fire.md

@@ -0,0 +1,12 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"agent-hale'
+min: 0
+max: 0
+---
+
+Relabelled by #25 and #24 after this case fired 0x on three runs. Nothing is left to decide and
+the code can simply be written, which is a turn the session can comply with unaided — and where
+it can, nothing should fire. Hale's own ground is a unit dispatched from a plan, built
+test-first, with the suite result reported back; this turn asks for none of that.

plugins/agent-hale/evals/trigger-queries.json

@@ -1,6 +1,6 @@
 {
   "skill": "agent-hale",
-  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.",
+  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. One validation label was corrected by hand (issues #25, #24): `the plan is agreed -- write it` is a negative here, not a positive. Its committed case states the agreed approach in full and still fired 0x across three runs, with spec-writer silent too, so nothing was stealing it. On the criterion settled for this marketplace -- a skill fires when the request needs what the assistant cannot do unaided -- writing the code that a settled plan describes is ordinary compliance in the current turn, and correct behaviour is that nothing fires. The compensating positive names what Hale adds beyond typing the code: a unit taken from the plan, built test-first, and a suite result reported back, which is the coordination limb of the criterion. It replaced `why is this endpoint slow`, whose line agent-page's set now draws from measurement rather than assertion. The set's remaining plain-implementation positives sit on the same boundary and are unmeasured; deciding them needs a run, not a blanket relabel.",
   "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.",
   "runs_per_query": 3,
   "trigger_rate_threshold": 0.5,
@@ -36,22 +36,22 @@
       "split": "train"
     },
     {
-      "query": "the plan is agreed -- write it",
+      "query": "fill in the handlers for these three routes",
       "should_trigger": true,
       "split": "validation"
     },
     {
-      "query": "fill in the handlers for these three routes",
+      "query": "write the unit tests for the cache layer",
       "should_trigger": true,
       "split": "validation"
     },
     {
-      "query": "write the unit tests for the cache layer",
+      "query": "make this compile against the new api",
       "should_trigger": true,
       "split": "validation"
     },
     {
-      "query": "make this compile against the new api",
+      "query": "take unit 3 from the plan, build it test-first, and report what the suite did",
       "should_trigger": true,
       "split": "validation"
     },
@@ -86,12 +86,12 @@
       "split": "train"
     },
     {
-      "query": "is this code secure",
+      "query": "the plan is agreed -- write it",
       "should_trigger": false,
       "split": "validation"
     },
     {
-      "query": "why is this endpoint slow",
+      "query": "is this code secure",
       "should_trigger": false,
       "split": "validation"
     },

plugins/agent-page/evals/endpoint-slow-in-prod/graders/does-not-fire.md

@@ -0,0 +1,14 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"agent-page'
+min: 0
+max: 0
+---
+
+Relabelled by #25 and #24 after this case fired 0x on three runs. Page reviews a change, not a
+running system: Phase 1 wants a `git diff`, Phase 4's performance list is a review checklist
+applied to that diff, and the phase stops outright when no `__REVIEW_VERDICT__` arrived. This
+turn supplies none of the three, so reading the pool-wait graph is ordinary work and nothing
+should load. The baseline fired nothing at all here, so there is still no second skill to rule
+out.

plugins/agent-page/evals/trigger-queries.json

@@ -1,6 +1,6 @@
 {
   "skill": "agent-page",
-  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.",
+  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. One validation label was corrected by hand (issues #25, #24): `why is this endpoint slow in production` is a negative here, not a positive. `make eval` on `dab7767` ran it three times as a realistic turn and Page fired 0x on every one, and the body says why -- Phase 1 reviews `the change, not the codebase` and needs a `git diff`, Phase 4's performance list is a review checklist applied to that diff, and the phase stops outright when no `__REVIEW_VERDICT__` arrived. A live latency regression supplies none of those, so complying means diagnosing a running system, which is not what this skill does. Its nine other positives are all review-of-a-change; this was the only diagnostic one, and the set already drew the same line with `fix the slow query`. That asserted negative was spent to restore the positive, because the relabelled query now draws the line with three measured runs behind it. This corrects what #24 and #25 both claim -- they read the 0/3 as a description gap.",
   "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.",
   "runs_per_query": 3,
   "trigger_rate_threshold": 0.5,
@@ -51,7 +51,7 @@
       "split": "validation"
     },
     {
-      "query": "why is this endpoint slow in production",
+      "query": "before we ship this change, check its timeout and retry handling",
       "should_trigger": true,
       "split": "validation"
     },
@@ -86,7 +86,7 @@
       "split": "train"
     },
     {
-      "query": "fix the slow query",
+      "query": "why is this endpoint slow in production",
       "should_trigger": false,
       "split": "validation"
     },

plugins/briefing/evals/options-table-recommendation/graders/does-not-fire.md

@@ -0,0 +1,12 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"briefing'
+min: 0
+max: 0
+---
+
+Relabelled by #25 and #24 after this case fired 1x on three runs, below the 0.5 threshold. "From
+here on" does set a format for six upcoming decisions, but putting options in a table with a
+recommendation is as easy to do six times as once, and briefing persists nothing that would carry
+the format past this session. The one run that did fire is what a threshold is for.

plugins/briefing/evals/result-and-options/graders/does-not-fire.md

@@ -0,0 +1,13 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"briefing'
+min: 0
+max: 0
+---
+
+Relabelled by #25 and #24 after this case fired 0x on three runs. A dozen more to come does make
+this a standing instruction rather than a request to redo the last answer — but briefing holds no
+state to set. Its `allowed-tools` is `Read` alone, with no config file and no session hook, so
+leading with the result for the rest of the afternoon is something the session can just do, and a
+standing instruction it can honour unaided is not a skill invocation.

plugins/briefing/evals/trigger-queries.json

@@ -1,6 +1,6 @@
 {
   "skill": "briefing",
-  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.",
+  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. Two labels were corrected by hand (issues #25, #24): `too long -- just the result and the options` (train) and `put the options in a table with a recommendation` (validation) are negatives here, not positives. Both committed cases were already written as standing instructions -- a dozen answers to come, `from here on` over six decisions -- and still fired 0x and 1x in three runs, with compactor silent on every run of both. Briefing holds no state: `allowed-tools` is `Read` alone, there is no config file and no session hook, so a request to answer more briefly is compliable in the current turn whether it is meant for one answer or for all of them, and nothing firing is correct. What does need this skill is the one thing the assistant cannot supply unaided -- what this project's output style actually says -- so both compensating positives ask for it to be loaded or read back, mirroring lingua's `what response language is set for this project`. Each replaced the negative that had asserted the same line the relabelled query now draws with runs behind it: `summarise the previous answer` and `condense this table`. The description still claims `when the user asks for shorter output`; that is a description-level mismatch and out of scope here -- #9 reverted a briefing rewrite and #25 does not re-authorise one.",
   "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.",
   "runs_per_query": 3,
   "trigger_rate_threshold": 0.5,
@@ -31,7 +31,7 @@
       "split": "train"
     },
     {
-      "query": "too long -- just the result and the options",
+      "query": "load the project's output style for this planning run",
       "should_trigger": true,
       "split": "train"
     },
@@ -46,17 +46,17 @@
       "split": "validation"
     },
     {
-      "query": "put the options in a table with a recommendation",
+      "query": "ask me one thing at a time",
       "should_trigger": true,
       "split": "validation"
     },
     {
-      "query": "ask me one thing at a time",
+      "query": "what output style do the planning agents use in this project",
       "should_trigger": true,
       "split": "validation"
     },
     {
-      "query": "summarise the previous answer",
+      "query": "too long -- just the result and the options",
       "should_trigger": false,
       "split": "train"
     },
@@ -86,7 +86,7 @@
       "split": "train"
     },
     {
-      "query": "condense this table",
+      "query": "put the options in a table with a recommendation",
       "should_trigger": false,
       "split": "validation"
     },

plugins/lingua/evals/trigger-queries.json

@@ -1,6 +1,6 @@
 {
   "skill": "lingua",
-  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation.",
+  "about": "Trigger eval queries for this skill's description. A query is labelled with whether the skill should activate for it; the negatives are near-misses that share vocabulary with the skill but need something else. Split fixed so successive description edits are compared on the same ground: tune against train, judge against validation. `stop refining my questions` (validation) was reviewed under the same criterion (issues #25, #24) and its positive label stands, though the case fires 0x. The criterion asks what complying requires, and here it requires a write: Phase 4 lists `stop refining` by name as a mid-session rule change that updates the config file with `Edit`, and the SessionStart hook reloads `refine_question` every session, so a stop that is not persisted does not stop anything past this one. `lingua off` is not the sharper phrasing to swap in -- it carries the magic word, so it fires whatever the description says, and the body has it suspend for the session without writing the file, making it the less persistent of the two. Relabelling would assert something the body contradicts, and replacing the query would delete the evidence, which is the asymmetry against `restate my question before you answer it` at 3/3: same skill, same vocabulary, opposite results. So this is recorded as a body-versus-description mismatch -- the description names the on-state and never the off-state -- and the failing case is left standing as the observation that would justify fixing it.",
   "how_to_run": "`make eval` is the committed runner, but it covers only the queries a 3-sample blind-router baseline actually missed, not this whole file -- see CLAUDE.md for why 13 and not 380. To judge the rest: `skillgrade` runs a labelled set like this one and exits non-zero when the pass rate falls below a threshold. Needing nothing installed: run each query with `claude -p --output-format json` and count the runs whose transcript contains a Skill tool_use naming this skill. Cheapest of all, and closest to the router's real conditions: give a model only the name and description of every installed skill, with no bodies and no repo access, and have it pick one skill or none per query. `claude plugin eval`, which `make eval` drives, is generally available rather than early access; what it needs instead is a credential, a network and real money -- $0.142 a run, measured -- which is why it sits outside `make test` and CI does not run it. It will not reproduce this file query-for-query either: a bare query does not fire a skill, so a case is one of these queries rewritten as a realistic turn. This file stays the cheap, free, offline record.",
   "runs_per_query": 3,
   "trigger_rate_threshold": 0.5,