From 9f9fc4d79ef62e2230514d8eca397f6628ca5754 Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Tue, 13 Oct 2026 14:35:51 +0100 Subject: [PATCH] fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec The step-2-classify output spec lists `bug-as-advertised-partial-fix` as a nature bucket but never says whether `nature` is counted as given on each verdict or re-derived from the `cases` array. On the case-1-mixed-verdicts fixture the model sometimes re-buckets the one verdict with a mixed `cases` array into the partial-fix nature, which drops `bug-as-advertised` from 4 to 3 and adds a fifth cross-table row. Observed on two of four full-suite `claude -p` runs on main; Opus and Sonnet passed the case. State the rule the fixture and classify.md already assume: count `classification` and `nature` exactly as given, and let a mixed `cases` array only add the key to `partial_fixes`. Generated-by: Claude Code (Fable 5.1) --- .../step-2-classify/fixtures/output-spec.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md b/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md index a49901553..982d22ab8 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md +++ b/tools/skill-evals/evals/issue-reassess-stats/step-2-classify/fixtures/output-spec.md @@ -27,5 +27,7 @@ Return ONLY valid JSON with this structure: ``` `cross_table` contains only non-zero combinations. +Count `classification` and `nature` exactly as given on each verdict; do not re-derive or re-bucket either. +A mixed `cases` array does not change a verdict's `nature`; it only adds the key to `partial_fixes`. `partial_fixes` contains keys where the `cases` array has mixed outcomes (some still-failing, some fixed). Do not include any text outside the JSON object.