Repository navigation
Conversation
thisisnic
marked this pull request as draft
October 13, 2026 13:38
thisisnic
force-pushed
the
fix/reassess-stats-classify-nature-spec
branch
from
October 13, 2026 13:40
48ec757 to
a475818
Compare
…l spec The step-2-classify output spec lists `bug-as-advertised-partial-fix` as a nature bucket but never says whether `nature` is counted as given on each verdict or re-derived from the `cases` array. On the case-1-mixed-verdicts fixture the model sometimes re-buckets the one verdict with a mixed `cases` array into the partial-fix nature, which drops `bug-as-advertised` from 4 to 3 and adds a fifth cross-table row. Observed on two of four full-suite `claude -p` runs on main; Opus and Sonnet passed the case. State the rule the fixture and classify.md already assume: count `classification` and `nature` exactly as given, and let a mixed `cases` array only add the key to `partial_fixes`. Generated-by: Claude Code (Fable 5.1)
thisisnic
force-pushed
the
fix/reassess-stats-classify-nature-spec
branch
from
October 13, 2026 13:41
a475818 to
9f9fc4d
Compare
thisisnic
marked this pull request as ready for review
October 13, 2026 13:51
thisisnic
added a commit
to thisisnic/magpie
that referenced
this pull request
Oct 13, 2026
CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which apache#1615 addresses. Generated-by: Claude Code (Fable 5.1)
thisisnic
added a commit
to thisisnic/magpie
that referenced
this pull request
Oct 13, 2026
CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which apache#1615 addresses. Generated-by: Claude Code (Fable 5.1)
2 of 11 tasks
thisisnic
added a commit
to thisisnic/magpie
that referenced
this pull request
Oct 13, 2026
CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which apache#1615 addresses. Generated-by: Claude Code (Fable 5.1)
thisisnic
added a commit
to thisisnic/magpie
that referenced
this pull request
Oct 13, 2026
CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which apache#1615 addresses. Generated-by: Claude Code (Fable 5.1)
github-actions
Bot
added
family:issue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
issue-reassess-statsstep-2-classify eval failed intermittently oncase-1-mixed-verdicts: the model sometimes re-bucketed the one verdict with a mixedcasesarray intobug-as-advertised-partial-fix, droppingbug-as-advertisedfrom 4 to 3 and adding a fifthcross_tablerow.natureis counted as given or re-derived fromcases.classify.mdand the fixture both assume "as given".classificationandnatureverbatim, and let a mixedcasesarray only add the key topartial_fixes.Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
Before the change, on the one failing case (
step-2-classify/case-1-mixed-verdicts):claude -pdefault (Fable 5.1), full suite onmainclaude -p --model opusexpected.jsonclaude -p --model sonnetAfter the change, full suite with
claude -pdefault (Fable 5.1) on this branch:case-1-mixed-verdictsreturned the expected JSON all 12 timesprek runpasses on the changed file(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner --cli "claude -p" tools/skill-evals/evals/issue-reassess-stats/, 12 consecutive runs)RFC-AI-0004 compliance
No principles touched — prompt-wording change in eval material only; no new mutation, host access, or LLM hop. No
SKILL.mdedit, so no token restamp.Linked issues
None.
Generated-by: Claude Code (Fable 5.1)
🤖 Generated with Claude Code