diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 9fbc76bc6..b3dc4ef60 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -510,6 +510,22 @@ reserve `judge` for properties that cannot be pinned down locally. For everything in between (decisions wrapped in explanatory prose like `rationale` or `reason`), `--grader-cli` adds a third mode: decision fields stay on exact equality, prose fields go to a cheap judge model that scores "does the candidate support the same conclusion?" See the "Field-aware grading" section above. +## Recording runs + +CI never runs a live model, so live runs are recorded by hand +in a *Run history* table in `evals//README.md`. + +```text +| Date | Ref | Model | Runs | Result | Notes | +``` + +- Ref is a short SHA on `main`; runs on a branch are not logged. +- Model is the name the CLI uses, such as `claude-fable-5-1`. +- One row per batch of identical runs; append, do not replace. +- Record a failure even if a rerun passed. + +Example: [`evals/issue-reassess-stats/README.md`](evals/issue-reassess-stats/README.md#run-history). + ## CI considerations Print mode supports manual review, while `--cli` runs the configured model and comparator automatically. diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md index 1dbd99f6a..2ff2fb789 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/README.md +++ b/tools/skill-evals/evals/issue-reassess-stats/README.md @@ -28,3 +28,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reassess-stats/step-3-aggregate/fixtures/case-1-healthy ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) |