From 1c3babd695f2250f5f53600035fed9906999c69c Mon Sep 17 00:00:00 2001 From: Nic Crane Date: Tue, 13 Oct 2026 16:17:56 +0100 Subject: [PATCH] docs(skill-evals): add a convention for recording live eval runs CI never runs the eval suites against a live model, so the only record of how a suite fares on a given model is whatever someone wrote down. Nothing was written down. Add a "Recording runs" section to the harness README defining a per-skill Run history table and four rules: main-only SHAs as the ref, the model name as the CLI uses it, one row per batch of identical runs, and record flakes. Seed the first table in issue-reassess-stats with the pre-fix flake observed on main, which #1615 addresses. Generated-by: Claude Code (Fable 5.1) --- tools/skill-evals/README.md | 16 ++++++++++++++++ .../evals/issue-reassess-stats/README.md | 9 +++++++++ 2 files changed, 25 insertions(+) diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index 9fbc76bc6..b3dc4ef60 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -510,6 +510,22 @@ reserve `judge` for properties that cannot be pinned down locally. For everything in between (decisions wrapped in explanatory prose like `rationale` or `reason`), `--grader-cli` adds a third mode: decision fields stay on exact equality, prose fields go to a cheap judge model that scores "does the candidate support the same conclusion?" See the "Field-aware grading" section above. +## Recording runs + +CI never runs a live model, so live runs are recorded by hand +in a *Run history* table in `evals//README.md`. + +```text +| Date | Ref | Model | Runs | Result | Notes | +``` + +- Ref is a short SHA on `main`; runs on a branch are not logged. +- Model is the name the CLI uses, such as `claude-fable-5-1`. +- One row per batch of identical runs; append, do not replace. +- Record a failure even if a rerun passed. + +Example: [`evals/issue-reassess-stats/README.md`](evals/issue-reassess-stats/README.md#run-history). + ## CI considerations Print mode supports manual review, while `--cli` runs the configured model and comparator automatically. diff --git a/tools/skill-evals/evals/issue-reassess-stats/README.md b/tools/skill-evals/evals/issue-reassess-stats/README.md index 1dbd99f6a..2ff2fb789 100644 --- a/tools/skill-evals/evals/issue-reassess-stats/README.md +++ b/tools/skill-evals/evals/issue-reassess-stats/README.md @@ -28,3 +28,12 @@ uv run --project tools/skill-evals skill-eval \ uv run --project tools/skill-evals skill-eval \ tools/skill-evals/evals/issue-reassess-stats/step-3-aggregate/fixtures/case-1-healthy ``` + +## Run history + +Live model runs over this whole eval directory, newest first. +Columns and rules: [Recording runs](../../README.md#recording-runs). + +| Date | Ref | Model | Runs | Result | Notes | +|---|---|---|---|---|---| +| 2026-10-13 | `dadb315c` | `claude-fable-5-1` | 4 | `step-2-classify/case-1-mixed-verdicts` failed 2 of 4 | fix: [#1615](https://github.com/apache/magpie/pull/1615) |