Skip to content

docs(skill-evals): add a convention for recording live eval runs - #1626

Draft
thisisnic wants to merge 1 commit into
apache:mainfrom
thisisnic:docs/eval-run-log
Draft

thisisnic wants to merge 1 commit into
apache:mainfrom
thisisnic:docs/eval-run-log

Conversation

@thisisnic

@thisisnic thisisnic commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Draft for design feedback before more families get logged.
  • The eval suites are never run against a live model in CI, so there was no record of which suites had been run on which model, or how they fared. Adds a short Recording runs convention to the harness README: a per-skill Run history table and four rules.
  • Seeds the first table in issue-reassess-stats with the pre-fix flake observed on main that fix(issue-reassess-stats): pin nature counting in step-2 classify eval spec #1615 addresses.

Design questions I'd like feedback on

  • Per-skill tables only, no cross-skill summary page. An earlier draft had a hand-maintained docs/eval-run-log.md listing all 81 suites; it was either a duplicate of the per-skill tables or a lossy digest, so it was dropped.
  • Refs are SHAs on main only; branch runs aren't logged. Simpler than tracking branch SHAs that rebases rewrite.
  • Model names are whatever the CLI calls them, no cross-provider normalisation.
  • Rows are append-only. No pruning rules yet; happy to add them if the tables get long.

Type of change

  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Skill change
  • Tool / bridge contract
  • Python package
  • Groovy reference impl
  • Cross-cutting
  • Project template
  • CI / dev loop
  • Other:

Test plan

  • prek run passes on the changed files
  • prek run --all-files
  • N/A for evals: no skill or fixture text changes

RFC-AI-0004 compliance

No principles touched — documentation only.

Linked issues

Refs #1615 (the fix the seeded row points at).


Generated-by: Claude Code (Fable 5.1)

🤖 Generated with Claude Code

@thisisnic

thisisnic commented Oct 13, 2026 •

Copy link
Copy Markdown
Member Author

@potiuk - I was going to run some more of these but was after a bit of feedback on the design of this before I get into it more. I've kept it intentionally minimal for the moment, and was a bit hesitant to add a summary page as it'd need to keep in sync with the individuals suites, but alternatively, I could ditch the individual ones in favour of an overall summary page - what do you think?

CI never runs the eval suites against a live model, so the only record
of how a suite fares on a given model is whatever someone wrote down.
Nothing was written down. Add a "Recording runs" section to the harness
README defining a per-skill Run history table and four rules: main-only
SHAs as the ref, the model name as the CLI uses it, one row per batch of
identical runs, and record flakes.

Seed the first table in issue-reassess-stats with the pre-fix flake
observed on main, which apache#1615 addresses.

Generated-by: Claude Code (Fable 5.1)
@github-actions github-actions Bot added family:tools tools/* family:docs Docs, MISSION.md, READMEs family:issue issue-* skills capability:stats Read-only dashboards, metrics, governance evidence substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:stats Read-only dashboards, metrics, governance evidence family:docs Docs, MISSION.md, READMEs family:issue issue-* skills family:tools tools/* substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant