Skip to content

Add the baseline workflow - #19

Merged
tamnd merged 1 commit into
mainfrom
m0/baseline
Nov 21, 2026
Merged

tamnd merged 1 commit into
mainfrom
m0/baseline

Conversation

@tamnd

@tamnd tamnd commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR adds the baseline of spec 08.7.1. The M0 exit criterion of spec 14 says that the numbers of the official kernel on our machine must be within 2% of the Arena numbers.

  • scripts/arena-fetch.sh downloads lean-arena-tests.tar.gz and results.json from the Arena. The Arena builds both files again on each deploy, so the script checks the SHA-256 of each export against the hash in results.json. It deletes a test with a different hash. It writes the hashes of the downloads and the Arena commit to sources.txt.
  • The bench workflow installs Lean v4.35.0 from the release tarball (pinned by SHA-256) and valgrind. It runs leanchecker --from-export on each test with rulean-bench, then rulean-bench compare --tolerance 2 against the official results of the Arena. It uploads the run file and the comparison.
  • The workflow runs each Monday, by hand, and on a pull request that changes the harness or the workflow.

GitHub-hosted runners do not expose the hardware counters, so the default counter is cachegrind. With --cachegrind-max 5, the job counts only the tests whose first run took at most 5 seconds. The large perf/ tests take more than an hour each under valgrind.

Result of the first run

The first run is 44912719616. docs/bench/baseline.md records it.

  • The verdicts match the Arena on all 225 tests.
  • The cachegrind counts are not within 2% of the Arena counts. The ratio of the sums is 1.0116, but 206 of the 212 counted tests are above 2%. Each small run counts about 19.5 M instructions fewer than the Arena (171 M against 190 M). perf/gf3-matrix-product counts 42% more.

So cachegrind cannot give the baseline. The compare step now applies the 2% limit only when the counter is perf. It still fails when a test gives a different status. The M0 acceptance item about the 2% limit needs a run with perf on a Linux x86-64 machine with hardware counters.

Test

  • actionlint and zizmor report no findings. shellcheck reports no findings on the script.
  • scripts/arena-fetch.sh _tmp/arena on macOS: 225 tests kept, 0 dropped, for the Arena commit dbb57ce of 2026-11-21.
  • The bench workflow ran on this PR in 41 minutes. The run of the official kernel took 233 s of wall time over the 225 tests, and the rest of the time was cachegrind.

Refers to #1

@tamnd tamnd added this to the M0 Foundations milestone Nov 21, 2026
@tamnd tamnd added area/bench Benchmarks and instruction counts area/ci Workflows, releases, policy checks and repository settings type/feature New function labels Nov 21, 2026
@tamnd tamnd mentioned this pull request Nov 21, 2026
9 of 12 tasks
@tamnd
tamnd marked this pull request as ready for review November 21, 2026 14:57
@tamnd
tamnd force-pushed the m0/baseline branch 3 times, most recently from 5006997 to fc4669e Compare November 21, 2026 16:35
The bench workflow runs the official kernel v4.35.0 on the Arena tests and compares the instruction counts with the Arena numbers. scripts/arena-fetch.sh downloads the tests, and docs/bench/baseline.md records the first run.
@tamnd
tamnd merged commit 881677a into main Nov 21, 2026
18 checks passed
@tamnd
tamnd deleted the m0/baseline branch November 21, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bench Benchmarks and instruction counts area/ci Workflows, releases, policy checks and repository settings type/feature New function

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant