Repository navigation
Add the baseline workflow - #19
Merged
Merged
Conversation
tamnd
marked this pull request as ready for review
November 21, 2026 14:57
tamnd
force-pushed
the
m0/baseline
branch
3 times, most recently
from
November 21, 2026 16:35
5006997 to
fc4669e
Compare
The bench workflow runs the official kernel v4.35.0 on the Arena tests and compares the instruction counts with the Arena numbers. scripts/arena-fetch.sh downloads the tests, and docs/bench/baseline.md records the first run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the baseline of spec 08.7.1. The M0 exit criterion of spec 14 says that the numbers of the official kernel on our machine must be within 2% of the Arena numbers.
scripts/arena-fetch.shdownloadslean-arena-tests.tar.gzandresults.jsonfrom the Arena. The Arena builds both files again on each deploy, so the script checks the SHA-256 of each export against the hash inresults.json. It deletes a test with a different hash. It writes the hashes of the downloads and the Arena commit tosources.txt.benchworkflow installs Lean v4.35.0 from the release tarball (pinned by SHA-256) and valgrind. It runsleanchecker --from-exporton each test withrulean-bench, thenrulean-bench compare --tolerance 2against theofficialresults of the Arena. It uploads the run file and the comparison.GitHub-hosted runners do not expose the hardware counters, so the default counter is cachegrind. With
--cachegrind-max 5, the job counts only the tests whose first run took at most 5 seconds. The largeperf/tests take more than an hour each under valgrind.Result of the first run
The first run is 44912719616.
docs/bench/baseline.mdrecords it.perf/gf3-matrix-productcounts 42% more.So cachegrind cannot give the baseline. The compare step now applies the 2% limit only when the counter is
perf. It still fails when a test gives a different status. The M0 acceptance item about the 2% limit needs a run withperfon a Linux x86-64 machine with hardware counters.Test
actionlintandzizmorreport no findings.shellcheckreports no findings on the script.scripts/arena-fetch.sh _tmp/arenaon macOS: 225 tests kept, 0 dropped, for the Arena commit dbb57ce of 2026-11-21.benchworkflow ran on this PR in 41 minutes. The run of the official kernel took 233 s of wall time over the 225 tests, and the rest of the time was cachegrind.Refers to #1