Repository navigation
Ci/benchmark tooling checks - #3
Merged
Merged
Conversation
perf_check.sh A/Bs the working tree against its merge-base on one machine and exits non-zero on a regression; perf_gate.py turns the two logs into a verdict; select_kernels.py resolves benchmarks/suites/kernels.tsv into the subset a given host can run. Rationale and usage are in benchmarks/README.md. Both sides are measured here and now rather than against a number stored in the repo, because the same binary varies run to run by more than the effects worth catching: on an Arc Pro B70, bf16_gemm 4096 cubed spans 0.874-0.994 ms (13.7%) across nine trials of one build while gqa_forward spans 1.3%. The two test suites need no GPU -- test_perf_gate.py synthesises its own logs, test_select_kernels.py checks selection and the committed manifest -- so they are the part that can run on a hosted runner. The next commit wires them in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Extends the existing lint job rather than adding a new one, since both additions are no-GPU Python and finish in seconds on a hosted runner. Syntax coverage follows the files that now exist: bash -n on perf_check.sh, py_compile on the three new tools. The two test suites are the substantive part. perf_gate.py decides whether a PR is blocked, so its verdict logic is tested rather than trusted. test_select_kernels.py checks selection and the committed manifest together, including that every kernel directory kernels.tsv names still has a Makefile -- renaming a kernel family then fails the PR instead of silently shrinking what the gate measures, which is the failure mode that would otherwise look like a pass. The GPU half of the gate is deliberately not wired up here: perf_check.sh needs a self-hosted runner with an Intel GPU and a fetch-depth 0 checkout to reach the merge-base. It already writes a GITHUB_STEP_SUMMARY table when that variable is set, so adding such a job later is a checkout plus one invocation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.