From 1d3a6c2edf5985850dc3daba5260ff72ec3e81f9 Mon Sep 17 00:00:00 2001 From: "Chen, Sheng S" Date: Mon, 19 Oct 2026 10:15:59 +0800 Subject: [PATCH 1/2] benchmarks: add perf gate tooling and its tests perf_check.sh A/Bs the working tree against its merge-base on one machine and exits non-zero on a regression; perf_gate.py turns the two logs into a verdict; select_kernels.py resolves benchmarks/suites/kernels.tsv into the subset a given host can run. Rationale and usage are in benchmarks/README.md. Both sides are measured here and now rather than against a number stored in the repo, because the same binary varies run to run by more than the effects worth catching: on an Arc Pro B70, bf16_gemm 4096 cubed spans 0.874-0.994 ms (13.7%) across nine trials of one build while gqa_forward spans 1.3%. The two test suites need no GPU -- test_perf_gate.py synthesises its own logs, test_select_kernels.py checks selection and the committed manifest -- so they are the part that can run on a hosted runner. The next commit wires them in. Co-Authored-By: Claude Opus 5 --- benchmarks/README.md | 359 ++++++++++++++++++++++++ benchmarks/suites/kernels.tsv | 56 ++++ benchmarks/suites/perf_check.sh | 356 +++++++++++++++++++++++ benchmarks/tools/parse_baseline.py | 242 ++++++++++++++++ benchmarks/tools/perf_gate.py | 269 ++++++++++++++++++ benchmarks/tools/select_kernels.py | 140 +++++++++ benchmarks/tools/test_perf_gate.py | 154 ++++++++++ benchmarks/tools/test_select_kernels.py | 151 ++++++++++ 8 files changed, 1727 insertions(+) create mode 100644 benchmarks/suites/kernels.tsv create mode 100755 benchmarks/suites/perf_check.sh create mode 100755 benchmarks/tools/parse_baseline.py create mode 100755 benchmarks/tools/perf_gate.py create mode 100755 benchmarks/tools/select_kernels.py create mode 100755 benchmarks/tools/test_perf_gate.py create mode 100755 benchmarks/tools/test_select_kernels.py diff --git a/benchmarks/README.md b/benchmarks/README.md index a990e14..91c85a2 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -4,6 +4,365 @@ This directory contains scheduler-independent benchmark contracts for the public kernel release. Platform allocation, host selection, scheduler options, and raw campaign evidence are deliberately kept outside this repository. +## Performance Check for a Change + +`benchmarks/suites/perf_check.sh` answers one question: did this branch make a +kernel slower? It builds the working tree *and* its merge-base, runs each +binary base-then-head back to back for several rounds, and exits non-zero on a +regression. Run it before opening a pull request. + +### Prerequisites + +- A local Intel GPU. The script detects the AOT target itself. +- `icpx` on `PATH`. The script sources `env.src` for you; nothing to do if that + already works for a normal kernel build. +- **Git history deep enough to reach the base commit.** A `--depth 1` clone has + no merge-base and the script will refuse to guess. `git fetch --unshallow`, or + pass `--base ` naming something you do have. +- oneMKL only for `--all`, which includes `decode`. The fast subset skips it. + +### Running it + +```bash +benchmarks/suites/perf_check.sh # fast subset, 5 trials per side +benchmarks/suites/perf_check.sh --all # every single-GPU case +benchmarks/suites/perf_check.sh --multi # the multi-GPU cases only +benchmarks/suites/perf_check.sh --all --multi # everything this host can run +benchmarks/suites/perf_check.sh --list # what would run here, no build +benchmarks/suites/perf_check.sh --trials 9 # tighter noise floor +benchmarks/suites/perf_check.sh --base v0.9.0 # against a tag instead of main +``` + +What it does, in order: resolve `git merge-base HEAD origin/main`; check that +commit out into a throwaway `git worktree`; build both trees, one family per +background job; run every binary base-then-head back to back, repeating for +`--trials` rounds; hand the two logs to `tools/perf_gate.py`; remove the +worktree. + +The head side is the **working tree, uncommitted changes included**, so it is +usable while still iterating. The base side is always a clean checkout. + +Measured on an Arc Pro B70, 2026.1 toolchain: + +| Scope | Binaries | Families | Cases | Trials | Wall clock | +|:--|--:|--:|--:|--:|--:| +| default | 5 | 4 | 16 | 5 | ~1m 25s (26s build, 57s run) | +| `--all` | 11 | 6 | 35 | 3 | ~1m 20s | + +Both builds dominate the fixed cost, so raising `--trials` is cheap: each extra +round adds only its own run time. + +The default subset is `bf16_gemm`, `qkv_gemm`, `gqa_forward`, `rmsnorm` and +`silu_mul` — the cases that are cheap, quiet, and cover the matrix, reduction +and elementwise paths. `--all` adds `gqa_backward`, `layernorm`, the three +`rotary` binaries and `decode`. + +### Single-GPU and multi-GPU scopes + +Single-GPU and multi-GPU benchmarks belong to different machines, so they are +selected separately and the same command is correct on both. A one-card +workstation runs the whole `fast`/`full` set and none of the collectives; a +12-tile host is the only place `all_to_all` means anything. + +| Scope | Flags | Tiers | Needs | +|:--|:--|:--|:--| +| PR gate default | *(none)* | `fast` | 1 GPU | +| whole single-GPU set | `--all` | `fast` + `full` | 1 GPU, oneMKL for `decode` | +| multi-GPU only | `--multi` | `multi` | 2+ GPUs, `ZE_FLAT_DEVICE_HIERARCHY=FLAT` | +| everything runnable | `--all --multi` | all three | as above | + +`--multi` alone **narrows** the run to the collectives; combined with `--all` it +widens instead. Nothing is hardcoded in the script: the kernel list, each +binary's device requirement and its make target are declared once in +[`suites/kernels.tsv`](suites/kernels.tsv), and `tools/select_kernels.py` +applies the filtering. Adding a kernel is a line in that file, not an edit to +three runners. + +The visible device count is detected from `ZE_AFFINITY_MASK` when a platform +adapter set one — that is the count the kernels will actually see — and +otherwise from `sycl-ls`. Override it with `--devices N` or `SK_NUM_DEVICES`. + +Check what a host will do before spending a build on it. `--list` prints the +selection and exits, with every skip and its reason on stderr: + +``` +$ benchmarks/suites/perf_check.sh --multi --list +select_kernels: skip collective/gemm_rs: needs 6 GPU(s), have 2 +select_kernels: skip collective/all_to_all: needs 12 GPU(s), have 2 +select_kernels: skip collective/ag_gemm: needs 12 GPU(s), have 2 +scope multi on 2 GPU(s), target intel_gpu_bmg_g31 +collective/all_reduce 2-12 GPU multi standalone +collective/all_gather 2-12 GPU multi standalone +collective/reduce_scatter 2-12 GPU multi standalone +context_attention/ring_attention 2-6 GPU multi all +context_attention/ring_attention_overlap 2-12 GPU multi all +context_attention/all_gather_attention 2-6 GPU multi all +``` + +Two kinds of device requirement show up in that `GPU` column. `2-12` scales: +the binary reads the topology (or `SK_NUM_DEVICES`) and adapts, so any count in +range works. A bare `6` or `12` is fixed: `NUM_DEVICES` is a compile-time +constant with no environment override, so the binary is skipped on every other +count rather than run wrong. Single-GPU rows read `1+` — they are *not* dropped +on a big host, they simply use one device. + +Skips are printed, never hidden, and a selection that comes out empty exits +non-zero. A run that measured nothing must not read as a pass. + +> **On a 2-card host the multi-GPU kernels do not currently run.** Verified on +> this box: `all_reduce` across two discrete B70s fails with +> `UR_RESULT_ERROR_DEVICE_LOST`; with `ZE_FLAT_DEVICE_HIERARCHY=FLAT` it instead +> busy-waits on its barrier forever; `ring_attention` reports `GPUs: 4` and then +> throws `Can't add devices across platforms to a single context`. These +> harnesses assume tiles of one multi-tile device, not separate cards. `--multi` +> therefore selects and builds correctly here but every trial times out, and the +> gate exits 2 (harness error) rather than 0. Treat the multi-GPU scope as +> valid only on a true multi-tile host for now. + +### Reading the output + +Real output, abridged to one row per verdict: + +``` +== perf check + base 9432113 For each kernel, add a standalone path to build the cpp source. (#1) + head 27b206a Add a local perf regression gate + uncommitted changes + target intel_gpu_bmg_g31 + devices 2 visible + scope fast (5 binaries, 4 families) + trials 5 per side, iters=50 warmup=5 + +== build base (9432113...) + base/activation: OK + base/attention: OK + base/gemm: OK + base/norm: OK +== build head + head/activation: OK + ... + note: base/gemm has 3 spilling kernel(s) + +== run + trial 1/5: bf16_gemm_4096x4096x4096 qkv_gemm gqa_forward rmsnorm silu_mul + ... + +== perf gate base 9432113 -> head 27b206a-dirty + target=intel_gpu_bmg_g31 iters=50 trials=5 floor=3% + + case base ms head ms delta noise verdict + -------------------------------------------------------------------------------------------------------------- + silu_mul|silu_mul T=2048 inter=14336 0.3288 0.3388 +3.0% 1.3% REGRESSION + rmsnorm|rmsnorm T=2048 D=4096 0.0545 0.0560 +2.8% 1.2% NEUTRAL + bf16_gemm_4096x4096x4096|GEMM 4096x4096x4096 (bf16 in, bf16 out) 18.0910 0.8700 -95.2% 6.6% IMPROVEMENT + rmsnorm|rmsnorm T=1 D=4096 0.0021 0.0028 +33.3% 461.9% BELOW-RES (best < 0.05 ms) + + REGRESSION=1 NEUTRAL=5 IMPROVEMENT=7 BELOW-RES=3 + + FAIL: 1 blocking case(s) + REGRESSION +3.0% silu_mul|silu_mul T=2048 inter=14336 + + logs: benchmarks/results/perf_check_20261014T074556Z.base.log + benchmarks/results/perf_check_20261014T074556Z.head.log +``` + +- `case` is `binary|case-label`, so the same shape measured by two binaries stays + distinguishable. +- `base ms` / `head ms` are **peak@N** — the fastest correct trial on each side. +- `delta` is the head/base latency change. Positive is slower. +- `noise` is the widest run-to-run spread seen on either side, as a fraction of + that side's best trial. It is the resolution limit of this particular run. +- Rows are sorted worst-news-first, not alphabetically. +- `-dirty` on the head commit means uncommitted changes were included. + +Compare the two middle rows to see the point: `+3.0%` against `1.3%` noise is a +call, while `+2.8%` against `1.2%` noise is not, because the floor is 3%. + +### What to do about each verdict + +| Verdict | Meaning | Blocks | What to do | +|:--|:--|:--|:--| +| `REGRESSION` | slower by more than the per-case threshold | yes | Confirm it, then fix or justify. See below. | +| `CORRECTNESS` | stopped passing its own numeric check | yes | Fix. Perf is not meaningful until it passes. | +| `MISSING` | measured on base, absent from head | yes | Either restore the case or say in the PR why it went. | +| `NUMERICS` | residual grew by more than `--error-factor` | `--fail-on-numerics` | Check you did not trade accuracy for speed by accident. | +| `NOISY` | spread exceeded the effect; inconclusive | `--fail-on-noisy` | Re-run with more `--trials`, or quiet the machine. | +| `NEUTRAL` | within noise, and the noise was small | no | Nothing. | +| `IMPROVEMENT` | faster by more than the threshold | no | Quote the number in the PR. | +| `BELOW-RES` | best trial under `--min-ms`; a launch-latency probe | no | Ignore. `T=1` runs at 0.003 ms and swings 30% on scheduling alone. | +| `NEW` | absent from base | no | Nothing. | + +`CORRECTNESS` outranks perf on purpose: a change that "wins" by breaking +numerics must not read as an improvement. `MISSING` blocks because silently +losing a benchmark is how coverage rots. + +Exit status is 0 for no regression, 1 for a blocking verdict, 2 for a harness +error — including a build failure on either side, since half an A/B would read +as a passing gate. + +#### Confirming a reported regression + +A verdict is a reason to look, not a conclusion. Re-run with more trials first; +if it holds, isolate it with a knob rather than a rebuild of the whole branch. +The `silu_mul T=2048` row above is a worked example — it survived `--trials 9`, +and one `LARGE_GRF` A/B on the head tree alone explained it: + +```bash +make -B -C kernels/activation all CONFIG=standalone LARGE_GRF=0 # 0.3294 ms +make -B -C kernels/activation all CONFIG=standalone LARGE_GRF=1 # 0.3426 ms (+4.0%) +``` + +`LARGE_GRF ?= auto` turns on 256-GRF mode for every Battlemage kernel. The +matrix kernels need it — that is where those `-95%` rows come from — but a +bandwidth-bound kernel only loses occupancy, and the cache-resident `T=2048` +shape pays 4% for it while DRAM-bound `T=8192` pays 0.4%. + +### Options + +| Flag | Env | Default | Effect | +|:--|:--|:--|:--| +| `--base REF` | `SK_PERF_BASE` | `origin/main` | What to take the merge-base against | +| `--trials N` | `SK_PERF_TRIALS` | 5 | Rounds per side. Below 3 the spread cannot be estimated and the gate warns. | +| `--all` | — | off | The whole single-GPU set instead of the fast subset | +| `--multi` | — | off | Multi-GPU cases only; with `--all`, added to the single-GPU set | +| `--devices N` | `SK_NUM_DEVICES` | detected | GPUs to select against. Detection prefers `ZE_AFFINITY_MASK`, then `sycl-ls`. | +| `--list` | — | off | Print the selection with skip reasons and exit; builds nothing | +| `--iters N` | `SK_ITERS` | 50 | Timed iterations inside each harness invocation | +| — | `SK_WARMUP` | 5 | Warmup iterations before timing starts | +| `--threshold F` | — | 0.03 | Smallest effect reportable, as a fraction. The per-case noise floor overrides it upward. | +| `--min-ms MS` | — | 0.05 | Below this a case is a latency probe and is not judged | +| `--error-factor X` | — | 2.0 | Residual growth that earns a `NUMERICS` flag | +| `--fail-on-noisy` | — | off | Treat inconclusive cases as failures | +| `--fail-on-numerics` | — | off | Treat a grown residual as a failure | +| `--json PATH` | — | — | Write the full verdict, including every threshold, as JSON | +| `--keep` | — | off | Leave the base worktree in place for inspection | +| — | `SK_TIMEOUT` | `300s` | Per-binary wall clock. A collective that cannot reach its peers busy-waits forever; this records it as a failure instead of hanging the gate. | +| — | `GPU_TARGET` | detected | AOT target, pinned identically on both sides | +| — | `SK_BENCH_RESULTS` | `benchmarks/results` | Where the two logs land | + +### Why `--trials` matters more than `--threshold` + +The same `silu_mul` effect reads differently depending only on how many rounds +were run: + +| Trials | delta | noise | Verdict | +|--:|--:|--:|:--| +| 3 | +3.1% | 4.4% | `NOISY` — does not block | +| 5 | +3.0% | 1.3% | `REGRESSION` — blocks | + +Nothing about the code changed between those two runs; the 3-trial run simply +happened to catch a slow round, and the gate then refused to call an effect +smaller than the spread it had just observed. That is the intended behaviour: +lowering `--threshold` would not have helped, because the noise floor, not the +fixed floor, was binding. More rounds is the only real fix. + +### Why it measures both sides instead of storing a baseline + +The same binary varies run to run by more than the effects worth catching. On +an Arc Pro B70, `bf16_gemm` 4096³ spans 0.874–0.994 ms — 13.7% — across nine +trials of one build, while `gqa_forward` spans 1.3%. A single committed +threshold is therefore simultaneously too loose for attention and too tight for +GEMM, and a stored number also drifts with the driver, the compiler and the +ambient temperature. Measuring both sides on one machine minutes apart cancels +all of that. The cost is building twice. + +Three rules keep the verdict honest: + +- **peak@N per side.** Interference is one-sided — it only ever makes a kernel + slower — so the fastest trial is the best estimator, matching the convention + used everywhere else in this directory. +- **No case is called by less than the noise it just showed.** The threshold is + `max(--threshold, observed spread)`. +- **The median must agree in sign with the minimum.** One lucky fast base trial + is not a regression. + +### Troubleshooting + +| Symptom | Cause and fix | +|:--|:--| +| `cannot resolve a merge-base` | Shallow clone, or the base branch was never fetched. `git fetch --unshallow`, or `--base `. | +| `working tree is identical to the merge-base` | Nothing to compare; exits 0. Commit or edit something first. | +| `BUILD FAILED` on the base side only | The base predates something the build now needs. Pick a newer `--base`. | +| `BUILD FAILED` on `decode` | oneMKL headers missing. Use the default scope, which excludes it. | +| `icpx not found` | `env.src` did not find a oneAPI install. Source your own environment first. | +| Everything is `NOISY` | Machine is busy, or `--trials` is too low. Close other GPU work and raise it. | +| Every case shows a huge delta | The two sides built for different architectures. Pin `GPU_TARGET` explicitly. | +| A worktree is left in `/tmp` | The run was killed. `git worktree prune` cleans the bookkeeping. | +| `no kernel in scope ... runs on N GPU(s)` | The scope needs more devices than are visible; exits 2. `--list` shows which and why. Drop `--multi`, or raise `--devices`. | +| `!` printed after a kernel name | That trial hit `SK_TIMEOUT`. Grep the log for `SK_TRIAL_TIMEOUT`. On a multi-card host this is the expected collective hang described above. | +| A multi-GPU case shows `MISSING` | Every trial timed out or exited non-zero, so there is nothing to compare. The log holds the harness's own stderr. | +| `skip ...: needs exactly 12 GPU(s)` | Fixed-topology binary; `NUM_DEVICES` is compiled in. Run it from `distributed.sh` on a host with that count. | + +The AOT target is pinned explicitly for both sides because `common.mk` only +learned to detect the local GPU recently; an older base checkout would otherwise +default to `intel_gpu_pvc` and the comparison would be across two +architectures. + +IGC spilling the mainloop is the single biggest perf factor on this hardware and +is only visible at compile time, so the script greps both build logs and prints +e.g. `note: base/gemm has 3 spilling kernel(s)`. A `-95%` row with that note +next to it is a spill being fixed, not an algorithmic win. + +### Tracking performance over time + +`perf_check.sh` compares two commits. To track absolute numbers across weeks, +capture a run and diff later ones against it: + +```bash +python3 benchmarks/tools/parse_baseline.py RUN.log --csv out.csv --json out.json +python3 benchmarks/tools/parse_baseline.py NEW.log --compare out.json +``` + +Those numbers are only comparable on the same host, driver and toolchain — which +is exactly why the PR gate does not use them. + +### Pieces + +| Path | Role | +|:--|:--| +| `suites/perf_check.sh` | build both sides, interleave the runs | +| `suites/kernels.tsv` | the kernel list: path, device range, tier, make target | +| `tools/select_kernels.py` | manifest → the subset this host can run | +| `tools/perf_gate.py` | paired statistics, verdict, exit code | +| `tools/parse_baseline.py` | log → CSV/JSON; also `--compare` against a stored run | +| `tools/test_perf_gate.py` | verdict-logic tests; no GPU required | +| `tools/test_select_kernels.py` | selection and manifest tests; no GPU required | + +`select_kernels.py` is usable on its own, which is how another runner or a +scheduler asks what a host supports without parsing the manifest itself: + +```bash +python3 benchmarks/tools/select_kernels.py --tier multi --devices 12 +python3 benchmarks/tools/select_kernels.py --tier fast --tier full --devices 1 \ + --what targets --quiet # ' ' pairs, for a build loop +``` + +It exits 0 with a selection, 3 when the selection is empty, and 1 on a +malformed manifest. + +`perf_gate.py` can also be re-run on its own against two logs you already have, +which is how you re-judge an old run at a different threshold without +remeasuring: + +```bash +python3 benchmarks/tools/perf_gate.py \ + --base results/perf_check_*.base.log \ + --head results/perf_check_*.head.log \ + --threshold 0.05 +``` + +### CI + +Wiring this into CI needs a self-hosted runner with an Intel GPU; the script +already writes a `GITHUB_STEP_SUMMARY` table when that variable is set, so the +job body is a checkout with `fetch-depth: 0` plus one invocation. A single-GPU +runner uses the default scope; a multi-tile runner adds `--multi`. + +The no-GPU parts run in the existing lint job on every pull request: +`test_perf_gate.py`, which synthesises its own logs, and +`test_select_kernels.py`, which also checks the committed manifest — including +that every kernel directory it names still has a `Makefile`, so a renamed +kernel family fails the PR instead of silently shrinking the gate. + ## Distributed Suites Run a suite after an external platform adapter has selected the devices: diff --git a/benchmarks/suites/kernels.tsv b/benchmarks/suites/kernels.tsv new file mode 100644 index 0000000..53d804b --- /dev/null +++ b/benchmarks/suites/kernels.tsv @@ -0,0 +1,56 @@ +# Which kernels exist, what they need, and how to build them. +# +# One source of truth for every benchmark runner, so the binary list is not +# duplicated across perf_check.sh, baseline.sh and distributed.sh. +# benchmarks/tools/select_kernels.py reads this and applies the filtering. +# +# Columns (whitespace-separated): +# path / as produced by the dir's Makefile +# min_dev fewest GPUs the binary will run on +# max_dev most GPUs it will use; 0 means no upper bound, and a value +# equal to min_dev means it accepts that count and no other +# tier fast = single-GPU, cheap and quiet; the PR gate default +# full = single-GPU, the rest +# multi = needs more than one GPU +# make_target target that produces it, as `make -C kernels/ ` +# +# A binary runs when min_dev <= visible GPUs. When max_dev == min_dev it also +# refuses any other count, because its NUM_DEVICES is compiled in with no +# environment override. Single-GPU kernels use max_dev 0: they run on a 12-tile +# host perfectly well, they just use one device. +# +# path min_dev max_dev tier make_target + +# -- single GPU, PR gate default ------------------------------------------ +# Covers the matrix, reduction and elementwise paths without the latency +# probes or the oneMKL dependency that make the rest flaky to build. +gemm/bf16_gemm_4096x4096x4096 1 0 fast all +gemm/qkv_gemm 1 0 fast all +attention/gqa_forward 1 0 fast all +norm/rmsnorm 1 0 fast all +activation/silu_mul 1 0 fast all +# -- single GPU, the rest ------------------------------------------------- +attention/gqa_backward 1 0 full all +norm/layernorm 1 0 full all +rotary/rotary 1 0 full all +rotary/rotary_qk 1 0 full all +rotary/rope_kvwrite 1 0 full all +# decode links oneMKL for its GEMV fallback; needs MKLROOT on CPATH. +decode/decode 1 0 full all +# -- multi GPU, scales with the device count ------------------------------ +# These three read SK_NUM_DEVICES, so they run on any count from 2 up. +collective/all_reduce 2 12 multi standalone +collective/all_gather 2 12 multi standalone +collective/reduce_scatter 2 12 multi standalone +# These three take min(visible, NUM_DEVICES_MAX), so they adapt on their own. +context_attention/ring_attention 2 6 multi all +context_attention/ring_attention_overlap 2 12 multi all +context_attention/all_gather_attention 2 6 multi all + +# -- multi GPU, fixed topology -------------------------------------------- +# NUM_DEVICES is compiled in with no environment override, so these refuse to +# run on anything but the exact count. benchmarks/suites/distributed.sh drives +# them, since they also need an externally supplied ZE_AFFINITY_MASK. +collective/gemm_rs 6 6 multi all +collective/all_to_all 12 12 multi standalone +collective/ag_gemm 12 12 multi all diff --git a/benchmarks/suites/perf_check.sh b/benchmarks/suites/perf_check.sh new file mode 100755 index 0000000..a1caada --- /dev/null +++ b/benchmarks/suites/perf_check.sh @@ -0,0 +1,356 @@ +#!/usr/bin/env bash +# Local performance check for a pull request: A/B the working tree against its +# merge-base on this machine, and fail if a kernel got slower. +# +# benchmarks/suites/perf_check.sh # fast subset, 5 trials/side +# benchmarks/suites/perf_check.sh --all # every single-GPU case +# benchmarks/suites/perf_check.sh --multi # the multi-GPU cases only +# benchmarks/suites/perf_check.sh --all --multi # everything this host can run +# benchmarks/suites/perf_check.sh --list # what would run here, no build +# benchmarks/suites/perf_check.sh --trials 9 # tighter noise floor +# benchmarks/suites/perf_check.sh --base v0.9.0 # against a tag, not main +# +# Which kernels exist and how many GPUs each needs is declared once, in +# benchmarks/suites/kernels.tsv. The visible device count is detected and +# anything that does not fit is skipped with a reason, so the same command is +# correct on a single-GPU workstation and on a 12-tile host. +# +# It checks out the merge-base into a throwaway git worktree, builds both trees, +# then runs each binary base-then-head back to back, repeating for --trials +# rounds. benchmarks/tools/perf_gate.py turns the two logs into a verdict. +# +# Measuring both sides here and now is what makes the result trustworthy: the +# same binary varies run to run by more than the effects worth catching (see the +# rationale in perf_gate.py), so a number stored in the repo cannot be compared +# against. The cost is building twice. +# +# The head side is the *working tree*, uncommitted changes included, which is +# what you want while iterating. The base side is always a clean checkout. +# +# Controls: +# SK_PERF_BASE=origin/main base ref (--base wins) +# SK_PERF_TRIALS=5 rounds per side (--trials wins) +# SK_BENCH_RESULTS=DIR output directory (default benchmarks/results) +# SK_ITERS=50 SK_WARMUP=5 harness timing arguments +# GPU_TARGET=... AOT target; detected when unset +# SK_NUM_DEVICES=N visible GPUs (--devices wins); detected when unset +# SK_TIMEOUT=300s per-binary wall clock; a stuck collective is +# recorded as a failure instead of hanging the run +# +# Exit status: 0 no regression, 1 regression or correctness break, 2 harness error. +set -uo pipefail + +SK_ROOT=$(git -C "$(dirname "${BASH_SOURCE[0]}")" rev-parse --show-toplevel 2>/dev/null) || { + echo "perf_check: not inside a git checkout" >&2; exit 2 +} + +BASE_REF=${SK_PERF_BASE:-origin/main} +TRIALS=${SK_PERF_TRIALS:-5} +ITERS=${SK_ITERS:-50} +WARMUP=${SK_WARMUP:-5} +TIMEOUT=${SK_TIMEOUT:-300s} +RESULTS_DIR=${SK_BENCH_RESULTS:-$SK_ROOT/benchmarks/results} +KEEP=0 +GATE_ARGS=() +DEVICES=${SK_NUM_DEVICES:-} +# A gate that nobody trusts gets ignored, so the default tier is the cases that +# are both cheap and quiet: no latency probes, no decode (which needs oneMKL and +# is the usual build failure). --all adds the rest of the single-GPU set, +# --multi the ones that need more than one device. +WANT_SINGLE_FULL=0 +WANT_MULTI=0 +ONLY_MULTI=0 +LIST_ONLY=0 + +usage() { sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//'; } + +while [[ $# -gt 0 ]]; do + case "$1" in + --base) BASE_REF=$2; shift 2 ;; + --trials) TRIALS=$2; shift 2 ;; + --all) WANT_SINGLE_FULL=1; shift ;; + # --multi on its own means "only the multi-GPU cases"; combined with + # --all it widens the run instead of narrowing it. + --multi) WANT_MULTI=1; ONLY_MULTI=1; shift ;; + --devices) DEVICES=$2; shift 2 ;; + --list) LIST_ONLY=1; shift ;; + --keep) KEEP=1; shift ;; + --iters) ITERS=$2; shift 2 ;; + # Everything perf_gate.py understands: --threshold, --min-ms, + # --fail-on-noisy, --fail-on-numerics, --json. + --threshold|--min-ms|--error-factor|--json) + GATE_ARGS+=("$1" "$2"); shift 2 ;; + --fail-on-noisy|--fail-on-numerics) + GATE_ARGS+=("$1"); shift ;; + -h|--help) usage; exit 0 ;; + *) echo "perf_check: unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[[ "$TRIALS" =~ ^[1-9][0-9]*$ ]] || { echo "perf_check: --trials must be a positive integer" >&2; exit 2; } +if [[ $TRIALS -lt 3 ]]; then + echo "perf_check: warning: --trials $TRIALS cannot estimate run-to-run spread;" \ + "the gate will fall back to its fixed threshold" >&2 +fi + +[[ $WANT_SINGLE_FULL == 1 && $WANT_MULTI == 1 ]] && ONLY_MULTI=0 + +TIER_ARGS=() +if [[ $ONLY_MULTI == 1 ]]; then + TIER_ARGS=(--tier multi) + SCOPE=multi +else + TIER_ARGS=(--tier fast) + SCOPE=fast + if [[ $WANT_SINGLE_FULL == 1 ]]; then + TIER_ARGS+=(--tier full); SCOPE=single-all + fi + if [[ $WANT_MULTI == 1 ]]; then + TIER_ARGS+=(--tier multi); SCOPE="$SCOPE+multi" + fi +fi + +# --- resolve the base commit ---------------------------------------------- +if ! BASE_SHA=$(git -C "$SK_ROOT" merge-base HEAD "$BASE_REF" 2>/dev/null); then + # A fresh clone has no origin/main; fall back to a local branch of that name. + FALLBACK=${BASE_REF##*/} + if ! BASE_SHA=$(git -C "$SK_ROOT" merge-base HEAD "$FALLBACK" 2>/dev/null); then + echo "perf_check: cannot resolve a merge-base with '$BASE_REF' or '$FALLBACK'." >&2 + echo " Fetch the base branch, or pass --base ." >&2 + exit 2 + fi + echo "perf_check: '$BASE_REF' not found, using '$FALLBACK'" >&2 + BASE_REF=$FALLBACK +fi + +HEAD_SHA=$(git -C "$SK_ROOT" rev-parse HEAD) +DIRTY=$(git -C "$SK_ROOT" status --porcelain --untracked-files=no) +if [[ "$BASE_SHA" == "$HEAD_SHA" && -z "$DIRTY" ]]; then + echo "perf_check: working tree is identical to the merge-base ($BASE_REF); nothing to check." + exit 0 +fi + +# --- toolchain ------------------------------------------------------------ +# Source the *current* tree's env.src for both sides. The base checkout may +# predate a change to it, and in any case the A/B is only meaningful if both +# sides see the same compiler. +# shellcheck disable=SC1091 +source "$SK_ROOT/env.src" +command -v icpx >/dev/null 2>&1 || { + echo "perf_check: icpx not found; source an oneAPI environment first" >&2; exit 2 +} +# decode includes ; setvars does not put MKL on CPATH. +[[ -n "${MKLROOT:-}" ]] && export CPATH="$MKLROOT/include${CPATH:+:$CPATH}" + +# The AOT target must be pinned explicitly. common.mk only learned to detect the +# local GPU recently, so an older base checkout would silently default to +# intel_gpu_pvc and the comparison would be between two architectures. +if [[ -z "${GPU_TARGET:-}" ]]; then + if [[ -x "$SK_ROOT/kernels/detect_gpu_target.sh" ]]; then + GPU_TARGET=$(sh "$SK_ROOT/kernels/detect_gpu_target.sh" icpx) + fi + if [[ -z "${GPU_TARGET:-}" ]]; then + probe=$(mktemp -d) + printf '#include \nint main(){return 0;}\n' > "$probe/p.cpp" + for c in intel_gpu_bmg_g31 intel_gpu_bmg_g21 intel_gpu_pvc spir64; do + if icpx -fsycl -fsycl-targets="$c" -std=c++20 -o "$probe/p" "$probe/p.cpp" >/dev/null 2>&1; then + GPU_TARGET=$c; break + fi + done + rm -rf "$probe" + fi + GPU_TARGET=${GPU_TARGET:-spir64} +fi +export GPU_TARGET + +# --- device count and kernel selection ------------------------------------ +# ZE_AFFINITY_MASK wins when a platform adapter set it: that is the count the +# kernels will actually see, regardless of what the machine holds. +if [[ -z "$DEVICES" ]]; then + if [[ -n "${ZE_AFFINITY_MASK:-}" ]]; then + DEVICES=$(awk -F, '{print NF}' <<< "$ZE_AFFINITY_MASK") + else + DEVICES=$(sycl-ls 2>/dev/null | grep -c "level_zero.*gpu") || DEVICES=0 + fi + [[ ${DEVICES:-0} -lt 1 ]] && DEVICES=1 +fi +[[ "$DEVICES" =~ ^[1-9][0-9]*$ ]] || { + echo "perf_check: --devices must be a positive integer, got '$DEVICES'" >&2; exit 2 +} + +SELECT="$SK_ROOT/benchmarks/tools/select_kernels.py" +# Skips are reported, not hidden: a run that silently measured less than the +# operator believes is worse than one that refused. +BINS=$(python3 "$SELECT" "${TIER_ARGS[@]}" --devices "$DEVICES") +select_rc=$? +if [[ $select_rc -eq 3 ]]; then + echo "perf_check: no kernel in scope '$SCOPE' runs on $DEVICES GPU(s)." >&2 + echo " Try --all for the single-GPU set, or run on more devices." >&2 + exit 2 +elif [[ $select_rc -ne 0 ]]; then + echo "perf_check: kernel selection failed" >&2; exit 2 +fi + +# dir -> make target, widest wins when a dir contributes several binaries. +TARGETS=$(python3 "$SELECT" "${TIER_ARGS[@]}" --devices "$DEVICES" --what targets --quiet) +FAMILIES=$(cut -d' ' -f1 <<< "$TARGETS") +# Needed at run time to decide whether a binary takes timing arguments. +MULTI_BINS=$(python3 "$SELECT" --tier multi --devices "$DEVICES" --quiet | tr '\n' ' ') + +# The scalable collectives read this; the fixed-topology ones ignore it. +export SK_NUM_DEVICES=$DEVICES + +if [[ $LIST_ONLY == 1 ]]; then + # Skips were already reported on stderr by the selection above. + echo "scope $SCOPE on $DEVICES GPU(s), target $GPU_TARGET" + python3 "$SELECT" "${TIER_ARGS[@]}" --devices "$DEVICES" --what table --quiet + exit 0 +fi + +mkdir -p "$RESULTS_DIR" +RUN_ID="perf_check_$(date -u +%Y%m%dT%H%M%SZ)" +BASE_LOG="$RESULTS_DIR/${RUN_ID}.base.log" +HEAD_LOG="$RESULTS_DIR/${RUN_ID}.head.log" +BASE_TREE=$(mktemp -d "${TMPDIR:-/tmp}/sk-perf-base.XXXXXX") + +cleanup() { + if [[ $KEEP == 1 ]]; then + echo "perf_check: keeping base worktree at $BASE_TREE" >&2 + return + fi + git -C "$SK_ROOT" worktree remove --force "$BASE_TREE" >/dev/null 2>&1 \ + || rm -rf "$BASE_TREE" +} +trap cleanup EXIT + +echo "== perf check" +echo " base $(git -C "$SK_ROOT" log --oneline -1 "$BASE_SHA")" +echo " head $(git -C "$SK_ROOT" log --oneline -1 HEAD)${DIRTY:+ + uncommitted changes}" +echo " target $GPU_TARGET" +echo " devices $DEVICES visible$([[ -n "${ZE_AFFINITY_MASK:-}" ]] && echo " (ZE_AFFINITY_MASK=$ZE_AFFINITY_MASK)")" +echo " scope $SCOPE ($(echo "$BINS" | wc -w) binaries, $(echo "$FAMILIES" | wc -w) families)" +echo " trials $TRIALS per side, iters=$ITERS warmup=$WARMUP" +echo + +rmdir "$BASE_TREE" # git worktree add wants to create the directory itself +# git narrates the checkout on stderr; the base commit is already in the header. +git -C "$SK_ROOT" worktree add --detach "$BASE_TREE" "$BASE_SHA" >/dev/null 2>&1 || { + echo "perf_check: could not create a worktree for $BASE_SHA" >&2; exit 2 +} + +# --- build ---------------------------------------------------------------- +# One family per background job, as baseline.sh does. A build failure on either +# side is fatal: half an A/B is worse than none, because the missing side reads +# as a passing gate. +build_tree() { + local tree=$1 side=$2 rc=0 + declare -A pid + while read -r fam tgt; do + [[ -z "$fam" ]] && continue + mkdir -p "$tree/kernels/$fam/build" + make -B -C "$tree/kernels/$fam" "$tgt" CONFIG=standalone GPU_TARGET="$GPU_TARGET" \ + > "$tree/kernels/$fam/build/perf_check_build.log" 2>&1 & + pid[$fam]=$! + done <<< "$TARGETS" + for fam in $FAMILIES; do + if wait "${pid[$fam]}"; then + echo " $side/$fam: OK" + else + echo " $side/$fam: BUILD FAILED" + tail -n 15 "$tree/kernels/$fam/build/perf_check_build.log" | sed 's/^/ | /' + rc=1 + fi + done + return $rc +} + +echo "== build base ($BASE_SHA)" +build_tree "$BASE_TREE" base || exit 2 +echo "== build head" +build_tree "$SK_ROOT" head || exit 2 + +# IGC spilling the mainloop is the single biggest perf factor on this hardware +# and it is only visible at compile time, so surface it rather than leaving it +# in a log nobody opens. +for fam in $FAMILIES; do + for side in base head; do + tree=$BASE_TREE; [[ $side == head ]] && tree=$SK_ROOT + n=$(grep -c "spilled around" "$tree/kernels/$fam/build/perf_check_build.log" 2>/dev/null) + [[ ${n:-0} -gt 0 ]] && echo " note: $side/$fam has $n spilling kernel(s)" + done +done + +# --- run ------------------------------------------------------------------ +write_header() { + local path=$1 side=$2 commit=$3 + cat > "$path" </dev/null | head -1) +SK_PERF_SIDE=$side +SK_PERF_COMMIT=$commit +EOF +} + +write_header "$BASE_LOG" base "$(git -C "$SK_ROOT" rev-parse --short "$BASE_SHA")" +write_header "$HEAD_LOG" head "$(git -C "$SK_ROOT" rev-parse --short HEAD)${DIRTY:+-dirty}" + +echo +echo "== run" +for ((t = 1; t <= TRIALS; t++)); do + printf ' trial %d/%d:' "$t" "$TRIALS" + for b in $BINS; do + label=${b##*/} + printf ' %s' "$label" + # base then head back to back, so thermal and ambient drift lands on + # both sides of each comparison equally. + for side in base head; do + if [[ $side == base ]]; then tree=$BASE_TREE; log=$BASE_LOG + else tree=$SK_ROOT; log=$HEAD_LOG; fi + exe="$tree/kernels/${b%%/*}/build/${b#*/}" + echo "SK_TRIAL label=$label trial=$t" >> "$log" + if [[ ! -x "$exe" ]]; then + echo "SK_CASE_SKIPPED label=$label reason=not_built" >> "$log" + continue + fi + # The distributed harnesses size themselves from the topology and + # take no timing arguments; the single-GPU ones want iters/warmup. + run_args=("$ITERS" "$WARMUP") + [[ " $MULTI_BINS " == *" $b "* ]] && run_args=() + # A collective that cannot reach its peers busy-waits forever, so + # every binary is bounded. Without this one wedged kernel hangs the + # whole gate and no verdict is ever produced. + # Launched as a job and reaped with wait: bash prints "Aborted + # (core dumped)" for a foreground child killed by a signal, which + # would land in the middle of the progress line. A crashing kernel + # is expected input here, not an emergency -- its stderr is already + # captured in the log. + timeout --kill-after=10s "$TIMEOUT" "$exe" \ + "${run_args[@]+"${run_args[@]}"}" >> "$log" 2>&1 & + wait $! 2>/dev/null + rc=$? + if [[ $rc -eq 124 || $rc -eq 137 ]]; then + echo "SK_TRIAL_TIMEOUT label=$label trial=$t after=$TIMEOUT" >> "$log" + printf '!' + elif [[ $rc -ne 0 ]]; then + echo "SK_TRIAL_NONZERO_EXIT label=$label trial=$t rc=$rc" >> "$log" + fi + done + done + printf '\n' +done + +# --- verdict -------------------------------------------------------------- +python3 "$SK_ROOT/benchmarks/tools/perf_gate.py" \ + --base "$BASE_LOG" --head "$HEAD_LOG" "${GATE_ARGS[@]+"${GATE_ARGS[@]}"}" +rc=$? + +echo +echo " logs: $BASE_LOG" +echo " $HEAD_LOG" +exit $rc diff --git a/benchmarks/tools/parse_baseline.py b/benchmarks/tools/parse_baseline.py new file mode 100755 index 0000000..d97db9d --- /dev/null +++ b/benchmarks/tools/parse_baseline.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Turn a SyclKittens benchmark log into a CSV/JSON baseline, or diff against one. + + python3 parse_baseline.py RUN.log --csv out.csv --json out.json + python3 parse_baseline.py NEW.log --compare baseline.json + +Per the benchmarks/README convention a case keeps its fastest correct trial +(peak@N), so repeated trials reduce to one number per case. + +`parse_samples` keeps every trial instead of reducing; `perf_gate.py` needs the +full set to measure run-to-run spread. +""" + +import argparse +import csv +import json +import re +import sys +from pathlib import Path + +# "