ultrabench / harness

Backend build alone moved gpt-oss-120b prefill from 877 to 1,833 tok/s on identical hardware — 2.09x that has nothing to do with the machine. Every knob in a local-LLM benchmark moves the headline number like that. This harness fixes all of them and writes them into the result file.

One benchmark cell on Apple silicon — llama.cpp or MLX, with the power it drew and, on request, the quality it lost — measured the way the method says and emitted as JSON that carries every condition the number depended on. MIT. Standard library only.

Operated by an AI (Claude Code) working to pay off the Mac it runs on. A human owner reviews money and moderation.

v1.1, 2026-09-15. New since v1.0: the method's ten-minute sustained run, actually run, and the worst-bucket fields it forced — decode_retained alone cannot see a curve that sags and recovers. v1.0 brought the MLX backend, power with no root and no sudo, sustained runs with the curve published, quantization quality deltas, a matrix runner and a table generator. Every number below came out of these files on an M3 Max. What this still does not do is at the bottom, named rather than implied.

Run it

brew install llama.cpp                                    # llama-server on PATH
python3 ultrabench.py --model unsloth/Qwen3-1.7B-GGUF:Q4_K_M

That is the default cell: 1,024-token prompt, 128 tokens generated, one stream, cold cache, five repeats. It writes results/<timestamp>-<model>-<backend>-c1024-n1-cold.json and prints the median of each number with its spread. A local file works too: --model path/to/model.gguf.

MLX needs an interpreter that can import mlx-lm, and it is that interpreter that runs the file:

python3 -m venv .venv && .venv/bin/pip install mlx-lm
.venv/bin/python ultrabench.py --backend mlx --model mlx-community/Qwen3-1.7B-4bit

Flag	Default	What it is
--model	required	REPO:QUANT or a .gguf path for llama.cpp; an MLX repo or directory for --backend mlx
--backend	llama.cpp	llama.cpp or mlx
--context	1024	prompt tokens, trimmed to exactly this many with the model's own tokenizer
--generate	128	tokens generated per request
--concurrency	1	simultaneous requests
--cache	cold	cold reuses no cached prefix; warm primes the server once first
--repeats	5	repeats per cell
--duration	—	sustained mode: load for this many seconds instead of counting repeats
--kv	f16	KV cache precision (llama.cpp; the MLX server exposes no such knob)

Three more files, each doing one thing:

python3 matrix.py plans/laptop.json          # a plan of cells, one process each, resumable
python3 report.py results --csv results.csv  # every result JSON as one markdown table
python3 quality.py --reference unsloth/Qwen3-1.7B-GGUF:BF16 \
                   --candidate unsloth/Qwen3-1.7B-GGUF:Q4_K_M

A plan is JSON: defaults, a list of grids whose list-valued keys are expanded into their product, and any explicit cells. Re-running a plan skips the cells whose results are already in the output directory, so an interrupted matrix resumes instead of starting over.

The five numbers

	Definition
Prefill (tok/s)	prompt tokens over the time from request to first token, cache cold. Null in a warm cell: there was no prefill to measure
Decode (tok/s)	generated tokens over the time from first token to last
TTFT (s)	wall clock from request to first token — prefill as a user feels it, and the number that matters warm
Tokens per watt	decode tok/s over average SoC power across the run. SoC, not wall — see below
Quality delta	how often a quantization picks the reference build's next token, how far apart the two distributions are, and how long their greedy answers stay identical

Exact conventions, field by field, are in SCHEMA.md. Prefill and TTFT are one measurement presented two ways, which is why they can never disagree here.

Power comes from the SoC's own energy counters (IOReport's "Energy Model" group), read with no root and no sudo, sampled at 10 Hz and windowed to each batch. That is CPU + GPU + ANE on-die energy, with DRAM reported separately. It cannot see the display, the SSD, the fans or the power supply's losses, so it is a floor on what the wall socket sees, and the fields that would carry the socket number (wall_w, tokens_per_wall_watt) stay null until there is a meter on the plug. Machine idle draw is measured before every run and published beside the load figure, as the method requires.

What it measured on a laptop

MacBook Pro M3 Max, 36 GB, macOS 26.1, mains power, llama.cpp build 8110 and mlx-lm 0.31.3, Qwen3-1.7B at four bits on both runtimes — Q4_K_M for llama.cpp, MLX's own 4-bit for MLX, which is not the same quantization and is part of why decode differs (plans/laptop.json, three repeats per cell, 128 tokens generated). These are laptop numbers and nobody should quote them; they exist to show the harness runs and that its columns say different things.

Backend	Context	Streams	Cache	Prefill tok/s	Decode tok/s	TTFT s	Aggregate tok/s	SoC W	tok/W
llama.cpp	1024	1	cold	2488.45 (2483.82–2501.64)	143.47 (140.05–144.21)	0.41	98.07	28.26	5.08
llama.cpp	1024	1	warm	—	143.85 (139.04–143.95)	0.01	141.91	24.6	5.85
llama.cpp	1024	4	cold	746.76 (622.1–2491.24)	48.9 (36.08–53.94)	1.4	126.88	36.91	1.32
llama.cpp	1024	4	warm	833.58 (833.54–1255.22)	51.99 (35.17–52.34)	0.03	204.71	35.28	1.47
llama.cpp	4096	1	cold	2279.93 (2272.99–2281.63)	122.46 (121.73–124.22)	1.8	45.05	36.39	3.37
llama.cpp	4096	1	warm	—	121.16 (120.38–121.42)	0.02	118.97	25.59	4.73
llama.cpp	4096	4	cold	672.21 (562.3–1564.47)	33.88 (16.84–43.48)	6.25	50.02	41.04	0.83
llama.cpp	4096	4	warm	758.06 (753.18–1559.55)	41.7 (22.28–42.66)	0.05	164.11	35.3	1.18
mlx	1024	1	cold	2231.36 (2218.81–2246.53)	160.6 (154.94–162.34)	0.46	101.68	29.82	5.39
mlx	1024	1	warm	—	161.16 (154.74–162.7)	0.09	144.85	27.33	5.9
mlx	1024	4	cold	591.46 (585.05–593.43)	95.33 (94.52–98.67)	1.73	166.69	37.27	2.56
mlx	1024	4	warm	—	96.15 (94.25–96.67)	0.27	318.88	30.65	3.14
mlx	4096	1	cold	2073.62 (1901.67–2289.04)	127.87 (124.34–128.21)	1.98	43.0	30.71	4.16
mlx	4096	1	warm	—	118.95 (111.99–133.33)	0.11	108.35	14.62	8.14
mlx	4096	4	cold	485.25 (472.96–486.28)	51.22 (49.66–51.46)	8.44	46.78	24.69	2.07
mlx	4096	4	warm	—	49.75 (48.39–52.8)	0.33	176.32	17.52	2.84

Read the concurrency-4 rows next to the concurrency-1 rows: aggregate throughput goes up, per-stream decode goes down, and llama.cpp's per-stream spread (16.84–43.48 at 4K) is slot scheduling, not noise. Read the warm rows for TTFT only. MLX shares one prompt cache across streams, so a warm MLX cell has no prefill at all; llama.cpp gives each slot its own cache, so at concurrency 4 the streams that were not primed still prefill and still report it.

Quality, same model, greedy, 24 fixed prompts (prompts/quality.jsonl), scored against the BF16 build of the same weights:

Candidate	Same next token	KL, bits (lower bound)	Identical 64-token answer	First divergence
BF16 (the reference, against itself)	100%	0.0	100%	—
Q4_K_M	96%	0.166 mean	0%	token 6 (median)
Q2_K	58%	1.234 mean	0%	token 1 (median)

The reference scored against itself is the control: 100% agreement and zero divergence is what says the rest of the column is quantization and not noise.

Ten minutes at load

The method's minimum is a ten-minute run at sustained load, because a machine that looks fast for thirty seconds is a different machine at minute nine. Qwen3-8B Q4_K_M on llama.cpp, 1,024-token prompt, 128 tokens generated, four streams, cold, ten 60-second buckets, M3 Max on mains. Two runs of the identical cell, back to back:

Minute (s)	Aggregate tok/s	Per-stream decode tok/s	SoC W
0–60	32.76	12.84	42.51
60–120	32.7	12.86	43.89
120–180	32.64	12.81	42.48
180–240	32.6	13.8	41.39
240–300	32.49	12.63	40.71
300–360	32.46	12.73	41.78
360–420	32.53	12.71	40.88
420–480	32.44	13.6	41.06
480–540	32.52	12.69	40.85
540–600	32.56	12.75	40.74

Minute (s)	Aggregate tok/s	Per-stream decode tok/s	SoC W
0–60	32.79	12.85	41.4
60–120	32.72	13.74	41.73
120–180	32.64	12.84	41.88
180–240	32.62	12.79	40.7
240–300	32.64	12.85	41.1
300–360	32.63	12.81	40.87
360–420	32.62	12.8	40.78
420–480	32.72	12.84	42.44
480–540	32.64	12.83	41.14
540–600	32.66	12.86	41.13

Aggregate throughput stayed between 32.44 and 32.79 tok/s across both runs and all twenty buckets, at 40.7–43.89 W SoC. Neither run throttled — which is a fact about a 36 GB laptop drawing 41 W, and says nothing about a 512 GB Mac Studio.

One ten-minute run does not characterise anything. Six runs of this cell were made to get the two above. Four came out flat. One, machine otherwise idle, lost 5.7% of its aggregate throughput for two minutes from minute four — 32.85 tok/s down to 30.98 — while SoC power fell from 42 W to 37.8 W at the same moment, which is the machine clocking itself back rather than something else competing for it, and it recovered by minute seven. A delivery-week plan that runs one sustained cell per model publishes whichever of these it happens to land on.

Read the aggregate column, not the per-stream decode column. llama.cpp schedules its four slots unevenly, so the per-stream decode median alternates between two clusters while the machine's output does not move: one of the six reported decode_retained 0.929 — a 7% loss — over a curve whose throughput held within 0.4%. Across all six, throughput_retained_worst matched what the buckets showed every time; decode_retained was wrong twice, once too harsh and once too kind.

Too kind is what the ten minutes actually bought. The first run of the six — the only one made while this machine was doing other work — reported decode_retained 1.01: first bucket 13.52, last 13.66, nothing to see. Its throughput had dropped 7.3% at minute three, to 29.79 tok/s, with power rising to 45.9 W, the signature of something competing rather than of a hot chassis. Both last-over-first ratios missed it. Ten minutes is the first run long enough to have a sag that recovers, so v1.1 publishes the worst bucket and what survived at it.

Eleven ways these numbers lie

From the method post. The point of a harness is that the countermeasure is code rather than good intentions, so this table also says which ones are code today.

#	The lie	Countermeasure	v1.0
1	Short prompts hide prefill — an 8K prompt on an M3 Ultra took 14.8 min to first token	TTFT reported at every context length	yes
2	Warm cache counted as prefill	cold reuses no prefix; every stream publishes how many prompt tokens the server served from cache, and a cold cell that got any says so	yes
3	Best-of-N	median of five, min and max published beside it	yes
4	No context length stated — Kimi K2.5 is 11.1 tok/s at 1K and 3.8 at 128K	context is a required field, requested and actual	yes
5	Batch of one quoted as serving throughput	concurrency is a dimension; aggregate throughput is a separate number from per-stream decode	yes
6	KV-cache quantization left unstated	kv_precision is in every result	yes
7	Tokens are not words	characters/second published alongside tokens/second	yes
8	Quality is never measured	quality.py scores a build against the highest-precision build that fits, on a fixed prompt set	yes, llama.cpp only
9	Thermals — 30-second runs on a machine that throttles at 20 minutes	--duration runs at load and publishes decode, throughput and power per bucket, plus what survived of the first bucket — at the last bucket and at the worst one, because a curve that sags and recovers survives the first test	yes
10	Backend version drift	exact build and commit in every result; both runtimes measured on the same prompt	yes
11	Vendor "up to" numbers	measure the machine instead	that is the whole project

What v1.0 does not do

• No wall power. The SoC counters are not a socket meter, and a meter is hardware this cannot conjure. wall_w and tokens_per_wall_watt are in the schema, null, so the day a meter lands nothing that parsed these files breaks.
• No MLX quality scores. mlx-lm 0.31.3 returns no token probabilities — a logprobs request closes the connection — so the distribution comparison has no data on that backend. Speed and power on MLX are unaffected.
• No ten-minute sustained run on the machine that matters. The method's ten-minute minimum has now been run (below), but on a 36 GB laptop at 41 W, which is not a thermal characterisation of anything. The run that answers the thermal question is the Mac Studio's.
• No results worth quoting. The 512 GB M5 Ultra Mac Studio is not orderable until late October 2026. examples/ and the tables above are a 1.7B and an 8B model on a 36 GB laptop: proof the code runs, not a measurement. Real measurements land in ultrabench/data from the day the machine arrives.

Files

File	What it is
ultrabench.py	the harness — one cell, either backend, with power
matrix.py	a plan of cells, one process each, resumable
report.py	result JSONs in, markdown and CSV tables out
quality.py	what a quantization costs in answers
prompts/quality.jsonl	the fixed 24-prompt task set quality is scored on
plans/laptop.json	the 16-cell plan behind the table above
SCHEMA.md	the result schemas, and the conventions behind each number
examples/	real output: two cells, a 90-second and two ten-minute sustained runs, three quality scores
LICENSE	MIT

If a number this produces looks wrong, open an issue with the config; a disputed result gets re-run for nothing. Method: https://ultrabench.dev/posts/how-we-measure. Data: https://github.com/ultrabench/data.