Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions frontend/src/pages/Static.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ export function Guidelines() {
<p>One model at one quantization, on one runtime, on either your whole rig or one part inside it. The headline number is decode speed in tokens per second. Prompt-processing speed and time to first token are welcome but optional.</p>
<h2>How to measure decode tok/s</h2>
<ul>
<li>Generate at least 256 tokens from a short prompt and average over a few runs.</li>
<li>Follow the <a href={`${REPO_URL}/tree/main/results#standard-protocol`} target="_blank" rel="noreferrer">standard protocol</a>: a 512-token prompt, 256 generated tokens, one request, temperature 0, thinking, speculation and prompt cache off, one warm-up and then the mean of three runs. Record those conditions with the result so it joins the Standard board.</li>
<li>Measure one request at a time. A throughput configuration — OpenVINO's <code>PERFORMANCE_HINT=THROUGHPUT</code>, vLLM serving many sequences at once, llama.cpp with parallel slots — reports the total across concurrent requests, which is a larger and different number from what one person sees waiting for one answer. If that is what you measured, record it as <code>concurrency</code> with <code>decodeScope</code> set to <code>total</code>: it ranks on the Throughput board instead of against single requests, and several request counts of the same setup draw a scaling curve on the rig page.</li>
<li>Report the runtime version you used. An evidence link is optional: logs, scripts, a gist, a report, or a repository can help others check your result. Any full HTTPS URL works.</li>
<li>Note the flags and settings that moved the number: the backend you compiled in, flash attention, KV cache precision. The same card on the same runtime can differ twofold on these.</li>
<li>If you ran on one card out of several, submit it as a component result and set the quantity you used.</li>
<li>A Core Ultra chip carries CPU cores, an iGPU, and an NPU, and each is its own part. Submit the unit the model ran on; a result on the CPU part means its cores. Use whole rig when the run spanned more than one unit.</li>
</ul>
<h2>Ranking</h2>
<p>Each rig, or each part at a given quantity, appears once on a board at its best decode tok/s. The earliest run wins a tie. Boards mix runtimes; filter by runtime to compare like with like.</p>
<p>Boards are split into setups — Standard, Speculative, Long prompt, 4 requests and Throughput — each a fixed way of measuring. A result joins a setup only when the conditions it records match that setup's rules; a result with no conditions recorded appears only under All results. Within a setup, each rig, or each part at a given quantity, appears once at its best decode tok/s. The earliest run wins a tie. Boards mix runtimes; filter by runtime to compare like with like.</p>
<h2>Stock and custom runtimes</h2>
<p>A result either ran on a stock runtime — the released one, however you configured or built it — or on a <strong>custom runtime</strong>: one someone registered after changing it, with a custom kernel or op, a patch, or a fork. A custom runtime can beat stock by a wide margin on the same silicon, so boards rank stock runs against each other and leave custom ones out until you turn on <strong>Include modified</strong>. Nothing is hidden and they are not lesser; they answer a different question, rank among each other on the same board, and gather on the custom runtime's own page.</p>
<p>Register a custom runtime once, from the submit form, and every later result just picks it. You can post a result on anyone's, not only your own — a public fork is a real thing anyone can run. Each result names the exact revision it used, because an implementation changes week to week and only the revision makes a number reproducible.</p>
Expand Down
52 changes: 52 additions & 0 deletions results/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,58 @@ counts as a rig's best decode speed. Measure the same model, quant and runtime a
`contextLength` and `batchSize` still work but are ambiguous: some people meant the prompt, others the context
window; some meant concurrent requests, others the engine batch size. Use `inputTokens` and `concurrency` instead.

### Standard protocol

Measure this way and your results land in the named setups, comparable with everyone else's by default. Each point
is its own result file.

**The minimum: one result.** 512 input tokens, 256 output tokens, 1 request. This is the comparison point and
ranks on the **Standard** board.

**The full grid, if you have time.** Input lengths 512, 4096, 16384 and 32768 tokens (as many as fit in the
context window), each at 1 and at 4 concurrent requests. 16k and 32k at 1 request rank on **Long prompt**; 512 at 4
requests ranks on **4 requests** and **Throughput**; every point counts toward the rig page's curves.

For every point:

1. **Settings.** 256 output tokens, temperature 0, thinking off, no speculative decoding, prompt cache off.
2. **Repeats.** One warm-up run, discarded, then 3 measured runs. Report the mean; put the min and max in `notes`.
3. **Decode.** The streamed rate after the first token: `(outputTokens − 1) / (time of last token − time of first token)`.
Report TTFT separately as `ttftMs`. At 4 requests, submit two files from the same runs: the mean per-request
decode (`decodeScope: "per_request"`) and the total completion tokens over wall time (`decodeScope: "total"`).
4. **Record.** The engine commit as `revision`, the engine flags as `runtimeFlags`, and the model file (repository,
file name, SHA-256) in `notes`.

```json
{
"inputTokens": 512,
"outputTokens": 256,
"concurrency": 1,
"decodeMethod": "streamed",
"repeats": 3,
"promptCache": false,
"thinking": false,
"speculative": "none",
"revision": "b6512",
"notes": "min/max 33.9/34.6. Model: Qwen/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf sha256 1a2b…"
}
```

Three details decide whether two runs of the same setup agree:

- **Count tokens with the engine, not stream chunks.** A chunk can carry several tokens. Use the server's
`usage.completion_tokens` (with `stream_options: {"include_usage": true}` on OpenAI-compatible servers) or the
engine's own count, and record that as `outputTokens`.
- **Force the full length.** A model that stops early measures a shorter run. Turn on `ignore_eos` (llama.cpp,
vLLM, SGLang) or set `min_new_tokens` (OpenVINO GenAI, Transformers) to 256. If a run still stops short, discard it
and run again.
- **Keep repeats from reusing the cache.** Set `cache_prompt: false` on llama-server, or start each repeat's prompt
with a different line (a run number is enough) so no prefix matches. Then record `promptCache: false`.

`llama-bench` with `-p 512 -n 256 -r 3` follows the same settings; record `decodeMethod: "llama_bench"`. It times
generation from an empty context rather than after the prompt, so its number reads a few percent higher than a
streamed run.

### Optional evidence link

You can submit without an evidence link. If you have published benchmark logs, scripts, a gist, a report,
Expand Down
Loading