Skip to content

Add the standard measuring protocol (#130 step 5) - #143

Merged
jackwsmth merged 1 commit into
mainfrom
issue-130-protocol
Nov 18, 2026
Merged

jackwsmth merged 1 commit into
mainfrom
issue-130-protocol

Conversation

@jackwsmth

Copy link
Copy Markdown
Collaborator

Step 5 of #130.

results/README.md gets a Standard protocol section:

  • Minimum: one result at 512 input / 256 output / 1 request, which lands on the Standard board.
  • Optional grid: 512, 4096, 16384 and 32768 input tokens, each at 1 and 4 requests, with the setup each point lands in.
  • For every point: temperature 0, thinking, speculation and prompt cache off; one warm-up and then the mean of 3 runs, with min/max in notes; decode is the streamed rate after the first token; at 4 requests, submit both a per-request file and a total file; record the revision, flags and model file.
  • The three questions the issue left open: count tokens with the engine (usage.completion_tokens), not by stream chunk; force the full length (ignore_eos / min_new_tokens) and rerun any run that stops early; keep repeats off the cache (cache_prompt: false or a different first line per repeat).

Guidelines page: the measuring advice links the protocol, and Ranking now describes setups instead of one best result per rig.

Docs and page copy only. No schema change or migration. Spread and the model file go in notes until there are fields for them (step 4).

results/README.md gains a Standard protocol section: the 512/256/1 comparison
point, the optional grid, settings, repeats, how decode is timed, and the three
details the issue left open (token counting, early stops, cache reuse).
The Guidelines page links it and describes ranking within setups.
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
intelinside Ready Ready Preview Nov 18, 2026 2:32pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
No submission files to import.

Site / sign up · Workflow details and retry

@jackwsmth
jackwsmth merged commit 90d99f4 into main Nov 18, 2026
5 checks passed

This branch was successfully deployed

1 active deployment
Preview — 0e2f80b8 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant