Measure Grok Voice Transcribe 2.0 on the benchmark page's samples - #22
Merged
Merged
Conversation
The Raven benchmark page publishes an "avemio" column measured on the first 100 rows of avemio/ASR-GERMAN-MIXED-TEST. This repo had no loader for it — german-mixed is flozi00/asr-german-mixed-evals, a different repo with a different row order and different references — so that column had no path to reproduction here, which ADR-app-0054 treats as a defect. The loader is pinned at the repo's last commit (2025-01-07); the page's existing numbers were measured at an unpinned HEAD that already was this commit. Checked row by row against the page's own loader: the first 100 references are identical and in the same order. Refs Philflow/flow.raven#7279 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A prediction line carried reference, prediction and latency — nothing that said which utterance it was or how long it ran. Two things need that: * a failed call is dropped from the list, so without the id nobody can see which rows are missing or join a line to the same row elsewhere; * RTFx and a per-minute price are ratios over audio length, so a downstream table (the benchmark page shows both) cannot derive them from latency alone. Both are properties of the input, recorded next to the output. Additive: older artifacts and verify.py, which reads only reference/prediction, are unaffected. Refs Philflow/flow.raven#7279 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmark page reads strict-de WER, length-weighted per dataset; this repo publishes flozi-strict. With only the latter committed, the page would have to compute its number from the predictions on its own — next to raven.eval, not by it. promote now writes wer_strict_de_pct per subset (raven_eval_core.corpus_wer_strict_de_pct) and verify re-scores it like every published value, so the page number is defended by this repo's CI. The corpus rule is Σ(wer_strict · words) / Σ words, not Σ edits / Σ words: for an empty hypothesis compute_wer reports wer_strict = 1.0 but zero edit counts, so the edit form would score a model that returns nothing as perfect. benchmark.config.yaml makes BLEU the headline for bleu+wer corpora, yet no run produced one — only the hand-built demo artifact carried the key. promote now adds BLEU and its signature for every subset whose dataset declares bleu+wer, taken from the dataset spec rather than a caller flag. Both keys are opt-in in verify, so artifacts promoted before them still pass. Refs Philflow/flow.raven#7279 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
REST POST /v1/stt, multipart. Two vendor rules change the result silently rather than failing: `file` must be the last field (options sent after it may be ignored), and `format=true` requires `language`. The adapter sends language=de + format=true and the test pins the field order. Concurrency 8 stays under the documented 10 requests/s. Registered as xai/grok-voice-transcribe-2.0 with XAI_API_KEY; the key's name is documented in TIER2-KEYS.md, its value lives only in the operator's env. Refs Philflow/flow.raven#7279 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on exactly the rows the Raven benchmark page compares every provider
on — the first 100 of avemio-german-mixed-test, fleurs, mls-de, voxpopuli-de
and the first 50 of spc-test and fhnw-all-dialects — so the row can stand next
to the other 49 there. 500 of 500 utterances returned, 1.50 h of audio,
USD 0.15 at the listed USD 0.10 per hour.
strict-de (the page's lens): 6.62 / 5.19 / 8.44 / 12.36 % on the four
read-aloud sets, mean 8.15 %. BLEU 43.42 on spc-test and 40.65 on
fhnw-all-dialects. On mls-de the model sometimes speaks the audiobook's
section marker ("Ende von Abschnitt eins") into the transcript; the reference
omits it, so it counts as insertions and CER exceeds WER there.
make verify: 29 rows reproduced, the six new ones included.
Refs Philflow/flow.raven#7279
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
xAI released
grok-voice-transcribe-2.0. Under ADR-app-0054 every number the Raven benchmark page publishes has to be reproducible from this repo, so the model is measured here, once, and the page reads the artifact (Philflow/flow.raven#7279).What
avemiocolumn is measured onavemio/ASR-GERMAN-MIXED-TEST, which is notgerman-mixed. Pinned at the repo's last commit; the first 100 references match the page's row for row.wer_strict_de_pct(strict-de, length-weighted, the page's rule) and, forbleu+wercorpora,bleu+ signature.make verifyre-scores both; both are opt-in, so older artifacts still pass.filelast,language=de,format=true.Result
strict-de 6.62 / 5.19 / 8.44 / 12.36 % (avemio / fleurs / mls-de / voxpopuli-de), mean 8.15 %; BLEU 43.42 (spc-test), 40.65 (fhnw). 500 of 500 utterances, 1.50 h audio, USD 0.15.
Checks
make verify: 29 rows reproduced (the six new ones included).pytest: relevant suites green (63 in the touched files).🤖 Generated with Claude Code