WER integrity: complete, pinned, per-utterance, with intervals - #30
Merged
Merged
Conversation
DER and WER are the same shape of estimator, a corpus sum of errors over a corpus sum of reference units, so one seeded resampler can serve both. It moves out of raven_diar.analysis into raven_eval_core.bootstrap with the draw order unchanged. Every committed DER interval and paired delta recomputes byte-identical, checked against a snapshot of all of them taken before the move. A paired comparison now fails when a file exists for only one model. Before, such files were dropped silently, and the interval then described a smaller corpus than either published number while looking just as confident. Refs #25 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three Sortformer artifacts were measured the day before the adapter began writing its latency preset into summary.json. Nothing stated which streaming setting their numbers belong to. At the time, the adapter hardcoded v2 to the exact values of today's "very-high-latency" preset (identical dict at f9dd436) and gave the offline v1 no streaming config. That is backfilled here in the same shape the runner writes today. A guard now fails any artifact of a diarizer that exposes run_config but does not record it, so a diagnostic run cannot be published as the shipped configuration. Refs #26 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A published WER must describe the complete subset it names, under one fixed counting method, with its uncertainty stated. Four gaps kept it from doing so: - Failed requests vanished. The runner dropped them before scoring, so a model that timed out on the hardest clips read *better*. Every attempted clip now gets a prediction line (a failure carries prediction null plus error), the summary counts n_attempted/n_ok/n_failed, a failed subset leaves no resume marker, and promote refuses the run. verify fails on a failure line or on a line count that differs from expected.json's n_samples. The N-model path goes through the same recorder. Before, it wrote no predictions at all. - Errors cancelled across clip boundaries. flozi's scorer concatenates a subset into one sentence before aligning, so a word missing from the end of one clip cancels against a surplus word at the start of the next. Every utterance is now aligned on its own. On flozi's own subsets and FLEURS the numbers are identical (largest move +0.001 pp). On VoxPopuli and the two Swiss sets the old count read 0.26-0.85 pp low, and six published Grok rows rise by that. - No interval. Every WER now carries a 95 % bootstrap interval over utterances (contract: wer.uncertainty). It is committed to expected.json, re-derived by verify and bound to the page. `make analyse COMPARE=` adds the paired gap. The page's Grok 2.0-vs-1.0 prose claimed more than its data holds: only FLEURS, spc-test (2.0 ahead) and VoxPopuli (1.0 ahead) exclude zero. - No provenance. Summaries now record model_revision and a per-result dataset_revision (or archive sha256). The flozi dataset is pinned to 9c34cbcc, the HEAD since 2026-01-06, whose shard row counts equal the published 414/3996/5389 exactly. Existing artifacts are backfilled from their pins, and a guard fails any WER artifact without them. Closes #24 Closes #25 Closes #26 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three commits, one work unit: a published WER must describe the whole subset it names, under a fixed counting method, with its uncertainty stated.
What changes for readers
fhnw-all-dialects38.81→39.66 / 43.50→43.92,spc-test49.38→50.07 / 41.08→41.77,voxpopuli-de11.01→11.27 / 12.68→12.99. The cause: the old scorer concatenated a subset into one sentence before aligning, so an error at one clip's end could cancel against an error at the next clip's start. Every utterance is now aligned on its own. On flozi's own subsets and FLEURS, nothing moves at the printed precision, so the flozi anchors stay comparable. BENCHMARKS.md documents this under How WER is counted.expected.json, re-derived bymake verifyand bound to the table.Guards added
verifyfails on a failure line or a line-count mismatch. The N-model runner path now writes predictions at all.raven_eval_coreserves DER and WER. All DER intervals are byte-identical after the move, and a paired comparison now refuses unpaired files instead of dropping them.model_revisionand a per-resultdataset_revisionare now in every WER summary. The flozi dataset is pinned to9c34cbcc, whose shard counts equal the published 414/3996/5389 exactly. Sortformer artifacts now carry their operating point. Guards fail any artifact that lacks them, and the negative check was run.Verification
make verify: 35/35 rows reproduce, intervals included. It now takes about 1 min longer because it recomputes the intervals.pytest: 374 passed. The one failure (module registry) is fixed in the first commit, and the affected modules were rerun: 137 passed.Follow-ups (not in this PR)
model_revisionis null for those today.Closes #24
Closes #25
Closes #26
🤖 Generated with Claude Code