Make the DiariZen adapter runnable again, and pin the streaming operating point - #21
Merged
Merged
Conversation
The `language` kwarg was added to the runner's adapter dispatch so a hosted diarizer cannot be sent a corpus in the wrong language. The DiariZen adapter landed in parallel and never grew it, so on master the model is registered, tested, documented and impossible to run: the first call raises TypeError. Nothing caught it, because every test builds the adapter directly and the dispatcher is the only caller that passes all four kwargs. The adapter now accepts and ignores it, like the other local models — it scores speaker turns, not words. The test that would have caught it reads the dispatcher's keyword names out of its own source with `ast` and asserts every registered adapter can bind them. Parsed rather than listed, so adding a kwarg to the dispatcher tightens the contract instead of leaving a stale list that still passes. Verified against the broken signature before the fix. Refs #5461 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A streaming diarizer does not have one DER, it has one per latency setting: the same Sortformer v2 checkpoint reads 8.98 on German CALLHOME at 30.4 s of input-buffer latency and 9.82 at 1.04 s. That is 0.84 pp, seventeen times the tolerance every number on the page is reproduced to, and it is the whole explanation of the disagreement with the ETH benchmark. benchmark.config.yaml documents publicly what every published number was produced under, and it pinned the collar, the overlap rule, the aggregation and the speaker mapping while leaving out the one convention that moved this row. It is pinned now, and asserted against the adapter rather than merely for presence, so the file and the code cannot drift apart. A new streaming checkpoint cannot slip in unpinned either. Refs #5454 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DiariZen now has a meeting row: 25.69 at collar 0.0, 23.14 at 0.25, all 16 AMI test meetings, artifact committed and re-scored by verify. It ties the streaming Sortformer (23.03) and loses to community-1 (13.10) by 10 pp, so the many-speaker argument built on VoxConverse does not transfer to meetings. Measuring it surfaced something that changes how two of its three rows may be read. Its model card names the training data verbatim, and it includes AMI and VoxConverse. Both of those rows therefore compare a model on its own training distribution against models that never saw the data — the 1.58 pp lead over community-1 is real as a number and is not evidence of better generalisation, which is what the surrounding paragraph had used it for. CALLHOME is not in that list, so the German ranking and the reconciliation with the ETH benchmark stay clean out-of-domain measurements. That is now stated per corpus beside the rows rather than in a footnote: a benchmark that quietly ranks a model on its own training data is the failure this page exists to prevent. Refs #5461, #5468 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects found while running the next measurement on master. Both are small; the first makes a registered model impossible to run.
1. The DiariZen adapter cannot be built
The
languagekwarg was added to the runner's adapter dispatch (PR #16) so a hosted diarizer cannot be sent a corpus in the wrong language. The DiariZen adapter (PR #20) landed in parallel and never grew it. On master the model is registered, tested, documented — and raisesTypeErroron the first call:Nothing caught it because every test builds an adapter directly, and the dispatcher is the only caller that passes all four kwargs. The adapter now accepts and ignores it, like the other local models: it scores speaker turns, not words.
The test that would have caught it reads the dispatcher's keyword names out of
reproduce.pywithastand asserts every registered adapter can bind them. Parsed rather than listed, so adding a kwarg to the dispatcher tightens the contract instead of leaving a stale list that still passes. Verified against the broken signature before the fix — it fails there and passes here.2. The streaming operating point was not in the public contract
benchmark.config.yamldocuments what every published number was produced under. It pinned the collar, the overlap rule, the aggregation and the speaker mapping — and left out the one convention that moves this row most: a streaming diarizer has one DER per latency setting, and the same checkpoint reads 8.98 at 30.4 s of input-buffer latency against 9.82 at 1.04 s. That is 0.84 pp, seventeen times the ±0.05 pp reproduction tolerance, and it is the entire explanation of our disagreement with the ETH benchmark (#5454).It is pinned now under
der.model_configuration, and asserted against the adapter rather than merely for presence, so the contract file and the code cannot drift apart. A new streaming checkpoint cannot slip in unpinned either — the test sweeps every model that has a configurable operating point.Verification
uv run --extra dev --extra asr pytest -q→ 330 passed, 2 skipped (2 new).uv run python scripts/verify.py→ OK: 22 row(s) reproduced.Refs #5454, #5461 (Forgejo
Philflow/flow.raven)🤖 Generated with Claude Code