Skip to content

Make the DiariZen adapter runnable again, and pin the streaming operating point - #21

Merged
PhilflowIO merged 3 commits into
masterfrom
fix/adapter-kwarg-contract
Sep 5, 2026
Merged

PhilflowIO merged 3 commits into
masterfrom
fix/adapter-kwarg-contract

Conversation

@PhilflowIO

Copy link
Copy Markdown
Owner

Two defects found while running the next measurement on master. Both are small; the first makes a registered model impossible to run.

1. The DiariZen adapter cannot be built

The language kwarg was added to the runner's adapter dispatch (PR #16) so a hosted diarizer cannot be sent a corpus in the wrong language. The DiariZen adapter (PR #20) landed in parallel and never grew it. On master the model is registered, tested, documented — and raises TypeError on the first call:

TypeError: DiariZenDiarizer.__init__() got an unexpected keyword argument 'language'

Nothing caught it because every test builds an adapter directly, and the dispatcher is the only caller that passes all four kwargs. The adapter now accepts and ignores it, like the other local models: it scores speaker turns, not words.

The test that would have caught it reads the dispatcher's keyword names out of reproduce.py with ast and asserts every registered adapter can bind them. Parsed rather than listed, so adding a kwarg to the dispatcher tightens the contract instead of leaving a stale list that still passes. Verified against the broken signature before the fix — it fails there and passes here.

2. The streaming operating point was not in the public contract

benchmark.config.yaml documents what every published number was produced under. It pinned the collar, the overlap rule, the aggregation and the speaker mapping — and left out the one convention that moves this row most: a streaming diarizer has one DER per latency setting, and the same checkpoint reads 8.98 at 30.4 s of input-buffer latency against 9.82 at 1.04 s. That is 0.84 pp, seventeen times the ±0.05 pp reproduction tolerance, and it is the entire explanation of our disagreement with the ETH benchmark (#5454).

It is pinned now under der.model_configuration, and asserted against the adapter rather than merely for presence, so the contract file and the code cannot drift apart. A new streaming checkpoint cannot slip in unpinned either — the test sweeps every model that has a configurable operating point.

Verification

  • uv run --extra dev --extra asr pytest -q330 passed, 2 skipped (2 new).
  • uv run python scripts/verify.py → OK: 22 row(s) reproduced.
  • The DiariZen AMI run that surfaced this is now proceeding on the AI box against this branch.

Refs #5454, #5461 (Forgejo Philflow/flow.raven)

🤖 Generated with Claude Code

PhilflowIO and others added 3 commits September 5, 2026 20:01
The `language` kwarg was added to the runner's adapter dispatch so a hosted
diarizer cannot be sent a corpus in the wrong language. The DiariZen adapter
landed in parallel and never grew it, so on master the model is registered,
tested, documented and impossible to run: the first call raises TypeError.
Nothing caught it, because every test builds the adapter directly and the
dispatcher is the only caller that passes all four kwargs.

The adapter now accepts and ignores it, like the other local models — it
scores speaker turns, not words.

The test that would have caught it reads the dispatcher's keyword names out
of its own source with `ast` and asserts every registered adapter can bind
them. Parsed rather than listed, so adding a kwarg to the dispatcher tightens
the contract instead of leaving a stale list that still passes. Verified
against the broken signature before the fix.

Refs #5461

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A streaming diarizer does not have one DER, it has one per latency setting:
the same Sortformer v2 checkpoint reads 8.98 on German CALLHOME at 30.4 s of
input-buffer latency and 9.82 at 1.04 s. That is 0.84 pp, seventeen times the
tolerance every number on the page is reproduced to, and it is the whole
explanation of the disagreement with the ETH benchmark.

benchmark.config.yaml documents publicly what every published number was
produced under, and it pinned the collar, the overlap rule, the aggregation
and the speaker mapping while leaving out the one convention that moved this
row. It is pinned now, and asserted against the adapter rather than merely
for presence, so the file and the code cannot drift apart. A new streaming
checkpoint cannot slip in unpinned either.

Refs #5454

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DiariZen now has a meeting row: 25.69 at collar 0.0, 23.14 at 0.25, all 16
AMI test meetings, artifact committed and re-scored by verify. It ties the
streaming Sortformer (23.03) and loses to community-1 (13.10) by 10 pp, so
the many-speaker argument built on VoxConverse does not transfer to meetings.

Measuring it surfaced something that changes how two of its three rows may
be read. Its model card names the training data verbatim, and it includes
AMI and VoxConverse. Both of those rows therefore compare a model on its own
training distribution against models that never saw the data — the 1.58 pp
lead over community-1 is real as a number and is not evidence of better
generalisation, which is what the surrounding paragraph had used it for.

CALLHOME is not in that list, so the German ranking and the reconciliation
with the ETH benchmark stay clean out-of-domain measurements. That is now
stated per corpus beside the rows rather than in a footnote: a benchmark that
quietly ranks a model on its own training data is the failure this page
exists to prevent.

Refs #5461, #5468

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@PhilflowIO
PhilflowIO merged commit 3cca0c6 into master Sep 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant