Skip to content

fix: state the WER/CER caps in every published POSEIDON formula (#291) - #294

Merged
RN0311 merged 2 commits into
mainfrom
fix/291-document-poseidon-caps
Sep 22, 2026
Merged

RN0311 merged 2 commits into
mainfrom
fix/291-document-poseidon-caps

Conversation

@RN0311

@RN0311 RN0311 commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #291. calculate_poseidon_score caps WER and CER at 1.0 before use while the artifact columns publish the raw values, and every statement of the formula (metrics.py docstring, docs/FAQ.md, benchmark/README.md) omitted the caps — so for any row with WER or CER above 1, recomputing from the published components gave a different number than the published score (the issue's row: −3.30 by the documented formula vs 0.033 in the artifact).

Direction taken (the call the issue left to maintainers)

The computation is unchanged; the documentation now states the real formula. The other two directions both damage something:

With the caps stated in the formula, a published poseidon_score is checkable against the components printed beside it — the property the issue asks for.

Changes

  • calculate_poseidon_score docstring: formula rewritten as w_wer·(1 − min(WER, 1)) + w_cer·(1 − min(CER, 1)) + w_sem·Similarity, clamped to [0, 1], with the raw-columns/capped-composite convention and its rationale spelled out; a comment at the cap lines marks them as part of the published formula.
  • benchmark/README.md: the "Missing values and metric ranges" convention section gains a bullet stating both sides — columns raw, composite capped — and the exact recomputation formula.
  • docs/FAQ.md: the POSEIDON row of the metric table carries the capped formula and the raw-columns note.
  • PoseidonScorer docstring: one sentence on the same convention.

Testing

  • New TestPublishedScoreIsCheckable in tests/unit/test_metrics.py: the issue's exact row (WER 6.67, CER 7.77, sim 0.0739, weights 0.35/0.20/0.45) equals both the artifact value it observed (0.45 × sim) and the now-documented formula applied to the raw components; a parametrized grid (perfect, ordinary, floor, WER-overshoot, CER-overshoot, the issue's row) pins the computation to the documented formula; an in-range row confirms no ordinary score moves.
  • test_metrics.py + test_scorer.py: 39 passed. ruff check, ruff format --check, ty check clean.

The composite caps WER and CER at 1.0 before use while the artifact
columns publish the raw values, and the formula was documented without
the caps — so for any row whose WER or CER exceeds 1 (ordinary
insertion behaviour) the published poseidon_score could not be arrived
at from the wer, cer and semantic_similarity printed beside it.

The computation is deliberately unchanged: capping the published
columns would misreport WER/CER themselves and shift wer_mean/cer_mean
(unlike out-of-range cosine similarity, a WER of 6.67 is a real
measurement, not noise — which is why the #107 clamp-at-computation
convention fits similarity but not the error rates), and uncapping the
composite would change every affected published score. Instead the
formula now carries the caps everywhere it is stated — the
calculate_poseidon_score docstring, docs/FAQ.md, benchmark/README.md
(with a new bullet in the metric-ranges convention section naming both
sides of the convention), and the PoseidonScorer docstring — so a
published score is checkable against its published parts. Regression
tests pin the documented formula to the computation, including the
issue's exact row.
@RN0311
RN0311 requested a review from AndyBoWu as a code owner September 22, 2026 22:00
@github-actions github-actions Bot added track: ops Governance docs, templates, repo operations track: toolkit Core psdn-sonar eval toolkit code and tests track: qa Evaluation correctness and regression testing labels Sep 22, 2026
@RN0311
RN0311 merged commit bdbe59e into main Sep 22, 2026
17 checks passed
@RN0311
RN0311 deleted the fix/291-document-poseidon-caps branch September 22, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

track: ops Governance docs, templates, repo operations track: qa Evaluation correctness and regression testing track: toolkit Core psdn-sonar eval toolkit code and tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] WER and CER are capped inside the POSEIDON composite but published raw, so a score cannot be checked against its own components

1 participant