Skip to content

fix: scan each transcript for a foreign script, not just the batch (#292) - #293

Merged
RN0311 merged 2 commits into
mainfrom
fix/292-per-row-script-check
Sep 22, 2026
Merged

RN0311 merged 2 commits into
mainfrom
fix/292-per-row-script-check

Conversation

@RN0311

@RN0311 RN0311 commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #292. The hypothesis script check added by #220 scans a model's predictions as one batch, so its majority gate divides by the run total while the condition happens per transcript: when one row of five comes back entirely in Latin script under --language bn, the foreign row is 21% of the batch, the check returns None, and the row is scored against a reference it shares no characters with — WER 1.68 averaged into wer_mean with an empty error column and warnings: [].

The same gates, applied per transcript:

The reference-side check (script_mismatch_warning) is deliberately untouched: it detects a mis-set --language, which is a run-level property, not a per-row one.

Testing

  • TestPerRowScriptMarkers: the issue's exact run shape — batch check silent, exactly the foreign row marked, summary names 1 of 5 and the column; plus clean runs, sub-threshold short rows, code-switched single rows, languages without an implied script, and empty predictions.
  • TestSingleForeignRowRecordedInArtifact, end-to-end through run_evaluation: the artifact carries the 1 of 5 warning in scores.json, the CSV marks exactly row 1, the batch warning does not double-fire; the all-foreign run keeps the [Bug] A transcript returned in a different writing system scores WER 1.0 with no warning, because the script-mismatch check reads references only #207 batch warning and marks every row; a clean run stays clean in both artifacts.
  • Smoke test with the issue's driver: whole batch → None, single row → marker, summary warning names the count and column.
  • test_script_check.py (34) plus every evaluator-adjacent suite (test_language_forwarding, test_zero_sample_runs, test_semantics_missing, test_submission_provenance, test_custom_eval, test_quality_models): 164 passed. ruff and ty clean.

)

The hypothesis script check from #220 computes its majority across the
whole run, while the condition it detects happens per transcript: one
row returned entirely in another writing system stays under the 50%
threshold when its neighbours are correct, is scored against a
reference it shares no characters with, and leaves no trace — its
inflated WER/CER averages into wer_mean/cer_mean unmarked.

The same gates (dedicated-normalizer language, minimum script-bearing
characters, clear majority) now also run over each transcript alone:
affected rows carry a foreign_script marker in a new script_warning CSV
column, and when the batch-level check stays silent, a run-level
warning naming the count and the column reaches the log and the
scores.json warnings array. Code-switching within a transcript still
does not trip the per-row gate, short rows remain too small to call,
and the whole-run-foreign case keeps the #207 batch warning while
gaining the per-row markers.
@RN0311
RN0311 requested a review from AndyBoWu as a code owner September 22, 2026 21:02
@github-actions github-actions Bot added track: toolkit Core psdn-sonar eval toolkit code and tests track: qa Evaluation correctness and regression testing labels Sep 22, 2026
@RN0311
RN0311 merged commit a81bdaf into main Sep 22, 2026
17 checks passed
@RN0311
RN0311 deleted the fix/292-per-row-script-check branch September 22, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

track: qa Evaluation correctness and regression testing track: toolkit Core psdn-sonar eval toolkit code and tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] The hypothesis script check runs over a whole batch, so a single transcript in another writing system leaves no trace

1 participant