Repository navigation
fix: scan each transcript for a foreign script, not just the batch (#292) - #293
Merged
Merged
Conversation
) The hypothesis script check from #220 computes its majority across the whole run, while the condition it detects happens per transcript: one row returned entirely in another writing system stays under the 50% threshold when its neighbours are correct, is scored against a reference it shares no characters with, and leaves no trace — its inflated WER/CER averages into wer_mean/cer_mean unmarked. The same gates (dedicated-normalizer language, minimum script-bearing characters, clear majority) now also run over each transcript alone: affected rows carry a foreign_script marker in a new script_warning CSV column, and when the batch-level check stays silent, a run-level warning naming the count and the column reaches the log and the scores.json warnings array. Code-switching within a transcript still does not trip the per-row gate, short rows remain too small to call, and the whole-run-foreign case keeps the #207 batch warning while gaining the per-row markers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #292. The hypothesis script check added by #220 scans a model's predictions as one batch, so its majority gate divides by the run total while the condition happens per transcript: when one row of five comes back entirely in Latin script under
--language bn, the foreign row is 21% of the batch, the check returnsNone, and the row is scored against a reference it shares no characters with — WER 1.68 averaged intower_meanwith an emptyerrorcolumn andwarnings: [].The same gates, applied per transcript:
hypothesis_row_script_warningsruns the existing scan over each transcript alone — same dedicated-normalizer-language gate, same 10-character minimum (short rows stay too small to call), same clear-majority rule (code-switching within a transcript still does not trip it, preserving fix: warn when a model's transcripts come back in a different writing system (#207) #220's deliberate gate: mixing inside one transcript is code-switching; one transcript entirely in another script is not).asr_detailed_<model>.csvgains ascript_warningcolumn; an affected row carries e.g.foreign_script: Latin script (100% of script-bearing characters; --language 'bn' expects Bengali script). The column is part of the canonical row builder so headers stay consistent on empty runs.Model '…' returned 1 of 5 transcripts in a different writing system …) names the count and the CSV column, is logged, and lands in the scores.jsonwarningsarray through the existing [Bug] A transcript returned in a different writing system scores WER 1.0 with no warning, because the script-mismatch check reads references only #207 plumbing.The reference-side check (
script_mismatch_warning) is deliberately untouched: it detects a mis-set--language, which is a run-level property, not a per-row one.Testing
TestPerRowScriptMarkers: the issue's exact run shape — batch check silent, exactly the foreign row marked, summary names1 of 5and the column; plus clean runs, sub-threshold short rows, code-switched single rows, languages without an implied script, and empty predictions.TestSingleForeignRowRecordedInArtifact, end-to-end throughrun_evaluation: the artifact carries the1 of 5warning inscores.json, the CSV marks exactly row 1, the batch warning does not double-fire; the all-foreign run keeps the [Bug] A transcript returned in a different writing system scores WER 1.0 with no warning, because the script-mismatch check reads references only #207 batch warning and marks every row; a clean run stays clean in both artifacts.None, single row → marker, summary warning names the count and column.test_script_check.py(34) plus every evaluator-adjacent suite (test_language_forwarding,test_zero_sample_runs,test_semantics_missing,test_submission_provenance,test_custom_eval,test_quality_models): 164 passed.ruffandtyclean.