Skip to content

fix: keep the prewarmed quality models off the meta device (#288) - #289

Merged
RN0311 merged 2 commits into
mainfrom
fix/288-utmos-meta-device
Sep 21, 2026
Merged

RN0311 merged 2 commits into
mainfrom
fix/288-utmos-meta-device

Conversation

@RN0311

@RN0311 RN0311 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #288. The evaluator prewarms the quality models (DNSMOS/UTMOS/SQUIM) on a background thread and joined it only after the ASR model factory ran. transformers implements from_pretrained(..., low_cpu_mem_usage=True) by opening init_empty_weights(), which replaces nn.Module.register_parameter on the class — process-wide, not per thread. A UTMOS predictor constructed by the prewarm thread inside that window lands its parameters on the meta device; loading real weights into meta tensors is a silent no-op, so _get_utmos() reported a healthy predictor whose every score then failed on first forward (Tensor on device meta is not on the expected device cpu!) and was discarded — an empty utmos column while dnsmos_ovrl/squim_pesq/snr_db populate from the same audio.

Both remedies the issue names, as two layers:

  1. The race is removed: the prewarm thread is joined immediately before _model_factory runs, so quality-model construction can never overlap the init_empty_weights() window. Joining an already-finished thread is a no-op, so later loop iterations pass straight through; the old post-factory join is gone.
  2. A predictor that cannot score is reported as unavailable, not as loaded: the UTMOS and SQUIM loaders now refuse a constructed model whose parameters sit on the meta device, with the reason recorded in _utmos_error/_squim_error — so it reaches mos_warnings → quality_warnings in the artifact (the [Bug] A UTMOS fetch failure empties the column with no reason recorded, bypassing the mos_metrics_unavailable marker #245 reporting path) rather than a debug line. This is the backstop for any other construction route, and covers the structurally identical SQUIM case the issue lists as not checked. The failure path clears the cached model reference so a refused predictor can never be returned. DNSMOS is ONNX-based with no torch parameters; the guard tolerates objects without parameters().

Testing

  • TestPrewarmJoinsBeforeTheModelFactory: end-to-end through run_evaluation with a delayed prewarm — fails against the pre-fix code with the exact reported race (factory before the prewarm events) and passes with the fix.
  • TestMetaDevicePredictorRefused: the guard raises for a meta-device module and names the window; CPU modules and parameterless objects pass; a meta UTMOS/SQUIM is recorded unavailable (_get_* returns None, error mentions the meta device and the issue); end-to-end compute_mos_metrics carries utmos_unavailable: ... meta device ... in mos_warnings.
  • Smoke test against the real mechanism: a module built inside transformers' actual init_empty_weights() context (transformers.integrations.accelerate) has parameters on ['meta'] and is refused by the guard; a normally-built module passes.
  • Full test_quality_models.py (29), plus test_language_forwarding.py, test_script_check.py, test_audio_quality.py: 91 passed. ruff check, ruff format --check clean.

The evaluator prewarms DNSMOS/UTMOS/SQUIM on a background thread and
joined it only after the ASR factory ran. transformers implements
from_pretrained(low_cpu_mem_usage=True) by opening init_empty_weights(),
which patches nn.Module.register_parameter on the class — process-wide,
not per thread — so a quality model constructing inside that window got
parameters on the meta device, loaded real weights as a silent no-op,
and then failed every score, leaving the utmos column empty with only a
debug-level trace.

Two layers, matching the issue's either/or:
- the prewarm thread is now joined before the model factory runs, so
  the constructions cannot overlap the window (joining again on later
  iterations is a no-op);
- the UTMOS and SQUIM loaders refuse a constructed model whose
  parameters sit on the meta device, recording the reason so it reaches
  mos_warnings/quality_warnings instead of reporting a healthy
  predictor — the backstop for any other construction route, and it
  covers the structurally identical SQUIM case the issue lists as
  untested. The failure path also clears the cached model reference so
  a refused predictor is never returned.
@RN0311
RN0311 requested a review from AndyBoWu as a code owner September 18, 2026 19:56
@github-actions github-actions Bot added track: toolkit Core psdn-sonar eval toolkit code and tests track: qa Evaluation correctness and regression testing labels Sep 18, 2026
@RN0311
RN0311 merged commit b64a1a9 into main Sep 21, 2026
17 checks passed
@RN0311
RN0311 deleted the fix/288-utmos-meta-device branch September 21, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

track: qa Evaluation correctness and regression testing track: toolkit Core psdn-sonar eval toolkit code and tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] A prewarmed quality model can be built on the meta device, so every score fails silently and its columns are empty

1 participant