Repository navigation
fix: keep the prewarmed quality models off the meta device (#288) - #289
Merged
Merged
Conversation
The evaluator prewarms DNSMOS/UTMOS/SQUIM on a background thread and joined it only after the ASR factory ran. transformers implements from_pretrained(low_cpu_mem_usage=True) by opening init_empty_weights(), which patches nn.Module.register_parameter on the class — process-wide, not per thread — so a quality model constructing inside that window got parameters on the meta device, loaded real weights as a silent no-op, and then failed every score, leaving the utmos column empty with only a debug-level trace. Two layers, matching the issue's either/or: - the prewarm thread is now joined before the model factory runs, so the constructions cannot overlap the window (joining again on later iterations is a no-op); - the UTMOS and SQUIM loaders refuse a constructed model whose parameters sit on the meta device, recording the reason so it reaches mos_warnings/quality_warnings instead of reporting a healthy predictor — the backstop for any other construction route, and it covers the structurally identical SQUIM case the issue lists as untested. The failure path also clears the cached model reference so a refused predictor is never returned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #288. The evaluator prewarms the quality models (DNSMOS/UTMOS/SQUIM) on a background thread and joined it only after the ASR model factory ran. transformers implements
from_pretrained(..., low_cpu_mem_usage=True)by openinginit_empty_weights(), which replacesnn.Module.register_parameteron the class — process-wide, not per thread. A UTMOS predictor constructed by the prewarm thread inside that window lands its parameters on the meta device; loading real weights into meta tensors is a silent no-op, so_get_utmos()reported a healthy predictor whose every score then failed on first forward (Tensor on device meta is not on the expected device cpu!) and was discarded — an emptyutmoscolumn whilednsmos_ovrl/squim_pesq/snr_dbpopulate from the same audio.Both remedies the issue names, as two layers:
_model_factoryruns, so quality-model construction can never overlap theinit_empty_weights()window. Joining an already-finished thread is a no-op, so later loop iterations pass straight through; the old post-factory join is gone._utmos_error/_squim_error— so it reachesmos_warnings→quality_warningsin the artifact (the [Bug] A UTMOS fetch failure empties the column with no reason recorded, bypassing the mos_metrics_unavailable marker #245 reporting path) rather than a debug line. This is the backstop for any other construction route, and covers the structurally identical SQUIM case the issue lists as not checked. The failure path clears the cached model reference so a refused predictor can never be returned. DNSMOS is ONNX-based with no torch parameters; the guard tolerates objects withoutparameters().Testing
TestPrewarmJoinsBeforeTheModelFactory: end-to-end throughrun_evaluationwith a delayed prewarm — fails against the pre-fix code with the exact reported race (factorybefore the prewarm events) and passes with the fix.TestMetaDevicePredictorRefused: the guard raises for a meta-device module and names the window; CPU modules and parameterless objects pass; a meta UTMOS/SQUIM is recorded unavailable (_get_*returns None, error mentions the meta device and the issue); end-to-endcompute_mos_metricscarriesutmos_unavailable: ... meta device ...inmos_warnings.init_empty_weights()context (transformers.integrations.accelerate) has parameters on['meta']and is refused by the guard; a normally-built module passes.test_quality_models.py(29), plustest_language_forwarding.py,test_script_check.py,test_audio_quality.py: 91 passed.ruff check,ruff format --checkclean.