Skip to content

Scope RunInference model_identifier annotation and preserve ModelManager metric_tag across model updates - #40671

Draft
AMOOOMA wants to merge 2 commits into
apache:masterfrom
AMOOOMA:fix-model-identifier-and-metric-tag
Draft

AMOOOMA wants to merge 2 commits into
apache:masterfrom
AMOOOMA:fix-model-identifier-and-metric-tag

Conversation

@AMOOOMA

@AMOOOMA AMOOOMA commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR addresses two related issues with model_identifier and ModelManager metrics during runtime side-input model updates in RunInference:

  1. Scope model_identifier transform annotation strictly to BeamML_RunInference:

    • Previously, RunInference.annotations() attached model_identifier to the outer RunInference composite PTransform, which caused all nested sub-transforms (such as WatchFilePattern side-input polling transforms, BatchElements, preprocess/postprocess transforms, etc.) to inherit the model_identifier annotation. Runners that group steps by model_identifier to look up num_loaded_models_<model_identifier> would then fail to resolve the metric on steps that never execute _RunInferenceDoFn.
    • Remove model_identifier from RunInference.annotations() and instead attach transform_annotations(model_identifier=self._model_tag) directly around BeamML_RunInference in RunInference.expand().
  2. Stick to reporting the original model_identifier (metric_tag) in ModelManager across runtime side-input model updates:

    • When RunInference updates its model via side input (ModelMetadata), _RunInferenceDoFn._load_model acquires the new model under the updated model_id tag (self._cur_tag) so ModelManager caches/evicts the new model version separately from the previous version, while the BeamML_RunInference transform annotation remains fixed at construction time (self._model_tag / model_identifier).
    • Pass metric_tag=self._model_tag through _SharedModelWrapper to ModelManager.acquire_model(..., metric_tag=...), and map metric_tag to the active version tag in ModelManager / ResourceEstimator so num_loaded_models_<metric_tag> and memory_estimate_mb_<metric_tag> strictly reuse and report under the original metric_tag rather than the new side-input version tag.

Testing

  • Added test_run_inference_only_attaches_model_identifier_to_inference_step in base_test.py.
  • Updated test_run_inference_dofn_model_update_acquires_and_releases_matching_tag in base_test.py to verify metric_tag preservation across model updates.
  • Added test_model_manager_metrics_with_metric_tag in model_manager_test.py verifying that num_loaded_models_<metric_tag> and memory_estimate_mb_<metric_tag> report exclusively under metric_tag.
  • Updated test_monitoring_outlet_clears_model_identifier_annotation in vertex_ai_model_monitoring_v2_test.py.

@AMOOOMA
AMOOOMA force-pushed the fix-model-identifier-and-metric-tag branch from 8f9def8 to ab4245c Compare October 19, 2026 22:43
@codecov

codecov Bot commented Oct 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 43.58974% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.00%. Comparing base (76f6e6d) to head (7ddd3f0).
⚠️ Report is 241 commits behind head on master.

Files with missing lines Patch % Lines
...s/python/apache_beam/ml/inference/model_manager.py 0.00% 20 Missing ⚠️
sdks/python/apache_beam/ml/inference/base.py 89.47% 2 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40671       +/-   ##
=============================================
+ Coverage     56.16%   80.00%   +23.83%     
=============================================
  Files          1124      490      -634     
  Lines        178336    82827    -95509     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    66268    -33902     
+ Misses        75645    16559    -59086     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 80.00% <43.58%> (+0.45%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant