Repository navigation
[OMNIML-6228] Run the GPTQ Hessian dp-sync test on the pooled gloo workers - #3064
Conversation
The test spawned its own two-process gloo group next to the persistent two-rank pool that `cpu_dist_workers` keeps for the other multi-rank unit tests. It was in flight in all 13 runner stops of the `torch_28` unit job since 10-26 (3.1% of its attempts), always at 97-99% of the run, where the 16 GB runner is nearly full. With a live pool the spawn adds two processes and about 2 GiB of RSS; the test now runs on the pool like the other multi-rank tests. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Use the CPU distributed workers fixture tests/unit/torch/quantization/test_gptq.py |
The test removes the partial and spawn_multiprocess_job imports. It uses cpu_dist_workers to run the helper with token counts (5, 0). |
Priority:
Estimated code review effort: 2 (Simple) | ~8 minutes
Change: Other
Suggested reviewers: kevalmorabia97, cjluo-nv
Merge Risk: 
e6fde
-
This change only reuses the persistent distributed worker fixture for the existing test, with no established product or production risk.
Pre-merge checks: 
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: running the GPTQ Hessian data-parallel synchronization test on the pooled Gloo workers. Docstring Coverage Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Security Anti-Patterns PASS. The pull request changes only tests/unit/torch/quantization/test_gptq.py. It does not change Python files under themodeloptpackage orexamples, so none of the specified security anti-pat…Full details: Docstring Coverage
-
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files.
Full details: Security Anti-Patterns
-
Explanation
PASS. The pull request changes only
tests/unit/torch/quantization/test_gptq.py. It does not change Python files under themodeloptpackage orexamples, so none of the specified security anti-pattern conditions apply to this pull request.
-
Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3064 +/- ##
=======================================
Coverage 79.99% 79.99%
=======================================
Files 661 661
Lines 75580 75580
=======================================
Hits 60458 60458
Misses 15122 15122
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
What does this PR do?
Type of change: CI (fewer runner stops in the unit job)
test_gptq_hessian_dp_syncspawned its own two-process gloo group (spawn_multiprocess_job) next to the persistent two-rank pool thatcpu_dist_workerskeeps for the other multi-rank unit tests. This PR runs the same job on the pool (cpu_dist_workers.run); the test body is unchanged.Why. The
torch_28unit job (16 GB hosted runner) is stopped by the runner ("The runner has received a shutdown signal"), and this test is in flight every time. These are not the stops #2947 fixed (the conv/IsInf ONNX tests, mid-run).multi-version (unit-3.12(torch_28, tf_latest)), attempts since 10-26 00:00 UTCtest_gptq_hessian_dp_syncin flight at the stopEffect. With a live pool the spawn adds two processes that re-import torch and modelopt; the pooled job adds none:
test_dist.py::test_data_parallel(builds the pool), then the dp-sync test; local CPU, two runs eachThe saving applies when the xdist worker that runs the test already has a live pool; if it has none, the pool is built instead of the spawn (the same cost) and stays for the rest of the run.
Testing
tests/unit/torch/quantization/test_gptq.py: 7 passed; run together withtest_dist.py(which shares the pool): passes.torch_28unit job locally. Whether the stop rate drops is seen in thetorch_28attempts of the next days.Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: N/AAdditional Information
Jira: [OMNIML-6228] (flaky-test cleanup, the
torch_28stops).Summary by CodeRabbit