Repository navigation
[OMNIML-6180] Add a nightly test marker and use it for the example PR skips - #2982
Conversation
…on PRs nightly tests are skipped unless --run-nightly is given, like manual and release. The unit, GPU and example workflows pass it off pull requests, and the llm_qat and torch_onnx example tests use it instead of checking GITHUB_REF themselves (the torch_onnx cases keep their test ids). Local runs now skip those cases unless --run-nightly is given. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files. (5 skipped: 5… | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main changes: adding a nightly test marker and applying it to example tests previously skipped in pull requests. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Security Anti-Patterns PASS. The PR changes only test-selection code in tests/examples,tests/conftest.py, workflows, documentation, and pytest configuration. The added Python code contains no `torch.load(..., weights_o…Full details: Security Anti-Patterns
-
Explanation
PASS. The PR changes only test-selection code in
tests/examples,tests/conftest.py, workflows, documentation, and pytest configuration. The added Python code contains notorch.load(..., weights_only=False),numpy.load(..., allow_pickle=True), hardcodedtrust_remote_code=True, external-inputeval()/exec(), or# nosec. Thepyproject.tomlchange registers a pytest marker and adds no PIP dependency.
-
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files. (5 skipped: 5 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2982 +/- ##
==========================================
- Coverage 79.50% 79.10% -0.40%
==========================================
Files 660 660
Lines 74771 74771
==========================================
- Hits 59448 59150 -298
- Misses 15323 15621 +298
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
What does this PR do?
Type of change: CI (test selection)
Step 1 of 2 of the follow-up to a review note on #2925 ("make it generic like unit and gpu tests:
pytest.mark.nightlyfor nightly ones andpytest.mark.manualfor the ones we may want to skip in nightly CI too").Test selection by tier was done three different ways: the unit and GPU workflows pass
--run-manualoff pull requests, somanualmeans "also nightly" there; the example workflow passes nothing, somanualmeans "never in CI" intests/examples(the two eagle cases intest_eagle.pynever run); and thellm_qatandtorch_onnxexample tests each decided "PR or not" themselves fromGITHUB_REF(two copies of the same line). There was no way to say "nightly only" in a test.This PR adds the
nightlymarker next tomanualandreleaseand uses it for the two example tests:pyproject.tomlregisters the marker;tests/conftest.pygets--run-nightlyand one entry in the existing marker/flag table.--run-nightlywith--run-manualoff pull requests; the example action passes it whengithub.refis not apull-request/Nref (the same testhf_ptquses for-n 4).test_llm_qat.py: the three backend cases that only run nightly arepytest.mark.nightlyparams; theGITHUB_REFcheck and itsosimport are gone.test_torch_quant_to_onnx.py: the three swinv2 cases that only run nightly arepytest.mark.nightlyparams of one flat(model_key, qformat)parametrization with the same ids and order as before; the in-testpytest.skiponGITHUB_REFis gone.CONTRIBUTING.mdnames the two marks.Behavior: pull-request runs and nightly runs select exactly the same tests as before. Local runs of those two example files now skip the six nightly cases unless
--run-nightlyis given (they used to run everything becauseGITHUB_REFis unset); that is whatmanualalready does for its tests.Step 2 (separate PR, needs the maintainers' agreement) makes the meaning uniform: the unit and GPU tests that are marked
manualtoday but run nightly becomenightly, the nightly workflows stop passing--run-manual, andmanualmeans "never in CI" everywhere. It will be guarded by a before/after comparison of the nightly selection.Usage
pytest tests/examples/llm_qat --run-nightly # also run the nightly-marked casesTesting
tests/examples/{llm_qat,torch_onnx,speculative_decoding}(156 tests) on main (67e7ac3) and on this branch under three setups, comparing which tests are not skipped by a mark: a PR run (GITHUB_REF=refs/heads/pull-request/7, no flags): 147 selected before and after, identical; a nightly run (GITHUB_REF=refs/heads/main,--run-nightlyafter): 153 before and after, identical; a local run (noGITHUB_REF, no flags): 153 before, 147 after, the difference being exactly the six nightly cases (test_qwen3_qat_nvfp4[ddp],test_qwen3_qat_nvfp4[deepspeed],test_qwen3_qad_nvfp4[deepspeed],test_torch_onnx[swinv2_tiny-int8],[swinv2_tiny-mxfp8],[swinv2_tiny-auto]). All 156 test ids are unchanged. The oldtorch_onnxskip happened inside the test, so for the "before" PR run those three ids were removed by hand.--strict-markerscollection passes with the new marker;--run-nightlyappears inpytest --help; a CPU unit file runs with--run-nightly --run-manual.llm_qat13 passed / 4 skipped,torch_onnx20 / 6,speculative_decoding109 / 4, identical to the last run before the change ([OMNIML-6180] Run the small example lanes of a job together #2924's head). Not run: the nightly side (--run-nightlyoffpull-request/*refs); the first nightly after the merge shows it (the last one before:llm_qat16 passed / 1 skipped,torch_onnx23 / 3).tests/conftest.py, and the example lanes exercise it.Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).--run-nightly(see Behavior)CONTRIBUTING.md: no new dependencyAdditional Information
Jira: [OMNIML-6180]. Follow-up (step 2): re-mark the unit and GPU
manualtests that run nightly asnightly; 24 marks in 10 files today (unit 17 in 4 files, gpu 6 in 5, gpu_megatron 1 in 1); the 2 intests/examples(the eagle cases that never run in CI) staymanual.