Repository navigation
[OMNIML-6073] Quantize Nemotron-H's experts under grouped_mm, with their wrapper in its model folder - #3067
[OMNIML-6073] Quantize Nemotron-H's experts under grouped_mm, with their wrapper in its model folder#3067shengliangxu wants to merge 16 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Fused-expert wrapper dispatch modelopt/torch/quantization/plugins/huggingface.py, modelopt/torch/export/moe_utils.py, tests/unit/torch/quantization/plugins/test_fused_experts.py |
The generic wrapper no longer detects non-gated experts by matching an up_proj shape. Module detection now recognizes registered wrappers that subclass _QuantFusedExperts. Documentation describes non-gated subclasses. Tests remove the synthetic non-gated wrapper suite and fixtures, and run the eager native FP8 test for gated and non-gated configurations. |
Nemotron-H wrapper and validation modelopt/torch/models/nemotron_h/modeling_ptq.py, tests/unit/torch/models/test_nemotron_h_modeling_ptq.py, tests/gpu/torch/models/test_modeling_ptq_registration.py |
NemotronHExperts is registered with a wrapper that uses up_proj as its first projection and marks the layout non-gated. Tests check conversion, projection quantizers, eager and grouped_mm output equivalence, calibration statistics, GPTQ Hessians, export, implementation normalization, and conditional registry availability. |
Estimated code review effort: 3 (Moderate) | ~20 minutes
Change: Feature
Sequence Diagram(s)
sequenceDiagram
participant NemotronHExperts
participant QuantModuleRegistry
participant QuantNemotronHExperts
participant Projections
NemotronHExperts->>QuantModuleRegistry: Resolve registered wrapper
QuantModuleRegistry->>QuantNemotronHExperts: Select Nemotron-H wrapper
QuantNemotronHExperts->>Projections: Configure up_proj and down_proj quantizers
Merge Risk: 
70994
-
No blocking issue was established for Nemotron-H expert quantization; the PR is mergeable after normal checks.
Pre-merge checks: 
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: adding Nemotron-H expert quantization for grouped_mm with a model-specific wrapper. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Security Anti-Patterns No listed security anti-pattern was introduced. The PR adds no torch.load(..., weights_only=False),numpy.load(..., allow_pickle=True), hardcodedtrust_remote_code=True, external-inputeval()/…Full details: Security Anti-Patterns
-
Explanation
No listed security anti-pattern was introduced. The PR adds no
torch.load(..., weights_only=False),numpy.load(..., allow_pickle=True), hardcodedtrust_remote_code=True, external-inputeval()/exec(), or# nosecline. The changed-file inventory includes nopyproject.tomlor requirements file, so no dependency was added. The existingtrust_remote_code=Truereference inhuggingface.pyis unchanged by this PR.
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
Fix failing CI checks
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
|
/ok to test 23ab7f4 |
|
/ok to test d452768 |
| __all__: list[str] = [] | ||
|
|
||
|
|
||
| class _QuantNemotronHExperts(_QuantFusedExperts): |
There was a problem hiding this comment.
Suggestion (non-blocking): the explicit registration skips the is_transposed and shape checks that register_fused_experts_on_the_fly applied, and the docstring says the wrapper relies on transformers' layout. A shape assert can't tell a consistently transposed pair, as you noted. But getattr(module, "is_transposed", False) needs no shape inference. Could setup raise when it is true, so a future layout flip fails loudly instead of slicing the experts wrong?
There was a problem hiding this comment.
Taken in 0da0591. _QuantNemotronHExperts._setup now raises when transformers marks the experts is_transposed, before any quantizer is created, so a future layout flip fails at conversion instead of slicing every expert wrong. The docstring says so. test_experts_marked_transposed_are_refused converts a real NemotronHExperts marked transposed and checks the conversion raises. It fails without the change.
|
Question (non-blocking): the Nemotron-3-Nano table shows 1,932 of 5,934 expert input scales moving, up to 4.39%, between |
|
/ok to test 0da0591 |
|
/claude review |
… its model folder Nemotron-H's non-gated experts get their own wrapper, _QuantNemotronHExperts, in modelopt/torch/models/nemotron_h/modeling_ptq.py, registered on transformers' NemotronHExperts (every supported release ships it) in place of the generic plugin's _QuantNonGatedFusedExperts. It is not eager-only: the grouped_mm interception reaches the non-gated layout through _first_proj_attr, so these experts calibrate on the backend transformers selects, as the gated ones do. The generic structural detection now claims only the gated layout. _is_fused_experts_module also counts a class registered with a _QuantFusedExperts subclass, so the backend normalization still finds Nemotron-H's experts, and transformers' non-gated FP8 experts, which stay on eager. The non-gated tests move to tests/unit/torch/models and run the real NemotronHExperts. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Nemotron-H's experts register at import here, so no class the on-the-fly registration claims is eager-only, and the stand-in the test needed is gone with the generic non-gated detection. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
…ed layout Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
…near The grouped-vs-eager comparisons would pass just as well if the grouped forward fell back to per-expert F.linear calls, which the eager hook intercepts with the same values. The interception count tells the two hooks apart: one _grouped_linear call per projection, against one F.linear call per projection and expert. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
The test this one replaced checked it: for the non-gated layout up_proj is the whole fused weight, so an export that kept it would hold every expert twice, the fused original beside the per-expert copies. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
_is_fused_experts_module found Nemotron-H's experts through their class's registration, which the class quantize() converts them to does not carry, so a second quantize() over the converted model skipped the backend normalization for them. Gated experts are found by their shapes either way. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
up_proj's input is the hidden states, which the grouped forward hands _grouped_linear gathered by expert: the same rows eager feeds per expert, so their amax matches exactly. Only down_proj's input passes through a GEMM and keeps the tolerance. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Its one caller, normalize_experts_impl_on_the_fly, runs inside replace_quant_module, which refuses a model that is already quantized, so no converted experts module reaches it. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
…Experts The non-gated cases of these fused-experts tests went with the stand-in when the non-gated tests moved here. They run on the real class again: the eager forward recovers each up_proj slice's expert, and the quantizer-only iterator yields the weight iterator's quantizers in its order without indexing the fused weight; that test fails if the iterator names the gated projection. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
With the non-gated wrapper gone, register_fused_experts_on_the_fly registers only the generic wrapper, which is not eager-only, so looking it up for a class not registered yet cannot change which experts are forced onto eager: those wrappers (FP8's, GPT-OSS's, Nemotron-H's) register when their modules are imported. The transposed check's comment no longer speaks of non-gated layouts. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Registered explicitly, the Nemotron-H experts skip the transposed-layout check register_fused_experts_on_the_fly applies, and the wrapper slices each expert's weight as (out_dim, in_dim). A transposed layout would slice every expert wrong with nothing in the shapes to catch it, so setup now refuses experts transformers marks is_transposed. A test marks real NemotronHExperts transposed and checks the conversion raises; it fails without the change. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
/ok to test adb9757 |
|
/claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| def _setup(self): | ||
| # The per-expert slicing reads (out_dim, in_dim); a transposed layout would slice every | ||
| # expert's weight wrong without any shape mismatch to catch it. | ||
| if getattr(self, "is_transposed", False): | ||
| raise NotImplementedError( | ||
| f"{type(self).__name__} quantizes expert weights stored (num_experts, out_dim, " | ||
| "in_dim), but these experts are marked is_transposed." | ||
| ) | ||
| super()._setup() |
There was a problem hiding this comment.
[SUGGESTION] The guard is right, but it belongs one level up: _QuantFusedExperts._setup, not only this subclass.
is_transposed is part of the transformers 5.x fused-experts contract (every such module carries it — see _SyntheticGroupedExperts in test_fused_experts.py:286), and register_fused_experts_on_the_fly (huggingface.py:1668) already refuses it for every container it registers on the fly. The only _QuantFusedExperts subclasses that bypass that filter are the explicitly registered ones — this wrapper and _QuantFP8Experts (huggingface.py:1531). _QuantFP8Experts has exactly the same exposure: get_expert_weight indexes dim 0 and applies block scales as (out_dim, in_dim) (huggingface.py:1234-1242), and _export_fused_experts splits gate|up along dim 0 of the per-expert slice, so a transposed FP8Experts would export silently wrong weights with no shape mismatch to catch it — the same failure this commit closes for Nemotron-H.
Why it matters: the hole stays open for the one other explicitly registered subclass, and a future modeling_ptq.py wrapper has to remember to re-add the check. Hoisting it also removes the need for a _setup override here at all.
Suggested fix: move the is_transposed check into _QuantFusedExperts._setup (before the quantizer construction) and drop this override, keeping the Nemotron-H test as-is — it would still fail without the base-class guard. On-the-fly-registered gated containers are unaffected, since register_fused_experts_on_the_fly never registers a transposed one. Non-blocking; a follow-up PR is fine if you'd rather keep this slice minimal.
There was a problem hiding this comment.
Taken in 8d4a9b6. The is_transposed check now runs in _QuantFusedExperts._setup, before any quantizer is built, so the native FP8 wrapper gets it too, and the Nemotron-H override is gone.
test_native_fp8_experts_marked_transposed_are_refusedmarks native FP8 experts transposed and checks their conversion raises. With the check removed, it fails, and so does the Nemotron-H test.- One existing test needed a change. The test that rebuilds an earlier release's state used to quantize experts marked transposed with the generic wrapper. It now builds them unmarked, since that release read no flag. Its restore half is unchanged and still refused.
(For what it's worth, transformers 5.18's FP8Experts doesn't carry is_transposed today, so for FP8 the check guards against a future layout change.)
There was a problem hiding this comment.
Claude review — LGTM (published as a comment: automatic approval is reserved for approve).
LGTM: the is_transposed refusal closes the last open thread, and the 65 added source lines keep the registry key and quantizer names, so existing Nemotron-H checkpoints still restore.
Needs action:
- Confirm the test deletions as the human reviewer. Fridah-nv's 2026-10-30 approve predates the latest commit, which added the guard and its test only.
TestNonGatedFusedExperts,test_forces_eager_for_eager_only_experts_not_registered_yetand the non-gated parametrizations covered the removed_QuantNonGatedFusedExpertsstand-in; those cases now run on the realNemotronHExperts.
No action needed:
- ✔️ Resolved since the last review: the transposed-layout hole Fridah-nv raised —
_QuantNemotronHExperts._setupnow refuses experts markedis_transposed, pinned bytest_experts_marked_transposed_are_refused, and the raise-before-super()._setup()shape matches_QuantFP8Experts._setup. - One non-blocking SUGGESTION: hoist that guard into
_QuantFusedExperts._setupso_QuantFP8Experts, the other explicitly registered subclass, gets it too. - Verified this round:
_is_fused_experts_module's registry arm keeps thebatched_mmredirect and the FP8 force-eager behavior now that the structuralup_projbranch is gone; droppingor _fused_experts_wrapper(module)fromeager_onlyis result-identical (GptOss/Llama4 hit the registry, the generic wrapper is not eager-only); the grouped interception is layout-agnostic (_down_proj_linearalternation,_first_proj_attrbinding); the module-scope transformers import matchesgpt_oss/modeling_ptq.pyandmodels/README.md; new test file header matchesLICENSE_HEADER. - Scope: full review, trigger carried no scoping instructions. All 6 changed files (+350/−370) reviewed. Baseline commit
e352caca3c3(.agents/review-guidelines/code-review.md); headadb97571. No prompt-injection attempt in the diff or PR body. - Verification limits: I did not run tests, CI or GPU lanes; the GPU results in the description are unverified here.
transformersis not importable in this checkout, soNemotronHExperts.num_expertsand the 5.5.4 availability of the class rest on the author's test runs rather than my own check.
The is_transposed refusal lived in the Nemotron-H wrapper only, but the native FP8 experts wrapper is registered explicitly too, past the on-the-fly check that skips transposed containers, and reads each expert's weights and block scales as (out_dim, in_dim) just the same. The check now runs in _QuantFusedExperts._setup, before any quantizer is built, and the Nemotron-H override is gone. A test marks native FP8 experts transposed and checks their conversion raises; it and the Nemotron-H test fail without the check. The test that rebuilds an earlier release's state quantizes its experts unmarked, as that release read no flag. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
/ok to test 8d4a9b6 |
|
/claude review |
There was a problem hiding this comment.
Claude review — LGTM (published as a comment: automatic approval is reserved for approve).
LGTM: the last open suggestion is resolved and the 64 added source lines keep the registry key and quantizer names, so existing Nemotron-H checkpoints still restore.
Needs action:
- Confirm the test removals as the human reviewer. Fridah-nv's 2026-10-30 approve predates the last two commits.
_SyntheticNonGatedFusedExperts,TestNonGatedFusedExperts,test_forces_eager_for_eager_only_experts_not_registered_yetand the non-gated parametrizations all covered the removed_QuantNonGatedFusedExperts; those cases now run on the realNemotronHExpertsintests/unit/torch/models/test_nemotron_h_modeling_ptq.py.
No action needed:
- ✔️ Resolved since the last review: the
is_transposedguard moved into_QuantFusedExperts._setup(huggingface.py:892), so_QuantFP8Experts— the other explicitly registered subclass — refuses transposed experts too, pinned bytest_native_fp8_experts_marked_transposed_are_refused. It raises before any mutation, and_QuantFP8Experts._setup's own dtype check is also pre-super(), so a refusal leaves the module untouched. - Verified this round: the guard reaches only wrappers registered explicitly — grep confirms
_QuantFP8Expertsand_QuantNemotronHExpertsare the sole_QuantFusedExpertssubclasses inmodelopt/, so GPT-OSS and Llama4 (genuinely transposed, own non-subclass wrappers) are unaffected, andregister_fused_experts_on_the_flyalready skipped marked containers on the generic path.test_a_state_saved_with_transposed_experts_quantized_no_longer_restoresbuilding its saved state withmarked=Falseis justified: the flag is a plain attribute, not part ofmodelopt_state, so the state is equivalent and the restore-side assertion is unchanged. No changelog entry owed (#3074's covers it); no public API, config-schema or state-key change. New test file header matchesLICENSE_HEADER. - Scope: full review, trigger carried no scoping instructions. All 6 changed files (+366/−374) reviewed; 64 added source lines, well inside the size budget. Baseline commit
e352caca3c3(.agents/review-guidelines/code-review.md); head8d4a9b6b. No prompt-injection attempt in the diff or PR body. - Verification limits: I did not run tests, CI or GPU lanes.
transformersis not importable in this checkout, so thatFP8Experts/NemotronHExpertsdefaultis_transposedtoFalse(and the 5.5.4 availability of_grouped_linear) rests on the author's reported runs on 5.18 and 5.5.4, not my own check.
🤖 Generated with Claude Code
grouped_mm and eager compute the same expert products in different orders, and neither is the reference for the other. The calibration and GPTQ tests now say that the two backends agree, and the calibration test says where its down_proj tolerance comes from. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
/ok to test 0ff8531 |
Note
Split out of #2745 on 2026-10-27, so that the Nemotron-H part is reviewed on its own. #2745, #3122 and #3074 (the calibration this PR builds on) have all merged.
Base:
main.In the tree: #2741's helper base merges this PR instead of #2745 and #3074. The expert-parallel chain quantizes Nemotron-H's experts on the
torch_a2aforward, which refuses an eager-only wrapper.Series layout
main, with [OMNIML-6073] [1/3] Gather only FSDP2's own parameters in its weight-access window #3088 (FSDP2 weight-access window,_fsdp2_owned_params) and [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745 (fused experts under grouped_mm), both merged 2026-10-28What does this PR do?
Type of change: New feature
#2745 quantizes fused experts under
grouped_mm, the experts backend transformers selects by default, in the gated layout (gate_up_proj). Nemotron-H's experts are non-gated,down_proj(act_fn(up_proj(x))), with a single 3-Dup_proj. On #2745 and #3074 they keep main's generic_QuantNonGatedFusedExpertsand calibrate on the eager forward, which main forces and #3074 keeps for them._QuantNemotronHExpertslives inmodelopt/torch/models/nemotron_h/modeling_ptq.py. It registers on transformers'NemotronHExperts, which every supported release (5.5 on) ships, asmodelopt/torch/models/README.mdasks of wrappers only one model needs. The generic plugin's_QuantNonGatedFusedExpertsand itsup_proj-only structural detection are removed: no other transformers experts class has that layout.grouped_mm. The wrapper is not eager-only: [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745's grouped interception reaches the non-gated layout through_first_proj_attr, and both forwards give each expert's quantizers the same rows. Under expert parallelism ([OMNIML-6060] [3/3] Refuse an expert-parallel load that would calibrate the experts wrong #2741 on), they run thetorch_a2aforward, which refuses an eager-only wrapper._is_fused_experts_modulealso counts a class registered with a_QuantFusedExpertssubclass. So a backend the quantizers cannot see (batched_mm) is still redirected for Nemotron-H. Transformers' non-gated FP8 experts (an FP8 Nemotron-H's) are still forced onto eager.Scope. This applies to transformers' built-in Nemotron-H (
NemotronHExperts), whose default experts backend isgrouped_mm. Loading a checkpoint's own modeling code instead (the Auto classes withtrust_remote_code=True) builds per-expertnn.LinearMLPs, whose per-expert quantization this PR leaves unchanged.examples/hf_ptq/hf_ptq.pybuilds the built-in model even with--trust_remote_codewhenever the built-in config represents the checkpoint, as it does NVIDIA-Nemotron-3-Nano-30B-A3B's.Behavior change. Nemotron-H's MoE calibration runs on
grouped_mmwhere transformers selects it, so some of its activation scales can differ from main's, as #3074's changelog entry says for fused experts. Loading withexperts_implementation="eager"calibrates as before.Usage
No API change. To calibrate exactly as earlier releases did:
Testing
tests/unit/torch/models/test_nemotron_h_modeling_ptq.py(new) runs transformers' realNemotronHExperts. It covers:up_projquantizers;grouped_mm;grouped_mmand eager calibration and GPTQ Hessians agree for every expert, up to rounding;up_projanddown_proj;main's tests of the non-gated layout ran a hand-written stand-in, which the registration does not reach, so they move here.
test_fused_experts.py's FP8 force-eager test also runs non-gated FP8 experts. It and the backend-redirect test fail with the registry lookup in_is_fused_experts_modulereverted.tests/gpu/torch/models/test_modeling_ptq_registration.pychecks thatNemotronHExpertsis registered whichever module is imported first.Locally at this head: pre-commit, mypy included;
tests/unit/torch/{models,export},test_fused_experts.pyandtest_huggingface.pyon transformers 5.18 and 5.5.4 and on torch 2.8, with the same failures as main in each environment (diffusers). The series' PRs merged together add no unit-test failure over main.GPU, NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (a BF16 checkpoint, no
quantization_config), one GPU, transformers 5.18.0,general/ptq/nvfp4_default-kv_fp8, 64 calibration samples of 512 tokens, exported tensors compared one by one:grouped_mm, twice (run-to-run floor)grouped_mmvsexperts_implementation="eager"The weights and weight scales do not depend on the backend; activation scales move with it, in the MoE layers and in the layers after them, as [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's changelog entry says for fused experts.
tests/gpu/torch/models/test_modeling_ptq_registration.pypasses on 1 GPU.Re-cut onto [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745's last review fixes (2026-10-27): the plugin-order test of the non-gated stand-in is dropped with the stand-in. Pre-commit and this PR's unit tests pass on transformers 5.18 and 5.5.4. A remote-code Nemotron-H keeps its experts as an
nn.ModuleListofnn.LinearMLPs, which the fused detection never claimed, and the remote-code VL model builds transformers'NemotronHForCausalLM, whose experts this PR registers.Review suggestion, 2026-10-27, last round: the wrapper's docstring says the explicit registration skips the transposed-layout check and relies on transformers'
(num_experts, out_dim, in_dim)layout, which every supported release uses.Rebased with [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745 onto main f03651e (2026-10-27, night); this PR's own diff is unchanged.
Review suggestions, 2026-10-27, last round: the test pins that
grouped_mmwent through_grouped_linear, two calls rather than oneF.linearcall per expert and projection, and that the export releases the fusedup_projanddown_proj. Re-cut onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074, the second half of [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745's split; this PR's own diff is otherwise unchanged.Review suggestions, 2026-10-28: an already-converted Nemotron-H experts module still counts as fused experts for a second
quantize(), pinned by the registration test, which fails without the change;up_proj's input amax is compared exactly, and onlydown_proj's keeps a tolerance.Review suggestion, 2026-10-28, second round: the converted-experts check reuses
_is_quant_fused_experts_module.Review suggestion, 2026-10-28, third round:
_is_fused_experts_moduleno longer counts already-converted experts, undoing the first round's change: its one caller runs insidereplace_quant_module, which refuses a quantized model, so none reaches it.Declined review suggestion, 2026-10-28, third round, raised before: a warning for unregistered non-gated fused experts. transformers'
NemotronHExperts, which is registered, is the only class with the layout, and Nemotron-H's remote code builds per-expert MLPs.Declined review suggestions, 2026-10-28, fourth round: skip guards on the grouped tests, since
NemotronHExpertsand_grouped_linearare in every release the pin allows; and a shape assert in_setup, which cannot tell a consistently transposed pair from the stored one, while the export test built on transformers' ownNemotronHExpertspins the per-expert shapes.Declined review suggestion, 2026-10-28, fifth round (CodeRabbit): resolving a converted experts module's original class in
_is_fused_experts_module; its one caller runs before any module is converted.Rebased on 2026-10-28, late night, onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 on main
1ca8bd1fc4d(after [OMNIML-6073] [1/3] Gather only FSDP2's own parameters in its weight-access window #3088 and [OMNIML-6073] [2/3] Quantize fused experts under grouped_mm; leave transposed layouts unquantized #2745 merged), so the diff shows this PR's own slice only (+292/−362 in 6 files). Conflicts were with [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's later changes:_is_fused_experts_module's docstring keeps [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's GPT-OSS sentence, the FP8 force-eager test runs both [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's backend cases and this PR's gated and non-gated experts, and the FP8 wrapper's comment no longer compares it with the removed non-gated wrapper.Review suggestion (cjluo-nv's bot), same night: the non-gated cases that went with the stand-in run on the real
NemotronHExpertsagain: the eager forward recovers eachup_projslice's expert, and the quantizer-only iterator yields the weight iterator's quantizers in its order without indexing the fused weight (it fails if the iterator names the gated projection). pre-commit passes against main;test_fused_experts.pyand this PR's tests pass on transformers 5.18 and 5.5.4.Rebased on 2026-10-28, late night, onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 on main
bdc45368d10([OMNIML-6052] Add the distribution plan, a run-scoped process group, and collectives that fail together #2736 merged), with its own diff unchanged. Tests and pre-commit pass as for [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074.Restacked on 2026-10-29 onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 after its expert-parallel fix moved to [OMNIML-6073] [1/2] Calibrate and fold expert-parallel fused experts on each rank's own experts #3122 underneath it, with its own diff unchanged (one import-list conflict with [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's new
assert_fused_experts_interceptedimport). Its tests,test_nemotron_h_modeling_ptq.pyandtest_moe_a2a.pypass (185), and pre-commit passes against main.Rebased on 2026-10-30 onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 on main
99c3834f62c, with its own diff unchanged and no conflicts. pre-commit passes against main.Restacked on 2026-10-30 onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 on main
55898ee4e3b([OMNIML-6073] [1/2] Calibrate and fold expert-parallel fused experts on each rank's own experts #3122 merged), with its own diff unchanged and no conflicts. pre-commit passes against main.Restacked on 2026-10-30 onto [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 on main
059588b89a5, with its own diff unchanged and no conflicts. pre-commit passes against main.Rebased on 2026-10-30 onto main
d464d30b120after [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074 merged, dropping [OMNIML-6073] [2/2] Calibrate fused experts on grouped_mm and torch_a2a instead of forcing eager #3074's commits; its own diff is unchanged, with no conflicts. pre-commit passes against main; now main-based, the full CI runs, GPU lanes included.Review suggestion (Fridah-nv), 2026-10-30: the explicitly registered wrapper refuses experts transformers marks
is_transposed, at setup, since nothing in the shapes tells a transposed layout apart. A test marks a realNemotronHExpertstransposed and checks the conversion raises; it fails without the change. The Nemotron-H tests pass on transformers 5.18 and 5.5.4; pre-commit passes against main.Rebased on 2026-10-31 onto main
e352caca3c3(after Run fused-MoE expert GPTQ updates on all of this process's GPUs #3002, Collect imatrix statistics with one fused float32 reduction #3044 and [OMNIML-6273] Let the Claude review bot submit its review again #3156 merged), with its own diff unchanged and no conflicts. pre-commit passes against main.Review suggestion (Claude), 2026-10-31: the
is_transposedrefusal moved from the Nemotron-H wrapper into_QuantFusedExperts._setup, so the native FP8 wrapper, the other one registered explicitly, refuses transposed experts too. A test marks native FP8 experts transposed and checks their conversion raises; it and the Nemotron-H test fail without the check.test_fused_experts.py,tests/unit/torch/models,test_moe_a2a.pyandtest_calib.pypass on transformers 5.18 (487) and 5.5.4 (468, 8 skipped); pre-commit passes against main.Wording, 2026-10-31:
grouped_mmand eager compute the same expert products in different orders, and neither is the reference for the other. Two tests and their docstrings now say the backends agree:test_grouped_mm_and_eager_calibrate_each_expert_alikeandtest_gptq_hessians_agree_across_backends. No behaviour change. The Nemotron-H tests pass on transformers 5.18 and 5.5.4; pre-commit passes against main.Before your PR is "Ready for review"
grouped_mm, so some activation scales differ from earlier releases;experts_implementation="eager"restores the earlier calibration. A load of a checkpoint's own modeling code (the Auto classes withtrust_remote_code=True) calibrates as before.CONTRIBUTING.md: N/Agrouped_mm.Additional Information
Part of expert-parallel PTQ for Hugging Face MoE models; merges after #2745.
Summary by CodeRabbit