[Lumea-art]

Summary

ExpertsRoutingBypass.batch_size is consumed behind one guard in the staged
executor:

if is_moe_subset and batch_size is not None and batch_size > 0 and execute_forward:

so a value that cannot batch is not an error - it is whatever the guard does
with it:

0, -3        fall straight through -> every expert quantized in one chunk,
             silently. That is the VRAM pressure the option exists to avoid.
"2"          reaches `batch_size > 0` and raises TypeError, deep in the
             pipeline, with no mention of the field that caused it.
integer       survives a JSON round trip as an int and is the supported form.

The first row is the bug: a knob advertised as "reduce VRAM pressure during MoE
quantization" quietly does nothing, and the OOM arrives anyway.

What Changed

• normalize_bypass_batch_size() validates the value and coerces numeric
  strings. Coercion matches ExpertsRoutingOverride, which already coerces
  num_experts_per_tok the same way, so a value that came through JSON works
  instead of being rejected for its type.
• ExpertsRoutingBypass.__post_init__ runs it, so a bad value fails at
  construction, where the user can still fix the typo.
• The staged executor runs it again, because the attribute is writable after
  construction and would otherwise slip past the guard above.

Accepted: None and positive ints, plus numeric strings coerced to int.

Rejected, all with one message: 0, negatives, floats, booleans, non-numeric
strings, and non-scalars. The field help says which is which.

Tests

$ python -m pytest tests/test_moe_bypass_batch_size.py -q
21 passed

Covered:

• None and positive ints pass through unchanged
• numeric strings are coerced ("2" -> 2, " 16 " -> 16)
• 0, -1, -3, 2.0, True, "0", "-3", "abc", a list and a dict raise
• the dict payload the fix is about:
  QuantizeConfig(moe={"routing": {"class": "ExpertsRoutingBypass", "batch_size": ...}})
  coerces "2" and rejects 0
• a value assigned after construction is still rejected by the consumer

Reverting the validation: 15 failed / 6 passed.

End to end on the tiny Qwen3-MoE fixture, through the dict payload:

"2"   quantizes, batch_size == 2
0     ValueError
-3    ValueError
2     quantizes

tests/test_moe_expert_batching.py used to document batch_size=0 as "batching
disabled" and only survived by assigning the attribute after construction; it now
asserts the rejection. That file is 9/9 failing in this environment for an
unrelated telemetry mock issue, and fails identically on unmodified main; the
replacement test passes in isolation.

ruff check is clean. The new test file carries # GPU=-1 so it lands in the
CPU CI bucket.

[ZX-ModelCloud]

Review: fix(moe): reject a routing-bypass batch size that cannot batch

Verdict: Approve with minor comments. The validation is correctly placed at construction time, the error is actionable, and valid configurations (None / positive int) are unchanged.

Verified

• tests/test_moe_bypass_batch_size.py: 14 passed.
• Against unmodified main: 9 failed / 5 passed, so the new tests have real regression value.
• ruff check gptqmodel/quantization/config.py tests/test_moe_bypass_batch_size.py: clean.
• Real dict/JSON path: QuantizeConfig(moe={"routing": {"class": "ExpertsRoutingBypass", "batch_size": ...}}) rejects "2", 2.0, 0, -3, True; accepts 4; to_dict() round-trips.
• Unit tests were run CPU-only with GPUs masked; the ROCm check used only devices 2 and 3 (2× Radeon AI PRO R9700).

Findings

1. tests/test_moe_expert_batching.py:105-111 — test_no_batching_when_batch_size_is_zero still documents batch_size=0 as "batching disabled", which contradicts the new contract. It only survives because it assigns routing.batch_size = 0 after construction, bypassing __post_init__. Please update it to assert rejection (or remove it). If post-construction mutation should also be impossible, consider frozen=True or a defensive check in stage_subset.py.
2. tests/test_moe_bypass_batch_size.py:41-43 — test_moe_config_wraps_the_validation does not actually exercise MoEConfig: the ValueError is raised by ExpertsRoutingBypass(batch_size=0) before MoEConfig(...) is called. A test through QuantizeConfig(moe={"routing": {"class": "ExpertsRoutingBypass", "batch_size": "2"}}) would cover the deserialization path this PR is about.
3. PR description: an integer survives a JSON round-trip as int, not "2"; and when is_moe_subset is true, a string reaches batch_size > 0 and raises TypeError rather than silently falling back to one chunk. Only 0/negative values are silent no-ops. The fix is still valid — just the rationale needs a small correction.
4. Consistency: ExpertsRoutingOverride.__post_init__ coerces numeric strings ("2" → 2), while this change rejects them. If strict rejection is intended, please document it in the field help; otherwise consider coercing for consistency.
5. Nits: the two ValueError branches could share one message; the new test is CPU-only, so adding # GPU=-1 before the imports would put it in the CPU CI bucket; the SPDX year is 2025 while new files use 2026.

Notes

• tests/test_moe_config.py::TestMoEConfig::test_moe_routing_bypass_batch_size got past fixture setup here but OOM'd during the clone stage on 2×32GB ROCm, so the end-to-end valid-path run could not complete on this host.
• tests/test_moe_expert_batching.py is 9/9 failing on both main and this PR due to an unrelated telemetry mock issue, so it is not a regression from this change.
• The Ruff CI run is currently action_required (fork approval), so there are no CI results yet.

[Lumea-art]

Review: fix(moe): reject a routing-bypass batch size that cannot batch

Verdict: Approve with minor comments. The validation is correctly placed at construction time, the error is actionable, and valid configurations (None / positive int) are unchanged.

Verified

• tests/test_moe_bypass_batch_size.py: 14 passed.
• Against unmodified main: 9 failed / 5 passed, so the new tests have real regression value.
• ruff check gptqmodel/quantization/config.py tests/test_moe_bypass_batch_size.py: clean.
• Real dict/JSON path: QuantizeConfig(moe={"routing": {"class": "ExpertsRoutingBypass", "batch_size": ...}}) rejects "2", 2.0, 0, -3, True; accepts 4; to_dict() round-trips.
• Unit tests were run CPU-only with GPUs masked; the ROCm check used only devices 2 and 3 (2× Radeon AI PRO R9700).

Findings

1. tests/test_moe_expert_batching.py:105-111 — test_no_batching_when_batch_size_is_zero still documents batch_size=0 as "batching disabled", which contradicts the new contract. It only survives because it assigns routing.batch_size = 0 after construction, bypassing __post_init__. Please update it to assert rejection (or remove it). If post-construction mutation should also be impossible, consider frozen=True or a defensive check in stage_subset.py.
2. tests/test_moe_bypass_batch_size.py:41-43 — test_moe_config_wraps_the_validation does not actually exercise MoEConfig: the ValueError is raised by ExpertsRoutingBypass(batch_size=0) before MoEConfig(...) is called. A test through QuantizeConfig(moe={"routing": {"class": "ExpertsRoutingBypass", "batch_size": "2"}}) would cover the deserialization path this PR is about.
3. PR description: an integer survives a JSON round-trip as int, not "2"; and when is_moe_subset is true, a string reaches batch_size > 0 and raises TypeError rather than silently falling back to one chunk. Only 0/negative values are silent no-ops. The fix is still valid — just the rationale needs a small correction.
4. Consistency: ExpertsRoutingOverride.__post_init__ coerces numeric strings ("2" → 2), while this change rejects them. If strict rejection is intended, please document it in the field help; otherwise consider coercing for consistency.
5. Nits: the two ValueError branches could share one message; the new test is CPU-only, so adding # GPU=-1 before the imports would put it in the CPU CI bucket; the SPDX year is 2025 while new files use 2026.

Notes

• tests/test_moe_config.py::TestMoEConfig::test_moe_routing_bypass_batch_size got past fixture setup here but OOM'd during the clone stage on 2×32GB ROCm, so the end-to-end valid-path run could not complete on this host.
• tests/test_moe_expert_batching.py is 9/9 failing on both main and this PR due to an unrelated telemetry mock issue, so it is not a regression from this change.
• The Ruff CI run is currently action_required (fork approval), so there are no CI results yet.

All five addressed in 27558d3.

1. test_moe_expert_batching now asserts the rejection instead of mutating to 0.
   I also re-validate in the staged executor rather than freezing the dataclass -
   freezing would break ExpertsRoutingOverride, which assigns its own field in
   post_init.

2. The MoEConfig test goes through QuantizeConfig(moe={"routing": {...}}) now,
   so it covers the dict payload rather than the constructor.

3. You are right about the rationale. Corrected: 0 and negatives fall through the
   batch_size > 0 guard silently, a non-numeric string raises TypeError, and a
   numeric string only "worked" because nothing checked it. The description is
   updated.

4. Chose coercion for consistency: "2" -> 2, matching num_experts_per_tok. Zero,
   negatives, floats, bools and non-numeric strings still raise, with one shared
   message, and the field help says which is which.

5. GPU=-1 marker, SPDX 2026, single message - all done.

   tests/test_moe_bypass_batch_size.py 21 passed
   reverting the validation 15 failed / 6 passed
   dict payload end to end "2" -> 2 (quantizes), 0 and -3 raise
   ruff clean

test_moe_expert_batching.py is 9/9 failing here too, same telemetry mock issue
you saw on main. The one test I added to it passes in isolation.