feat: multi-GPU quantization improvements -- batched device-local sea… by avtc · Pull Request #2357 · intel/auto-round

avtc · 2026-09-14T18:37:41Z

Description

Multi-GPU tuning for large MoE models (validated end-to-end on a 300B-A21B MoE, 8x3090, W4A16): device-local batched quantization searches, calibration-data pool placement, atomic expert placement with per-device grouped-experts plans, and a memory-model-driven auto pick of the experts implementation. Plus four upstream bugs the new instrumentation surfaced (fixes benefit single-GPU users too).

Features

Batched device-local quantization searches (iters=0 OptRTN and iters>0 SignRound V2). Scale searches for modules with identical shape and quantization config stack into one call per device; when a block's weights span several GPUs, the devices run their batches in parallel. In the iters=0 lane stacked batches may also run on other CUDA GPUs with free memory, including GPUs outside the device map.
--calibration_data_device auto | off | cpu | cuda:1,cuda:2 | 1,2 (CLI and Python API). Per-block placement of the calibration input/output pools: they stay on the primary GPU when they fit beside the activation window, and are spread across the other GPUs otherwise, placed so that every GPU's peak VRAM during tuning stays close to the same level. Placement decisions come from a memory model of the block (activation window, collection retention, tuning state, and activation estimates computed per GPU from the modules whose weights live there).
Best-params snapshots next to their weights. On every improving iteration the snapshot was copied to the single cache device (the primary GPU by default, host with --low_gpu_mem_usage) - a cross-device copy for each module hosted on another GPU, with all snapshot bytes concentrated on that one device. Each parameter's snapshot now stays on the device that already hosts it: every copy stays local, and the snapshot footprint spreads across the hosting GPUs like the weights themselves. Under memory pressure it falls back to host.
Reference outputs on the loss device. The fp reference pool was previously gathered across GPUs on every tune iteration; it now moves once to the loss device before the loop when it fits, eliminating the per-iteration gather.
FP input pool released. The fp input pool is released at tune-loop start, freeing its VRAM for the loop.
Background shard writes. Shard saves and the AR_RESUME_DIR marks are written in the background, overlapping with the next block's tuning.
Atomic expert placement. The data-driven allocator places whole experts on single devices; chunks split by whole experts when no single device has budget. Previously the projections of one expert landed on different GPUs because expert detection matched only nn.ModuleList containers, and the MoE unfuse path builds a plain container.
Per-device grouped-experts plans. The grouped-GEMM fast path ([high risk]Speedup 2.5-6X moe tuning and pin ngram to cpu #2303) required all active experts of a layer on one device; experts are now grouped by home device and run one stacked grouped GEMM per group, so the fast path stays active when a layer's experts span several GPUs.
AR_MOE_EXPERTS_IMPL=auto (new value of the existing env var, now the default). Predicts the per-device VRAM of grouped tuning before the first MoE block and selects linear_loop for the run when grouped exceeds the free memory; explicit values take precedence.

Upstream fixes (single-GPU users benefit too)

layer_config rebuilt the full compression-plan dict on every read (~50 ms on a 300B plan); the pack loops read it once per module, which cost ~27 s per MoE block. The reads are hoisted.
Immediate pack ran after the block moved to CPU; it now runs while the block is still on the GPUs, matching the model-free path. Packed artifacts still land in host RAM.
With --enable_torch_compile, dynamo/FX caches retain tracing artifacts across blocks (~5 GiB in the block-3 OOM census on the 300B lane) until the recompile limit forces an eager fallback. Blocks wrap fresh modules, so no compiled graph is reused across blocks; the block-boundary dynamo reset (when compilation is enabled) frees the dead caches and keeps retention from accumulating.
The block memory estimator crashed on V2-wrapped modules (act_bits was read off the wrapper instead of the wrapped layer); it now reads quantization attributes through orig_layer.

Instrumentation (opt-in)

AR_PERF_COUNTERS emits a per-block phase summary; an OOM tensor census with holder attribution is installed by the CLI; pool-placement decisions are logged. The three upstream fixes were found with these tools.

Measured results (300B-A21B MoE, 80 blocks; iters=0 lane = optimized RTN, symmetric scheme; iters=20 lane = SignRound V2)

Per-block phase walls on 8x3090, serial baseline (with fixes unlocking iters=20) vs this branch:

Phase	Before	After	Speedup
quant scale search (iters=0)	61.4 s	9.4 s	6.5x
wrap scale search (iters=20)	80 s	7.7 s	10.4x
tune loop (iters=20), block 1	62.8 s	38.1 s	1.6x
tune loop (iters=20), blocks 2+	99.3 s	47.7 s	2.1x
pack (layer_config fix)	29 s	1.2 s	24x
move block to CPU (pack order)	4.5 s	1.2 s	3.8x

Full runs, same rig and model:

Configuration	main	this branch	Speedup
8x3090, iters=0	120 s/block, 2.7 h	26 s/block, 34 min	4.6x
8x3090, iters=20	CUDA OOM in the first MoE block	112 s/block, 2.5 h	unlocked
4x3090, iters=0	CUDA OOM in block 2	28.5 s/block, 38 min	unlocked
4x3090, iters=20	-	166 s/block, 3.7 h (auto linear_loop)	-

VRAM stays flat across blocks on the branch's full runs. On 5 and more GPUs the grouped-experts path stays active.

Numerical parity: iters=0 outputs are bit-identical to the serial path; at iters=20 the tuning losses match the serial path within run noise (7.305e-11 to 2.670e-11 batched, 2.671e-11 serial).

New knobs

--calibration_data_device auto | off | cpu | cuda:1,cuda:2 | 1,2 (CLI and Python API): calibration pool placement
AR_MOE_EXPERTS_IMPL=auto: new value of the existing env var, picks grouped vs linear_loop by predicted memory fit
AR_DISABLE_BATCHED_SEARCH=1: serial search fallback
AR_DISABLE_CROSS_DEVICE_SEARCH=1: keeps search batches on the devices hosting the weights instead of optimized search placement
AR_SEARCH_BATCH_GB: VRAM cap for stacked search batches
AR_PERF_COUNTERS=1: per-block phase timing

All documented in docs/environments.md and its CN counterpart.

Tests

About 3,100 lines of new CPU-runnable tests: search batching and dispatch, pool placement, grouped-experts plans and atomic placement (including a CUDA-gated two-device test), the auto linear_loop pick, pool release and writer ordering.

Type of Change

New feature (performance) + bug fix

Related Issues

Relates to #2303 (keeps the grouped-experts fast path active when experts span several GPUs)

Checklist Before Submitting

My code has been tested locally.
Documentation has been updated as needed.
New or updated tests are included where applicable.
The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

AutoRoundBot · 2026-09-14T22:12:12Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-09-14T22:12:23Z

Azure Pipelines successfully started running 1 pipeline(s).

AutoRoundBot · 2026-09-15T09:59:37Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-09-15T09:59:46Z

Azure Pipelines successfully started running 1 pipeline(s).

…eserved on the tune device Rework of the huge-layer snapshot parking along two axes. ONE HOME, STRUCTURAL (replaces the separate utils/snapshot_parking.py module): compressors/utils.py now carries collect_best_params_local and snapshot_best_params byte-identical to the multi-GPU improvements PR (intel#2357) plus this lane's ladder in the same file, so the two snapshot implementations meet in one place at merge time -- git, not a kit note, forces the analysis. SAME-GPU SLOT WITH WINDOW ACCOUNTING (the 24 GB expectation): the snapshot is now a BestParamsSlot reserved BEFORE the tune loop. The device ladder prefers the layer's own device whenever the snapshot fits beside the row-window floor (free - snapshot >= 1024 rows x in_features x fp32 x ~6 live arrays, the row-block machinery's own anchors), then an idle CUDA peer (10% headroom), then the host. Reserving upfront means row_block_bounds' per-forward free-memory probe sees the reduced pool from iteration 0: windows shrink in favor of keeping the snapshot, which removes both the multi-second host copy per improving iteration and the transient OOM of windows sized on the pre-snapshot pool. Refreshing an on-device slot copies into existing buffers; reservation failure degrades to the historical host path. quantize_layer_outside_block reserves the slot for layers above the chunking budget; small layers keep the exact cache_device path. _best_param_device is removed (the slot subsumes it); its contract tests move to the ladder tests. Signed-off-by: avtc <avtc@users.noreply.github.com>

…-> host) Adopting the streaming branch's fallback philosophy, one step further: the clone attempt itself is the test. The snapshot first duplicates beside the weights; on failure it falls through to the freest visible accelerator of the same type (most free first, every candidate still guarded by a real attempt-and-catch at clone time); when no accelerator can hold it, it parks on the host with a WARNING. The successful route is sticky per block (retried first on later improving iterations) and the host is terminal - no mid-tune flip-flopping back onto accelerators. The activation-floor routing is gone from the block path: placement no longer consults any prediction, so the estimator's tune-loop role ends (placement keeps it). act_floor_bytes stays in the signature for call-site compatibility and no longer routes anything. This also answers the observed gap: the floor-based ladder routed the predictor tree to an idle peer by prediction (its WARNING paths never fired in any real run), while the clone-time catches inherited from intel#2357 only covered the local-copy failure. Now every placement decision is made by attempting it. Signed-off-by: avtc <avtc@users.noreply.github.com>

avtc · 2026-09-27T11:02:53Z

rebased on latest main, tested (with fixes) on qwen3.8-27B iters=0, iters>0, neuqi

AutoRoundBot · 2026-09-27T11:42:56Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-09-27T11:43:06Z

Azure Pipelines successfully started running 1 pipeline(s).

wenhuach21 · 2026-09-30T02:31:37Z

I just noticed this PR. Really nice work! The speedup looks very promising.

I have a few questions, though:

Where does the speedup come from?

Based on the PR description, I'm not quite sure where the speedup comes from. The only change that seems to directly reduce the time cost is “Best-params snapshots next to their weights,” but this is already the default behavior in AutoRound, as far as I know.
How does this PR affect RAM and VRAM usage?
Per-device grouped-experts plans

If I remember correctly, the current behavior in main branch is that if the experts can fit on a single GPU, they are placed on one GPU; otherwise, they are split across multiple GPUs.

avtc · 2026-09-30T05:53:11Z

@wenhuach21 Thanks! Answers to the three questions:

1. Where the speedup comes from

The phase table maps each term to its source. The largest terms:

quant scale search 61.4 s -> 9.4 s and wrap scale search 80 s -> 7.7 s come from the batched device-local searches: modules with identical shape and quantization config stack into one batched call per device, the devices run their batches in parallel, and in the iters=0 lane batches may also borrow idle GPUs outside the device map. These phases dominate the iters=0 wall, which is where the 120 s -> 26 s per block comes from.
tune loop 99.3 s -> 47.7 s comes from three tune-lane changes together: the reference pool moves once to the loss device before the loop (it was gathered across GPUs every iteration), the fp input pool is released at loop start, and the best-params snapshot is taken on the device hosting its weight.
pack 29 s -> 1.2 s is the layer_config fix (main rebuilds the compression-plan dict on every read; the pack loops read it once per module).
block-to-CPU move 4.5 s -> 1.2 s is the pack-order change (packing runs while the block is still on the GPUs).

On the snapshot point: main's default cache device is the primary GPU (compress.py: cache_device = cpu if low_gpu_mem_usage else device_manager.device), so with a block spread over several GPUs every improving iteration copied each module's params to that one device — a cross-device copy for modules hosted elsewhere, and all snapshot bytes concentrated on the primary GPU. This branch duplicates each parameter on the device that already hosts it: every copy stays local, the snapshot footprint spreads across the hosting GPUs like the weights themselves, and the unwrap copy-back is local. On a single GPU the two behave the same, so it is a multi-GPU tune-lane improvement and a modest one next to the search batching.

2. RAM and VRAM

The calibration pools are placed by a per-block memory model (activation window, collection retention, tuning state, per-GPU activation estimates); pools stay on the primary GPU when they fit beside the activation window and spread across the other GPUs otherwise, placed so each GPU's predicted peak stays close to the same level. Measured full runs hold VRAM flat across blocks. --calibration_data_device cpu|off|<devices> overrides the automatic choice.
Stacked search batches are capped by AR_SEARCH_BATCH_GB; a mid-batch OOM falls back to per-module searches on the same device.
The best-params snapshot falls back to host when a local copy fails.
With --enable_torch_compile, the block-boundary dynamo reset frees tracing caches that main retains across blocks (~5 GiB in the block-3 census that motivated the fix).
AR_MOE_EXPERTS_IMPL=auto predicts the grouped-tuning VRAM before the first MoE block and picks linear_loop for the run when grouped exceeds the free memory.
Host RAM stays comparable to main: packed artifacts land in host RAM on both, and the background shard writes change timing rather than footprint.

3. Per-device grouped-experts plans

Your model of main matches the intent for models whose experts sit in an nn.ModuleList. The MoE unfuse path builds a plain container, so expert detection missed those and split them at the projection level - each expert's gate/up/down projections landed on different GPUs. With a layer split that way, the grouped-GEMM fast path (#2303), which wants a layer's active experts together on one device, fell back to linear_loop.

This branch keeps the fit-driven spreading and changes the unit: whole experts place atomically on one device, and chunks split at whole-expert boundaries when the unit exceeds each device's remaining budget. The experts of a layer are then grouped per home device, and each group runs its own stacked grouped GEMM - the fast path stays active per group when the layer spans several GPUs.

At run time the layer works like this: the router output is flattened into (token, top_k) pairs and sorted by expert id, so each active expert owns a contiguous range of rows. For a projection slot (gate/up/down), the rows of the experts hosted on one device travel to that device, pass through the activation-quantization wrapper, and run in a single grouped GEMM covering that device's experts - each expert keeps its own weights and its rows inside the call, addressed by per-expert offsets. The outputs return to a shared buffer on the layer's device, written back at their sorted positions. Gate and up run this way, the activation is applied, down runs the same way, and the final combine - multiply by the routing weights, sum over top_k - is the same code as the single-device path. One guard: experts sharing a group must agree on quantization scheme and weight layout; a layer that mixes them falls back to the per-expert loop.

wenhuach21

Really appreciate this PR, though I don’t fully understand it yet!

Please expect a slower response from most of us, as we’ll be taking a week-long national holiday starting tomorrow.

avtc · 2026-10-18T20:54:25Z

Follow-up:

I pushed the changes from my reply as one commit on top of the reviewed head.
It also reworks how the batched searches are sized and placed: chunk sizes now come from live free-memory probes and each search's transient footprint, modules stay on their home device when it has room and only the overflow goes elsewhere, and a chunk that turns out too big for its device splits at execution time.
The NeuQQI anchors now run as one stacked call per same-shape batch, with a VRAM cap per chunk.
Two tuning-loop fixes are in there as well, both found while validating multi-GPU MoE runs: a tune-loss tensor stayed on the first GPU while the loss itself ran on another device (multi-GPU runs crashed on the first MoE block), and a one-shot hook that records the routing shape attached to the expert weights instead of the router and just sat there - its presence alone made the fast grouped-experts path decline, so MoE blocks tuned through the slow per-expert loop. The tensor now follows the loss, and the hook attaches to the router only.

I'll rebase onto main when requested or after approval.

Copilot

🟡 Changes recommended

Unresolved batching correctness, memory-policy, and fallback errors can alter quantized weights or abort supported runs.

7 open issues

What changed in this PR

Improves AutoRound’s multi-GPU quantization throughput and memory use, extending the grouped-experts work from #2303.

Changes:

Adds batched searches and device-aware calibration-pool handling.
Improves expert placement, grouped execution, snapshots, and memory diagnostics.
Adds regression tests and matching English/Chinese documentation.

File	Description
test/unit/test_cpu/quantization/test_release_pool_inplace.py	Tests shared-pool release.
test/unit/test_cpu/quantization/test_pull_pool_if_fits.py	Tests pool-pull memory decisions.
test/unit/test_cpu/quantization/test_one_deep_writer.py	Tests background-write ordering and snapshots.
test/unit/test_cpu/quantization/test_neuqi.py	Tests compilation concurrency and batching parity.
test/unit/test_cpu/quantization/test_auto_linear_loop.py	Tests automatic expert implementation selection.
test/unit/test_cpu/quantization/test_attach_pool_placement.py	Tests calibration placement policies.
test/unit/test_cpu/modeling/test_grouped_experts_plans.py	Tests grouped routing and atomic placement.
test/unit/common/utils/test_device.py	Updates device-memory and compilation tests.
README.md	Documents calibration placement.
README_CN.md	Mirrors README documentation in Chinese.
docs/step_by_step.md	Adds calibration-placement usage.
docs/step_by_step_CN.md	Mirrors usage guidance in Chinese.
docs/environments.md	Documents search and memory controls.
docs/environments_CN.md	Mirrors environment documentation in Chinese.
auto_round/wrapper.py	Adds deferred searches and shared write-back helpers.
auto_round/utils/resume.py	Adds pool-container snapshots for asynchronous saves.
auto_round/utils/perf_counters.py	Adds optional phase timing.
auto_round/utils/oom.py	Adds memory-failure diagnostics.
auto_round/utils/device.py	Improves memory estimates and atomic allocation.
auto_round/utils/device_manager.py	Scales compilation limits by device count.
auto_round/modeling/fused_moe/grouped_experts.py	Adds per-device projection groups.
auto_round/export/formats/base.py	Instruments immediate packing.
auto_round/export/export_to_autoround/export.py	Records packing-phase timings.
auto_round/envs.py	Registers search and timing controls.
auto_round/data_type/utils.py	Exposes deferred scale-search callables.
auto_round/data_type/neuqi.py	Guards compilation setup and improves warnings.
auto_round/context/compress.py	Stores calibration-placement policy.
auto_round/compressors/utils.py	Adds device-local snapshots and packing counters.
auto_round/compressors/diffusion/tuning_cache.py	Explains rejected cache signatures.
auto_round/compressors/base.py	Accepts calibration-placement configuration.
auto_round/cli/parser.py	Adds the calibration-placement argument.
auto_round/cli/main.py	Forwards placement settings and installs diagnostics.
auto_round/autoround.py	Exposes placement through public APIs.
auto_round/algorithms/quantization/sign_roundv2/quantizer.py	Defers eligible initialization searches.
auto_round/algorithms/quantization/sign_round/quantizer.py	Integrates pool pulls, local snapshots, and timing.
auto_round/algorithms/quantization/rtn/quantizer.py	Dispatches staged RTN searches.
auto_round/algorithms/quantization/rtn/batched_search.py	Implements batched search and fallback execution.
auto_round/algorithms/quantization/base.py	Adds deferred RTN and memory-policy hooks.
auto_round/algorithms/composer.py	Releases consumed pools and records routing shapes.
auto_round/algorithms/block_runner.py	Supports distributed pool storage and batch gathering.

_{Review effort: Balanced}

avtc · 2026-10-19T12:26:12Z

The automated review's findings are addressed in a follow-up commit.

…rches, memory-aware pool placement, auto experts-impl pick Device-local batched searches (iters=0 OptRTN and iters>0 SignRound): same-shape search calls stack per (device, shape, dtype, bits, group, resolved callable) and run on the GPUs the weights already live on -- idle devices take the overflow. Rows bit-identical by construction; mid-batch OOM keeps applied results; the serial lane's OOM->CPU fallback is honored at every site; singletons fall back to serial. Calibration-data pool placement (--calibration_data_device auto|off|cpu|csv): per-block water-fill placement across GPUs under a first-principles need model (computed working allowance, measured collection-window term, deduped 14 B/param tuning state, per-device activation charge with expert-home attribution). Unblocks MoE blocks whose primary cannot host the pools beside the activation window. Pure memory behavior; chunk values bit-identical. Atomic MoE expert placement + per-device grouped-experts plans: the data-driven allocator places whole experts (atomic groups) instead of scattering projection Linears; the grouped-GEMM fast path runs one stacked GEMM per (slot, device) group so it stays eligible under mapped multi-GPU placement. Alignment-only accelerate hooks are exempted from its hook check; calibration hooks still fall back by design. One-shot auto pick of the experts implementation: when grouped tuning cannot fit (per-device qdq stack retention + guaranteed in-loop state vs the live free probe), the run switches to linear_loop BEFORE the first MoE loop, with per-device term logging. Explicit AR_MOE_EXPERTS_IMPL values are never overridden. Upstream fixes surfaced by the new instrumentation: layer_config rebuilt the whole compression-plan dict per read (~27 s per MoE block of invisible dict-copying in the pack loops; now cached/hoisted, pack 29 -> 1.2 s); immediate pack ran after mv_module_from_gpu (CPU packing; reordered, mv 4.5 -> 1.2 s, bit-identical); compiled blocks pinned FX/dynamo example tensors (~2-4 GiB monotonic growth; reset at block end); estimator act_bits read off orig_layer for V2 wrappers. Instrumentation (opt-in): [perf] block-phase summary, OOM tensor census with holder attribution, AR_PERF_COUNTERS, routed-shape recorders. Measured on a 300B-A21B MoE (8x3090): iters=0 quant searches 61.4 -> 9.40 s, MoE block 123 -> ~26 s, full run ~34 min; iters=20 wrap search 80 -> 7.7 s, tune loop 62.8 -> 38.1 s (lean) / 99.3 -> 47.7 s (spread), full 80-block run 2 h 29 m; tuning losses identical to serial. 4x3090 capacity unlock via the auto pick (166 s/block, linear_loop). 3,100+ test lines across 10 files. Signed-off-by: avtc <tarasenkov@gmail.com>

Add type annotations across the tuning pipeline (batched search, search dispatch, pool placement, oom and resume, device and grouped experts, algorithm and wrapper layers, orchestration), drop the unreachable iters > 0 gate before the tune-loop pool pulls (iters = 0 selects RTN at config normalization, so SignRound's quantize_block always runs with iters > 0), and move the tune-loop memory policy into a shared tune_memory module exposed as BaseQuantizer hooks (maybe_adapt_moe_implementation, pull_tuning_pool) so algorithm implementations stay free of fleet-memory policy. Signed-off-by: avtc <tarasenkov@gmail.com>

Capacity-planned batched searches: chunk sizes derive from probed free VRAM with execute-time headroom re-checks and recursive halving, wrap/RTN/anchor lanes share one planner with a residency-aware spread, and NeuUQI grid anchors batch with a per-chunk VRAM cap. Review feedback from the rounds on the published head: the dead expert-batch path removed, find_best_device documented as never-None/never-veto, comment provenance purged to derivations, tune_memory failure paths at WARNING with the consequence named, the dynamo reset at the block boundary removed after an A/B measured its cost, and the dense allocator unified. Follow-ups validated on a full smoke run: one AR_PERF_COUNTERS recorder in auto_round/utils serves the SignRound tune phases, the RTN phases and the orchestrator block phases; the batched-search summaries share the wrap grammar and are gated behind the switch at INFO; unexpected-fallback handlers log at WARNING, expected version gates at debug once per process. Signed-off-by: avtc <tarasenkov@gmail.com>

- stack only row-group weight layouts in the batched RTN search: per-tensor scales, 2-D block groups, and activation-only layers finish per-module through the unwrapper, exactly like the serial lane (parity tested per case) - seed the grouped-experts signature check from the first projection so a plain expert preceding a quantized wrapper can no longer latch its signature without comparison and accept the mixed group - carry the optimized-RTN and NeuQI settings through the per-module CPU fallback so an OOM changes only where the search runs, not which search runs - move batch materialization inside the OOM guard and release the partial batch before the per-module retry - splice pulled tuning pools into the shared pool object so the source tensors are freed instead of staying resident as duplicates - keep pools host-parked when the memory policy asks for it (low-memory mode or host-pinned calibration data) - require three CUDA devices for the third-device routing test Signed-off-by: avtc <tarasenkov@gmail.com>

- f-string formatting in the tune-phase and rtn perf lines (UP031) - direct boolean return in the grouped hook gate (SIM103) - collections.abc/typing imports for annotations flagged by the stricter default rule set (F821/UP035) - .items()/.values() and dict-loop cleanups (PERF102/PLC0206) - merge nested with statements in tests (SIM117) - apply the rule set's autofixes across the touched files Signed-off-by: avtc <tarasenkov@gmail.com>

AutoRoundBot · 2026-10-20T10:52:55Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-10-20T10:53:06Z

Azure Pipelines successfully started running 1 pipeline(s).

Habana lazy tensors execute on per-thread device contexts, so a stacked search spread onto worker threads reaches the write-back host copy with lazy tensors that have no storage attached (data_ptr fails with 'no storage tensor attached to lazy tensor'). HPU lazy mode now collapses the search worker set to the home device, which also keeps every chunk inline on the calling thread -- the configuration the hpu lazy CI ran green before cross-device spreading generalized beyond cuda. Stacked batching still engages on hpu; compile mode (eager pipeline) keeps spreading. Signed-off-by: avtc <tarasenkov@gmail.com>

AutoRoundBot · 2026-10-20T16:26:00Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-10-20T16:26:11Z

Azure Pipelines successfully started running 1 pipeline(s).

avtc · 2026-10-20T16:42:54Z

@xin3he Hi, i have fixed issues found by Copilot, rebased on main, fixed failed HPU test, CI should be green now.

xin3he · 2026-10-21T05:18:12Z

Hi @wenhuach21
I think this change is quite promising; perhaps we could merge this PR first and then focus on further optimizations.

wenhuach21 · 2026-10-21T05:42:28Z

Hi @wenhuach21 I think this change is quite promising; perhaps we could merge this PR first and then focus on further optimizations.

I agree that the results are very promising, and I’m fine with merging this PR. I’ll leave the final decision to you, as I won’t be focusing on this project for a while.

Given the size of this PR and our limited headcount and bandwidth, I’m a bit concerned about maintaining this feature long-term if @avtc is no longer available to support it. I suggest that one of us review the code carefully and develop a solid understanding of the implementation so we can maintain it going forward.

xin3he

Nice enhancement.

xin3he · 2026-10-21T06:43:22Z

@lvliang-intel Please have a look.

This comment has been minimized.

Sign in to view

avtc marked this pull request as draft September 14, 2026 19:28

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch from 5b49ec9 to 0f0ea17 Compare September 14, 2026 21:32

avtc marked this pull request as ready for review September 14, 2026 21:33

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch from 0f0ea17 to a44eb52 Compare September 15, 2026 09:19

chensuyue requested review from n1ck-guo and wenhuach21 September 16, 2026 03:49

avtc removed the request for review from wenhuach21 September 16, 2026 08:03

avtc requested a review from wenhuach21 September 21, 2026 22:49

avtc marked this pull request as draft September 27, 2026 09:25

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch from a44eb52 to 7b11263 Compare September 27, 2026 10:30

avtc marked this pull request as ready for review September 27, 2026 10:31

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch from 7b11263 to d29d1f0 Compare September 27, 2026 11:01

wenhuach21 requested review from lvliang-intel, xin3he and yiliu30 September 30, 2026 02:32

wenhuach21 added this to the 0.17.0 milestone Sep 30, 2026

wenhuach21 reviewed Sep 30, 2026

View reviewed changes

Comment thread auto_round/algorithms/quantization/rtn/batched_search.py Outdated

wenhuach21 reviewed Sep 30, 2026

View reviewed changes

Comment thread auto_round/algorithms/quantization/sign_round/quantizer.py Outdated

Comment thread auto_round/algorithms/quantization/sign_round/quantizer.py Outdated

lvliang-intel reviewed Oct 15, 2026

View reviewed changes

Comment thread auto_round/algorithms/quantization/sign_round/quantizer.py Outdated

lvliang-intel reviewed Oct 15, 2026

View reviewed changes

Comment thread auto_round/algorithms/quantization/tune_memory.py

lvliang-intel reviewed Oct 15, 2026

View reviewed changes

Comment thread auto_round/algorithms/quantization/tune_memory.py

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch 2 times, most recently from 7f820a9 to 9e87eb0 Compare October 17, 2026 17:07

xin3he requested a balanced review from Copilot October 19, 2026 07:38

Copilot started reviewing on behalf of xin3he October 19, 2026 07:38

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

avtc added 5 commits October 20, 2026 12:06

avtc force-pushed the feature/improvements-to-multi-gpu-quantization branch from bbc1031 to da75315 Compare October 20, 2026 10:10

xin3he requested review from lvliang-intel and wenhuach21 October 21, 2026 05:16

xin3he approved these changes Oct 21, 2026

View reviewed changes

xin3he merged commit a7656a2 into intel:main Oct 22, 2026
47 checks passed

avtc mentioned this pull request Oct 22, 2026

feat: data-parallel block tuning via --parallel_quantization #2351

Open

3 of 4 tasks

Conversation

avtc commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Features

Upstream fixes (single-GPU users benefit too)

Instrumentation (opt-in)

Measured results (300B-A21B MoE, 80 blocks; iters=0 lane = optimized RTN, symmetric scheme; iters=20 lane = SignRound V2)

New knobs

Tests

Type of Change

Related Issues

Checklist Before Submitting

Uh oh!

This comment has been minimized.

AutoRoundBot commented Sep 14, 2026

Uh oh!

azure-pipelines Bot commented Sep 14, 2026

Uh oh!

AutoRoundBot commented Sep 15, 2026

Uh oh!

azure-pipelines Bot commented Sep 15, 2026

Uh oh!

avtc commented Sep 27, 2026

Uh oh!

AutoRoundBot commented Sep 27, 2026

Uh oh!

azure-pipelines Bot commented Sep 27, 2026

Uh oh!

wenhuach21 commented Sep 30, 2026

Uh oh!

avtc commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

wenhuach21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avtc commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avtc commented Oct 19, 2026

Uh oh!

AutoRoundBot commented Oct 20, 2026

Uh oh!

azure-pipelines Bot commented Oct 20, 2026

Uh oh!

AutoRoundBot commented Oct 20, 2026

Uh oh!

azure-pipelines Bot commented Oct 20, 2026

Uh oh!

avtc commented Oct 20, 2026

Uh oh!

xin3he commented Oct 21, 2026

Uh oh!

wenhuach21 commented Oct 21, 2026

Uh oh!

xin3he left a comment

Choose a reason for hiding this comment

Uh oh!

xin3he commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

avtc commented Sep 14, 2026 •

edited

Loading

avtc commented Sep 30, 2026 •

edited

Loading

avtc commented Oct 18, 2026 •

edited

Loading