diff --git a/examples/vllm_serve/README.md b/examples/vllm_serve/README.md index 9556cebc2a6..6a8b0fda042 100644 --- a/examples/vllm_serve/README.md +++ b/examples/vllm_serve/README.md @@ -14,6 +14,8 @@ fake quantizers are enabled and retains it when they are disabled. The fakequant launcher does not support vLLM 0.9.0. Use one of the tested releases above. +See [Layer and quantizer support](#layer-and-quantizer-support) for available hooks and backend limitations. + ## Prepare environment Run the commands below from the ModelOpt repository root (`/workspace/Model-Optimizer` @@ -378,7 +380,8 @@ quantize: Plugins can add quantizers that turn their own configs into such arguments with `register_indexer_scorer_kwargs_quantizer` of `modelopt.torch.quantization.plugins.vllm_indexer`. -Stock DeepGEMM takes `logits_dtype` (on SM100 GPUs); other arguments need a DeepGEMM build that takes +Stock DeepGEMM takes `logits_dtype`; BF16 scorer reduction requires a compatible DeepGEMM kernel +and GPU, and ModelOpt does not check GPU compatibility. Other arguments need a DeepGEMM build that takes them and declares them, with their values, in `deep_gemm.MQA_LOGITS_NUMERICS`, such as a DeepGEMM fork. ModelOpt ships none. vLLM imports an installed `deep_gemm` before its own copy, so install such a build into the serving environment and serve: @@ -553,18 +556,103 @@ python3 examples/vllm_serve/vllm_serve_sparse_attn.py -tp 8 \ The installer supports both FlashInfer and FlashAttention, and the worker prints the installed adapter counts. Pass `--attention-backend FLASHINFER` or `--attention-backend FLASH_ATTN` only when an explicit override is needed. -This attention-only path applies a fixed dynamic block-16 NVFP4 fakequant format to Q/K/P/V. Q is dynamic; missing K/V scales default to global scale 1.0, and P defaults to amax 1.0. Existing scalar attention amax values are preserved, but this path does not calibrate or restore them itself. It does not re-quantize realquant Linear or MoE weights. An optional checkpoint `sparse_attention_config` is still honored for N:M sparse softmax; calibrated skip-softmax groups are rejected in combination with attention quantization, because quantized Q/K/P change the score distribution the skip thresholds were calibrated on. +This attention-only path defaults to dynamic block-16 NVFP4 Q/DQ for Q/K/P/V. Each operand +can independently use `nvfp4` or per-tensor FP8 E4M3 through `--additional-config`. +For example, keep Q/K in NVFP4 and select FP8 for P/V: + +```bash +python3 examples/vllm_serve/vllm_serve_sparse_attn.py -tp 8 \ + --no-enable-prefix-caching \ + --worker-cls sparse_attn_worker.QuantSparseAttnWorker \ + --additional-config '{"modelopt_attn_quant":{"q_format":"nvfp4","k_format":"nvfp4","p_format":"fp8","v_format":"fp8"}}' +``` + +The accepted keys are `q_format`, `k_format`, `p_format` and `v_format`; omitted keys +default to `nvfp4`. For NVFP4, Q is dynamic, missing K/V scales default to global scale +1.0, and P defaults to amax 1.0. For FP8, missing Q/K/V amax values default to 448 and +P to 1.0. Existing scalar attention amax values are preserved, but this worker does not +calibrate or restore them itself. + +This worker does not re-quantize realquant Linear or MoE weights. Optional checkpoint +`sparse_attention_config` metadata is honored for N:M sparse softmax. Skip-softmax groups +are rejected with attention quantization because Q/K/P Q/DQ changes the score distribution +used to calibrate the skip thresholds. -Decode uses a fixed 32-split, 128-key-tile schedule. P QDQ consumes split-local, -unnormalized online-softmax probabilities, so changing that schedule can change -quantized results; split count is part of the numerical contract. +When P or V uses NVFP4, decode uses a fixed 32-split, 128-key-tile schedule. P Q/DQ +consumes split-local, unnormalized online-softmax probabilities, so changing that schedule +can change quantized results. When both P and V use FP8, decode uses the general paged Triton path. -K is QDQ before its cache write, while V is written pristine. Complete 16-token V groups are finalized once in cache; an incomplete tail remains pristine and is QDQ on read. P@V therefore sees uniform fakequant values without re-quantizing the tail. +K is Q/DQ before its cache write. FP8 V is also Q/DQ before the cache write. NVFP4 V is +written pristine: complete 16-token groups are finalized once in cache, while an incomplete +tail remains pristine and is Q/DQ on read. This preserves consistent NVFP4 values for P@V +without re-quantizing the tail. Supported configurations are regular decoder self-attention with FlashInfer or FlashAttention, fp16/bf16 model and KV cache, equal Q/K/V head dimensions that are multiples of 16, and DCP 1. The FlashInfer adapter preserves both NHD and HND cache strides and separates mixed decode/prefill launches so each phase keeps its own kernel contract. The default `FULL_AND_PIECEWISE` mode remains enabled for fixed N:M and attention-only NVFP4; checkpoints with calibrated decode `threshold_scale_factor` must use a non-`FULL` decode graph mode such as `--enforce-eager` because the live sequence length is not replayed as a Python scalar. Unsupported features are sliding window, ALiBi, softcap, sinks, FP8 KV cache, cross/encoder/MLA attention, KV sharing or transfer, prefix caching, speculative decoding, DBO/ubatching, and `FULL` mixed/prefill CUDA graphs. +## Layer and quantizer support + +The available hooks depend on the vLLM layer class and selected backend. ModelOpt recipes +enable the hooks listed below; a model is covered only where its layers use a supported +class and implementation. Q/DQ means quantize then dequantize. + +| Layer / operation | vLLM layer class | Available quantizers | Limitations | +| --- | --- | --- | --- | +| Attention projections | `QKVParallelLinear`, `ColumnParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear` | `input_quantizer`, `weight_quantizer` | Requires exactly `UnquantizedLinearMethod`. Other methods pass through only with these quantizers disabled, except for the Kimi MLP path below. | +| Dense MLP and shared-expert projections | `ColumnParallelLinear`, `MergedColumnParallelLinear`, `RowParallelLinear` | `input_quantizer`, `weight_quantizer` | Same linear-method restriction. Custom projection methods require separate integration. | +| Kimi K3 dense and shared-expert MLP projections | Parallel linear children of `KimiMLP` using `KimiK3LowLatencyLinearMethod` | `input_quantizer`, `weight_quantizer` | Requires BF16 strided 2D weights. ModelOpt bypasses fused GEMM collective shortcuts when needed. This exception does not cover Kimi custom attention projections. | +| Expert gate/up GEMM | `FusedMoE`, `SharedFusedMoE`, `RoutedExperts` | `w13_input_quantizer`, `w13_weight_quantizer` | Use `--moe-backend triton` with exactly `UnquantizedFusedMoEMethod` and unquantized expert weights. See [expert GEMM backend requirements](#expert-gemm-backend-requirements). | +| Expert down GEMM | `FusedMoE`, `SharedFusedMoE`, `RoutedExperts` | `w2_input_quantizer`, `w2_weight_quantizer` | Same backend and unquantized-method requirements as the gate/up GEMM above. | +| Regular attention Q/K/V | `Attention` | `q_bmm_quantizer`, `k_bmm_quantizer`, `v_bmm_quantizer` | Hooks operands before native attention. Does not expose P inside the native fused kernel. | +| Cross-attention and encoder attention Q/K/V | `CrossAttention`, `EncoderOnlyAttention` | `q_bmm_quantizer`, `k_bmm_quantizer`, `v_bmm_quantizer` | Registered only when the installed vLLM exposes these classes. The compact attention and sparse-attention adapters do not support these layouts. | +| Compact attention Q/K/P/V | `Attention` with `ModelOptSparseAttentionImpl` or `ModelOptSparseFlashInferImpl` | `q_bmm_quantizer`, `k_bmm_quantizer`, `p_bmm_quantizer`, `v_bmm_quantizer` | Each operand supports NVFP4 or FP8. Requires CUDA SM8.9+, regular decoder self-attention, FP16/BF16 model and KV cache, and equal Q/K/V head dimensions divisible by 16. Unsupported: sliding-window attention, cross-layer KV sharing, FP8 KV cache, ALiBi, logits soft cap, and attention sinks. See the [compact worker restrictions](#compact-nvfp4-attention-worker). | +| MLA query, compressed latent and positional key | `MLAAttention` | `q_bmm_quantizer`, `kv_c_bmm_quantizer`, `k_pe_bmm_quantizer` | Custom MLA/CSA implementations are not automatically covered. No P hook; the compact adapter does not support MLA. See [MLA KV-cache configuration](#fake-quantize-the-mla-kv-cache). | +| DeepSeek V4 indexer query and key cache | `DeepseekV4Indexer` from `vllm.models.deepseek_v4.attention` | `indexer_q_quantizer`, `indexer_k_quantizer` | Applies Q/DQ after native FP8 quantization and requantizes to FP8, retaining the original FP8 rounding. MXFP4 indexer cache is rejected. | +| GLM 5.3 Flash indexer query and key cache | `Indexer` from `vllm.models.glm5next.common.attention` or `vllm.models.glm5next.nvidia.attention` | `indexer_q_quantizer`, `indexer_k_quantizer` | Same FP8 round-trip restriction. Applied in the rotated basis used by the native indexer. | +| GLM indexer scorer | GLM `Indexer` | `indexer_scorer_kwargs_quantizer` | Defaults off. Passes options to the scorer through `backend: scorer_kwargs`. Stock DeepGEMM supports `logits_dtype`; BF16 scorer reduction requires a compatible DeepGEMM kernel and GPU. ModelOpt does not check GPU compatibility. Additional options require a compatible custom backend. DeepSeek V4 rejects scorer controls. | +| Expert-parallel dispatch and combine | `RoutedExperts` modular MoE path | `dispatch_quantizer`, `combine_quantizer` | Independent of expert GEMM quantizers. Combine needs an accessible modular finalize payload; padded backends require a constant global scale or MX format. `flashinfer_moe_ep_*` combine is rejected. Reduction and residual addition remain unquantized. | +| DeepGEMM Mega-MoE dispatch and combine | `DeepseekV4MegaMoEExperts`, `KimiK3MegaMoEExperts` | `dispatch_quantizer`, `combine_quantizer` | Supports communication Q/DQ with `--moe-backend deep_gemm_mega_moe`, not expert weight/GEMM Q/DQ. Dispatch requires `VLLM_DISABLE_DSV4_MEGAMOE_SHARED_EXPERT_FUSION=1`. Combine requires a calibrated (`algorithm: max`) or constant global scale, or an MX format; only `fp8xfp4` buffer layouts are supported. | + +Linear and expert output quantizers are omitted from this table because they default to +disabled. Recipes can explicitly enable `output_quantizer`, `w13_output_quantizer` +and `w2_output_quantizer` after layer conversion. + +### Expert GEMM backend requirements + +The `w13_*` and `w2_*` quantizers on `FusedMoE`, `SharedFusedMoE` and `RoutedExperts` +intercept the decomposed Triton expert GEMMs. All three classes require the same expert +compute path; the layer class alone does not establish backend support. + +| Expert compute backend selection | Expert input/weight fakequant | Requirement or limitation | +| --- | --- | --- | +| `--moe-backend triton` | Supported | Requires exactly `UnquantizedFusedMoEMethod`, unquantized expert weights, and the intercepted Triton kernel entry points. | +| `--moe-backend deep_gemm_mega_moe` | Not supported by the expert GEMM hooks | DeepSeek/Kimi construct dedicated Mega-MoE expert modules instead of the regular `RoutedExperts` path. Mega-MoE runs its existing FP8/FP4 expert kernel; ModelOpt adds separate dispatch/combine fakequant. | +| `--moe-backend flashinfer_moe_ep_*` | Not supported by the expert GEMM hooks | The fused expert kernel is not intercepted by the Triton hooks. Dispatch Q/DQ remains available through its registered path; combine Q/DQ is rejected. | +| Other expert compute backends | No dedicated expert GEMM fakequant integration | Their kernels are not intercepted unless execution actually falls back to the supported decomposed Triton path. | + +ModelOpt does not verify that the intercepted expert GEMM kernels ran, so expert input +Q/DQ may be silently skipped. Do not rely on this path for expert activation fakequant +with an unsupported backend. + +The expert compute backend is separate from the EP communication backend. DeepEP, NIXL +or another communication backend may be used with the supported modular expert path, +subject to the dispatch/combine restrictions above. Communication-only recipes do not +require Triton expert GEMMs. Separately executed shared-expert MLP projections use their +registered linear hooks rather than the routed-expert `w13_*`/`w2_*` hooks. + +Additional limits: + +- All `ReplicatedLinear` layers (including routers and some MLA latent projections), + normalization, embeddings and LM-head layers have no dedicated + runtime quantizer registration. GDN/KDA recurrent state and decay also have no dedicated + vLLM hooks. Projection layers that use the registered linear execution paths are covered + by the linear-layer support above. +- N:M sparse softmax can combine with compact attention Q/DQ during prefill; N:M is not + applied during decode. Skip-softmax cannot combine with active attention Q/K/P/V Q/DQ. +- Whole-model `FakeQuantWorker` and the sparse-attention workers are separate entry points. + Combining live projection/expert fakequant with sparse attention requires a combined integration. + ## Known Problems 1. **MCore reload uses export sidecars rather than `MODELOPT_STATE_PATH`**. Current exports write both `quantizer_state.pth` and `quant_recipe.yaml`; serving the export directory through this launcher auto-detects both. For a legacy export without the YAML sidecar, pass `QUANT_FILE_PATH` and set `QUANT_CFG` to match the original MCore quantization recipe.