From 023a6024f80390c6c8c451a03508ec33f552870e Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 14:34:59 -0800 Subject: [PATCH] docs(qwen3_omni): record why FP8 is not shipped Add a README for the qwen3_omni family with the FP8 self-quantization study of the Qwen3-Omni-30B-A3B Thinker: recipe (FP8 attention q/k/v with calibrated static activation scales, FP8 routed-expert stacks with one scale per expert), method, plan size, speed and quality against the bf16 engine and Hugging Face. On TensorRT 11.1 (B200) the FP8 expert path gives wrong output (logit cosine about -0.09) and is not faster (decode 0.93x, 64-token prefill 0.66x); FP8 attention alone is correct but covers under 2% of the weights (decode 1.05x, perplexity +1.8%). A tiny-model check on TensorRT 11.4 (B200, B300) shows the FP8 Dequantize + Gather expert graph gives NaN there. FP8 is not shipped. NVFP4 was not tried. No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/qwen3_omni/README.md | 158 ++++++++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 families/qwen3_omni/README.md diff --git a/families/qwen3_omni/README.md b/families/qwen3_omni/README.md new file mode 100644 index 0000000000..e3a961b2ec --- /dev/null +++ b/families/qwen3_omni/README.md @@ -0,0 +1,158 @@ +# Qwen3-Omni + +This family builds the Thinker of `Qwen/Qwen3-Omni-30B-A3B-Instruct` for +`text_generation` as one TensorRT engine: the Qwen3 MoE text decoder (48 layers, +hidden size 2048, 32 query and 4 key-value heads, 128 routed experts with top-8 softmax routing, expert +intermediate size 768), bf16, batch 1. The audio and vision encoders, the talker +and the code-to-wave decoder are not part of the engine. The family has no +quantized build: `quantization` other than `none` is rejected. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the Thinker was built and measured on TensorRT 11.1 on +B200 (the expert path was also checked on a tiny model on TensorRT 11.4, B200 and +B300). It is not shipped: FP8 routed experts give wrong output (logit cosine +about -0.09 against the bf16 engine), and even before that the expert path is +not faster (decode 0.93x, 64-token prefill 0.66x of bf16). FP8 attention q/k/v +alone is correct but covers under 2% of the weights: decode 1.05x, plan -0.8%, +perplexity +1.8%. This section records the recipe, the method and the numbers, +so that the study does not have to be repeated. The FP8 code was not merged; +this family has no FP8 build. + +### Summary and decision + +B200, TensorRT 11.1, one job, max sequence length 256 (the manifest value). +Speed ratio is bf16 time over FP8 time. + +| Item | bf16 (manifest) | FP8 attention only | FP8 experts only | FP8 both | +|---|---|---|---|---| +| Plan size | 61.1 GB | 60.6 GB (-0.8%) | 32.1 GB (-47%) | 31.6 GB (-48%) | +| Decode step, ms (ratio) | 7.81 | 7.41 (1.05x) | 8.39 (0.93x) | 7.99 (0.98x) | +| Prefill of 64 tokens, ms (ratio) | 157.2 | 157.5 (1.00x) | 236.7 (0.66x) | 236.5 (0.66x) | +| Perplexity, 12 x 128 wikitext tokens (Hugging Face bf16: 14.945) | 14.889 | 15.150 | 545,097 | 653,873 | +| Logit cosine vs bf16 engine, mean | 1 | 0.9964 | -0.084 | -0.097 | +| Top-1 agreement vs bf16 engine | 1 | 92.6% | 0% | 0% | +| Non-finite logits | 0 | 0 | 0 | 0 | + +Decision: do not ship FP8 for Qwen3-Omni. + +- Correctness: the FP8 expert path (FP8 stack, Dequantize, then Gather of the + selected experts) gives wrong output at full size. The same graph on a tiny + random model already loses accuracy on TensorRT 11.1 and gives NaN on 11.4 + (see "Tiny-model check"), while the same FP8-rounded weights stored as a bf16 + constant are exact. The problem is the + run-time Dequantize + Gather of an FP8 constant, not the scale values. +- Speed: even ignoring correctness, the FP8 expert engine is not faster. Decode + is flat or slower and prefill is 1.5x slower. A second execution context of + the FP8 engine asked for 19.3 GB of scratch memory (about 403 MB per layer), + so TensorRT seems to write the dequantized expert stacks to memory instead of + reading FP8 weights in the matmul. +- FP8 attention only: correct, but q, k and v hold 0.5B of the 30.5B Thinker + parameters; the gain (decode 1.05x, 0.5 GB) does not pay for the quality + loss (perplexity 14.889 to 15.150, greedy 24-token match 69% vs the bf16 + engine). + +### Recipe that was measured + +No ModelOpt (`nvidia/*`) FP8 or NVFP4 checkpoint of Qwen3-Omni exists on the +Hub; the quantized repositories are community exports (128 x 128 block FP8 or +compressed-tensors FP8 with dynamic activation scales, third-party NVFP4, AWQ, +GPTQ, GGUF, MLX). So the original bf16 checkpoint was quantized at build time. + +Parameter split of the Thinker (30.5B): routed experts 29.0B (95%), attention +0.9B (q, k, v 0.5B), embedding and `lm_head` 0.6B. + +| Part | Decision | +|---|---| +| Attention q, k, v (W8A8) | FP8 weight, per-tensor scale amax / 448; static per-tensor activation scale, max calibration on 192 prompts of `nvidia/Nemotron-Post-Training-Dataset-v1` run through the bf16 engine (144 scales, keyed by model config); `Quantize -> Dequantize(fp32) -> MatMul(fp32) -> Cast` | +| Routed experts gate, up, down (W8A16) | one FP8 constant per projection and layer, `[experts, in, out]`, one scale per expert (amax / 448); `Dequantize(axis 0)` on the whole stack, then the existing Gather of the top-8 experts and batched matmuls in bf16 | +| Kept bf16 | attention output projection, attention, router, embedding, `lm_head`, norms | + +TensorRT does not accept a Gather directly on an FP8 constant, and it folds a +Dequantize of a constant into a bf16 constant at build time (no size gain). To +keep the stack in FP8, the expert scale was multiplied by a value that is 1 at +run time but is computed from the router indices, so it cannot be folded. + +### Method + +- GPU: B200, one GPU per job; TensorRT 11.1.0.106. The engines were built from + the family code at `4f9ab925` with the FP8 change applied; the bf16 engine is + the unchanged family build. +- Quality: 12 wikitext-2 test texts (longer than 600 characters, no headings), + 128 tokens each, teacher-forced one token per step: perplexity, logit cosine, + top-1 and KL against the bf16 engine and against the Hugging Face + `Qwen3OmniMoeForConditionalGeneration` Thinker in bf16. Free running: 24 + greedy tokens after a 32-token prefix of each text. Non-finite scan of all + logits. +- Speed: decode step at a cache length of a few tokens, mean of 60 steps after + 5 warm-up steps (the step includes the Python input copy; engine-only + re-execution gives about the same ratios: 6.91, 6.52, 7.48, 7.09 ms); prefill of 64 + tokens, best of 5 runs. All variants in the same job, built and timed one + after the other (not interleaved). + +### Results + +Quality against the Hugging Face bf16 Thinker (same 12 x 128 tokens): + +| Engine | Perplexity | Cosine mean / min | Top-1 | KL mean | Greedy match (24 tokens) | +|---|---|---|---|---|---| +| Hugging Face bf16 | 14.945 | - | - | - | - | +| bf16 engine | 14.889 | 0.9983 / 0.939 | 94.8% | 0.0104 | 79.9% | +| FP8 attention only | 15.150 | 0.9964 / 0.874 | 92.6% | 0.0199 | 55.6% | +| FP8 experts only | 545,097 | -0.084 / -0.515 | 0% | 11.05 | 0% | +| FP8 both | 653,873 | -0.096 / -0.542 | 0% | 11.24 | 0% | + +Build, B200, TensorRT 11.1: + +| Engine | Weight load s | Build s | Plan bytes | +|---|---|---|---| +| bf16 | 78 | 310 | 61,117,691,308 | +| FP8 attention only | 76 | 292 | 60,619,197,004 | +| FP8 experts only | 167 | 292 | 32,125,842,996 | +| FP8 both | 160 | 282 | 31,627,355,468 | + +### Tiny-model check of the expert path + +Random Thinker with 2 layers, hidden size 64, 8 experts (top 2), expert size +64; FP8 experts only; logit cosine against the bf16 engine of the same weights. + +| Expert graph | TensorRT 11.1, B200 and B300 | TensorRT 11.4, B200 and B300 | +|---|---|---| +| FP8 stack, Dequantize with per-expert scale x run-time 1, Gather (the recipe) | 0.9921 | NaN | +| Same, run-time value added as `scale + 0` or multiplied as ones | 0.9921 | NaN | +| FP8 stack, single scalar scale, Gather (checks the plumbing; wrong scale values) | 0.9977 | 0.9976 | +| FP8 stack, Dequantize of a plain constant scale (TensorRT folds it to bf16) | 0.99997 | 0.99997 | +| FP8-rounded weights dequantized on the host, bf16 constant | 0.99997 | 0.99997 | + +Storing the FP8-rounded weights as a bf16 constant keeps the result exact, so +the FP8 rounding is not the cause. Every graph that keeps the stack in FP8 and +dequantizes it at run time before the Gather loses accuracy on TensorRT 11.1, +with any form of the scale; at full size (128 experts, 48 layers) the error is +total. On TensorRT 11.4 the same graph gives NaN logits on the tiny model +(B200 and B300), so 11.4 does not fix it. The full model was not rebuilt on +11.4. + +### NVFP4 (not tried) + +The expert path is the limit for any low-precision weight format here, and the +NVFP4 exports on the Hub are third-party, not `nvidia/*`, with a different +expert layout. + +### What was not tried + +| Item | Note | +|---|---| +| B300 and TensorRT 11.4 on the full model | not measured (only the tiny-model check above ran there) | +| Interleaved timing rounds, longer sequences, batch above 1 | not measured | +| FP8 attention output projection, FP8 `lm_head` | not measured | +| An FP8 grouped GEMM (MoE) path for the experts | not available in this family | + +### When to revisit + +- A TensorRT release where Dequantize + Gather of an FP8 constant gives + correct results and reads FP8 weights inside the expert matmul (no + dequantized copy in scratch memory). +- An FP8 grouped-GEMM expert path in this family. + +The bf16 baseline itself builds on main, runs and matches Hugging Face (cosine +0.9983, perplexity 14.889 vs 14.945); no problem on main was found.