diff --git a/families/qwen3_omni/README.md b/families/qwen3_omni/README.md new file mode 100644 index 0000000000..e3a961b2ec --- /dev/null +++ b/families/qwen3_omni/README.md @@ -0,0 +1,158 @@ +# Qwen3-Omni + +This family builds the Thinker of `Qwen/Qwen3-Omni-30B-A3B-Instruct` for +`text_generation` as one TensorRT engine: the Qwen3 MoE text decoder (48 layers, +hidden size 2048, 32 query and 4 key-value heads, 128 routed experts with top-8 softmax routing, expert +intermediate size 768), bf16, batch 1. The audio and vision encoders, the talker +and the code-to-wave decoder are not part of the engine. The family has no +quantized build: `quantization` other than `none` is rejected. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the Thinker was built and measured on TensorRT 11.1 on +B200 (the expert path was also checked on a tiny model on TensorRT 11.4, B200 and +B300). It is not shipped: FP8 routed experts give wrong output (logit cosine +about -0.09 against the bf16 engine), and even before that the expert path is +not faster (decode 0.93x, 64-token prefill 0.66x of bf16). FP8 attention q/k/v +alone is correct but covers under 2% of the weights: decode 1.05x, plan -0.8%, +perplexity +1.8%. This section records the recipe, the method and the numbers, +so that the study does not have to be repeated. The FP8 code was not merged; +this family has no FP8 build. + +### Summary and decision + +B200, TensorRT 11.1, one job, max sequence length 256 (the manifest value). +Speed ratio is bf16 time over FP8 time. + +| Item | bf16 (manifest) | FP8 attention only | FP8 experts only | FP8 both | +|---|---|---|---|---| +| Plan size | 61.1 GB | 60.6 GB (-0.8%) | 32.1 GB (-47%) | 31.6 GB (-48%) | +| Decode step, ms (ratio) | 7.81 | 7.41 (1.05x) | 8.39 (0.93x) | 7.99 (0.98x) | +| Prefill of 64 tokens, ms (ratio) | 157.2 | 157.5 (1.00x) | 236.7 (0.66x) | 236.5 (0.66x) | +| Perplexity, 12 x 128 wikitext tokens (Hugging Face bf16: 14.945) | 14.889 | 15.150 | 545,097 | 653,873 | +| Logit cosine vs bf16 engine, mean | 1 | 0.9964 | -0.084 | -0.097 | +| Top-1 agreement vs bf16 engine | 1 | 92.6% | 0% | 0% | +| Non-finite logits | 0 | 0 | 0 | 0 | + +Decision: do not ship FP8 for Qwen3-Omni. + +- Correctness: the FP8 expert path (FP8 stack, Dequantize, then Gather of the + selected experts) gives wrong output at full size. The same graph on a tiny + random model already loses accuracy on TensorRT 11.1 and gives NaN on 11.4 + (see "Tiny-model check"), while the same FP8-rounded weights stored as a bf16 + constant are exact. The problem is the + run-time Dequantize + Gather of an FP8 constant, not the scale values. +- Speed: even ignoring correctness, the FP8 expert engine is not faster. Decode + is flat or slower and prefill is 1.5x slower. A second execution context of + the FP8 engine asked for 19.3 GB of scratch memory (about 403 MB per layer), + so TensorRT seems to write the dequantized expert stacks to memory instead of + reading FP8 weights in the matmul. +- FP8 attention only: correct, but q, k and v hold 0.5B of the 30.5B Thinker + parameters; the gain (decode 1.05x, 0.5 GB) does not pay for the quality + loss (perplexity 14.889 to 15.150, greedy 24-token match 69% vs the bf16 + engine). + +### Recipe that was measured + +No ModelOpt (`nvidia/*`) FP8 or NVFP4 checkpoint of Qwen3-Omni exists on the +Hub; the quantized repositories are community exports (128 x 128 block FP8 or +compressed-tensors FP8 with dynamic activation scales, third-party NVFP4, AWQ, +GPTQ, GGUF, MLX). So the original bf16 checkpoint was quantized at build time. + +Parameter split of the Thinker (30.5B): routed experts 29.0B (95%), attention +0.9B (q, k, v 0.5B), embedding and `lm_head` 0.6B. + +| Part | Decision | +|---|---| +| Attention q, k, v (W8A8) | FP8 weight, per-tensor scale amax / 448; static per-tensor activation scale, max calibration on 192 prompts of `nvidia/Nemotron-Post-Training-Dataset-v1` run through the bf16 engine (144 scales, keyed by model config); `Quantize -> Dequantize(fp32) -> MatMul(fp32) -> Cast` | +| Routed experts gate, up, down (W8A16) | one FP8 constant per projection and layer, `[experts, in, out]`, one scale per expert (amax / 448); `Dequantize(axis 0)` on the whole stack, then the existing Gather of the top-8 experts and batched matmuls in bf16 | +| Kept bf16 | attention output projection, attention, router, embedding, `lm_head`, norms | + +TensorRT does not accept a Gather directly on an FP8 constant, and it folds a +Dequantize of a constant into a bf16 constant at build time (no size gain). To +keep the stack in FP8, the expert scale was multiplied by a value that is 1 at +run time but is computed from the router indices, so it cannot be folded. + +### Method + +- GPU: B200, one GPU per job; TensorRT 11.1.0.106. The engines were built from + the family code at `4f9ab925` with the FP8 change applied; the bf16 engine is + the unchanged family build. +- Quality: 12 wikitext-2 test texts (longer than 600 characters, no headings), + 128 tokens each, teacher-forced one token per step: perplexity, logit cosine, + top-1 and KL against the bf16 engine and against the Hugging Face + `Qwen3OmniMoeForConditionalGeneration` Thinker in bf16. Free running: 24 + greedy tokens after a 32-token prefix of each text. Non-finite scan of all + logits. +- Speed: decode step at a cache length of a few tokens, mean of 60 steps after + 5 warm-up steps (the step includes the Python input copy; engine-only + re-execution gives about the same ratios: 6.91, 6.52, 7.48, 7.09 ms); prefill of 64 + tokens, best of 5 runs. All variants in the same job, built and timed one + after the other (not interleaved). + +### Results + +Quality against the Hugging Face bf16 Thinker (same 12 x 128 tokens): + +| Engine | Perplexity | Cosine mean / min | Top-1 | KL mean | Greedy match (24 tokens) | +|---|---|---|---|---|---| +| Hugging Face bf16 | 14.945 | - | - | - | - | +| bf16 engine | 14.889 | 0.9983 / 0.939 | 94.8% | 0.0104 | 79.9% | +| FP8 attention only | 15.150 | 0.9964 / 0.874 | 92.6% | 0.0199 | 55.6% | +| FP8 experts only | 545,097 | -0.084 / -0.515 | 0% | 11.05 | 0% | +| FP8 both | 653,873 | -0.096 / -0.542 | 0% | 11.24 | 0% | + +Build, B200, TensorRT 11.1: + +| Engine | Weight load s | Build s | Plan bytes | +|---|---|---|---| +| bf16 | 78 | 310 | 61,117,691,308 | +| FP8 attention only | 76 | 292 | 60,619,197,004 | +| FP8 experts only | 167 | 292 | 32,125,842,996 | +| FP8 both | 160 | 282 | 31,627,355,468 | + +### Tiny-model check of the expert path + +Random Thinker with 2 layers, hidden size 64, 8 experts (top 2), expert size +64; FP8 experts only; logit cosine against the bf16 engine of the same weights. + +| Expert graph | TensorRT 11.1, B200 and B300 | TensorRT 11.4, B200 and B300 | +|---|---|---| +| FP8 stack, Dequantize with per-expert scale x run-time 1, Gather (the recipe) | 0.9921 | NaN | +| Same, run-time value added as `scale + 0` or multiplied as ones | 0.9921 | NaN | +| FP8 stack, single scalar scale, Gather (checks the plumbing; wrong scale values) | 0.9977 | 0.9976 | +| FP8 stack, Dequantize of a plain constant scale (TensorRT folds it to bf16) | 0.99997 | 0.99997 | +| FP8-rounded weights dequantized on the host, bf16 constant | 0.99997 | 0.99997 | + +Storing the FP8-rounded weights as a bf16 constant keeps the result exact, so +the FP8 rounding is not the cause. Every graph that keeps the stack in FP8 and +dequantizes it at run time before the Gather loses accuracy on TensorRT 11.1, +with any form of the scale; at full size (128 experts, 48 layers) the error is +total. On TensorRT 11.4 the same graph gives NaN logits on the tiny model +(B200 and B300), so 11.4 does not fix it. The full model was not rebuilt on +11.4. + +### NVFP4 (not tried) + +The expert path is the limit for any low-precision weight format here, and the +NVFP4 exports on the Hub are third-party, not `nvidia/*`, with a different +expert layout. + +### What was not tried + +| Item | Note | +|---|---| +| B300 and TensorRT 11.4 on the full model | not measured (only the tiny-model check above ran there) | +| Interleaved timing rounds, longer sequences, batch above 1 | not measured | +| FP8 attention output projection, FP8 `lm_head` | not measured | +| An FP8 grouped GEMM (MoE) path for the experts | not available in this family | + +### When to revisit + +- A TensorRT release where Dequantize + Gather of an FP8 constant gives + correct results and reads FP8 weights inside the expert matmul (no + dequantized copy in scratch memory). +- An FP8 grouped-GEMM expert path in this family. + +The bf16 baseline itself builds on main, runs and matches Hugging Face (cosine +0.9983, perplexity 14.889 vs 14.945); no problem on main was found.