From 0fe04980191b250c6710d30e7e9413b516d4f799 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Fri, 6 Nov 2026 19:10:01 -0800 Subject: [PATCH] docs(m2m_100): record why FP8 is not shipped Add a README for the m2m_100 family with the FP8 self-quantization study of nllb-200-distilled-600M and 1.3B on TensorRT 11.4 (B200, B300): recipe, measurement method, speed and quality against Transformers fp32. End-to-end translation is 0.94x to 0.96x of fp16 at batch 1 and the decode step 1.00x to 1.02x, so FP8 is not shipped. NVFP4 was not built. Also record problems on main found during the study, not fixed here: the runtime frames the source as text src_lang instead of src_lang text (chrF -0.5, BLEU -0.7), the shared embedding is stored three times, cross-attention K/V are recomputed every decode step, and the host enqueue per step is close to its GPU time. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/m2m_100/README.md | 219 +++++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 families/m2m_100/README.md diff --git a/families/m2m_100/README.md b/families/m2m_100/README.md new file mode 100644 index 0000000000..3fc53eebe2 --- /dev/null +++ b/families/m2m_100/README.md @@ -0,0 +1,219 @@ +# M2M-100 / NLLB + +This family builds M2M-100 and NLLB-200 translation checkpoints (`model_type` +`m2m_100`) for `text_generation` as two TensorRT engines: the encoder and a +one-token decoder step with a static KV cache. The manifest builds +`facebook/nllb-200-distilled-600M` in fp16 with `max_sequence_length` 256. +Batch size is 1. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the encoder and decoder projections was built and +measured on TensorRT 11.4 for `facebook/nllb-200-distilled-600M` and +`facebook/nllb-200-distilled-1.3B`. It is not shipped: end-to-end translation +is slower than fp16 (0.94x to 0.96x) and the decode step is not faster (1.00x +to 1.02x). Quality is close to fp16. This section records the recipe, the +method and the numbers, so that the study does not have to be repeated. The FP8 +code was not merged; this repository has no FP8 build for M2M-100 or NLLB. + +### 1. Summary and decision + +| Item | fp16 | FP8 | Change | +|---|---|---|---| +| End to end (encoder + 64 decode steps), 600M, B200 / B300 | 83.7 / 74.9 ms | 89.0 / 78.6 ms | 0.94x / 0.95x | +| End to end, 1.3B, B200 / B300 | 160.5 / 145.7 ms | 169.2 / 151.0 ms | 0.95x / 0.96x | +| Decode step (CUDA graph), 600M, B200 / B300 | 0.934 / 0.850 ms | 0.923 / 0.846 ms | 1.01x / 1.00x | +| Decode step, 1.3B, B200 / B300 | 1.840 / 1.692 ms | 1.798 / 1.668 ms | 1.02x / 1.01x | +| Encoder (128 tokens), 600M, B200 / B300 | 0.488 / 0.404 ms | 0.429 / 0.402 ms | 1.14x / 1.00x | +| Encoder, 1.3B, B200 / B300 | 1.117 / 1.076 ms | 0.914 / 0.860 ms | 1.22x / 1.25x | +| chrF2 / BLEU vs human reference, 600M (B200; Transformers fp32 58.59 / 33.49) | 58.53 / 33.42 | 58.46 / 33.43 | -0.07 / +0.01 | +| chrF2 / BLEU, 1.3B (B200; Transformers fp32 60.78 / 35.86) | 60.74 / 35.81 | 60.79 / 35.95 | +0.05 / +0.14 | +| KL vs Transformers fp32, 600M / 1.3B (B200) | 3e-6 / 3e-6 | 0.0037 / 0.0028 | small | +| Top-1 agreement, 600M / 1.3B (B200) | 99.9% / 99.9% | 98.1% / 98.3% | -1.8 / -1.6 points | +| Non-finite sentences (300 per model and GPU) | 0 | 0 | none | +| Bundle size, 600M / 1.3B | 2201 / 3644 MiB | 1901 / 2661 MiB | -14% / -27% | + +Decision: do not ship FP8 for M2M-100 / NLLB. + +- Speed: the decode step at batch 1 is bound by the number of small kernels and + by the host enqueue, not by weight reads. FP8 halves the projection weights + but adds 48 (600M) and 96 (1.3B) Quantize layers (decoder plan 355 to 403 and + 703 to 799 layers). The step is 1.00x to 1.02x in a CUDA graph and 0.93x with + a host sync per step (B300: 1.21 to 1.30 ms; TensorRT enqueue 627 to 666 us). + The native runtime enqueues each step without a CUDA graph, so end to end FP8 + is 4% to 6% slower. Only the encoder of the 1.3B model gains (1.22x to + 1.25x), and the encoder runs once per sentence. +- Quality: small loss, similar to the merged t5 FP8 and smaller than in the + bart study. Top-1 agreement is 98.1% to 98.4%; 61% to 65% of greedy + translations equal the reference output (fp16: 97% to 98%). +- The only clear gain is the 14% to 27% smaller bundle. The bundle is dominated + by the 256206 x 1024 shared embedding, stored three times (section 7). + +### 2. Models and engines + +| Item | `facebook/nllb-200-distilled-600M` | `facebook/nllb-200-distilled-1.3B` | +|---|---|---| +| Revision | `f8d333a098d19b4fd9a8b18f94170487ad3f821d` (manifest revision) | `7be3e24664b38ce1cac29b8aeed6911aa0cf0576` | +| Model config | hidden 1024, 12 + 12 layers, MLP 4096, 16 heads, vocab 256206 | hidden 1024, 24 + 24 layers, MLP 8192, 16 heads, vocab 256206 | +| Quantized projections | 156 | 312 | +| `max_sequence_length` (decoder cache) | 256 (manifest) | 256 | +| Engines | encoder (128 source tokens, fixed), decoder step (fp16) | same | + +No official FP8 or NVFP4 checkpoint exists for NLLB-200 or M2M-100 on the Hub +(only CTranslate2 int8 and GGUF conversions), so the weights were +self-quantized. + +### 3. The recipe + +No official FP8 config exists for NLLB. The recipe follows the ModelOpt +guidelines for encoder-decoder text models (as the merged t5 FP8 and the bart +study): quantize the linear projections of the encoder and decoder blocks, keep +everything else in fp16. Every quantized projection is built as: + +``` +activation -> Quantize(FP8) -> Dequantize(fp16) -> MatMul <- Dequantize(fp16) <- FP8 weight constant +``` + +| Choice | Decision | Reason | +|---|---|---| +| Encoder projections | q, k, v, `fc1`, `fc2` of every layer | The GEMMs of the encoder block. | +| Decoder projections | self-attention q, k, v; cross-attention q, k, v; `fc1`, `fc2` of every layer | Cross-attention k and v project the encoder output on every decode step. | +| Attention output projections (`out_proj` of self- and cross-attention) | fp16 | TensorRT 11.4 does not build an FP8 Quantize right after the fused attention: on a tiny model the encoder and decoder self-attention output projections fail ("Input 0's element type (fp8_e4m3) differs from input 1's element type (half)") and the cross-attention output projection aborts the process. TensorRT 11.1 builds all three. | +| Embeddings, tied `lm_head`, sinusoidal positions, LayerNorms | fp16 | Not quantized. | +| Activation scale | static, per tensor, one per projection input | ModelOpt `FP8_DEFAULT_CFG` (max calibration) on PyTorch fp32 while the model greedily translates 576 WMT16 validation sentences (cs, de, fi, ro, ru, tr with en, both directions, 48 each, forced target-language token), so the decoder sees decode-time activations. Scale = amax / 448. | +| Weight scale | per tensor, amax / 448, at build time | FP8 weights are quantized from the original checkpoint at build time. A checkpoint that is already quantized is rejected. | +| Dequantize type | engine type (fp16 for the measured builds) | The FP8 GEMM writes fp16 and the bias stays in the epilogue. No non-finite value in 1200 evaluated sentences. The largest input scale is 0.255; the inputs are LayerNorm or ReLU outputs. fp32 Dequantize was not needed. | +| Outlier rule | a scale above 20x the median of the same projection in the other layers is stored as `null` and stays fp16 | Measured: no outlier; 156 of 156 (600M) and 312 of 312 (1.3B) projections quantized. | +| Scale table key | model config plus a weight fingerprint | The two models need their own scales. A missing scale is a build error. | + +### 4. How the results were measured + +| Item | Value | +|---|---| +| GPUs | B200 and B300, one GPU per job, all builds of one GPU type in one job | +| TensorRT | 11.4.0.106 (target); 11.1.0.106 for a build and a 60-sentence quality check on B200 (section 5) | +| Calibration data | WMT16 validation splits (576 sentences) | +| Evaluation data | WMT16 test splits (newstest2016), 6 pairs, both directions, 25 sentences per direction = 300 sentences per model | +| Reference | Transformers fp32 greedy output, forced target-language token, up to 128 new tokens, Transformers tokenizer framing | +| Harness | Python, the engines of the bundle called directly (not the native runtime); encoder and decoder masks as in the native runtime | + +| Metric | Definition | +|---|---| +| KL | mean KL(reference, engine) per token, with the reference greedy output teacher-forced through the engine (forced language token skipped) | +| Top-1 | fraction of those tokens where the engine argmax equals the reference token | +| Greedy equal | fraction of sentences where the free-running engine greedy output equals the reference output | +| chrF2 / BLEU | sacrebleu 2.6.0 corpus scores of the free-running output against the human reference | +| Non-finite scan | NaN and Inf in every encoder output and every logit (teacher-forced and free-running) | + +Speed: median of 7 interleaved rounds per build. Encoder: one engine call (128 +tokens). Decode step: one replayed CUDA graph of the decoder engine (200 +replays per round). End to end: encoder plus 64 decode steps with a host argmax +per step, as the native runtime does. Host enqueue: host wall time of +`execute_async_v3`. + +### 5. Results + +#### Speed + +TensorRT 11.4, batch 1, ms, fp16 / FP8 (ratio = fp16 time / FP8 time). + +| Model | GPU | Encoder | Decode step (graph) | End to end (64 steps) | +|---|---|---|---|---| +| 600M | B200 | 0.488 / 0.429 (1.14x) | 0.934 / 0.923 (1.01x) | 83.7 / 89.0 (0.94x) | +| 600M | B300 | 0.404 / 0.402 (1.00x) | 0.850 / 0.846 (1.00x) | 74.9 / 78.6 (0.95x) | +| 1.3B | B200 | 1.117 / 0.914 (1.22x) | 1.840 / 1.798 (1.02x) | 160.5 / 169.2 (0.95x) | +| 1.3B | B300 | 1.076 / 0.860 (1.25x) | 1.692 / 1.668 (1.01x) | 145.7 / 151.0 (0.96x) | + +Host cost per step, 600M, B300, fp16 / FP8: enqueue 627 / 666 us; step with a +host sync 1.21 / 1.30 ms (0.93x); end to end at 16 / 32 / 64 / 128 steps +20.5 / 21.9, 40.3 / 43.1, 79.8 / 85.4 and 159.0 / 170.2 ms (0.93x to 0.94x at +every length). + +The FP8 decoder plans have 132 (600M) and 264 (1.3B) layers with e4m3 inputs. +The fp16 `lm_head` (256206 x 1024, 0.088 ms in the per-layer profile) is the +largest single kernel of the decode step and stays fp16. + +#### Quality + +TensorRT 11.4, 300 held-out sentences per row. + +| Model | GPU | Build | chrF2 | BLEU | KL | KL p99 | Top-1 | Greedy equal | Non-finite | +|---|---|---|---|---|---|---|---|---|---| +| 600M | - | Transformers fp32 | 58.59 | 33.49 | - | - | - | - | - | +| 600M | B200 | fp16 | 58.53 | 33.42 | 3e-6 | 2e-5 | 99.89% | 97.7% | 0 | +| 600M | B200 | FP8 | 58.46 | 33.43 | 0.0037 | 0.028 | 98.11% | 61.0% | 0 | +| 600M | B300 | fp16 | 58.54 | 33.45 | 6e-6 | 4e-5 | 99.88% | 97.0% | 0 | +| 600M | B300 | FP8 | 58.43 | 33.32 | 0.0036 | 0.026 | 98.26% | 61.0% | 0 | +| 1.3B | - | Transformers fp32 | 60.78 | 35.86 | - | - | - | - | - | +| 1.3B | B200 | fp16 | 60.74 | 35.81 | 3e-6 | 2e-5 | 99.94% | 98.3% | 0 | +| 1.3B | B200 | FP8 | 60.79 | 35.95 | 0.0028 | 0.020 | 98.32% | 63.3% | 0 | +| 1.3B | B300 | fp16 | 60.70 | 35.77 | 4e-6 | 3e-5 | 99.89% | 97.0% | 0 | +| 1.3B | B300 | FP8 | 60.45 | 35.60 | 0.0027 | 0.020 | 98.35% | 65.0% | 0 | + +Per direction (600M, B200) chrF2 changes from fp16 to FP8 by -1.32 (fi-en) to ++0.81 (en-tr); no direction breaks. Encoder output cosine against Transformers: +min 0.993, mean 0.999. + +TensorRT 11.1 (B200, 600M, 60 sentences): fp16 equals Transformers on 100% of +tokens; FP8 KL 0.0037, top-1 98.2%, chrF2 57.02 (fp16 56.63). + +The FP8 code did not change the unquantized builds: the fp16 plans of main and +of the study code have the same layer count (decoder 355, encoder 100), the +same size within 96 bytes, and bit-identical outputs on the same inputs (B200 +and B300). + +### 6. NVFP4 + +NVFP4 was not built. NVFP4 pays only when decode is bound by weight reads. Here +the decode step does not get faster when FP8 halves the projection weights +(1.00x to 1.02x), so 4-bit weights would not help either: the step is set by +the kernel count, the host enqueue and the fp16 `lm_head`. The encoder (128 +tokens) is compute bound and FP8 already gives at most 1.25x there. + +### 7. Found on the way (main, independent of FP8; not fixed here) + +1. The native runtime frames the source as `text src_lang` (the legacy + NLLB framing, see `runtime/request_tokens.h`). The Transformers NLLB + tokenizer frames it as `src_lang text ` (its default since + `legacy_behaviour=False`). Transformers fp32 600M on the 300 test sentences + with the runtime framing: chrF2 58.59 to 58.10 (-0.5), BLEU 33.49 to 32.76 + (-0.7); 84 of 300 outputs differ; worst direction en-tr, BLEU 20.12 to + 18.49. +2. The shared 256206 x 1024 embedding is stored three times (encoder gather + table, decoder gather table, decoder `lm_head`): about 1.5 GiB of the + 2.2 GiB 600M bundle (encoder plan 790 MiB, decoder plan 1389 MiB). +3. Cross-attention K and V are projected from the encoder output on every + decode step (24 GEMMs of 128 x 1024 x 1024 per step for 600M, see + `_add_m2m100_decoder_layer` in `model.py`) instead of once per sentence. + The cost was not timed separately. +4. Each decode step costs about 0.63 ms of host enqueue (B300), close to its + GPU time (0.85 ms in a CUDA graph); the native runtime does not use a CUDA + graph. + +### 8. What was not tried + +| Item | Why it might matter | +|---|---| +| FP8 attention output projections | Do not build on TensorRT 11.4 (section 3); retest on newer releases. | +| fp32 Dequantize, percentile calibration, per-channel weight scales | Could lower the drift. | +| Batch above 1 | The family supports batch 1 only; larger GEMMs would favor FP8. | +| M2M-100 checkpoints, NLLB 3.3B | No scales calibrated. | +| H100, TensorRT 11.1 speed | Not measured. | +| Native runtime with an FP8 bundle | Not run; the Python harness called the engines. | + +### 9. How to reproduce + +1. Calibrate per-tensor activation scales with ModelOpt `FP8_DEFAULT_CFG` on + PyTorch fp32 during greedy translation of the calibration sentences. +2. Build the fp16 and FP8 bundles of the checkpoints above with the recipe of + section 3. +3. Time the engines and run the quality metrics of section 4 against + Transformers fp32 on the evaluation sentences. + +### 10. When to revisit + +- A TensorRT release that builds FP8 after the fused attention and gives a + decode-step gain at batch 1. +- Batch support above 1 in the family, or a native runtime that replays the + decode step as a CUDA graph. +- A need for the 14% to 27% smaller bundle that outweighs the small drift.