From 314ba5edde5c6a35471dff9ed1447f2c4fdbd969 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 15:26:35 -0800 Subject: [PATCH] docs(deberta_v2): record why FP8 is not shipped Add a README for the deberta_v2 family with the FP8 self-quantization study of microsoft/deberta-v3-base and deberta-v3-large on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-tensor FP8 for the six projections of every layer on the fp16 engine, ModelOpt-calibrated static activation scales), measurement method, speed, plan size, feature quality and MNLI head agreement against PyTorch fp32. At the manifest window (256 tokens, batch 1) FP8 runs at 0.94x to 0.99x of fp16 on B200 and 1.08x to 1.11x on B300, the plan is 21% to 30% smaller, and the feature error grows from 0.2% to 6% to 7%. In the FP8 engine TensorRT fuses the attention of one layer only (all layers in fp16), which takes the GEMM gain back. FP8 is not shipped. NVFP4 was not tried. No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/deberta_v2/README.md | 226 ++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 families/deberta_v2/README.md diff --git a/families/deberta_v2/README.md b/families/deberta_v2/README.md new file mode 100644 index 0000000000..188cb0baa3 --- /dev/null +++ b/families/deberta_v2/README.md @@ -0,0 +1,226 @@ +# DeBERTa-v2 / v3 + +This family builds DeBERTa-v2 and DeBERTa-v3 encoders (`DebertaV2Model`, for +example `microsoft/deberta-v3-base` and `microsoft/deberta-v3-large`) as one +TensorRT engine for the `text_to_token_features` task. The engine takes token ids +and an attention mask over a fixed window (`max_sequence_length`, 256 in the +manifests) and returns the final hidden rows. Batch size is 1. The build accepts +precision fp32 (default) and fp16; the manifests build both. The family has no +quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the encoder projections, on top of the fp16 engine, was +built and measured on TensorRT 11.4 (and 11.1) for `microsoft/deberta-v3-base` +and `microsoft/deberta-v3-large`. It is not shipped: at the manifest window (256 +tokens, batch 1) FP8 is 0.94x to 0.99x of fp16 speed on B200 and 1.08x to 1.11x on +B300, and the feature error grows from about 0.2% to 6% to 7%. FP8 is only +clearly faster at a 512-token window (1.18x to 1.37x). This section records the +recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. All +numbers come from a local study. + +### Summary and decision + +Batch 1, window 256 (the manifest window), TensorRT 11.4; x is fp16 time over +FP8 time. + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| base latency, B200 / B300 | 0.636 / 0.682 ms | 0.674 / 0.633 ms | 0.94x / 1.08x | +| large latency, B200 / B300 | 1.397 / 1.468 ms | 1.407 / 1.325 ms | 0.99x / 1.11x | +| base / large plan size (B200) | 386.1 / 935.8 MiB | 306.0 / 650.8 MiB | -21% / -30% | +| token feature relative error vs PyTorch fp32, base / large | 0.0021 / 0.0023 | 0.070 / 0.060 | about 30x | +| MNLI head prediction agreement with PyTorch fp32, base / large (B200) | 100.00% / 100.00% | 99.25% / 99.40% | -0.6 to -0.75 point | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for DeBERTa-v2/v3. + +- Speed: at batch 1 and windows 128 and 256 FP8 is slower than fp16 on B200 + and at most 1.11x faster on B300. In the FP8 engine TensorRT no longer fuses + the attention of most layers into one kernel (see + [Why there is no speedup](#why-there-is-no-speedup)), which takes the GEMM + gain back. +- Quality: the token features move by 6% to 7% (relative error) against 0.2% + for fp16; MNLI predictions change for 0.45% to 0.75% of pairs; on one + short-input build of the large model the features lose most of their + direction (token cosine 0.63, see [Quality](#quality)). +- Size: the plan is 21% (base) and 30% (large) smaller, because the fp16 word + table (128100 rows) stays. A size-only gain is not enough to ship. +- Against the fp32 default engine FP8 is 3.7x to 6.1x faster, but most of that + is the fp16 gain, which comes with no quality loss, so the comparison that + counts is against fp16. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of a DeBERTa-v2/v3 model exists on the Hub, and no +official quantization config exists for one. The recipe is the usual one for +small text encoders: quantize the attention and feed-forward linears, keep the +embeddings, position tables, norms and the attention core in high precision. +Each quantized projection is built as +`input -> Quantize -> Dequantize(fp16) -> MatMul`, with the weight +`FP8 constant -> Dequantize(fp16)`. + +| Choice | Decision | +|---|---| +| Quantized (6 per layer; 72 in base, 144 in large) | `attention.self.query_proj`, `key_proj`, `value_proj` (applied to the hidden rows), `attention.output.dense`, `intermediate.dense`, `output.dense` | +| fp16 | word embeddings, relative position table, the positional products of the disentangled attention (`query_proj` and `key_proj` applied to the relative table; they do not depend on the input), layer norms (statistics in float32), GELU, attention matmuls and softmax, c2p / p2c gathers | +| Weight scale | per tensor, amax / 448, computed at build time one projection at a time; FP8 weights are never stored | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the PyTorch model, 800 calibration texts (400 WikiText-2 train paragraphs, 400 MNLI train pairs), framed like the runtime (`[CLS] a [SEP]`, pairs `[CLS] a [SEP] b [SEP]`, up to 512 tokens) | +| Scale table key | model config plus two weight sums, so a fine-tune needs its own calibration | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier (largest ratio 4.3x base, 2.5x large) | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | +| Builder | the dual-GEMM fusion was turned off for FP8 builds | + +With `share_att_key` the reference also feeds the relative table through +`query_proj` and `key_proj`, so the calibrated input range of these two also +covers the table. The stored scale is at most 1.12x the range of the hidden rows +(measured against `value_proj`, which has the same input); this costs a little +resolution and never clips. + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of a model built and timed + in the same job; TensorRT 11.4.0.106 (target) and 11.1.0.106 (B200). +- Speed: replayed CUDA graph, median of 9 interleaved rounds of 100 calls; + spread between rounds at most 1.5%. Batch 32 is an experiment-only graph + (the family builds batch 1 only). +- Feature quality: 800 held-out texts (400 WikiText-2 test paragraphs, 400 MNLI + validation-matched pairs; calibration used the train splits), each framed by + itself, against Transformers `DebertaV2Model` in float32 (TF32 off). Token + cosine is per real token; relative error is + `||engine - reference|| / ||reference||` per text. +- Task metric: the MNLI heads of `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` + and `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` (fine-tunes of + the two checkpoints, calibrated the same way for this check only) applied to + the engine features, first 2000 pairs of MNLI validation-matched, window 256, + against the same head on the PyTorch features. +- Non-finite scan: every output above (all rows, padded rows included) and 11 + stress inputs at window 512 (repeated tokens, random ids, `[CLS] [SEP]` only, + one token, the longest texts). +- Kernel time: TensorRT layer profiler, window 256, average of 20 calls. The + per-layer sum is larger than the CUDA graph time (per-layer timing adds + overhead), so only the split between groups is used. + +### Results + +Speed and size, batch 1 (ms; x is fp16 time over FP8 time; plan MiB fp16 to FP8): + +| Model | Window | B200 11.4 fp16 / FP8 | x | B300 11.4 fp16 / FP8 | x | B200 11.1 x | plan MiB (B200 11.4) | +|---|---|---|---|---|---|---|---| +| base | 128 | 0.457 / 0.502 | 0.91 | 0.489 / 0.469 | 1.04 | 0.91 | 381.4 to 301.9 | +| base | 256 | 0.636 / 0.674 | 0.94 | 0.682 / 0.633 | 1.08 | 0.99 | 386.1 to 306.0 | +| base | 512 | 1.084 / 0.920 | 1.18 | 1.093 / 0.873 | 1.25 | 1.19 | 404.0 to 325.5 | +| large | 128 | 1.109 / 1.154 | 0.96 | 1.153 / 1.097 | 1.05 | 0.98 | 929.4 to 644.5 | +| large | 256 | 1.397 / 1.407 | 0.99 | 1.468 / 1.325 | 1.11 | 1.03 | 935.8 to 650.8 | +| large | 512 | 2.507 / 2.002 | 1.25 | 2.607 / 1.906 | 1.37 | 1.22 | 959.5 to 676.3 | + +An earlier run of the same code (one job per GPU type, TensorRT 11.4) gave the +same ratios within 0.02 (for example base 256: 0.95x B200, 1.07x B300). +In that earlier run the fp32 default engine took 2.534 ms (base) and 7.239 ms +(large) at window 256 on B200. + +Batch 32 (experiment-only graph, TensorRT 11.4, ms fp16 / FP8): + +| Model | Window | B200 | x | B300 | x | +|---|---|---|---|---|---| +| base | 128 | 2.505 / 2.352 | 1.07 | 2.755 / 2.259 | 1.22 | +| base | 512 | 16.677 / 16.207 | 1.03 | 17.706 / 15.575 | 1.14 | +| large | 128 | 6.321 / 5.790 | 1.09 | 7.404 / 5.524 | 1.34 | +| large | 512 | 46.659 / 43.636 | 1.07 | 49.787 / 41.863 | 1.19 | + +#### Quality + +Feature quality against PyTorch fp32, 800 held-out texts, TensorRT 11.4: + +| Model | Window | GPU | Engine | Token cosine mean (min) | Relative error mean (p99) | Max abs error | Non-finite | +|---|---|---|---|---|---|---|---| +| base | 256 | B200 | fp16 | 0.999998 (0.9999) | 0.0021 (0.0025) | 0.14 | 0 | +| base | 256 | B200 | FP8 | 0.997525 (0.9334) | 0.070 (0.085) | 5.12 | 0 | +| base | 256 | B300 | FP8 | 0.997520 (0.9517) | 0.070 (0.086) | 4.01 | 0 | +| base | 512 | B200, B300 | FP8 | 0.997516 (0.9337) | 0.070 (0.086) | 4.40 | 0 | +| large | 256 | B200 | fp16 | 0.999997 (0.9971) | 0.0023 (0.0058) | 0.66 | 0 | +| large | 256 | B200 | FP8 | 0.998138 (0.8519) | 0.060 (0.086) | 6.24 | 0 | +| large | 256 | B300 | FP8 | 0.998154 (0.8748) | 0.060 (0.080) | 5.31 | 0 | +| large | 512 | B200 | FP8 | 0.998121 (0.8248) | 0.060 (0.089) | 6.16 | 0 | +| large | 512 | B300 | FP8 | 0.998162 (0.8468) | 0.060 (0.079) | 5.84 | 0 | + +TensorRT 11.1 (B200, 400 texts, window 256): FP8 relative error 0.074 (base) +and 0.062 (large); fp16 0.0022 and 0.0014. + +MNLI heads, 2000 validation-matched pairs, window 256, TensorRT 11.4 (PyTorch +fp32 accuracy: base 90.65%, large 91.90%): + +| Model | GPU | Engine | Accuracy | Agreement with PyTorch | KL mean (p99) | Max logit difference | +|---|---|---|---|---|---|---| +| base | B200 | fp16 | 90.65% | 100.00% | 1.4e-06 (3.5e-05) | 0.024 | +| base | B200, B300 | FP8 | 91.00% | 99.25% | 1.7e-03 (2.9e-02) | 1.03 | +| large | B200 | fp16 | 91.90% | 100.00% | 9.9e-07 (2.4e-05) | 0.036 | +| large | B200 | FP8 | 92.10% | 99.40% | 2.3e-03 (4.4e-02) | 1.30 | +| large | B300 | FP8 | 92.05% | 99.55% | 2.6e-03 (4.7e-02) | 1.63 | + +TensorRT 11.1 (B200, 1000 pairs): FP8 agreement 99.70% (base) and 99.60% +(large). On 11.4 the FP8 accuracy moves by +0.15 to +0.35 point, inside the +noise of 2000 pairs; the KL is about 1000x that of fp16. + +Short inputs: on stress inputs FP8 keeps token cosine 0.986 to 0.9995 (base) +and 0.994 to 0.9997 (large), except the one-token input (`[CLS] the [SEP]`) on +the large model, which depends on the build: token cosine 0.631 (worst token +0.078) in one B300 build at window 512, 0.944 (worst 0.885) in two more B300 +builds, 0.978 at window 256 (B300), and 0.973 to 0.995 in five B200 builds at +windows 256 and 512. fp16 stays at 0.9999 or better. No value was non-finite. + +#### Why there is no speedup + +Kernel time per group, window 256, batch 1, TensorRT 11.4, B200 (layer +profiler, ms (kernel count)): + +| Kernel group | base fp16 | base FP8 | large fp16 | large FP8 | +|---|---|---|---|---| +| GEMMs (projections, attention matmuls not fused) | 0.679 (73) | 0.746 (85) | 1.428 (145) | 1.553 (171) | +| fused attention | 0.120 (12) | 0.010 (1) | 0.245 (24) | 0.011 (1) | +| separate softmax | - | 0.083 (11) | - | 0.175 (23) | +| transposes, concat, add, other | 0.076 (14) | 0.352 (58) | 0.156 (26) | 0.720 (118) | +| c2p / p2c gathers | 0.221 (25) | 0.199 (25) | 0.490 (49) | 0.406 (49) | +| layer norms (with Quantize in FP8) | 0.169 (25) | 0.181 (25) | 0.338 (49) | 0.339 (49) | +| sum | 1.264 | 1.571 | 2.658 | 3.204 | + +B300 and TensorRT 11.1 show the same split. + +- In the fp16 engine TensorRT runs the attention of every layer as one fused + kernel. In the FP8 engine it does so for one layer only; in the other 11 (base) + or 23 (large) layers the attention runs as separate matmuls, a softmax and + extra transposes, concats and adds (pointwise kernels grow from 14 to 58 in + base and from 26 to 118 in large). The cause was not isolated. +- The Quantize of the activations is fused into neighboring kernels (the layer + norm kernel count and time do not change), so it costs little. +- At 128 and 256 tokens the projection GEMMs are small (256 x 768 x 768 in base), + so FP8 saves little GEMM time, and the lost attention fusion costs more. At + 512 tokens the GEMM saving is larger and FP8 wins (1.18x to 1.37x). + +### NVFP4 (not tried) + +DeBERTa-v3 is not bound by weight reads at batch 1: reading all FP8 encoder +weights (85 MB base, 302 MB large) at about 8 TB/s takes 0.01 to 0.04 ms of a +0.6 to 1.5 ms call, so 4-bit weights could save at most that. On other small +encoders (`timm_swin`) NVFP4 was slower than FP8 at batch 1 to 32 because of +the extra dynamic quantize kernels. No NVFP4 engine was built. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 attention matmuls (BMM1 / BMM2), FP8 for the positional products | not measured | +| Keeping `attention.output.dense` (or q/k/v) in fp16 to keep the attention fusion | not measured | +| Per-channel weight scales, SmoothQuant, percentile or entropy calibration | not measured | +| H100, the native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release that keeps the fused attention kernel when the projections + around it are FP8: the GEMM saving would then show up at 128 and 256 tokens. +- The family supports batch greater than 1 or a 512-token window in practice: + FP8 is 1.14x to 1.34x faster at batch 32 on B300 and 1.18x to 1.37x at 512 + tokens. +- A recipe that brings the feature error near fp16 (per-channel scales, + SmoothQuant) and removes the short-input loss on the large model.