From 35f419bc0c7a8a798c13570e5b36ab185f6c37f1 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:00:34 -0800 Subject: [PATCH] docs(segformer): record why FP8 is not shipped Add a README for the segformer family with the FP8 self-quantization study of SegFormer-B0 (512) and B5 (640) on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-tensor FP8 on the query, key/value and Mix-FFN linears of every encoder block; attention output projection, convolutions, norms, attention and the fp32 decode head unchanged), measurement method, speed, plan size and ADE20K quality. The plan is 19% (b0) to 28% (b5) smaller, but latency is 0.97x to 1.00x of fp16 at batch 1: the encoder matmuls are small (hidden size 32 to 512) and take the same time in FP8 and fp16. 1% to 2% of the pixels change class against the fp16 engine. FP8 is not shipped. NVFP4 was not tried. No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/segformer/README.md | 159 +++++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 families/segformer/README.md diff --git a/families/segformer/README.md b/families/segformer/README.md new file mode 100644 index 0000000000..194df68bd5 --- /dev/null +++ b/families/segformer/README.md @@ -0,0 +1,159 @@ +# SegFormer + +This family builds SegFormer semantic segmentation models (SegFormer-B0 to B5, +for example `nvidia/segformer-b0-finetuned-ade-512-512`) for the +`segmentation` task as one TensorRT engine: input `pixel_values` (batch 1, the +checkpoint input size), output `logits` (class scores at 1/4 of the input +resolution). The encoder is a hierarchical transformer (overlap patch +embedding, efficient self-attention with sequence reduction, Mix-FFN); the +decode head is a light all-MLP head. The b0 premerge manifest uses fp16. The +family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the encoder linears was built and measured on TensorRT +11.4 (and 11.1) for `nvidia/segformer-b0-finetuned-ade-512-512` and +`nvidia/segformer-b5-finetuned-ade-640-640`. It is not shipped: the plan is 19% +(b0) to 28% (b5) smaller, but latency is 0.97x to 1.00x of fp16 at batch 1 on +B200 and B300, and 1% to 2% of the pixels change class against the fp16 +engine. This section records the recipe, the method and the numbers, so that +the study does not have to be repeated. The FP8 code was not merged; this +family has no FP8 build. + +### Summary and decision + +TensorRT 11.4, batch 1, B200 / B300 (x is fp16 time over FP8 time): + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| b0 512 latency | 0.456 / 0.433 ms | 0.471 / 0.437 ms | 0.97x / 0.99x | +| b5 640 latency | 3.749 / 3.484 ms | 3.874 / 3.498 ms | 0.97x / 1.00x | +| b0 / b5 plan size (B200) | 10.1 / 178.3 MiB | 8.1 / 129.2 MiB | -19% / -28% | +| b0 / b5 ADE20K mIoU (B200) | 0.3260 / 0.4482 | 0.3285 / 0.4486 | +0.003 / +0.000 | +| b0 / b5 pixels equal to the fp16 engine (B200) | 1.0000 | 0.9790 / 0.9915 | 2.1% / 0.9% changed | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for SegFormer. + +- Speed: at batch 1 every encoder matmul is small (hidden size 32 to 512) and + takes about 9 to 13 us in fp16 and in FP8 alike (layer profiler). The FP8 + GEMM saves no time, and the extra Quantize and layout kernels make FP8 + slightly slower. +- Quality: mIoU against the ground truth does not drop, but 2.1% (b0) and 0.9% + (b5) of the pixels change class against the fp16 engine (mIoU of the FP8 map + against the fp16 map: 0.87 and 0.95). The fp32 engine changes 0.05%. +- The only gain is plan size (2 MiB for b0, 49 MiB for b5). + +### Recipe that was measured + +No FP8 or NVFP4 SegFormer checkpoint exists on the Hub (`nvidia/` and the model +repos), and no official quantization config exists for it. The recipe follows +the vision encoder guidance used for the `timm_swin` FP8 build: quantize the +linears of the transformer blocks, keep convolutions, norms, attention and the +head. Each quantized projection is built as +`input -> Quantize -> Dequantize(fp16) -> MatMul`, with the weight +`FP8 constant -> Dequantize(fp16)`. + +| Choice | Decision | +|---|---| +| Quantized, per encoder block | query (`attn.q`), key and value (`attn.kv`, one shared input scale, own weight scales), Mix-FFN `mlp.fc1` and `mlp.fc2`: 40 matmuls in b0 (8 blocks), 260 in b5 (52 blocks) | +| fp16 | attention output projection, overlap patch embedding, sequence-reduction and depthwise convolutions, layer norms, attention; the decode head runs in fp32 as in the fp16 build | +| Attention output projection | stays fp16: TensorRT 11.4 cannot compile an FP8 Quantize after the fused attention (the Quantize moves into the attention and an fp8 operand reaches the softmax: "Could not infer output types for operation: softmax"; b0 and b5, B200 and B300) | +| Weight scale | per tensor, amax / 448 of the original weight, at build time; FP8 weights are not stored | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration), 200 ADE20K validation images; stored per model config (depths, hidden sizes, heads, reduction ratios, input size and a weight fingerprint) | +| Outlier rule | a scale above 20x the median of the same projection in the other blocks stays fp16; measured: no outlier (largest ratio 3.3, `mlp.fc2` of b5; largest input amax 38) | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job; TensorRT 11.4.0.106 (target) and + 11.1.0.106. All variants of a model were built and timed in the same job. +- Variants: fp32 (the build default), fp16 (the manifest precision) and fp16 + with FP8, from the same graph code. +- Data: ADE20K validation (`zhoubolei/scene_parse_150`, 2000 images). Images + 0, 10, ..., 1990 (200) were used for calibration; quality uses 300 of the + other 1800 images, resized to the model input size like the engine input. +- Quality: mIoU and pixel accuracy against the ground truth labels (150 + classes, logits upsampled to the label size); pixel agreement and mIoU of + each engine's class map against the fp16 engine's class map. +- Non-finite scan: the logits of the 300 held-out images. +- Speed: batch 1 (the only batch the family builds), CUDA graph replay and + eager enqueue, median of 5 interleaved rounds of 300 runs. +- Checkpoints: b0 at the manifest revision; b5 from the Hub (only a + `pytorch_model.bin`, converted to safetensors for the study). + +### Results + +Speed and size (x is fp16 time over FP8 time, CUDA graph; eager in the last +latency column): + +| Model | GPU | TensorRT | fp32 ms | fp16 ms | FP8 ms | x | x eager | plan MiB fp16 to FP8 | +|---|---|---|---|---|---|---|---|---| +| b0 512 | B200 | 11.4 | 0.607 | 0.456 | 0.471 | 0.97 | 0.99 | 10.1 to 8.1 | +| b0 512 | B300 | 11.4 | 0.574 | 0.433 | 0.437 | 0.99 | 0.97 | 10.0 to 8.2 | +| b5 640 | B200 | 11.4 | 5.276 | 3.749 | 3.874 | 0.97 | 0.97 | 178.3 to 129.2 | +| b5 640 | B300 | 11.4 | 5.007 | 3.484 | 3.498 | 1.00 | 0.99 | 174.4 to 125.3 | +| b0 512 | B200 | 11.1 | 0.619 | 0.488 | 0.499 | 0.98 | 1.00 | 10.2 to 8.3 | +| b0 512 | B300 | 11.1 | 0.578 | 0.453 | 0.461 | 0.98 | 0.99 | 10.2 to 8.3 | +| b5 640 | B200 | 11.1 | 5.408 | 3.950 | 4.017 | 0.98 | 0.98 | 178.6 to 129.6 | +| b5 640 | B300 | 11.1 | 5.063 | 3.534 | 3.586 | 0.99 | 0.98 | 175.4 to 126.4 | + +An earlier run of the same code on TensorRT 11.4 gave the same picture (FP8 +0.97x to 1.00x of fp16). + +Quality, 300 held-out images, TensorRT 11.4, B200 (B300 in brackets where it +differs; TensorRT 11.1 is within 0.001): + +| Model | Engine | mIoU | pixel accuracy | pixels equal to fp16 engine | mIoU vs fp16 engine map | non-finite | +|---|---|---|---|---|---|---| +| b0 | fp32 | 0.3260 | 0.7737 | 0.9995 | 0.997 (0.996) | 0 | +| b0 | fp16 | 0.3260 | 0.7737 | 1 | 1 | 0 | +| b0 | FP8 | 0.3285 (0.3279) | 0.7740 (0.7733) | 0.9790 (0.9794) | 0.872 (0.876) | 0 | +| b5 | fp32 | 0.4482 | 0.8210 | 0.9997 | 0.998 | 0 | +| b5 | fp16 | 0.4482 (0.4481) | 0.8210 | 1 | 1 | 0 | +| b5 | FP8 | 0.4486 (0.4491) | 0.8215 | 0.9915 (0.9918) | 0.946 (0.947) | 0 | + +mIoU against the ground truth stays the same, but the FP8 class map moves away +from the fp16 engine. The b0 premerge case compares the engine class map with +the PyTorch reference (pixel agreement at least 0.99, mIoU at least 0.94). On +the held-out images the FP8 b0 map is at 0.979 and 0.87 against the fp16 map, +so an FP8 b0 bundle would likely fail that case (not run). + +Kernel time per group, TensorRT 11.4, batch 1, B200 (layer profiler, ms +(count); the profiler adds a fixed cost per layer, so only the comparison +between the columns is meaningful): + +| Kernel group | b0 fp16 | b0 FP8 | b5 fp16 | b5 FP8 | +|---|---|---|---|---| +| matmuls | 0.410 (42) | 0.415 (42) | 2.531 (261) | 2.523 (261) | +| attention | 0.085 (8) | 0.086 (8) | 0.765 (52) | 0.763 (52) | +| convolutions | 0.243 (20) | 0.247 (20) | 1.399 (107) | 1.495 (107) | +| GELU pointwise (with the next Quantize in FP8) | 0.062 (8) | 0.063 (8) | 0.557 (52) | 0.559 (52) | +| norms and other pointwise | 0.253 (29) | 0.242 (29) | 1.359 (160) | 1.358 (160) | +| layout and Quantize | 0.134 (46) | 0.166 (50) | 0.901 (97) | 0.639 (63) | + +The matmul time does not change: in b5 the matmul kernels take 8.6 to 12.7 us +each in both engines. With hidden sizes of 32 to 512 these GEMMs are too small +to gain from FP8 tensor cores at batch 1. B300 shows the same picture (b0 +matmuls 0.417 fp16, 0.421 FP8; b5 2.504 fp16, 2.500 FP8). + +### NVFP4 (not tried) + +The matmuls are the same speed in FP8 and fp16, so a lower weight precision +cannot make them faster at batch 1; NVFP4 would only reduce plan size further, +with more quality loss. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 attention output projection | does not compile on TensorRT 11.4 (see the recipe table) | +| Batch above 1, tensor-parallel FP8 | the family builds batch 1 only | +| SegFormer-B1 to B4, H100, the native runtime with an FP8 bundle, the full 2000-image validation set | not measured | +| Percentile or entropy calibration | not measured | + +### When to revisit + +- A use case that needs larger batches or larger inputs, where the encoder + matmuls become compute bound. +- A plan size limit that matters more than latency (the only measured gain).