From b904fabcb8e71a372f7116bf1f326673ddd9ef92 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:01:41 -0800 Subject: [PATCH] docs(timm_mobilenetv4): record why FP8 is not shipped Add a section to the timm_mobilenetv4 README with the FP8 self-quantization study of mobilenetv4_conv_large on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-channel FP8 weights on the 54 pointwise convolutions, static per-tensor activation scales from ModelOpt max calibration, stem, depthwise, head and classifier kept fp16), measurement method, speed, plan size, ImageNet top-1 against PyTorch fp32, kernel time, recipe variants and the other widths. The FP8 plan is 43% smaller (63.6 to 36.0 MiB) and top-1 stays at the fp16 level (81.40% / 81.60% vs 81.53% / 81.47%), but FP8 is only 1.01x (B200) and 1.04x (B300) the fp16 speed at batch 1, 0.92x to 0.96x at batch 32 and 128, and 0.99x on TensorRT 11.1. With builder optimization level 3 (#2123) fp16 is 1.36x to 1.40x faster and FP8 is slower than fp16 (0.93x to 0.94x). conv_medium and conv_small are slower than fp16 in FP8 at every batch. A smaller plan with no speed gain is not a reason to ship FP8. NVFP4 was not tried (no NVFP4 convolution in TensorRT). Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_mobilenetv4/README.md | 171 ++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) diff --git a/families/timm_mobilenetv4/README.md b/families/timm_mobilenetv4/README.md index 2bb7058708..b72686b9ae 100644 --- a/families/timm_mobilenetv4/README.md +++ b/families/timm_mobilenetv4/README.md @@ -64,3 +64,174 @@ SDK benchmark worker. The existing reference policy still excludes input preparation (`task-model-call-wall`), while the native public Task call includes family preprocessing. This fixes reduction/reporting placement, not that existing scope difference, and does not establish a performance improvement. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the pointwise convolutions was built and measured on +TensorRT 11.4 (and 11.1) for `timm/mobilenetv4_conv_large.e500_r256_in1k`. It is +not shipped: the plan is 43% smaller, but at batch 1 FP8 is only 1.01x (B200) +and 1.04x (B300) the fp16 speed, it is slower than fp16 at batch 32 and 128 +(0.92x to 0.96x), and with TensorRT optimization level 3 it is slower than fp16 +at batch 1 too (0.93x to 0.94x). A smaller plan with no real speed gain is not a +reason to ship FP8. This section records the recipe, the method and the +numbers, so that the study does not have to be repeated. The FP8 code was not +merged; this family has no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4, optimization level 1) | 0.480 / 0.465 ms | 0.475 / 0.449 ms | 1.01x / 1.04x | +| Latency, batch 1, B200 / B300 (TensorRT 11.4, optimization level 3) | 0.360 / 0.334 ms | 0.386 / 0.354 ms | 0.93x / 0.94x | +| Latency, batch 32, B200 / B300 (TensorRT 11.4, level 1) | 1.092 / 1.038 ms | 1.160 / 1.111 ms | 0.94x / 0.93x | +| Latency, batch 128, B200 / B300 (TensorRT 11.4, level 1) | 2.908 / 2.711 ms | 3.030 / 2.940 ms | 0.96x / 0.92x | +| Plan size, batch 1 | 63.6 MiB | 36.0 MiB | -43% | +| ImageNet top-1, B200 / B300 (PyTorch fp32 81.53%) | 81.53% / 81.47% | 81.40% / 81.60% | -0.13 / +0.13 points | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for timm MobileNetV4. + +- Speed: at batch 1 the engine is bound by launches and small kernels (170 + layers, about 0.47 ms). The FP8 1x1 convolutions are small (48 to 2048 + channels on 8x8 to 64x64 maps) and take about the same kernel time as the fp16 + ones, and 12 to 15 Quantize layers do not fuse into the producing layer (for + example after a residual add). The 1% to 4% gain at batch 1 is within the + spread between jobs (B200: 1.004x to 1.018x over three jobs). At batch 32 and + 128 the extra Quantize layers cost more than the convolutions save. +- Optimization level 3 (#2123, open: build this family at level 3) changes the + fp16 baseline. It makes the fp16 engine 1.36x (B200, 0.354 ms) and 1.38x + (B300, 0.332 ms) faster at batch 1 in the #2123 measurement, 1.36x and 1.40x in + the job below, and removes about 60 layers. FP8 at level 3 is 1.26x to 1.30x + faster than FP8 at level 1, but slower than fp16 at level 3 (0.93x to 0.94x). +- Quality is not the problem: top-1 stays at the fp16 level (2 of 1500 images + change), KL 0.011 against PyTorch. PyTorch fake quantization of the same recipe + gives the same KL, so the small loss comes from the recipe, not from TensorRT. +- `conv_medium` and `conv_small` with the same recipe are slower than fp16 at + every batch (0.86x to 0.97x), and `conv_small` loses 3.2 top-1 points. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of these models exists on the Hub, and no official +quantization config exists for them. Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16, axis 0)`. + +| Choice | Decision | +|---|---| +| Quantized (54 of 102 convolutions in `conv_large`) | every 1x1 convolution inside the blocks: `blocks.0.0.conv_pwl`, `pw_exp.conv` and `pw_proj.conv` of every universal inverted bottleneck, `blocks.4.0.conv` | +| fp16 | the 3x3 stem, the 3x3 `conv_exp` of the edge residual block, the 45 depthwise convolutions, the head convolution (on the pooled 1x1 map), the classifier | +| Weight scale | per output channel, amax / 448 of the batch-norm-folded kernel, at build time | +| Activation scale | static, per tensor, ModelOpt `FP8_DEFAULT_CFG` (max calibration) on the timm PyTorch model in fp32 over 500 calibration images: amax / 448 (54 scales, 0.0085 to 0.30, median 0.050) | +| Outlier rule | a scale above 20x the median stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of one GPU type built and + timed in the same job; TensorRT 11.4.0.106 (target) and 11.1.0.106. +- Data: 2000 ImageNet validation images from `clip-benchmark/wds_imagenet1k` + (every 25th image of each shard, shuffled with seed 0): the first 500 for + calibration, the other 1500 held out. Family preprocessing (256x256, bicubic, + crop 0.95). +- Quality: top-1 against the labels, top-1 agreement and KL(PyTorch || engine) + of the softmax against the timm PyTorch fp32 model on the 1500 held-out + images, batch 1. +- Non-finite scan: the 2000 images and 29 stress inputs (Gaussian noise with + standard deviation 1, 3 and 10, constants 0, +-1, +-10), B200 and B300. +- Speed: CUDA graph replay, median of 9 interleaved rounds of 200 calls, spread + at most 0.8%. The family builds batch 1 only; batch 32 and 128 are scratch + builds of the same family graph with the batch dimension patched in. The + optimization level 3 rows come from a separate job per GPU that built fp16 + and FP8 at levels 1 and 3 and timed them together. + +### Results + +Speed and size, `conv_large` (ms per engine call; x is fp16 time over FP8 time; +optimization level 1 unless noted): + +| GPU | TensorRT | Batch | fp32 ms | fp16 ms | FP8 ms | x | plan MiB fp16 to FP8 | +|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1 | 0.639 | 0.480 | 0.475 | 1.01 | 63.6 to 36.0 | +| B200 | 11.4 | 32 | 2.064 | 1.092 | 1.160 | 0.94 | 66.6 to 39.3 | +| B200 | 11.4 | 128 | 7.407 | 2.908 | 3.030 | 0.96 | 78.6 to 51.2 | +| B300 | 11.4 | 1 | 0.632 | 0.465 | 0.449 | 1.04 | 63.2 to 35.6 | +| B300 | 11.4 | 32 | 1.983 | 1.038 | 1.111 | 0.93 | 66.7 to 39.3 | +| B300 | 11.4 | 128 | 7.147 | 2.711 | 2.940 | 0.92 | 78.7 to 51.3 | +| B200 | 11.1 | 1 | - | 0.476 | 0.479 | 0.99 | 63.2 to 35.8 | + +Optimization level 3, batch 1 (one job per GPU; level 1 measured again in the +same job): + +| GPU | TensorRT | fp16 level 1 | FP8 level 1 | fp16 level 3 | FP8 level 3 | x at level 3 | layers fp16 / FP8 at level 3 | +|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 0.490 | 0.485 | 0.360 | 0.386 | 0.93 | 109 / 154 | +| B300 | 11.4 | 0.467 | 0.445 | 0.334 | 0.354 | 0.94 | 109 / 154 | +| B200 | 11.1 | 0.489 | 0.495 | 0.354 | 0.381 | 0.93 | 111 / 155 | + +At level 3 the FP8 plan is 37.3 MiB against 64.6 MiB for fp16, and FP8 top-1 is +81.40% (fp16 81.53%). + +Quality (1500 held-out images, batch 1, level 1): + +| Engine | Top-1 | Agreement with PyTorch top-1 | KL mean | KL p99 | +|---|---|---|---|---| +| PyTorch fp32 | 81.53% | - | - | - | +| fp16 engine (B200 / B300) | 81.53% / 81.47% | 100.00% / 99.93% | 0.00001 | 0.0001 | +| FP8 engine, TensorRT 11.4 (B200 / B300) | 81.40% / 81.60% | 97.13% / 97.33% | 0.0107 / 0.0105 | 0.127 / 0.125 | +| FP8 engine, TensorRT 11.1 (B200) | 81.00% | 97.27% | 0.0104 | 0.116 | +| PyTorch fake quantization of the same recipe | 81.53% | 97.33% | 0.0110 | 0.126 | + +Kernel time per group, B200, TensorRT 11.4, level 1 (layer profiler, ms; the +profiler adds time per layer, so the sums are larger than the latencies above): + +| Kernel group | fp16 batch 1 | FP8 batch 1 | fp16 batch 128 | FP8 batch 128 | +|---|---|---|---|---| +| convolutions | 0.871 | 0.372 fp16 + 0.488 FP8 = 0.860 | 3.398 | 1.992 fp16 + 1.161 FP8 = 3.153 | +| other (pointwise, Quantize, pooling, classifier) | 0.428 | 0.530 | 0.083 | 0.685 | +| layers in the plan | 170 | 185 | 108 | 155 | + +Variants, `conv_large` (TensorRT 11.4, B200; x is fp16 time over variant time): + +| Variant | x batch 1 / 32 / 128 | plan MiB batch 1 | Top-1 | KL mean | +|---|---|---|---|---| +| recipe above | 1.01 / 0.94 / 0.96 | 36.0 | 81.40% | 0.0104 | +| + 3x3 stem and edge residual `conv_exp` in FP8 | 1.02 / 0.88 / 0.86 | 36.2 (139.2 at batch 128) | 81.53% | 0.0217 | +| per-tensor weight scale | 1.02 / 0.94 / 0.96 | 36.0 | 81.40% | 0.0124 | +| batch norm kept as a scale layer | 1.01 / 0.94 / 0.96 | 36.0 | 81.33% | 0.0107 | + +Other widths with the same recipe (TensorRT 11.4, level 1; x at batch 1 / 32 / +128; quality on the same 1500 images, B200): + +| Model | x B200 | x B300 | plan MiB fp16 to FP8 (B200) | Top-1 fp16 to FP8 | KL mean | +|---|---|---|---|---|---| +| `mobilenetv4_conv_medium.e500_r256_in1k` | 0.94 / 0.91 / 0.94 | 0.97 / 0.92 / 0.91 | 19.7 to 13.5 | 79.67% to 80.33% | 0.0150 | +| `mobilenetv4_conv_small.e2400_r224_in1k` | 0.92 / 0.86 / 0.89 | 0.92 / 0.87 / 0.88 | 8.2 to 7.3 | 73.40% to 70.20% | 0.179 | + +The depthwise convolutions were not quantized: in FP8, TensorRT 11.4 splits a +depthwise convolution into one convolution per group (measured on EfficientNet-B0, +see the timm EfficientNet README: 157 to 2462 layers, 15x slower). + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (block-scaled FP4 needs a 2-D +GEMM), and the engine is not limited by weight reads (36 MiB FP8 plan, about +0.45 ms per image). + +### What was not tried + +| Item | Note | +|---|---| +| Percentile or entropy calibration, quantization-aware training | not measured; quality is already at the fp16 level | +| FP8 at batch 32 and 128 with optimization level 3 | not measured; FP8 is already slower at level 1 | +| `conv_small_050`, H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release with FP8 1x1 convolution kernels that are clearly faster + than fp16 for small channel counts, and that fuses the Quantize into the + producing layer after residual adds, so that FP8 beats fp16 at the family's + optimization level. +- A TensorRT FP8 kernel for depthwise (grouped) convolutions that keeps one + layer per convolution. +- A use case where only the plan size matters (43% smaller at the fp16 top-1 + level); the speed rule does not allow it today.