From 7d7ced6665cd1888ed28e5e124596d86e9471919 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 14:01:24 -0800 Subject: [PATCH] docs(timm_efficientnet): record why FP8 is not shipped Add a section to the timm_efficientnet README with the FP8 self-quantization study of efficientnet_b0 on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-channel FP8 weights on the 32 pointwise convolutions, static per-tensor activation scales, depthwise and squeeze-excite convolutions kept fp16), measurement method, speed, plan size, ImageNet top-1 against PyTorch fp32 and the recipe variants. FP8 is slower than fp16 at batch 1 (0.94x to 0.95x) and at batch 32 and 128 (0.95x to 0.99x), and top-1 drops from 79.20% to 77.60%: the small FP8 1x1 convolutions are not faster than fp16 and the unfused Quantize layers add time. In FP8, TensorRT 11.4 splits each depthwise convolution into one convolution per group (157 to 2462 layers, build 16 s to 144 s, 15x slower), so depthwise stays fp16. FP8 is not shipped. NVFP4 was not tried (no NVFP4 convolution in TensorRT). Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_efficientnet/README.md | 132 +++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/families/timm_efficientnet/README.md b/families/timm_efficientnet/README.md index 8194b44d66..e8c9a44e46 100644 --- a/families/timm_efficientnet/README.md +++ b/families/timm_efficientnet/README.md @@ -65,3 +65,135 @@ SDK benchmark worker. The existing reference policy still excludes input preparation (`task-model-call-wall`), while the native public Task call includes family preprocessing. This fixes reduction/reporting placement, not that existing scope difference, and does not establish a performance improvement. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the convolutions was built and measured on TensorRT +11.4 (and 11.1) for `timm/efficientnet_b0.ra_in1k`. It is not shipped: FP8 is +slower than fp16 at batch 1 (0.94x to 0.95x) and at batch 32 and 128 (0.95x to +0.99x), and ImageNet top-1 drops by 1.6 points. This section records the +recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4) | 0.414 / 0.419 ms | 0.439 / 0.441 ms | 0.94x / 0.95x | +| Latency, batch 32, B200 / B300 (TensorRT 11.4) | 1.041 / 0.994 ms | 1.091 / 1.034 ms | 0.95x / 0.96x | +| Latency, batch 128, B200 / B300 (TensorRT 11.4) | 2.807 / 2.688 ms | 2.830 / 2.723 ms | 0.99x / 0.99x | +| Plan size, batch 1 | 11.6 MiB | 8.9 MiB | -23% | +| ImageNet top-1, B200 (PyTorch fp32 79.13%) | 79.20% | 77.60% | -1.6 points | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for timm EfficientNet. + +- Speed: the FP8 1x1 convolutions take about the same kernel time as the fp16 + ones. They are small (16 to 1280 channels) and limited by activation traffic, + not by math. The Quantize layers that do not fuse into the producing layer + (after residual adds and squeeze-excite products) add 11 to 14 layers and make + the FP8 engine slower. +- The 16 depthwise convolutions stay fp16. In FP8, TensorRT 11.4 splits each + depthwise (grouped) convolution into one convolution per group: the plan grows + from 157 to 2462 layers, the build takes 144 s instead of 16 s, and batch 1 is + 15x slower (6.43 ms instead of 0.42 ms). +- Quality: per-tensor 8-bit activations cost 1.6 top-1 points on this mobile + network. PyTorch fake quantization of the same recipe loses the same, so the + loss comes from the recipe, not from TensorRT. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of this model exists on the Hub, and no official +quantization config exists for it. Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16, axis 0)`. + +| Choice | Decision | +|---|---| +| Quantized (32 of 81 convolutions) | every pointwise (1x1) convolution on a spatial map larger than 1x1 | +| fp16 | the 3x3 stem (3 input channels), the 16 depthwise convolutions, the 32 squeeze-excite convolutions (on a 1x1 map), the classifier | +| Weight scale | per output channel, amax / 448 of the batch-norm-folded kernel, at build time | +| Activation scale | static, per tensor, max \|input\| / 448 over 500 calibration images (the statistic of ModelOpt max calibration), read from an fp32 engine of the family graph | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job; TensorRT 11.4.0.106 (target) and + 11.1.0.106. +- Data: 2000 ImageNet validation images from `clip-benchmark/wds_imagenet1k` + (every 25th image of each shard, shuffled with seed 0): the first 500 for + calibration, the other 1500 held out. Family preprocessing. +- Quality: top-1 against the labels, top-1 agreement and KL divergence against + the timm PyTorch fp32 model on the 1500 held-out images, batch 1. +- Non-finite scan: the 2000 images and 29 stress inputs (noise with standard + deviation 1, 3 and 10, constants), all variants, B200 and B300. +- Speed: CUDA graph replay, median of 7 interleaved rounds of 200 calls, spread + at most 1.4%. Batch 32 and 128 are scratch builds (the family builds batch 1 + only). + +### Results + +Speed and size (ms per engine call; x is fp16 time over FP8 time): + +| GPU | TensorRT | Batch | fp32 ms | fp16 ms | FP8 ms | x | plan MiB fp16 to FP8 | +|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1 | 0.522 | 0.414 | 0.439 | 0.94 | 11.6 to 8.9 | +| B200 | 11.4 | 32 | 1.831 | 1.041 | 1.091 | 0.95 | 14.1 to 12.4 | +| B200 | 11.4 | 128 | 5.897 | 2.807 | 2.830 | 0.99 | 23.4 to 24.6 | +| B300 | 11.4 | 1 | 0.496 | 0.419 | 0.441 | 0.95 | 11.5 to 9.0 | +| B300 | 11.4 | 32 | 1.793 | 0.994 | 1.034 | 0.96 | 14.3 to 12.5 | +| B300 | 11.4 | 128 | 5.709 | 2.688 | 2.723 | 0.99 | 23.5 to 24.6 | +| B200 | 11.1 | 1 | - | 0.420 | 0.432 | 0.97 | 11.5 to 8.9 | + +Quality (1500 held-out images, batch 1): + +| Engine | Top-1 | Agreement with PyTorch top-1 | KL mean | KL p99 | +|---|---|---|---|---| +| PyTorch fp32 | 79.13% | - | - | - | +| fp16 engine (B200) | 79.20% | 99.87% | < 0.0001 | < 0.001 | +| FP8 engine (B200) | 77.60% | 93.13% | 0.0424 | 0.358 | +| FP8 engine (B300) | 77.67% | 93.27% | 0.0420 | - | +| PyTorch fake quantization of the same recipe | 77.93% | 93.40% | 0.0417 | 0.346 | + +Kernel time per group, B200, TensorRT 11.4 (layer profiler, ms; the profiler +adds time per layer, so the sums are larger than the latencies above): + +| Kernel group | fp16 batch 1 | FP8 batch 1 | fp16 batch 128 | FP8 batch 128 | +|---|---|---|---|---| +| convolutions | 0.674 | 0.399 fp16 + 0.282 FP8 = 0.681 | 2.284 | 1.332 fp16 + 0.926 FP8 = 2.258 | +| other (pointwise, Quantize, pooling, classifier) | 0.540 | 0.619 | 1.238 | 1.359 | +| layers in the plan | 159 | 170 | 136 | 150 | + +Variants (TensorRT 11.4, B200; x is fp16 time over variant time): + +| Variant | x batch 1 / 32 / 128 | plan MiB batch 1 | Top-1 | KL mean | +|---|---|---|---|---| +| recipe above | 0.94 / 0.95 / 0.99 | 8.9 | 77.60% | 0.0424 | +| + 3x3 stem in FP8 | 0.96 / 0.89 / 0.88 | 9.1 (92.0 at batch 128) | 77.53% | 0.0558 | +| per-tensor weight scale | 0.95 / 0.95 / 0.99 | 8.9 | 77.20% | 0.0576 | +| batch norm kept as a scale layer | 0.95 / 0.95 / 0.99 | 8.9 | 77.67% | 0.0421 | +| + depthwise convolutions in FP8 (B300, 100 images) | 0.07 (batch 1 and 8) | 23.4 | - | 0.117 | + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (block-scaled FP4 needs a 2-D +GEMM), FP8 already loses 1.6 top-1 points, and the engine is not limited by +weight reads (plan under 12 MiB). + +### What was not tried + +| Item | Note | +|---|---| +| Percentile or entropy calibration, quantization-aware training | not measured; could recover accuracy, not speed | +| Other EfficientNet sizes (b1 to b7), H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release with FP8 1x1 convolution kernels that are faster than + fp16 for small channel counts, and that fuses the Quantize into the producing + layer after residual adds and squeeze-excite products. +- A TensorRT FP8 kernel for depthwise (grouped) convolutions that keeps one + layer per convolution. +- A larger, compute-bound model in this family: at batch 128 FP8 is already at + 0.99x, so the gain, if any, would come from larger convolutions.