From 2d545796deea99d98273a13c1a6f50e112ffbe76 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:01:15 -0800 Subject: [PATCH] docs(timm_ghostnet): record why FP8 is not shipped Add a section to the timm_ghostnet README with the FP8 self-quantization study of ghostnet_100 on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-channel FP8 weights on the 38 pointwise convolutions, static per-tensor activation scales, depthwise, squeeze-excite and stem convolutions kept fp16), measurement method, speed, plan size, ImageNet top-1 against PyTorch fp32 and the recipe variants. FP8 is slower than fp16 at batch 1 (0.89x to 0.91x) and at batch 32 and 128 (0.86x to 0.90x), the plan shrinks by only 0.7 MiB, and top-1 drops from 74.00% to 73.67% (93.60% agreement with PyTorch): the small FP8 1x1 convolutions are not faster than fp16 and the unfused Quantize layers add time. With the depthwise convolutions in FP8, the TensorRT 11.4 build fails (no tactic for a Ghost module concatenation feeding an FP8 depthwise convolution), so depthwise stays fp16. The section also notes that builder optimization level 3 (#2123) makes the fp16 engine 1.12x to 1.15x faster at batch 1. FP8 is not shipped. NVFP4 was not tried (no NVFP4 convolution in TensorRT). Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_ghostnet/README.md | 149 +++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/families/timm_ghostnet/README.md b/families/timm_ghostnet/README.md index 29229516a3..e16962ca5f 100644 --- a/families/timm_ghostnet/README.md +++ b/families/timm_ghostnet/README.md @@ -64,3 +64,152 @@ SDK benchmark worker. The existing reference policy still excludes input preparation (`task-model-call-wall`), while the native public Task call includes family preprocessing. This fixes reduction/reporting placement, not that existing scope difference, and does not establish a performance improvement. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the convolutions was built and measured on TensorRT +11.4 (and 11.1) for `timm/ghostnet_100.in1k`. It is not shipped: FP8 is slower +than fp16 at batch 1 (0.89x to 0.91x) and at batch 32 and 128 (0.86x to 0.90x), +and ImageNet top-1 drops by 0.1 to 0.3 points, with only 93% top-1 agreement +with PyTorch. This section records the recipe, the method and the numbers, so +that the study does not have to be repeated. The FP8 code was not merged; this +family has no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4) | 0.364 / 0.362 ms | 0.407 / 0.397 ms | 0.89x / 0.91x | +| Latency, batch 32, B200 / B300 (TensorRT 11.4) | 0.828 / 0.711 ms | 0.918 / 0.822 ms | 0.90x / 0.86x | +| Latency, batch 128, B200 / B300 (TensorRT 11.4) | 1.824 / 1.734 ms | 2.047 / 2.022 ms | 0.89x / 0.86x | +| Plan size, batch 1, B200 | 11.4 MiB | 10.7 MiB | -7% | +| ImageNet top-1, B200 (PyTorch fp32 74.00%) | 74.00% | 73.67% | -0.33 points | +| Top-1 agreement with PyTorch, B200 | 99.73% | 93.60% | -6.1 points | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for timm GhostNet. + +- Speed: the FP8 1x1 convolutions are not faster than the fp16 ones. They are + small (8 to 960 output channels) and limited by activation traffic, not by + math. The Quantize layers that do not fuse into the producing layer (after + the Ghost module concatenations, residual adds and squeeze-excite products) + add 22 to 28 layers and make the FP8 engine slower at every batch size. +- Size: the plan shrinks by 0.7 MiB at batch 1; the weights are small, so FP8 + gives no useful size gain. +- Quality: per-tensor 8-bit activations change the top-1 class of 6% to 7% of + the images on this mobile network. PyTorch fake quantization of the same + recipe gives the same loss, so the loss comes from the recipe, not from + TensorRT. +- The 41 depthwise convolutions stay fp16. With them in FP8, the TensorRT 11.4 + build fails on B300 at batch 1 and 8: the autotuner finds no tactic for a + Ghost module concatenation that feeds an FP8 depthwise convolution ("no + tactics to implement operation", then "Could not find any implementation for + node"). + +TensorRT builder optimization level 3 (proposed for this family in #2123, not +merged when this study was done) makes the fp16 engine faster: at batch 1, +0.379 to 0.338 ms (1.12x) on B200 and 0.368 to 0.319 ms (1.15x) on B300 with +TensorRT 11.4, measured with a separate harness, so its absolute numbers +differ slightly from the tables here. FP8 was not measured at level 3; the +fp16 baseline that FP8 would have to beat only gets faster. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of this model exists on the Hub, and no official +quantization config exists for it. Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16, axis 0)`. + +| Choice | Decision | +|---|---| +| Quantized (38 of 95 convolutions) | every pointwise (1x1) convolution on a spatial map larger than 1x1: the primary convolutions of the Ghost modules and the shortcut 1x1 convolutions | +| fp16 | the 3x3 stem (3 input channels), the 41 depthwise convolutions (Ghost cheap operations, strided and shortcut depthwise), the 15 squeeze-excite and head convolutions (on a 1x1 map), the classifier | +| Weight scale | per output channel, amax / 448 of the batch-norm-folded kernel, at build time | +| Activation scale | static, per tensor, max \|input\| / 448 over 500 calibration images (the statistic of ModelOpt max calibration), read from an fp32 engine of the family graph | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job; TensorRT 11.4.0.106 (target) and + 11.1.0.106. +- Data: 2000 ImageNet validation images from `clip-benchmark/wds_imagenet1k` + (every 25th image of each shard, shuffled with seed 0): the first 500 for + calibration, the other 1500 held out. Family preprocessing. +- Quality: top-1 against the labels, top-1 agreement and KL divergence against + the timm PyTorch fp32 model on the 1500 held-out images, batch 1. +- Non-finite scan: the 2000 images and 29 stress inputs (noise with standard + deviation 1, 3 and 10, constants), all variants, B200 and B300. +- Speed: CUDA graph replay, median of 7 interleaved rounds of 200 calls, all + variants built and timed in one job per GPU type, spread at most 3.1% + (B200 fp16 batch 128; all others at most 1.5%). Batch 32 and 128 are scratch + builds (the family builds batch 1 only). + +### Results + +Speed and size (ms per engine call; x is fp16 time over FP8 time): + +| GPU | TensorRT | Batch | fp32 ms | fp16 ms | FP8 ms | x | plan MiB fp16 to FP8 | layers fp16 to FP8 | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1 | 0.443 | 0.364 | 0.407 | 0.89 | 11.4 to 10.7 | 168 to 190 | +| B200 | 11.4 | 32 | 1.272 | 0.828 | 0.918 | 0.90 | 14.9 to 14.2 | 151 to 174 | +| B200 | 11.4 | 128 | 3.525 | 1.824 | 2.047 | 0.89 | 27.2 to 26.5 | 151 to 174 | +| B300 | 11.4 | 1 | 0.423 | 0.362 | 0.397 | 0.91 | 10.8 to 10.2 | 158 to 186 | +| B300 | 11.4 | 32 | 1.148 | 0.711 | 0.822 | 0.86 | 15.0 to 14.4 | 152 to 175 | +| B300 | 11.4 | 128 | 3.315 | 1.734 | 2.022 | 0.86 | 27.3 to 26.6 | 152 to 175 | +| B200 | 11.1 | 1 | - | 0.363 | 0.407 | 0.89 | 11.4 to 10.6 | 162 to 182 | + +Quality (1500 held-out images, batch 1): + +| Engine | Top-1 | Agreement with PyTorch top-1 | KL mean | KL p99 | +|---|---|---|---|---| +| PyTorch fp32 | 74.00% | - | - | - | +| fp16 engine (B200) | 74.00% | 99.73% | < 0.0001 | < 0.001 | +| fp16 engine (B300) | 73.93% | 99.73% | < 0.0001 | < 0.001 | +| FP8 engine (B200) | 73.67% | 93.60% | 0.0297 | 0.228 | +| FP8 engine (B300) | 73.87% | 93.20% | 0.0297 | 0.240 | +| FP8 engine (B200, TensorRT 11.1) | 73.33% | 93.60% | 0.0299 | 0.220 | +| PyTorch fake quantization of the same recipe | 73.73% | 93.47% | 0.0294 | 0.228 | + +Kernel time per group, B200, TensorRT 11.4 (layer profiler, ms; the profiler +adds time per layer, so the sums are larger than the latencies above): + +| Kernel group | fp16 batch 1 | FP8 batch 1 | fp16 batch 128 | FP8 batch 128 | +|---|---|---|---|---| +| convolutions | 0.708 | 0.411 fp16 + 0.354 FP8 = 0.766 | 1.885 | 1.174 fp16 + 0.760 FP8 = 1.934 | +| other (pointwise, Quantize, concatenation, pooling, classifier) | 0.472 | 0.595 | 0.711 | 1.011 | +| layers in the plan | 168 | 190 | 151 | 174 | + +Variants (TensorRT 11.4, B200; x is fp16 time over variant time): + +| Variant | x batch 1 / 32 / 128 | plan MiB batch 1 (batch 128) | Top-1 | KL mean | +|---|---|---|---|---| +| recipe above | 0.89 / 0.90 / 0.89 | 10.7 (26.5) | 73.67% | 0.0297 | +| + 3x3 stem in FP8 | 0.89 / 0.82 / 0.76 | 10.8 (93.9) | 73.07% | 0.0552 | +| per-tensor weight scale | 0.89 / 0.90 / 0.89 | 10.6 (26.5) | 73.53% | 0.0421 | +| batch norm kept as a scale layer | 0.89 / 0.90 / 0.89 | 10.6 (26.5) | 73.33% | 0.0297 | +| + depthwise convolutions in FP8 (B300) | build fails at batch 1 and 8 | - | - | - | +| + depthwise convolutions, PyTorch fake quantization | - | - | 72.47% | 0.0923 | + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (block-scaled FP4 needs a 2-D +GEMM), FP8 already loses accuracy, and the engine is not limited by weight +reads (plan under 12 MiB). + +### What was not tried + +| Item | Note | +|---|---| +| Percentile or entropy calibration, quantization-aware training | not measured; could recover accuracy, not speed | +| FP8 at builder optimization level 3 (#2123) | not measured | +| Other GhostNet sizes or versions (GhostNetV2, V3), H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release with FP8 1x1 convolution kernels that are faster than + fp16 for small channel counts, and that fuses the Quantize into the producing + layer after concatenations, residual adds and squeeze-excite products. +- A TensorRT FP8 kernel for depthwise (grouped) convolutions that builds after + a concatenation and keeps one layer per convolution. +- A larger, compute-bound model in this family.