From 12d30e1aa055d583e692da4931f870ffca1935a1 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 19:30:16 -0800 Subject: [PATCH] docs(timm_hrnet): record why FP8 is not shipped Add a README to the timm_hrnet family with the FP8 self-quantization study of hrnet_w18 on TensorRT 11.4 (B200, B300) and 11.1 (B200): recipe (per-channel FP8 weights on 324 of 325 convolutions, static per-tensor activation scales from ModelOpt max calibration, fp16 Dequantize), measurement method, speed, plan size, ImageNet top-1 against PyTorch fp32 and the kernel time per group. FP8 is slower than fp16 at batch 1 (0.92x to 0.93x) and at batch 32 (0.89x to 0.94x), and top-1 drops by 0.7 to 0.8 point; the only gain is a 38% smaller plan. The narrow FP8 convolutions gain only 5%, while the Quantize of most convolution inputs does not fuse: those inputs are residual or multi-resolution fuse sums that also feed an fp16 consumer, so 142 extra pointwise kernels carry the Quantize. FP8 is not shipped. NVFP4 was not tried (no NVFP4 convolution in TensorRT). Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_hrnet/README.md | 136 ++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 families/timm_hrnet/README.md diff --git a/families/timm_hrnet/README.md b/families/timm_hrnet/README.md new file mode 100644 index 0000000000..7686950c28 --- /dev/null +++ b/families/timm_hrnet/README.md @@ -0,0 +1,136 @@ +# timm HRNet + +This family builds `timm/hrnet_w18.ms_aug_in1k` (architecture `hrnet_w18`) for +`image_to_class_scores` as one TensorRT engine: 224 x 224 input, batch 1, fp16 +or fp32 (the manifest uses fp16), builder optimization level 1. The family has +no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the convolutions was built and measured on TensorRT +11.4 (and 11.1) for `timm/hrnet_w18.ms_aug_in1k`. It is not shipped: FP8 is +slower than fp16 at batch 1 (0.92x to 0.93x) and at batch 32 (0.89x to 0.94x), +and ImageNet top-1 drops by 0.7 to 0.8 point. The only gain is a 38% smaller +plan. This section records the recipe, the method and the numbers, so that the +study does not have to be repeated. The FP8 code was not merged; this family has +no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4) | 1.298 / 1.234 ms | 1.418 / 1.334 ms | 0.92x / 0.93x | +| Latency, batch 32, B200 / B300 (TensorRT 11.4) | 3.344 / 3.019 ms | 3.554 / 3.390 ms | 0.94x / 0.89x | +| Plan size, batch 1 | 43.1 MiB | 26.9 MiB | -38% | +| ImageNet top-1, B200 / B300 (PyTorch fp32 78.53%) | 78.60% / 78.40% | 77.80% / 77.73% | -0.7 to -0.8 point | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for timm HRNet. + +- Speed: the convolutions do run in FP8 (308 FP8 convolution kernels in the + plan), but they are narrow (18 to 144 channels per branch) and gain only 5% + at batch 1 (2.88 to 2.73 ms of kernel time on B200). The Quantize of most + convolution inputs is not fused into the producing layer and runs in extra + pointwise kernels that cost more than the gain (see "Kernel time per group"). +- Why the Quantize does not fuse: in HRNet most convolution inputs are the + output of a residual add + ReLU, or of a multi-resolution fuse sum, that also + feeds an fp16 shortcut or another branch. That fp16 tensor must still be + written, so TensorRT adds a second pointwise kernel (or a second output) for + the FP8 copy. +- The only gain is plan size (16 MiB less). Plan size alone is not a reason to + ship. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of timm HRNet exists on the Hub (one INT8 Lite-HRNet +for a mobile runtime, a different model), and no official quantization config +exists for it. Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16)`. + +| Choice | Decision | +|---|---| +| Quantized (324 of 325 convolutions) | every convolution except the stem convolution (3 input channels) | +| fp16 | the stem convolution, the classifier | +| Weight scale | per output channel, amax / 448 of the batch-norm-folded kernel, at build time (per tensor measured as a variant) | +| Activation scale | static, per tensor, from ModelOpt `FP8_DEFAULT_CFG` (max calibration) on the timm PyTorch model, 500 calibration images | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants built and timed in the same + job; TensorRT 11.4.0.106 (target) and 11.1.0.106 (B200 check). +- Graph: the family's own graph with the family builder settings, with only the + Q/DQ convolution wrapper added. Batch 32 is a scratch build (the family builds + batch 1 only). +- Data: 2000 ImageNet validation images from `clip-benchmark/wds_imagenet1k` + (every 25th image of each shard, shuffled with seed 0): every 4th of them + (500) for calibration, the other 1500 held out. timm preprocessing of the + checkpoint. +- Quality: top-1 against the labels, top-1 agreement and KL divergence against + the timm PyTorch fp32 model on the 1500 held-out images, batch 1. Batch 32 is + checked for top-1 and non-finite outputs. +- Non-finite scan: all outputs of all variants, batch 1 and 32, B200 and B300. +- Speed: CUDA graph replay, median of 5 interleaved rounds (200 calls per round + at batch 1, 50 at batch 32), spread at most 0.1%. + +### Results + +Speed and size (TensorRT 11.4, ms per engine call; x is fp16 time over FP8 +time): + +| GPU | Batch | fp32 ms | fp16 ms | FP8 per channel ms | x | FP8 per tensor ms | x | plan MiB fp16 to FP8 | +|---|---|---|---|---|---|---|---|---| +| B200 | 1 | 1.669 | 1.298 | 1.418 | 0.92 | 1.429 | 0.91 | 43.1 to 26.9 | +| B200 | 32 | 4.422 | 3.344 | 3.554 | 0.94 | 3.543 | 0.94 | 47.6 to 33.2 | +| B300 | 1 | 1.650 | 1.234 | 1.334 | 0.93 | 1.317 | 0.94 | 42.7 to 26.7 | +| B300 | 32 | 4.528 | 3.019 | 3.390 | 0.89 | 3.420 | 0.88 | 48.0 to 33.6 | + +TensorRT 11.1.0.106 check (B200, FP8 per channel, same method): + +| GPU | Batch | fp16 ms | FP8 ms | x | plan MiB fp16 to FP8 | Top-1 fp16 / FP8 | +|---|---|---|---|---|---|---| +| B200 | 1 | 1.300 | 1.441 | 0.90 | 43.1 to 26.9 | 78.60% / 77.80% | +| B200 | 32 | 3.441 | 3.561 | 0.97 | 47.5 to 33.2 | - | + +Quality (1500 held-out images, batch 1, TensorRT 11.4): + +| Engine | Top-1 B200 / B300 | Agreement with PyTorch top-1 | KL mean | KL p99 | +|---|---|---|---|---| +| PyTorch fp32 | 78.53% | - | - | - | +| fp16 | 78.60% / 78.40% | 99.93% / 99.73% | < 0.0001 | < 0.001 | +| FP8, per-channel weights | 77.80% / 77.73% | 95.67% / 95.60% | 0.021 / 0.020 | 0.20 | +| FP8, per-tensor weights | 77.73% / 78.20% | 95.33% / 95.53% | 0.022 / 0.023 | 0.22 to 0.26 | + +Kernel time per group (TensorRT 11.4, FP8 per channel, layer profiler, ms; the +profiler adds time per layer, so the sums are larger than the latencies above): + +| Kernel group | B200 fp16 b1 | B200 FP8 b1 | B300 fp16 b1 | B300 FP8 b1 | B300 fp16 b32 | B300 FP8 b32 | +|---|---|---|---|---|---|---| +| convolutions (309 layers) | 2.877 | 2.729 | 2.951 | 2.766 | 4.440 | 4.421 | +| pointwise kernels that contain a Quantize | - | 0.948 (142 layers) | - | 0.936 (142) | - | 1.305 (142) | +| of these, Quantize only (no add or ReLU) | - | 0.714 (107) | - | 0.693 (107) | - | 0.847 (107) | +| other (add, ReLU, resize, pooling, classifier) | 0.450 (66) | 0.219 (31) | 0.460 (70) | 0.216 (31) | 0.809 (94) | 0.258 (31) | +| layers in the plan | 375 | 482 | 379 | 482 | 403 | 482 | + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (block-scaled FP4 needs a 2-D +GEMM), FP8 already loses 0.7 to 0.8 top-1 point, and the engine is not limited +by weight reads. + +### What was not tried + +| Item | Note | +|---|---| +| Builder optimization level 3 | the family builds at level 1; FP8 was measured at level 1 only | +| Percentile or entropy calibration, quantization-aware training | not measured; could recover accuracy, not speed | +| Wider HRNet models (W30 to W64), H100, native runtime with an FP8 bundle | not measured; the family qualifies only `hrnet_w18` | + +### When to revisit + +- A TensorRT release that fuses the Quantize into a residual add or fuse sum + whose fp16 output has a second consumer (one kernel that writes both the fp16 + and the FP8 tensor). +- The family qualifies the wide HRNet models (W48, W64), where the convolutions + are wider and take a larger share of the time.