From 39810fc2e4a61e80525b0a54ef40a26e73893156 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:00:20 -0800 Subject: [PATCH] docs(timm_dpn): record why FP8 is not shipped Add a README for the timm_dpn family with the FP8 self-quantization study of dpn68b and dpn92 on TensorRT 11.4 (B200, B300) and 11.1 (B200): recipe (per-channel FP8 weights, ModelOpt-calibrated static per-tensor activation scales, fp16 Dequantize), measurement method, speed, plan size, layer count and ImageNet top-1 against PyTorch fp32. With every convolution in FP8, TensorRT 11.4 runs each FP8 grouped 3x3 convolution as one convolution per group (131 to 1059 layers) and dpn68b is 7x slower. With the 1x1 convolutions only, dpn68b runs at 0.95x to 0.99x of fp16 (0.67x to 0.87x on 11.1), and both models lose 2.2 to 3.0 top-1 points. dpn92 is faster in FP8 only because its fp16 baseline is slow: at optimization level 1 TensorRT also splits its fp16 grouped convolutions per group (2515 layers). The note lists the measured level-3 fp16 speed from the optimization-level change, which changes that baseline. FP8 is not shipped. NVFP4 was not tried (no NVFP4 convolution in TensorRT). Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_dpn/README.md | 178 ++++++++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 families/timm_dpn/README.md diff --git a/families/timm_dpn/README.md b/families/timm_dpn/README.md new file mode 100644 index 0000000000..0bfff0592f --- /dev/null +++ b/families/timm_dpn/README.md @@ -0,0 +1,178 @@ +# timm DPN + +This family builds timm Dual Path Network classifiers (`timm/dpn68b.ra_in1k`, +`timm/dpn92.mx_in1k` in the manifests; dpn68, 98, 107 and 131 build from the +same code path) as one TensorRT engine: 224 x 224 input, batch 1, fp16 or fp32 +(the manifests use fp16), builder optimization level 1. The output contains +every class logit in checkpoint order. The family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the convolutions was built and measured on TensorRT +11.4 (and 11.1) for `timm/dpn68b.ra_in1k` and `timm/dpn92.mx_in1k`. It is not +shipped: with every convolution in FP8, dpn68b is 7x slower than fp16, because +TensorRT 11.4 runs each FP8 grouped 3x3 convolution as one convolution per +group. With the 1x1 convolutions only in FP8, dpn68b runs at 0.95x to 0.99x of +fp16, and ImageNet top-1 drops by 2.2 to 3.0 points on both models. This +section records the recipe, the method and the numbers, so that the study does +not have to be repeated. The FP8 code was not merged; this family has no FP8 +build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8, all convolutions | FP8, 1x1 only | FP8 1x1 only vs fp16 | +|---|---|---|---|---| +| dpn68b latency, batch 1, B200 / B300 (TensorRT 11.4) | 0.424 / 0.416 ms | 3.073 / 2.817 ms | 0.436 / 0.422 ms | 0.97x / 0.99x | +| dpn68b latency, batch 32, B200 / B300 (TensorRT 11.4) | 1.109 / 1.036 ms | 7.180 / 6.635 ms | 1.140 / 1.087 ms | 0.97x / 0.95x | +| dpn68b plan size, batch 1, B200 | 25.4 MiB | 17.0 MiB | 16.1 MiB | -37% | +| dpn68b ImageNet top-1, B200 (PyTorch fp32 78.60%) | 78.60% | 75.73% | 76.20% | -2.4 points | +| dpn92 latency, batch 1, B200 / B300 (TensorRT 11.4) | 5.512 / 4.919 ms | - | 4.396 / 3.876 ms | 1.25x / 1.27x | +| dpn92 ImageNet top-1, B200 (PyTorch fp32 80.93%) | 80.87% | - | 77.93% | -2.9 points | +| Non-finite outputs | 0 | 0 | 0 | none | + +Decision: do not ship FP8 for timm DPN. + +- dpn68b (the premerge model): no speed gain in any layout. With the grouped + 3x3 convolutions in FP8, TensorRT 11.4 lowers each of them (32 groups) into + one FP8 convolution per group: the plan grows from 131 to 1059 layers (774 + FP8 convolution kernels) and batch 1 is 7x slower. With the 1x1 convolutions + only, FP8 is 0.95x to 0.99x of fp16 (0.67x to 0.87x on TensorRT 11.1). +- dpn92: the 1x1-only FP8 engine is 1.25x faster at batch 1 only because the + fp16 baseline is slow (see "fp16 grouped convolutions in dpn92" below), and it + loses 2.6 to 2.9 top-1 points. At batch 32 on B300 it is at 1.00x. +- Quality: 2.2 to 3.0 top-1 points lost against PyTorch fp32 with per-channel + weights (up to 3.7 with per-tensor weights), KL divergence 0.09 to 0.13. + The input of each 1x1 convolution is batch norm + ReLU of the concatenated + dual-path state, whose channels span a wide range; one static per-tensor + activation scale per convolution loses the small channels. +- The only gain is plan size. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of any timm DPN exists on the Hub, and no official +quantization config exists for one. Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16, axis 0)`, on the family's own graph. + +| Choice | Decision | +|---|---| +| Variant A, all convolutions (92 in dpn68b) | every convolution except the stem: 1x1 `c1x1_a`, projections `c1x1_w_*`, grouped 3x3 `c3x3_b` (32 groups), block outputs `c1x1_c` / `c1x1_c1` / `c1x1_c2` | +| Variant B, 1x1 only (70 in dpn68b, 64 in dpn92) | the 1x1 convolutions; the grouped 3x3 convolutions stay fp16 | +| fp16 | the 7x7 stem (3 input channels), batch norm + ReLU, pooling, the classifier | +| Weight scale | per output channel, amax / 448 (per tensor measured as a variant) | +| Activation scale | static, per tensor; ModelOpt `FP8_DEFAULT_CFG` (max calibration) on the timm PyTorch model, 500 ImageNet validation images | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of a job built and timed + together; TensorRT 11.4.0.106 (target) and 11.1 (sanity check, B200). + Family builder settings (optimization level 1). +- Data: 2000 ImageNet validation images: 500 for calibration, the other 1500 + held out. Family preprocessing. +- Quality: top-1 against the labels, top-1 agreement and KL divergence against + the timm PyTorch fp32 model on the 1500 held-out images, batch 1. +- Non-finite scan: the 1500 held-out images at batch 1 and at batch 32, all + variants, B200 and B300. +- Speed: CUDA graph replay, median of 5 interleaved rounds of 200 calls, spread + at most 0.6%. Batch 32 is a scratch build (the family builds batch 1 only). + +### Results + +Speed and size, dpn68b (ms per engine call; fp16 and variant B from one job, +fp32 and variant A from another job, whose fp16 is in brackets): + +| GPU | TensorRT | Batch | fp32 ms | fp16 ms | FP8 A ms | FP8 B ms | B vs fp16 | plan MiB fp16 / A / B | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1 | 0.629 | 0.424 (0.437) | 3.073 | 0.436 | 0.97x | 25.4 / 17.0 / 16.1 | +| B200 | 11.4 | 32 | 2.703 | 1.109 (1.103) | 7.180 | 1.140 | 0.97x | 29.2 / 22.4 / 19.4 | +| B300 | 11.4 | 1 | 0.608 | 0.416 (0.418) | 2.817 | 0.422 | 0.99x | 24.9 / 16.7 / 15.7 | +| B300 | 11.4 | 32 | 2.706 | 1.036 (1.035) | 6.635 | 1.087 | 0.95x | 29.3 / 22.6 / 19.5 | +| B200 | 11.1 | 1 | - | 0.451 | - | 0.519 | 0.87x | 25.4 / - / 16.3 | +| B200 | 11.1 | 32 | - | 1.464 | - | 2.195 | 0.67x | 29.6 / - / 19.9 | + +Speed and size, dpn92 (variant B only): + +| GPU | TensorRT | Batch | fp16 ms | FP8 B ms | B vs fp16 | plan MiB fp16 / B | layers fp16 / B | +|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1 | 5.512 | 4.396 | 1.25x | 78.7 / 46.6 | 2515 / 1784 | +| B200 | 11.4 | 32 | 7.926 | 7.020 | 1.13x | 82.1 / 50.0 | 2515 / 1784 | +| B300 | 11.4 | 1 | 4.919 | 3.876 | 1.27x | 77.8 / 46.1 | 2259 / 1656 | +| B300 | 11.4 | 32 | 5.277 | 5.265 | 1.00x | 77.6 / 46.8 | 979 / 1016 | +| B200 | 11.1 | 1 | 5.561 | 4.431 | 1.25x | 78.8 / 46.8 | 2515 / 1811 | +| B200 | 11.1 | 32 | 8.916 | 8.714 | 1.02x | 82.6 / 50.6 | 2515 / 1811 | + +TensorRT 11.1 gives the same top-1 as 11.4 within 0.1 point (dpn68b fp16 +78.67%, FP8 B 76.20%; dpn92 fp16 80.87%, FP8 B 78.20%, B200). + +Layers in the plan, dpn68b, batch 1, TensorRT 11.4 (B200 / B300): + +| Engine | Layers | Convolution kernels | FP8 convolution kernels | +|---|---|---|---| +| fp16 | 131 / 130 | 93 | 0 | +| FP8 A (all convolutions) | 1059 / 1058 | 775 | 774 | +| FP8 B (1x1 only) | 153 / 152 | 93 | 70 | + +Quality (1500 held-out images, batch 1; B200 / B300): + +| Model | Engine | Top-1 | Agreement with PyTorch top-1 | KL mean | KL p99 | +|---|---|---|---|---|---| +| dpn68b | PyTorch fp32 | 78.60% | - | - | - | +| dpn68b | fp16 | 78.60% / 78.53% | 99.6% | 0.00006 | 0.0004 | +| dpn68b | FP8 A, per-channel weights | 75.73% / 76.33% | 86.2% / 86.8% | 0.095 | 0.81 | +| dpn68b | FP8 A, per-tensor weights | 74.87% / 75.53% | 84.4% / 85.1% | 0.107 | 0.81 | +| dpn68b | FP8 B, per-channel weights | 76.20% / 76.07% | 86.6% / 86.6% | 0.091 | 0.74 | +| dpn92 | PyTorch fp32 | 80.93% | - | - | - | +| dpn92 | fp16 | 80.87% / 80.87% | 99.5% / 99.7% | 0.00006 | 0.0009 | +| dpn92 | FP8 B, per-channel weights | 77.93% / 78.27% | 89.5% / 90.3% | 0.133 | 1.5 | + +### fp16 grouped convolutions in dpn92 (TensorRT 11.4 behavior) + +Independent of FP8: at builder optimization level 1, TensorRT 11.4 splits each +fp16 grouped 3x3 convolution of dpn92 into per-group convolutions with slice +and concatenation kernels (2515 layers on B200, 2259 on B300; 740 group +convolutions, 640 slice and 608 concatenation kernels). Batch 1 is launch bound: +5.51 ms (B200) and 4.92 ms (B300), almost as long as batch 32 (7.93 / 5.28 ms). +dpn68b, also with 32 groups but fewer channels per group, builds 131 layers and +runs batch 1 in 0.42 ms. TensorRT 11.1 builds the same 2515-layer fp16 plan on +B200. The FP8 B speedup on dpn92 comes from this slow baseline, not from FP8 +math. + +Builder optimization level 3 (proposed in #2123, not merged with this note) +changes the fp16 baseline. Measured on TensorRT 11.4, batch 1, fp16 (separate +job, 500 held-out images): + +| Model | GPU | Level 1 ms (layers) | Level 3 ms (layers) | Level 3 vs level 1 | +|---|---|---|---|---| +| dpn68b | B200 | 0.436 (131) | 0.377 (128) | 1.16x | +| dpn68b | B300 | 0.418 (130) | 0.358 (128) | 1.17x | +| dpn92 | B200 | 5.685 (2515) | 3.932 (1232) | 1.45x | +| dpn92 | B300 | 4.881 (2259) | 3.721 (1233) | 1.31x | + +At level 3 the fp16 engines are faster than the level-1 FP8 B engines measured +above on dpn68b, and close to them on dpn92 (3.93 vs 4.40 ms on B200, 3.72 vs +3.88 ms on B300; FP8 was not rebuilt at level 3). + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (block-scaled FP4 needs a 2-D +GEMM), FP8 already loses 2.2 to 3.0 top-1 points, and the engines are small +(plan under 80 MiB). + +### What was not tried + +| Item | Note | +|---|---| +| Per-channel activation scales, percentile or entropy calibration, quantization-aware training | not measured; could recover accuracy, not speed | +| FP8 at builder optimization level 3 | not measured | +| dpn68, dpn98, dpn107, dpn131, H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release with an FP8 kernel for grouped convolutions that keeps + one layer per convolution (variant A). +- A calibration that keeps the small channels of the concatenated dual-path + state (for example per-channel activation scales), so that the loss drops + well below 1 top-1 point. +- A TensorRT release that builds the fp16 grouped convolutions of dpn92 as one + layer each; then compare FP8 with that baseline.