From 4b3150996f4674b38ff4164ceaeb030d4de93f48 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:21:37 -0800 Subject: [PATCH] docs(timm_mobilevit): record why FP8 is not shipped Add a README for the timm_mobilevit family with the FP8 self-quantization study of mobilevit_xxs, mobilevit_xs and mobilevit_s on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-tensor FP8 weights and static activation scales on attn.qkv, mlp.fc1 and mlp.fc2 of the 9 transformer layers; convolutions and attn.proj kept fp16), measurement method, speed and plan size at builder optimization levels 1 and 3, ImageNet top-1 against PyTorch fp32 and the fp16 engine. FP8 is slower than fp16 (0.89x to 0.96x at level 1, 0.94x to 0.97x at level 3) and top-1 agreement with fp16 drops to 95.7% to 98.7%; the plan is 4% to 19% smaller. The transformer GEMMs are tiny and the extra Quantize and Dequantize layers cost more than they save. FP8 is not shipped. NVFP4 was not tried. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_mobilevit/README.md | 143 ++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 families/timm_mobilevit/README.md diff --git a/families/timm_mobilevit/README.md b/families/timm_mobilevit/README.md new file mode 100644 index 0000000000..6a7dcd0f09 --- /dev/null +++ b/families/timm_mobilevit/README.md @@ -0,0 +1,143 @@ +# timm MobileViT + +This family builds the timm MobileViT image classifiers (`timm/mobilevit_xxs`, +`mobilevit_xs` and `mobilevit_s`, `cvnets_in1k`) for `image_to_class_scores` as +one TensorRT engine: 256 x 256 input, batch 1, fp16 or fp32 (the manifests use +fp16). Most of the network is MobileNetV2 inverted residual blocks; three stages +end in a MobileViT block (convolutions plus a small transformer over 2 x 2 +patches). The family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the transformer linears was built and measured on +TensorRT 11.4 (and 11.1) for all three checkpoints. It is not shipped: FP8 is +slower than fp16 on B200 and B300 (0.89x to 0.96x at builder optimization +level 1, 0.94x to 0.97x at level 3), and top-1 agreement with the fp16 engine +drops to 95.7% to 98.7%. The plan is 4% to 19% smaller. This section records the +recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +TensorRT 11.4, batch 1, builder optimization level 1 (the level of this family); +x is fp16 time over FP8 time. + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| mobilevit_s latency, B200 / B300 | 0.431 / 0.420 ms | 0.462 / 0.442 ms | 0.93x / 0.95x | +| mobilevit_xs latency, B200 / B300 | 0.408 / 0.406 ms | 0.449 / 0.425 ms | 0.91x / 0.96x | +| mobilevit_xxs latency, B200 / B300 | 0.357 / 0.363 ms | 0.403 / 0.385 ms | 0.89x / 0.94x | +| Same, builder optimization level 3 | - | - | 0.94x to 0.97x | +| Plan size s / xs / xxs | 12.6 / 6.3 / 4.3 MiB | 10.2 / 5.6 / 4.0 MiB | -19% / -12% / -6% | +| Top-1 agreement with the fp16 engine, s / xs / xxs (B200) | 100% | 98.5% / 97.7% / 95.8% | - | +| Non-finite outputs on ImageNet images | 0 | 0 | none | + +Decision: do not ship FP8 for timm MobileViT. + +- Speed: the quantized GEMMs are tiny (token width 64 to 240, 16 to 256 + tokens per sequence, 9 transformer layers per model), and the engine + is dominated by the convolutions and by per-layer launch time (about 0.4 ms + for 150 to 165 layers). The FP8 engine adds 9 to 12 layers (Quantize and + Dequantize that do not fuse), which cost more than the FP8 GEMMs save. +- Builder optimization level 3 makes both engines faster, but FP8 stays slower + than fp16 at that level too. +- Quality: per-tensor FP8 activations change the top-1 class of 1.5% (s) to + 4.2% (xxs) of the images compared with the fp16 engine; the smaller the + model, the larger the loss. +- The only gain is plan size (2.4 MiB less for mobilevit_s). + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of these models exists on the Hub, and no official +quantization config exists for them. The recipe follows the ModelOpt ViT recipe +(attention and MLP linears quantized, convolutions and patch embedding kept). +Each quantized linear is built as +`input -> Quantize -> Dequantize(fp16) -> MatMul`, with the weight +`FP8 constant -> Dequantize(fp16)`. + +| Choice | Decision | +|---|---| +| Quantized | `attn.qkv` (one shared input scale), `mlp.fc1`, `mlp.fc2` of every transformer layer (9 layers in each model: 2, 4 and 3 per MobileViT stage) | +| fp16 | stem, inverted residual convolutions, MobileViT 1x1 and fusion convolutions, attention matmuls and softmax, layer norms, classifier, `attn.proj` | +| `attn.proj` | a Quantize on its input makes the TensorRT 11.4 attention fusion try an FP8 softmax and the build fails ("Could not infer output types for operation softmax"; xxs, xs, s); kept fp16 | +| Weight scale | per tensor, amax / 448, at build time (no FP8 weights stored) | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the timm PyTorch model, 500 ImageNet validation images | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of one model built and + timed in the same job; TensorRT 11.4.0.106 (target) and 11.1.0.106. +- Builds: the family's own engine build at builder optimization level 1 (this + family) and level 3 (the TensorRT default), fp32, fp16 and FP8. +- Data: 2000 ImageNet validation images from `clip-benchmark/wds_imagenet1k` + (every 25th image of the class-sorted set): 500 spread over the classes for + calibration, 1500 of the others held out. Checkpoint preprocessing (timm). +- Quality: top-1 against the labels, top-1 agreement and logit cosine against + the fp16 engine and against the timm PyTorch fp32 model, batch 1. +- Non-finite scan: the 1500 held-out images and 8 stress inputs (constants 0, + +-1, +-10 and Gaussian noise with standard deviation 1, 3 and 10). +- Speed: CUDA graph replay, median of 15 interleaved rounds of 200 calls, + spread at most 0.1%. + +### Results + +Speed and size, batch 1 (ms per engine call; x is fp16 time over FP8 time; +L1 and L3 are builder optimization levels 1 and 3): + +| Model | GPU | TensorRT | fp32 L1 | fp16 L1 | FP8 L1 | x L1 | fp16 L3 | FP8 L3 | x L3 | plan MiB fp16 to FP8 (L1) | +|---|---|---|---|---|---|---|---|---|---|---| +| s | B200 | 11.4 | 0.929 | 0.431 | 0.462 | 0.93 | 0.398 | 0.414 | 0.96 | 12.6 to 10.2 | +| s | B300 | 11.4 | 0.897 | 0.420 | 0.442 | 0.95 | 0.371 | 0.388 | 0.96 | 12.7 to 10.3 | +| xs | B200 | 11.4 | 0.749 | 0.408 | 0.449 | 0.91 | 0.385 | 0.401 | 0.96 | 6.3 to 5.6 | +| xs | B300 | 11.4 | 0.714 | 0.406 | 0.425 | 0.96 | 0.355 | 0.367 | 0.97 | 6.4 to 5.6 | +| xxs | B200 | 11.4 | 0.614 | 0.357 | 0.403 | 0.89 | 0.338 | 0.358 | 0.94 | 4.3 to 4.0 | +| xxs | B300 | 11.4 | 0.583 | 0.363 | 0.385 | 0.94 | 0.313 | 0.331 | 0.95 | 4.3 to 4.0 | +| s | B200 | 11.1 | 0.915 | 0.447 | 0.462 | 0.97 | 0.412 | 0.410 | 1.01 | 12.6 to 10.3 | + +Layers in the plan (TensorRT 11.4, B200, fp16 to FP8): s 154 to 163, xs 165 to +176, xxs 153 to 164 at level 1; 152 to 161, 162 to 173, 149 to 161 at level 3. + +Quality, TensorRT 11.4, level 1, 1500 held-out images (B300 in brackets where +it differs by more than 0.1 point): + +| Model | PyTorch fp32 top-1 | fp16 engine top-1 | FP8 engine top-1 | FP8 agreement with fp16 top-1 | FP8 logit cosine to fp16, mean / min | +|---|---|---|---|---|---| +| s | 78.00% | 78.00% | 78.07% | 98.5% | 0.9971 / 0.896 (0.854) | +| xs | 74.33% | 74.47% | 74.20% (74.33%) | 97.7% | 0.9961 / 0.931 | +| xxs | 68.87% | 68.73% | 68.40% | 95.8% | 0.9951 / 0.951 | + +The fp16 engine agrees with the PyTorch top-1 on 99.7% to 99.8% of the images +(logit cosine 0.99995 or higher on average). Level 3 gives the same quality as +level 1 (FP8 agreement with fp16 95.7% to 98.7%). + +Non-finite values: none on the 1500 images for any engine. One of the 8 +stress inputs gives non-finite logits on mobilevit_xxs in fp16 and in FP8 alike +(level 1 and 3, B200 and B300; PyTorch fp32 is finite); mobilevit_xs and +mobilevit_s are finite. This is fp16 behavior of the existing build, not +caused by FP8. + +### NVFP4 (not tried) + +The only GEMMs are the small transformer linears (K = 64 to 480); FP8 is +already slower than fp16 there, and NVFP4 would add block-scale handling on the +same tiny GEMMs and lose more accuracy. TensorRT does not support NVFP4 on the +convolutions that dominate the time. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 convolutions, FP8 attention matmuls, percentile or entropy calibration | not measured | +| Batch size above 1 | the family builds batch 1 only | +| H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A TensorRT release that fuses the Quantize into the producing layer (layer + norm, GELU) for small GEMMs, so that FP8 adds no layers. +- FP8 for `attn.proj` once the TensorRT attention fusion accepts a Quantize on + its output. +- A batched or larger-resolution variant of this family, where the transformer + GEMMs would be large enough to be limited by math.