From 835402bfce6c33d3c8b3e1cfe87bae81b29ef811 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 17:01:55 -0800 Subject: [PATCH] docs(timm_xcit): record why FP8 is not shipped Add the FP8 self-quantization study of timm XCiT to the family README: recipe (FP8 for the attention and MLP linears, per-tensor weight scales, ModelOpt-calibrated static activation scales, fp16 Dequantize), method, speed, plan size and ImageNet quality against PyTorch fp32 on TensorRT 11.4 (B200, B300) and 11.1. At batch 1 the FP8 engine is 0.94x to 0.95x of fp16 for xcit-nano and xcit-tiny at the family's builder optimization level 1 and 1.01x to 1.03x at level 3; the plan is 19% to 31% smaller and FP8 changes the top class of 2% to 5% of the images. The engine is launch bound at batch 1. Larger configs reach at most 1.07x at batch 1. FP8 is not shipped. NVFP4 was not tried. No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/timm_xcit/README.md | 156 +++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) diff --git a/families/timm_xcit/README.md b/families/timm_xcit/README.md index d94f22f4ac..b9cacb579f 100644 --- a/families/timm_xcit/README.md +++ b/families/timm_xcit/README.md @@ -32,3 +32,159 @@ Inference and complete host scores are timed; argmax, finite scans and JSON reporting are not. The existing reference policy excludes preprocessing while the native Task includes family preprocessing; this scope difference is not claimed resolved and no performance improvement is claimed. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the attention and MLP linears was built and measured on +TensorRT 11.4 (and 11.1) for the two manifest checkpoints and five larger XCiT +configs. It is not shipped: at batch 1 (the only batch the family builds) the +FP8 engine is 0.94x to 0.95x of the fp16 engine for xcit-nano-12-p16 and +xcit-tiny-12-p16 at the family's builder optimization level 1, and 1.01x to +1.03x at level 3. The plan is 19% to 31% smaller. This section records the +recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +Manifest checkpoints, batch 1, TensorRT 11.4 (x is fp16 time over FP8 time): + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| nano latency, level 1, B200 / B300 | 0.580 / 0.534 ms | 0.613 / 0.570 ms | 0.95x / 0.94x | +| tiny latency, level 1, B200 / B300 | 0.614 / 0.575 ms | 0.647 / 0.611 ms | 0.95x / 0.94x | +| nano latency, level 3, B200 / B300 | 0.498 / 0.453 ms | 0.493 / 0.447 ms | 1.01x / 1.01x | +| tiny latency, level 3, B200 / B300 | 0.557 / 0.514 ms | 0.539 / 0.498 ms | 1.03x / 1.03x | +| Plan size, level 1, B200, nano / tiny | 7.4 / 14.5 MiB | 5.6 / 9.9 MiB | -24% / -31% | +| Top-1 agreement with PyTorch fp32, nano / tiny, B200 | 99.8% / 99.6% | 96.4% / 98.4% | -3.4 / -1.2 points | +| Minimum logit cosine vs PyTorch fp32, nano / tiny, B200 | 0.9997 / 0.9991 | 0.985 / 0.974 | lower | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for timm XCiT. + +- Speed: at batch 1 the engine is bound by kernel launches, not by math. The + token count is 196 and the linears are 128 to 768 wide, so each fp16 or FP8 + GEMM takes about 10 microseconds in the layer profiler, at either precision. + At level 1 the 12 extra Quantize layers (264 instead of 252 layers for nano) + are not fused and make FP8 slower. At level 3 they are fused (252 layers in + both plans) and FP8 is 1% to 3% faster than the level-3 fp16 engine, which is + itself 1.10x to 1.18x faster than level 1 without any quantization. +- Larger configs (not in the manifests) gain at most 1.07x at batch 1 and 1.15x + at batch 32 (scratch build; the family builds batch 1 only). +- Quality: top-1 stays within noise of the fp16 engine, but FP8 changes the top + class of 2% to 5% of the images on nano and tiny (near ties), and the minimum + logit cosine against PyTorch drops from above 0.999 to 0.96 to 0.99. +- The only clear gain is plan size, which is small for the manifest checkpoints + (2 to 5 MiB). + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of any timm XCiT classifier exists on the Hub, and no +official quantization config exists for one. The recipe follows the guidelines +for small vision encoders (the `timm_vit` flow): all attention and MLP linears, +nothing else. Each quantized linear is built as +`input -> Quantize -> Dequantize(fp16) -> MatMul`, with the weight +`FP8 constant -> Dequantize(fp16)`, then the bias add. + +| Choice | Decision | +|---|---| +| Quantized (per block) | `attn.qkv` (q, k and v share one input scale, each keeps its own weight scale), `attn.proj`, `mlp.fc1`, `mlp.fc2`; per class-attention block the key and value (`attn.kv`, shared input) | +| fp16 | stem and depthwise convolutions, position encoding, layer norms, the channel-attention matmuls and softmax, class-attention query, output and MLP projections (one-token GEMVs), classifier head | +| Weight scale | per tensor, amax / 448, at build time; FP8 weights are never stored | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the timm PyTorch model, 500 ImageNet validation images | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job; TensorRT 11.4.0.106 (target) and + 11.1.0.106 (B200 sanity check). torch 2.14, timm 1.0.28, ModelOpt 0.46.1. +- Builds: the family's `build()` with `quantization=fp8` on an fp16 build. The + level-3 rows replace only the builder optimization level (1 on main, 3 in the + pending level change); everything else is the family code. +- Data: ImageNet validation images from `clip-benchmark/wds_imagenet1k`. + Calibration: 500 images (every 25th image, the first 500 of a 2000-image set). + Study 1 (all configs): the other 1500 images of that set. Study 2 (manifest + checkpoints, levels 1 and 3): 500 images of a separate 4000-image set (every + 12th image, shuffled with seed 0, indices 1100 to 1599; 6 of them are also + calibration images). +- Quality: top-1 against the labels, top-1 agreement and logit cosine against + the timm PyTorch fp32 model, batch 1. The fp16 engine is the floor. +- Non-finite scan: the evaluation images and 17 to 26 stress inputs (noise with + standard deviation 1, 3 and 10, constants), all variants, B200 and B300. +- Speed: CUDA graph replay at batch 1. Study 2: all four variants of a model + built in one job, median of 21 interleaved rounds of 100 replays (spread at + most 2.5%). Study 1: variants built one after another in one job, median of + 5 runs of 200 replays, level 1 only. + +### Results + +Study 2, manifest checkpoints, batch 1 (x is fp16 time over FP8 time at the +same level; plan MiB fp16 to FP8): + +| Model | GPU | TensorRT | fp16 L1 ms | FP8 L1 ms | x L1 | fp16 L3 ms | FP8 L3 ms | x L3 | plan MiB L1 | plan MiB L3 | +|---|---|---|---|---|---|---|---|---|---|---| +| nano | B200 | 11.4 | 0.580 | 0.613 | 0.95 | 0.498 | 0.493 | 1.01 | 7.4 to 5.6 | 9.6 to 7.7 | +| nano | B300 | 11.4 | 0.534 | 0.570 | 0.94 | 0.453 | 0.447 | 1.01 | 6.9 to 5.2 | 10.7 to 7.9 | +| tiny | B200 | 11.4 | 0.614 | 0.647 | 0.95 | 0.557 | 0.539 | 1.03 | 14.5 to 9.9 | 16.7 to 12.1 | +| tiny | B300 | 11.4 | 0.575 | 0.611 | 0.94 | 0.514 | 0.498 | 1.03 | 14.1 to 9.6 | 17.8 to 12.2 | +| nano | B200 | 11.1 | 0.555 | 0.602 | 0.92 | 0.485 | 0.484 | 1.00 | 7.4 to 5.6 | 9.5 to 7.7 | +| tiny | B200 | 11.1 | 0.596 | 0.648 | 0.92 | 0.538 | 0.535 | 1.01 | 14.5 to 9.9 | 16.6 to 11.9 | + +Study 2 quality, 500 images, TensorRT 11.4, B200 (B300 in brackets where it +differs; PyTorch fp32 top-1: nano 72.0%, tiny 78.8%): + +| Model | Engine | Top-1 | Agreement with PyTorch top-1 | Mean / min logit cosine | Non-finite rows | +|---|---|---|---|---|---| +| nano | fp16 L1 | 72.2% | 99.8% | 0.99995 / 0.9997 | 0 | +| nano | FP8 L1 | 73.0% (72.4%) | 96.4% (96.2%) | 0.9964 / 0.985 (0.981) | 0 | +| nano | fp16 L3 | 72.4% | 99.6% | 0.99995 / 0.9997 | 0 | +| nano | FP8 L3 | 72.4% | 97.0% (96.4%) | 0.9964 / 0.988 (0.987) | 0 | +| tiny | fp16 L1 | 79.2% | 99.6% | 0.99996 / 0.9991 | 0 | +| tiny | FP8 L1 | 79.4% (78.4%) | 98.4% (98.2%) | 0.9966 / 0.974 (0.969) | 0 | +| tiny | fp16 L3 | 79.0% | 99.8% | 0.99995 / 0.9975 | 0 | +| tiny | FP8 L3 | 78.8% | 98.4% | 0.9966 / 0.965 | 0 | + +TensorRT 11.1 gives the same picture (FP8 agreement 96.0% to 98.8%, no +non-finite output). + +Study 1, all configs, level 1, batch 1, TensorRT 11.4, 1500 held-out images +(x is fp16 time over FP8 time; top-1 on B200): + +| Config | x B200 | x B300 | plan MiB fp16 to FP8 (B200) | Top-1 PyTorch / fp16 / FP8 | FP8 agreement with PyTorch | FP8 mean logit cosine | +|---|---|---|---|---|---|---| +| nano_12_p16 | 0.93 | 0.95 | 7.4 to 5.6 | 67.87 / 68.00 / 67.67 | 95.0% | 0.9962 | +| tiny_12_p16 | 0.95 | 0.94 | 14.5 to 9.9 | 74.87 / 74.93 / 75.40 | 96.5% | 0.9965 | +| small_12_p8 | 1.02 | 1.04 | 52.3 to 32.2 | 81.40 / 81.53 / 81.60 | 98.9% | 0.9980 | +| small_24_p16 | 1.07 | 1.05 | 93.0 to 53.6 | 81.20 / 81.20 / 81.27 | 98.9% | 0.9984 | +| medium_24_p16 | 1.01 | 0.97 | 163.0 to 91.7 | 80.33 / 80.40 / 80.20 | 99.1% | 0.9986 | +| large_24_p16 | 1.00 | 1.01 | 362.8 to 200.0 | 82.47 / 82.47 / 82.40 | 99.3% | 0.9990 | +| large_24_p8 | 1.05 | 1.04 | 363.5 to 200.7 | 84.40 / 84.40 / 84.47 | 99.7% | 0.9989 | + +Batch 8 and 32 (scratch builds, level 1, TensorRT 11.4): FP8 is 1.00x to +1.04x for nano and tiny and 1.01x to 1.15x for the larger configs (best: +large_24_p8 at batch 32, B200, 23.1 ms fp16 and 20.1 ms FP8). Study 1 found no +non-finite output on 1500 images and 26 stress inputs. + +### NVFP4 (not tried) + +FP8 already gives no speedup at batch 1, the manifest plans are 7 to 15 MiB, so +weight reads are not the limit, and FP8 already moves the top class of 2% to 5% +of the images on the small configs. NVFP4 would lose more accuracy for no +expected speed gain. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 for the depthwise convolutions, the one-token class-attention GEMVs or the channel-attention matmuls | not measured; small or launch-bound work | +| Layer-norm output Q/DQ, fixed softmax-output scale, percentile calibration | not measured | +| 384 px configs, H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- The family builds batch sizes above 1: FP8 is 1.08x to 1.15x at batch 32 for + the larger configs. +- Larger configs (small_24 and up) become manifest checkpoints and the family + builds at level 3, if FP8 then gains clearly more than the 1% to 3% measured + for nano and tiny. +- A TensorRT release that cuts the per-kernel overhead of small GEMMs at batch 1.