From 75b24471e5b7f8da304d8dc7bfa736119628e160 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 15:06:11 -0800 Subject: [PATCH] docs(detr): record why FP8 is not shipped Add a README for the detr family with the FP8 self-quantization study of facebook/detr-resnet-50 on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-tensor FP8 for the encoder and decoder MLP linears, the backbone, attention projections and heads kept fp16), measurement method, speed, plan size, COCO AP and agreement with PyTorch fp32, and each part measured alone. The best recipe is 1.026x to 1.033x faster on B200 and 1.013x to 1.014x on B300 at batch 1, the plan is 14.5% smaller and AP is unchanged, but precision and recall against PyTorch drop from 0.995 to 0.975. The model is launch bound: a third of the time is the ResNet-50 convolutions and the GEMMs have at most 1050 rows. Every other part loses quality, gives no speed or does not build on 11.4. FP8 is not shipped. NVFP4 was not tried (not limited by weight reads). No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/detr/README.md | 156 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 families/detr/README.md diff --git a/families/detr/README.md b/families/detr/README.md new file mode 100644 index 0000000000..3999fbff6d --- /dev/null +++ b/families/detr/README.md @@ -0,0 +1,156 @@ +# DETR + +This family builds DETR object detectors (`facebook/detr-resnet-50`) for +`image_to_boxes` as one TensorRT engine: ResNet-50 backbone, 1x1 input +projection, sine position embeddings, a 6-layer encoder and a 6-layer decoder +(d_model 256, 100 queries) and the class and box heads. The manifest builds +batch 1, fp16, with image capacity 1333 x 1333. The family has no quantized +build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the transformer linears was built and measured on +TensorRT 11.4 (and 11.1) for `facebook/detr-resnet-50`. It is not shipped: the +best recipe (the encoder and decoder MLPs) is only 1.01x to 1.03x faster than +fp16 at batch 1, and detection agreement with PyTorch drops from 0.995 to 0.975 +(precision and recall), although COCO AP does not change. This section records +the recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +Recipe: FP8 for fc1 and fc2 of the 6 encoder and 6 decoder layers (24 linears). +TensorRT 11.4, batch 1, 800 x 1067 input (the range over 800 x 800, 800 x 1067 +and 800 x 1333 is in brackets). + +| Item | fp16 (manifest) | FP8 MLPs (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, B200 | 1.223 ms | 1.192 ms | 1.03x (1.026x to 1.033x) | +| Latency, B300 | 1.113 ms | 1.098 ms | 1.01x (1.013x to 1.014x) | +| Plan size, B200 / B300 | 82.06 / 82.10 MB | 70.16 / 70.24 MB | -14.5% / -14.4% | +| COCO AP, B200 / B300 (PyTorch fp32 0.4241) | 0.4243 / 0.4248 | 0.4250 / 0.4242 | unchanged | +| Precision / recall vs PyTorch, B200 | 0.9951 / 0.9961 | 0.9764 / 0.9726 | -0.02 | +| Precision / recall vs PyTorch, B300 | 0.9953 / 0.9965 | 0.9756 / 0.9743 | -0.02 | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for DETR. + +- Speed: 1 to 3 percent on B200 and about 1 percent on B300. The model is + launch bound: about 250 small plan layers, the GEMMs have at most 1050 rows, and + the ResNet-50 convolutions (a third of the time) have no FP8 path in this + family. +- Quality: AP is unchanged, but 2 to 3 percent of the detections at score 0.5 + differ from PyTorch (the fp16 engine: 0.5 percent). +- Every other part either loses quality (encoder attention, decoder + cross-attention, box head), gives no speed (decoder self-attention, class + head) or does not build on TensorRT 11.4 (attention output projection). +- The plan is 14.5 percent smaller; that alone is not a reason to ship. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of DETR exists on the Hub, and no official +quantization config exists for it. The start was the guideline for small +encoders and vision models: all attention and MLP linears, per-tensor static +activation scales, ModelOpt max calibration. Each part was then measured alone, +and every part that loses quality or gives no speed was put back to fp16. + +| Choice | Decision | +|---|---| +| Quantized (24 of 100 linears) | `fc1`, `fc2` of the 6 encoder and 6 decoder layers | +| fp16 | ResNet-50 backbone and 1x1 input projection (convolutions), attention q, k, v and `out_proj`, class and box heads, layer norms, position embedding, attention matmuls and softmax | +| Graph | `input -> Quantize -> Dequantize(fp16) -> MatMul`; weight `FP8 constant -> Dequantize(fp16)` | +| Weight scale | per tensor, amax / 448 of each linear, at build time (FP8 weights not stored) | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the transformers PyTorch model, 300 COCO validation images at native size, batch 1 | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: none. The decoder layer-0 self-attention `v_proj` reads an all-zero input (the queries start at zero) and has no scale | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +TensorRT picks real FP8 GEMMs for the MLPs (on B300: +`cutlass3x_sm100 ... gemm_e4m3_e4m3_f32_f16_f16 ... bias_f16_relu`, 36 FP8 +layers in the plan). + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of one TensorRT version + built and timed in the same job; TensorRT 11.4.0.106 (target) and 11.1.0.106. + Family code at `94fcf143`, before the Task SDK migration (#2019). +- Data: the first 800 validation images of `detection-datasets/coco` with their + COCO 2017 boxes. Calibration: every second image of the first 600 (300 + images). Quality: 500 other images (held out). +- Reference: transformers `DetrForObjectDetection` in fp32 with the same + preprocessed input. AP is `pycocotools` COCO box AP on the 500 images + (comparable only inside these tables). Precision and recall: detections at + score 0.5 matched to the PyTorch detections (same class, IoU at least 0.5). + Box IoU is the mean IoU of the matched boxes. +- Non-finite scan: the 500 images and noise, white and black inputs, every + variant. +- Speed: batch 1, random input of the given size, 30 warm-up runs, then 200 + runs timed with CUDA events; 3 interleaved rounds over all variants, minimum + per variant. + +### Results + +Speed and size, both MLPs (x is fp16 time over FP8 time): + +| GPU | TensorRT | size | fp32 ms | fp16 ms | FP8 ms | x | plan MB fp16 to FP8 | +|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 800x1067 | 2.160 | 1.223 | 1.192 | 1.026 | 82.06 to 70.16 | +| B200 | 11.4 | 800x1333 | 2.491 | 1.338 | 1.296 | 1.033 | | +| B200 | 11.4 | 800x800 | 1.734 | 1.113 | 1.078 | 1.032 | | +| B300 | 11.4 | 800x1067 | 2.012 | 1.113 | 1.098 | 1.013 | 82.10 to 70.24 | +| B300 | 11.4 | 800x1333 | 2.341 | 1.205 | 1.189 | 1.014 | | +| B300 | 11.4 | 800x800 | 1.617 | 1.013 | 1.000 | 1.013 | | +| B200 | 11.1 | 800x1067 | 2.154 | 1.253 | 1.223 | 1.025 | 82.09 to 70.24 | +| B300 | 11.1 | 800x1067 | 2.055 | 1.153 | 1.139 | 1.013 | 82.16 to 70.25 | + +Each part alone (TensorRT 11.4, 500 held-out images; quality on B300, speed x +at 800 x 1067 on B200 / B300): + +| Engine | FP8 linears | AP | prec / rec vs PyTorch | box IoU | x B200 / B300 | +|---|---|---|---|---|---| +| PyTorch fp32 | - | 0.4241 | - | - | - | +| fp16 engine | 0 | 0.4248 | 0.9953 / 0.9965 | 0.991 | 1 / 1 | +| encoder MLP | 12 | 0.4246 | 0.979 / 0.977 | 0.976 | 1.011 / 1.010 | +| decoder MLP | 12 | 0.4245 | 0.988 / 0.988 | 0.985 | 1.022 / 1.007 | +| **both MLPs (the recipe)** | 24 | **0.4242** | 0.976 / 0.974 | 0.974 | **1.026 / 1.013** | +| decoder self-attention q, k, v | 17 | 0.4248 | 0.988 / 0.990 | 0.984 | 0.995 / 0.996 | +| both MLPs + decoder self-attention | 41 | 0.4241 | 0.975 / 0.970 | 0.971 | 1.007 / 1.000 | +| encoder self-attention q, k, v | 18 | 0.4178 | 0.969 / 0.961 | 0.936 | 0.992 / 0.989 | +| decoder cross-attention q, k, v | 18 | 0.4177 | 0.945 / 0.943 | 0.917 | 0.997 / 0.995 | +| class head | 1 | 0.4233 | 0.981 / 0.988 | 0.989 | 1.004 / 1.003 | +| box head (3 linears) | 3 | 0.3760 | 0.921 / 0.922 | 0.845 | 1.005 / 1.000 | +| every part that builds (`out_proj` excluded) | 81 | 0.3666 | 0.861 / 0.849 | 0.835 | 0.999 / 0.990 | +| attention `out_proj` (TensorRT 11.1 only) | 18 | 0.4231 | 0.981 / 0.983 | 0.968 | 0.983 / 0.989 | + +B200 quality is within 0.003 AP and 0.006 precision or recall of the B300 row; +11.1 gives the same picture. All variants were finite on every input. + +Time split (B300, TensorRT 11.4, fp16, 800 x 1067, layer profiler, which adds +overhead): 2.37 ms in total, of which 0.80 ms (34 percent) is the fused ResNet +convolutions; the rest is spread over about 250 other plan layers of 10 to 50 +microseconds each. The GEMMs +have 850 rows (800 x 1067) to 1050 rows (800 x 1333) and 256 or 2048 columns, +too small to keep the GPU busy, so FP8 saves only a few microseconds per GEMM. + +### NVFP4 (not tried) + +The transformer has 17.3 M weights (34.6 MB in fp16, a few microseconds of +reads), against a 1.0 to 1.4 ms run, so the model is not limited by weight +reads. FP8 gives 1 to 3 percent, and NVFP4 would lose more quality than FP8. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 for the backbone convolutions | no FP8 convolution path in this family | +| Batch above 1 | the family builds batch 1 only | +| Percentile or entropy calibration, per-channel weight scales | not measured | +| Full 5000-image COCO validation, H100, native runtime with an FP8 bundle | not measured | + +### When to revisit + +- A larger batch or a larger DETR (wider d_model, more queries), where the + transformer GEMMs are large enough for FP8 to pay. +- FP8 convolutions for the ResNet-50 backbone, which is a third of the time. +- A TensorRT release that builds FP8 `out_proj` behind the fused attention (11.4 + fails with "Could not find any implementation for node"; 11.1 builds it but + gives no speed).