From 3c4b8f74397ead0014d5c6076816e5c00295fed7 Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 15:25:16 -0800 Subject: [PATCH] docs(rt_detr_v2): record why FP8 is not shipped Add a README for the rt_detr_v2 family with the FP8 self-quantization study of PekingU/rtdetr_v2_r18vd on TensorRT 11.4 (B200, B300) and 11.1: recipe (per-tensor FP8 on the linear layers, split in seven parts; convolutions kept fp16), measurement method, speed, plan size, COCO-style AP and detection agreement against PyTorch fp32. Latency stays at the fp16 level (0.98x to 1.03x for every part and combination, 1.01x / 1.00x for the measured default on B200 / B300 with 11.4), the plan is 3% smaller, and each part moves 2 to 3 percent of the detections at the 0.3 threshold with AP kept. The linears are 22% of the kernel time of a 0.7 ms engine of 182 small kernels; the added Quantize kernels take back the GEMM gain. FP8 is not shipped. On TensorRT 11.4 the two sampling projections of the deformable attention give non-finite outputs when both are FP8 (finite on 11.1). NVFP4 was not tried. No problem on main was found. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/rt_detr_v2/README.md | 210 ++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 families/rt_detr_v2/README.md diff --git a/families/rt_detr_v2/README.md b/families/rt_detr_v2/README.md new file mode 100644 index 0000000000..5996837b05 --- /dev/null +++ b/families/rt_detr_v2/README.md @@ -0,0 +1,210 @@ +# RT-DETR v2 + +This family builds RT-DETR v2 detectors (`PekingU/rtdetr_v2_r18vd`) for +`object_detection` as one TensorRT engine: 640 x 640 input, batch 1, fp16 or +fp32 (the manifest uses fp16). The engine holds the ResNet-D backbone, the +hybrid encoder (one AIFI transformer layer on the stride-32 level, then the CCFM +fusion of RepVGG convolutions) and the decoder (query selection on 8400 anchors, +3 layers with self-attention, deformable cross-attention and a feed-forward, box +and class heads). The family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the linear layers was built and measured on TensorRT +11.4 (and 11.1) for `PekingU/rtdetr_v2_r18vd`. It is not shipped: no part and no +combination of parts is faster than fp16 beyond the run-to-run spread (0.98x to +1.03x on B200 and B300), the plan is only 3% smaller with the default parts, +and each quantized part moves 2 to 3 percent of the detections at the runtime +score threshold (COCO AP stays at the fp16 level). This section records the +recipe, the method and the numbers, so that the study does not have to be +repeated. The FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 default, 12 linears (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4) | 0.727 / 0.705 ms | 0.723 / 0.703 ms | 1.01x / 1.00x | +| Latency, all 44 linears, B200 / B300 (TensorRT 11.4) | 0.727 / 0.705 ms | 0.726 / 0.698 ms | 1.00x / 1.01x | +| Plan size, default / all 44 linears | 42.8 MiB | 41.6 / 39.0 MiB | -3% / -9% | +| COCO-style AP, B200 / B300 (PyTorch fp32 0.5080) | 0.5077 / 0.5077 | 0.5085 / 0.5094 | at the fp16 level | +| Detections at score 0.3 matching PyTorch (prec / rec), B200 | 0.992 / 0.990 | 0.976 / 0.974 | 2 percent move | +| Non-finite outputs | 0 | 0 (sampling pair on 11.4: all) | see Results | + +Numbers of this table: one TensorRT 11.4 job per GPU on the current family code +(the spot check in Results). The per-part tables come from earlier jobs on the +same family code and the same TensorRT versions. + +Decision: do not ship FP8 for RT-DETR v2. + +- Speed: the engine runs in about 0.7 ms as 182 small kernels. The linear GEMMs + (the FP8 candidates) are 22% of the kernel time, about 9 microseconds each, + close to the launch floor. FP8 makes them 4% faster (355 to 342 + microseconds), and the added Quantize and cast kernels (about 45 + microseconds) take that back. +- The convolutions (42% of the kernel time) have no FP8 path in this family and + stay fp16. +- Size: the weights are 42 MiB; reading them takes about 5 microseconds, under + 1% of the latency, so the engine is not weight-bandwidth bound. The plan is + 3% smaller with the default parts and 9% smaller with all 44 linears. +- Quality: AP is kept, but the detection set at the 0.3 threshold drifts by 2 + to 3 percent per part and 5 percent with all parts, for no speed gain. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of RT-DETR v2 exists on the Hub, and no official +quantization config exists for it. The layer list follows the usual guidance +for small vision encoders and detectors: attention, MLP and head linears with +static per-tensor activation scales; convolutions, norms and position tables +stay fp16. Each quantized linear is built as +`input -> Quantize -> Dequantize(fp16) -> MatMul (+bias)`, with the weight +`FP8 constant -> Dequantize(fp16)`. Linears that read the same tensor with the +same scale share one Quantize/Dequantize pair. + +| Choice | Decision | +|---|---| +| Parts (55 linears in total) | `aifi` (6), `decoder_self` (q, k, v, out of 3 layers; 12), `decoder_value` (value and output projection of the deformable attention; 6), `decoder_sampling` (sampling offsets and attention weights; 6), `decoder_ffn` (6), `query_select` (encoder output projection, class and box heads; 4), `refine` (query position head, per-layer box deltas and class heads; 10) | +| Measured default | `aifi` + `decoder_value` (12 linears), the two parts that move the detections least together | +| fp16 | all convolutions (backbone, CCFM, input projections), norms, attention matmuls and softmax, grid sample, top-k; a linear whose input or output width is not a multiple of 16 (position head input of 4, last box-head layers with 4 outputs) | +| Weight scale | per tensor, amax / 448, at build time | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the transformers `RTDetrV2ForObjectDetection` model, 256 COCO validation images resized like the runtime | +| Outlier rule | a scale above 20x the median of its kind stays fp16; measured: no outlier | +| Dequantize type | fp16; no non-finite value except the sampling pair on 11.4 (below), fp32 Dequantize was not needed | +| Scale table key | model config (hidden size, layers, feed-forward sizes, queries, labels, input size) and a numeric weight fingerprint | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants of a table built and timed + in that job; TensorRT 11.4.0.106 (target) and 11.1.0.106. Builder + optimization level 3 (the family default), strongly typed networks. +- Reference: PyTorch fp32 `RTDetrV2ForObjectDetection` (transformers 5.17.0) on + the same preprocessed pixels. +- Quality: COCO validation images 256 to 555 of `detection-datasets/coco` + (300 held out; 0 to 255 are the calibration set), resized to 640 x 640 without + normalization, as the runtime does. Detections are the transformers + post-processing (sigmoid, top 300 of queries x classes). AP is a COCO-style AP + (101-point, IoU 0.50 to 0.95) from a small numpy implementation on these 300 + images, comparable only inside these tables. Agreement: detections at score + 0.3 (the runtime threshold) matched greedily to the reference detections (same + class, IoU at least 0.5); `prec` is the share of the engine's detections that + match, `rec` the share of the reference's. +- Non-finite scan: the 300 images and 3 stress inputs (noise, all white, all + black). +- Speed: `execute_async_v3` loop, 400 calls after 50 warm-up, 5 interleaved + rounds; the table shows the fastest round (the median round differs by at + most 0.0004 ms). x is fp16 time over FP8 time. + +### Results + +Spot check, TensorRT 11.4 (one job per GPU, all variants built and timed in +that job; x is fp16 time over variant time): + +| GPU | fp32 ms | fp16 ms | FP8 default ms (x) | FP8 all 44 ms (x) | plan MiB fp16 / default / 44 | AP fp16 / default / 44 | +|---|---|---|---|---|---|---| +| B200 | 1.085 | 0.727 | 0.723 (1.01) | 0.726 (1.00) | 42.8 / 41.6 / 39.0 | 0.5077 / 0.5085 / 0.5085 | +| B300 | 1.046 | 0.705 | 0.703 (1.00) | 0.698 (1.01) | 42.8 / 41.8 / 39.2 | 0.5077 / 0.5094 / 0.5073 | + +Default build against its fp16 control, earlier jobs (TensorRT 11.4 and 11.1): + +| GPU | TensorRT | fp32 ms | fp16 ms | FP8 default ms | x | plan MiB fp16 / default | prec / rec vs PyTorch, default | AP default | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 1.086 | 0.734 | 0.726 | 1.01 | 42.7 / 41.7 | 0.974 / 0.973 | 0.5092 | +| B300 | 11.4 | 1.030 | 0.688 | 0.682 | 1.01 | 42.7 / 41.5 | 0.972 / 0.973 | 0.5094 | +| B200 | 11.1 | 1.081 | 0.723 | 0.719 | 1.01 | 42.4 / 41.5 | 0.973 / 0.975 | 0.5080 | +| B300 | 11.1 | 1.034 | 0.688 | 0.687 | 1.00 | 42.6 / 41.3 | 0.971 / 0.975 | 0.5089 | + +The fp16 engine of the same GPU and TensorRT version varied by 1.5% between +jobs by up to 2.4% (B200, 11.4: 0.727 to 0.744 ms in five jobs), which is as +large as every FP8 gain measured. + +Each part alone and all parts together, x against the fp16 engine of the same +job: + +| FP8 linears | B200 11.4 | B300 11.4 | B200 11.1 | B300 11.1 | +|---|---|---|---|---| +| fp16 ms | 0.744 | 0.689 | 0.730 | 0.686 | +| `aifi` (6) | 1.010 | 0.999 | 0.996 | 1.002 | +| `decoder_self` (12) | 1.016 | 0.998 | 0.995 | 0.995 | +| `decoder_value` (6) | 1.010 | 0.991 | 1.013 | 0.992 | +| `decoder_ffn` (6) | 1.017 | 0.998 | 0.998 | 0.998 | +| `query_select` (4) | 1.013 | 0.991 | 0.995 | 0.978 | +| `refine` (10) | 1.004 | 0.994 | 1.000 | 0.989 | +| all except `decoder_sampling` (44) | 0.995 | 0.994 | 1.001 | 0.988 | +| `decoder_sampling` (6) | non-finite | non-finite | 0.984 | 0.982 | +| all 50 | non-finite | not run | 0.983 | 0.992 | + +Combinations, B200, TensorRT 11.4, one more job (fp16 0.741 ms): +`decoder_value` + `aifi` 1.012, `decoder_value` + `decoder_ffn` 1.025, +`decoder_value` + `decoder_ffn` + `aifi` 1.015, the same plus `decoder_self` (30 +linears) 1.028, the best seen, with 4.5 percent of the detections moved. + +Quality of each part, B200, TensorRT 11.4 (300 held-out images; the first row +is the floor): + +| Engine (FP8 linears) | AP | prec / rec vs PyTorch | prec / rec vs fp16 engine | box IoU vs fp16 | score error vs fp16 | +|---|---|---|---|---|---| +| PyTorch fp32 | 0.5080 | - | - | - | - | +| fp16 engine (floor) | 0.5077 | 0.991 / 0.991 | - | - | - | +| `aifi` (6) | 0.5098 | 0.979 / 0.980 | 0.978 / 0.979 | 0.994 | 0.0056 | +| `decoder_self` (12) | 0.5088 | 0.971 / 0.971 | 0.972 / 0.972 | 0.991 | 0.0067 | +| `decoder_value` (6) | 0.5088 | 0.982 / 0.982 | 0.982 / 0.983 | 0.993 | 0.0050 | +| `decoder_ffn` (6) | 0.5090 | 0.975 / 0.976 | 0.975 / 0.976 | 0.992 | 0.0064 | +| `query_select` (4) | 0.5068 | 0.975 / 0.976 | 0.977 / 0.979 | 0.990 | 0.0055 | +| `refine` (10) | 0.5088 | 0.980 / 0.972 | 0.981 / 0.973 | 0.992 | 0.0066 | +| all except `decoder_sampling` (44) | 0.5090 | 0.951 / 0.946 | 0.950 / 0.946 | 0.983 | 0.0140 | +| `decoder_sampling` (6), TensorRT 11.1 | 0.5087 | 0.987 / 0.986 | 0.987 / 0.988 | 0.993 | 0.0034 | +| `decoder_sampling` (6), TensorRT 11.4 | 0 (non-finite) | | | | | + +B300 and TensorRT 11.1 give the same picture (AP 0.507 to 0.510 for every +part). Each part moves 2 to 3 percent of the detections at the 0.3 threshold +and the effect adds up; the boxes stay (IoU 0.98 to 0.99). + +Non-finite scan: 0 non-finite outputs on the 300 images and the 3 stress inputs +for every variant, except `decoder_sampling` (and every set that contains it) on +TensorRT 11.4: the outputs are non-finite on every input, on B200 and B300, +while each of the two sampling linears alone is finite and TensorRT 11.1 is +finite with both. This part is not in the default. + +Kernel time per group of the fp16 engine and of the FP8 engine with 44 linears, +B200, TensorRT 11.4 (layer profiler; the profiler adds overhead, so the sums are +above the wall time; microseconds (count)): + +| Kernel group | fp16 | FP8, 44 linears | +|---|---|---| +| convolutions | 686 (65) | 673 (65) | +| linear GEMMs | 355 (41) | 342 (41) | +| norms, softmax, casts, adds, resizes, Quantize | 347 (49) | 392 (54) | +| deformable attention tail | 78 (9) | 87 (10) | +| grid sample | 76 (9) | 77 (9) | +| attention matmuls (AIFI) | 39 (4) | 40 (4) | +| pooling, top-k | 59 (5) | 59 (5) | +| sum | 1640 | 1668 | + +B300 and TensorRT 11.1 show the same split (linear GEMMs 20% to 22% of the +kernel time). The GEMMs are small (M = 300 queries or 400 tokens for most, M = +8400 for the value projections and the query-selection heads). + +### NVFP4 (not tried) + +The engine is launch and latency bound, not weight-bandwidth bound, and FP8 on +all linears gives no gain. NVFP4 cannot save more than the linears' own time +(22%) while adding dynamic-quantize kernels, and TensorRT has no NVFP4 +convolution for the 42% of the time spent in convolutions. + +### What was not tried + +| Item | Note | +|---|---| +| FP8 convolutions (backbone, CCFM) | no FP8 convolution path in this family; the largest share of the time | +| FP8 attention matmuls or softmax, percentile calibration, per-channel weights | not measured | +| Larger checkpoints (r34vd, r50vd, r101vd), batch greater than 1 | not measured; the family builds batch 1 only | +| Full 5000-image COCO validation with `pycocotools`, H100, native runtime with an FP8 bundle | not measured | +| Isolating the TensorRT 11.4 non-finite output of the sampling pair | a tiny engine with two FP8 linears of 192 and 96 outputs on one activation does not reproduce it; separate Quantize/Dequantize pairs do not avoid it | + +### When to revisit + +- An FP8 convolution path for this family: the convolutions are 42% of the + kernel time, so this is the only part large enough to give a real speed-up. +- Larger RT-DETR v2 checkpoints or larger batches, where the linears become a + larger share of the time. +- A TensorRT release that fuses the Quantize into the producer of each FP8 + linear, so the added kernels no longer take back the GEMM gain.