From 8ffa9796c9e97d9d951601ca6d5dbc9cec29809c Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 14:01:10 -0800 Subject: [PATCH] docs(yolov8): record why FP8 is not shipped Add a family README that records the FP8 self-quantization study of yolov8n on TensorRT 11.4 and 11.1: FP8 is slower than fp16 (0.89x to 0.97x), the plan saves 2.7 MiB and COCO AP drops by 0.5 point, because bias, SiLU and the next Quantize are not fused into the FP8 convolution. Documentation only. Signed-off-by: Zhenshan Xie --- families/yolov8/README.md | 125 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 families/yolov8/README.md diff --git a/families/yolov8/README.md b/families/yolov8/README.md new file mode 100644 index 0000000000..18df4c41c7 --- /dev/null +++ b/families/yolov8/README.md @@ -0,0 +1,125 @@ +# YOLOv8 + +This family builds the YOLOv8 detector `yolov8n` (`yolov8n.pt` of +`Ultralytics/YOLOv8`) for `image_to_boxes` as one TensorRT engine: 640 x 640 +input, batch 1, fp16 or fp32 (the manifest uses fp16). The head is not end to +end: the engine reports every anchor with its box and strongest class, and the +runtime drops weak scores and runs greedy per-class non-max suppression. The +family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the convolutions was built and measured on TensorRT +11.4 (and 11.1) for `yolov8n`. It is not shipped: FP8 is slower than fp16 +(0.92x to 0.97x at batch 1, 0.89x to 0.91x at batch 8), the plan saves only +2.7 MiB, and COCO AP drops by 0.5 point. This section records the recipe, the +method and the numbers, so that the study does not have to be repeated. The +FP8 code was not merged; this family has no FP8 build. + +### Summary and decision + +| Item | fp16 (manifest) | FP8 (fp16 engine) | FP8 vs fp16 | +|---|---|---|---| +| Latency, batch 1, B200 / B300 (TensorRT 11.4) | 0.317 / 0.287 ms | 0.335 / 0.312 ms | 0.95x / 0.92x | +| Latency, batch 8, B200 / B300 (TensorRT 11.4) | 0.612 / 0.578 ms | 0.673 / 0.643 ms | 0.91x / 0.90x | +| Plan size, B200 | 7.36 MiB | 4.60 MiB | -37% | +| COCO AP, B200 (PyTorch fp32 0.3971) | 0.3969 | 0.3922 | -0.5 point | +| Detections that match PyTorch (prec / rec) | 0.998 / 1.000 | 0.926 / 0.888 | 7% to 11% changed | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for YOLOv8. + +- Speed: the FP8 convolutions run on FP8 kernels, but bias, SiLU and the + Quantize of the next convolution are not fused into the FP8 convolution. They + run as extra pointwise kernels (23 in the fp16 plan, 30 in the FP8 plan; 89 + to 99 plan layers) and take the gain back. `yolov8n` is small, so it is + launch and pointwise bound, and FP8 ends up slower than fp16. +- Depthwise convolutions have no FP8 kernel on TensorRT 11.4 (the build fails + with "no tactics"). `yolov8n` has no depthwise convolution, so this does not + limit it. +- The only gain is plan size (2.7 MiB less). +- The sibling families `yolov5` (`yolov5n`) and `yolo11` (`yolo11n`) were + measured with the same recipe and gave the same result: FP8 at 0.89x to 0.95x + and 0.90x to 0.93x of fp16 at batch 1, AP 0.3075 to 0.2991 and 0.4158 to + 0.4034. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of any YOLO detector exists on the Hub, and no +official quantization config exists for one. The recipe is the one of the +`yolov10` study: quantize the dense convolutions, keep the input convolution. +Each quantized convolution is built as +`input -> Quantize -> Dequantize(fp16) -> Convolution(+bias)`, with the kernel +`FP8 constant -> Dequantize(fp16, axis 0)`. + +| Choice | Decision | +|---|---| +| Quantized (62 of 63 convolutions) | every dense convolution whose input and output channels are multiples of 16: Conv blocks, C2f, bottlenecks, SPPF, head box and class convolutions | +| fp16 | stem `model.0` (3 input channels), DFL decode, top-1 class | +| Weight scale | per output channel, amax / 448 of the batch-norm-folded kernel, at build time | +| Activation scale | static, per tensor; max calibration (the statistic ModelOpt `FP8_DEFAULT_CFG` keeps) on the family's PyTorch reference, 256 letterboxed COCO 2017 validation images | +| Outlier check | 64 convolution inputs, median amax 11.0, largest 178 (16x): no outlier | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job; TensorRT 11.4.0.106 (target) and + 11.1.0.106. +- Quality: COCO 2017 validation images 300 to 799 of `detection-datasets/coco` + (500 held out; 0 to 255 are the calibration set), letterboxed like the + runtime. AP is `pycocotools` COCO AP (IoU 0.50 to 0.95), comparable only + inside these tables. Agreement: detections at score 0.25 matched to the + PyTorch fp32 detections (same class, IoU at least 0.5, NMS IoU 0.7 on both + sides). +- Non-finite scan: the 500 images and 3 stress inputs (noise, black, white). +- Speed: CUDA graph replay, median of 9 interleaved rounds, all variants of a + row in one job. Batch 8 is a scratch build (the family builds batch 1 only). + +### Results + +Speed and size (x is fp16 time over FP8 time): + +| GPU | TensorRT | fp32 ms | fp16 ms | FP8 ms | x batch 1 | batch 8 fp16 / FP8 ms | x batch 8 | plan MiB fp16 to FP8 | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 0.477 | 0.317 | 0.335 | 0.95 | 0.612 / 0.673 | 0.91 | 7.36 to 4.60 | +| B300 | 11.4 | 0.455 | 0.287 | 0.312 | 0.92 | 0.578 / 0.643 | 0.90 | 7.38 to 4.72 | +| B200 | 11.1 | - | 0.321 | 0.332 | 0.97 | 0.597 / 0.672 | 0.89 | 7.33 to 4.62 | + +Quality, B200, TensorRT 11.4 (B300 and TensorRT 11.1 within 0.0003 AP): + +| Engine | AP | AP50 | prec / rec vs PyTorch | box IoU | score error | +|---|---|---|---|---|---| +| PyTorch fp32 | 0.3971 | 0.5389 | - | - | - | +| fp32 engine | 0.3967 | 0.5391 | 0.999 / 1.000 | 0.9996 | 0.0005 | +| fp16 engine | 0.3969 | 0.5387 | 0.998 / 1.000 | 0.995 | 0.0008 | +| FP8 engine | 0.3922 | 0.5336 | 0.926 / 0.888 | 0.955 | 0.042 | +| FP8 without the six last head convolutions | 0.3931 | 0.5338 | 0.931 / 0.892 | 0.962 | 0.040 | + +The variant without the six last head convolutions (56 quantized) runs at 0.95x +(B200) and 0.91x (B300) of fp16 at batch 1, so keeping the head in fp16 does +not help either. + +### NVFP4 (not tried) + +TensorRT does not support NVFP4 on convolutions (4-D tensors), and the model +has no large matmul. The engine is launch bound, not weight-bandwidth bound, +so 4-bit weights would not help. + +### What was not tried + +| Item | Note | +|---|---| +| Percentile or entropy calibration, per-tensor weight scales | not measured | +| A SiLU form that TensorRT might fuse into the FP8 convolution epilogue | not measured | +| Larger widths (s, m, l, x), full 5000-image COCO validation, H100, native runtime with an FP8 bundle | not measured; the family builds only `yolov8n` | + +### When to revisit + +- A TensorRT release that fuses bias, SiLU and the next Quantize into the FP8 + convolution epilogue (as it does for most fp16 convolutions). +- The family adds a larger width: on `yolov10x` the same recipe reached fp16 + speed with a 43% smaller plan, so a large YOLOv8 may be worth a new + measurement. + +Batch 8 gave a lower ratio than batch 1, so larger batches alone do not change +the result.