From 517c9131dc10ecfea5caa056bd83f2499d3647ac Mon Sep 17 00:00:00 2001 From: Zhenshan Xie Date: Mon, 9 Nov 2026 14:36:29 -0800 Subject: [PATCH] docs(sam3): record why FP8 is not shipped Add a README for the sam3 family with the FP8 self-quantization study on TensorRT 11.4 (B200, B300) and 11.1: recipe (q/k/v, out_proj, fc1 and fc2 of the 24 text encoder blocks in FP8, static per-tensor scales from ModelOpt), measurement method, speed, size and mask quality against PyTorch fp32. The text encoder is 1.4x faster and the bundle 31% smaller at the default mask quality, but the text encoder is about 8% of a request, so a request is only 1.02x faster; a plain fp16 text plan is faster than the FP8 one. FP8 of the vision plan lowers mask IoU from 0.9975 to 0.96 for at most 1.11x. FP8 is not shipped. NVFP4 was measured and is slower than FP8 with lower quality. Documentation only; no code change. Signed-off-by: Zhenshan Xie --- families/sam3/README.md | 197 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 families/sam3/README.md diff --git a/families/sam3/README.md b/families/sam3/README.md new file mode 100644 index 0000000000..3d60c5c5d6 --- /dev/null +++ b/families/sam3/README.md @@ -0,0 +1,197 @@ +# SAM3 + +This family builds `facebook/sam3` for `text_prompted_segmentation` as a bundle +of TensorRT plans: the CLIP text encoder (24 layers, hidden 1024, 32 tokens), +the vision plan (ViT-L, 32 blocks, 1008 x 1008 image, plus the FPN necks), the +detector core (DETR encoder and decoder, mask head) and the video tracker plans. +Batch size is 1. The family builds `fp32` only: the text plan is fp32, and the +vision and core plans keep the residual stream, norms and activations in fp32 +and run most GEMMs and attention in fp16. The family has no quantized build. + +## FP8 quantization study (not shipped) + +FP8 self-quantization was built and measured on TensorRT 11.4 (B200, B300) and +11.1 (B300). It is not shipped: only the text encoder keeps the mask quality, +and the text encoder is about 8% of the request time, so a request is only +1.02x faster. The vision plan, where the time is, loses mask quality with FP8. +This section records the recipe, the method and the numbers, so that the study +does not have to be repeated. The FP8 code was not merged; this family has no +FP8 build. + +### Summary and decision + +Measured FP8 build: q/k/v, `out_proj`, `fc1` and `fc2` of the 24 text encoder +blocks in FP8; vision plan, core and tracker plans unchanged. + +| Item | default bundle | FP8 text encoder | FP8 vs default | +|---|---|---|---| +| Request time (text + vision + core), B200 / B300, TensorRT 11.4 | 18.601 / 17.118 ms | 18.215 / 16.759 ms | 1.02x / 1.02x | +| Text encoder time, B200 / B300, TensorRT 11.4 | 1.385 / 1.314 ms | 1.005 / 0.932 ms | 1.38x / 1.41x | +| Text plan size | 1348.7 MiB | 485.5 MiB | -64% | +| Bundle size (all plans) | 2769.3 MiB | 1905.9 MiB | -31% | +| Mask IoU vs PyTorch fp32, B200 / B300 (mean) | 0.9992 / 0.9975 | 0.9981 / 0.9973 | -0.001 / -0.0002 | +| Non-finite outputs | 0 | 0 | none | + +Decision: do not ship FP8 for SAM3. + +- Speed: the text encoder is 1.4x faster, but a request is 1.02x faster on + B200 and B300 (11.1: 1.03x). This is the same level as the yolov10 study, + which was not shipped. +- The text encoder runs 32 tokens per GEMM and is launch bound. A plain fp16 + text plan (no FP8) is faster than the FP8 one (0.806 ms against 1.005 ms on + B200) and keeps the default quality. The family cannot build it today: + `precision` is one value for the bundle and the vision plan refuses fp16. +- Vision plan: FP8 of its projections gives at most 1.11x for that plan and + lowers the mask IoU from 0.9975 to 0.96. No subset of the projections keeps + the quality. +- Detector core: about 470 small kernels, no kernel above 0.14 ms; not worth + quantizing. +- The only clear gain is size: the bundle is 31% smaller. + +### Recipe that was measured + +No FP8 or NVFP4 checkpoint of SAM3 exists, and no official quantization config +exists for it. The recipe follows the public ModelOpt and Edge-LLM guidelines +for text encoders (attention and MLP linears quantized; embeddings, norms and +the output projection kept) and for multi-tower models (the image tower kept in +high precision unless it holds quality). + +| Choice | Decision | +|---|---| +| Quantized (text encoder) | q, k, v, `out_proj`, `fc1`, `fc2` of the 24 blocks: 96 modules, 144 weight tensors; q, k and v share one Quantize on the layer-norm output | +| Not quantized | token and position embeddings, layer norms, attention matmuls, `text_projection`; the vision plan, the core and the tracker plans | +| Graph | `fp32 activation -> Cast fp16 -> Quantize -> Dequantize(fp16) -> MatMul -> Cast fp32`; residual stream, norms, GELU and bias adds stay fp32 | +| Weight scale | per tensor, amax / 448, at build time (nothing FP8 stored) | +| Activation scale | static, per tensor; ModelOpt 0.46.1 `FP8_DEFAULT_CFG` (max calibration) on the PyTorch `Sam3Model` text encoder with 304 prompts (80 COCO object names, each also with an adjective and with a context phrase, and 40 longer noun phrases), padded to 32 tokens | +| Outlier rule | a scale above 20x the median of its kind stays fp32; measured: no outlier (largest ratio 12.4x, `layers.0.mlp.fc2`) | +| Scale key | model config plus two weight sums, so another checkpoint does not reuse the table | +| Dequantize type | fp16; no non-finite value, fp32 Dequantize was not needed | + +### Method + +- GPUs: B200 and B300, one GPU per job, all variants built and timed in that + job through the family's `build()`; TensorRT 11.4.0.106 (target) and + 11.1.0.106 (B300 only). Builder optimization level 1, precision fp32 (the + family default). +- Reference: PyTorch fp32 `Sam3Model` (Transformers 5.17.0) on the same + preprocessed pixels and tokens. The default bundle is the quality floor. +- Data: `detection-datasets/coco` validation, 400 images at a fixed stride; + 100 held-out images give 361 prompt and image pairs: 181 present-category + prompts, 100 absent-category prompts and 80 pairs with noun phrases that are + not in the calibration set. The calibration used no image. +- Quality: 288 x 288 mask logits, sigmoid, bilinear to the image size, + threshold 0.5; score = sigmoid(logit) x sigmoid(presence), kept above 0.5; + instances compared by query slot. +- Non-finite scan: the 361 pairs, the 304 calibration prompts plus the held-out + phrases through the text plan, and five stress images (noise with std 1, 3 and + 10, all black, all white). +- Speed: the three plans of one request captured in one CUDA graph, median of + 9 interleaved rounds (default, FP8, fp16 text control). The fp16 text control + is a scratch build of the text plan with `precision="fp16"`. + +### Results + +Speed, batch 1, ms (x is default time over FP8 time): + +| GPU | TensorRT | request default | request FP8 | x | text default | text FP8 | text fp16 control | request with fp16 text control | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | 18.601 | 18.215 | 1.02 | 1.385 | 1.005 | 0.806 | 18.029 | +| B300 | 11.4 | 17.118 | 16.759 | 1.02 | 1.314 | 0.932 | 0.767 | 16.594 | +| B300 | 11.1 | 17.181 | 16.735 | 1.03 | 1.352 | 0.913 | 0.786 | 16.604 | + +Parts on their own (CUDA graph, median of 5 rounds): vision plan 13.718 ms +(B200) and 12.585 ms (B300), core 3.732 ms and 3.347 ms, TensorRT 11.4. + +Size, MiB (same on all three runs within 1 MiB): text plan 1348.7 to 485.5 +(the 193 MiB token table stays fp32), bundle 2769.3 to 1905.9. + +Quality against PyTorch fp32, 361 pairs: + +| GPU, TensorRT | Build | mask IoU mean (p5) | semantic IoU | box IoU | score abs err | same instance count | top-10 logit abs err | text feature cos (min) | +|---|---|---|---|---|---|---|---|---| +| B200, 11.4 | default | 0.9992 (0.9964) | 0.9994 | 0.9985 | 0.0011 | 0.992 | 0.011 | 1.0000 (1.0000) | +| B200, 11.4 | FP8 text | 0.9981 (0.9959) | 0.9990 | 0.9973 | 0.0027 | 0.986 | 0.032 | 0.9988 (0.9968) | +| B200, 11.4 | fp16 text control | 0.9993 (0.9964) | 0.9995 | 0.9985 | 0.0011 | 0.992 | 0.010 | 1.0000 (1.0000) | +| B300, 11.4 | default | 0.9975 (0.9955) | 0.9994 | 0.9970 | 0.0020 | 0.995 | 0.014 | 1.0000 (1.0000) | +| B300, 11.4 | FP8 text | 0.9973 (0.9951) | 0.9992 | 0.9964 | 0.0034 | 0.997 | 0.036 | 0.9988 (0.9968) | +| B300, 11.4 | fp16 text control | 0.9975 (0.9954) | 0.9993 | 0.9971 | 0.0020 | 0.992 | 0.015 | 1.0000 (1.0000) | +| B300, 11.1 | default | 0.9988 (0.9952) | 0.9993 | 0.9983 | 0.0027 | 0.989 | 0.017 | 1.0000 (1.0000) | +| B300, 11.1 | FP8 text | 0.9987 (0.9946) | 0.9990 | 0.9983 | 0.0033 | 0.986 | 0.037 | 0.9989 (0.9971) | + +FP8 text against the default engine: mask IoU 0.9986 (B200), 0.9979 (B300), +0.9980 (B300, 11.1); the fp16 text control: 0.9998 to 0.9999. The text feature +cosine on the held-out phrases is 0.9988, as on the calibration prompts. No +non-finite output in any build. + +Kernels (plan inspector, text plan, B200): TensorRT 11.4 runs 96 e4m3 x e4m3 +CUTLASS GEMMs (q, k and v fused into one GEMM, 4 per block); TensorRT 11.1 runs +144 (q, k and v separate). The 48 attention matmuls stay fp16. + +Vision plan, FP8 of q/k/v, `fc1` and `fc2` of all 32 blocks (scratch, B300, +TensorRT 11.4, 281 pairs, text plan default): 11.24 ms against 12.48 ms (1.11x), +653 MiB against 1044 MiB, mask IoU 0.9606 against 0.9975. TensorRT 11.1: 11.55 +ms against 12.53 ms (1.08x), mask IoU 0.9634 against 0.9990. Subsets (same job, +TensorRT 11.4): + +| FP8 projections of the vision plan | vision ms | mask IoU | IoU >= 0.9 | score abs err | +|---|---|---|---|---| +| none (default) | 12.48 | 0.9975 | 0.9986 | 0.0020 | +| q/k/v | 12.19 | 0.9797 | 0.9788 | 0.0243 | +| `fc1` | 12.04 | 0.9745 | 0.9774 | 0.0249 | +| `fc2` | 11.86 | 0.9891 | 0.9929 | 0.0161 | +| `fc1` + `fc2` | 11.45 | 0.9709 | 0.9774 | 0.0237 | +| q/k/v + `fc1` + `fc2` | 11.21 | 0.9607 | 0.9605 | 0.0346 | +| all, first 4 blocks kept | 11.35 | 0.9729 | 0.9746 | 0.0285 | +| all, the 4 global-attention blocks kept | 11.36 | 0.9638 | 0.9676 | 0.0293 | + +A fake-quant run of the same recipe on the PyTorch model (16 images) gives a +relative error of the deepest FPN map of 0.111 (the engine: 0.111; weights only +0.060, activations only 0.101; half the activation scale 0.158), so TensorRT +runs the recipe as written and the loss comes from static per-tensor FP8 on +this ViT-L. + +### Why there is no gain + +- The text encoder is about 8% of a request (1.3 ms of 17 to 18.6 ms). Even a + free text encoder would give at most 1.08x. +- At 32 tokens the text GEMMs are small and the text plan is launch bound. The + fp32 to fp16 change alone gives more (0.77 to 0.81 ms) than FP8 (0.91 to + 1.01 ms). +- The vision plan is about 73% of a request but does not hold mask quality in + FP8, and a plain fp16 vision graph (scratch, no fp32 residual, norms or GELU) + is 9.40 ms against 11.11 ms (1.18x) at mask IoU 0.9974, more than FP8 gives. + +### NVFP4 + +Measured (scratch, B300, TensorRT 11.4, 281 pairs): NVFP4 MLP projections +(block-16 scales) with FP8 attention projections. Text encoder 1.037 ms against +0.927 ms for FP8 (slower), mask IoU 0.9954 against 0.9975; vision plan 11.39 +ms against 11.24 ms for FP8, mask IoU 0.9337 (5th percentile 0.63). All outputs +finite. Not pursued: the text encoder is not weight-bandwidth bound at 32 rows, +and the vision plan is compute and elementwise bound at 5184 tokens. + +### TensorRT note + +On TensorRT 11.4 an FP8 text graph built in fp16 (scratch) fails to build when +`out_proj` is quantized (error `Input 0's element type (fp8_e4m3) differs from +input 1's element type (half)`); it builds on 11.1 (0.676 ms on B300) and on +11.4 without `out_proj` (0.745 ms on B200, 0.694 ms on B300). The fp32 graph of +the measured recipe builds on both versions. + +### What was not tried + +| Item | Note | +|---|---| +| Percentile or entropy calibration, per-channel weights for the vision plan | not measured | +| Batch greater than 1 (the family builds batch 1 only), the video tracker plans | not measured | +| H100, the native runtime with an FP8 bundle, other SAM3 checkpoints | not measured | +| Segmentation benchmark against ground truth | not measured; quality is against PyTorch fp32 | + +### When to revisit + +- The family gets an fp16 text plan or fp16 vision plan: FP8 would then have + to beat those, not the fp32 default. +- A recipe that keeps the vision plan at the floor (mask IoU about 0.997) with + a clear speedup; that plan is where the request time is. +- A caller that encodes many prompts per image: the text encoder share grows, + and so does the FP8 gain.