diff --git a/families/wav2vec2/README.md b/families/wav2vec2/README.md index f87c9c6085..8f1e4d2c83 100644 --- a/families/wav2vec2/README.md +++ b/families/wav2vec2/README.md @@ -44,3 +44,159 @@ The new direct E2E uses the existing non-silent English waveform fixture and requires an exact transcript and CTC-token match to the official Japanese checkpoint. It fails on an empty reference transcript. This is an inference parity workload, not a Japanese WER/CER evaluation; it has not been run locally. + +## FP8 quantization study (not shipped) + +FP8 self-quantization of the encoder projections of `facebook/wav2vec2-base-960h` +was built and measured on TensorRT 11.4 (B200, B300) and 11.1 (B200). It is not +shipped. The engine of this family is fp32; an fp16 build of the same graph +(convolutions, projections and attention in fp16, every normalization in fp32) +is 4.5x to 6.9x faster than fp32, and FP8 on top of that fp16 build adds only +1.01x to 1.02x on B200 and 1.05x to 1.08x on B300. The plan is 44% smaller than +the fp16 plan, but 13% of the transcripts change against the reference model and +the exact E2E transcript check fails. This section records the recipe, the method +and the numbers, so that the study does not have to be repeated. The FP8 code was +not merged; this family has no FP8 build. + +### Summary and decision + +TensorRT 11.4, batch 1, B200 / B300. FP8 is built on the fp16 graph. + +| Item | fp32 (default) | fp16 graph | FP8 | FP8 vs fp16 | +|---|---|---|---|---| +| Latency, 5 s clip (ms) | 5.605 / 5.358 | 1.167 / 1.187 | 1.150 / 1.100 | 1.02x / 1.08x | +| Latency, 30 s clip (ms) | 20.861 / 20.418 | 3.044 / 3.058 | 2.973 / 2.873 | 1.02x / 1.06x | +| Plan size (MiB) | 362.5 | 183.6 / 185.1 | 103.4 | -44% | +| LibriSpeech test WER (Transformers fp32: 5.68%) | 5.68% | 5.70% / 5.72% | 5.58% / 5.58% | -0.1 point | +| Transcripts identical to the Transformers fp32 model | 399 of 400 | 393 / 394 of 400 | 348 of 400 | 45 / 46 more changed | +| E2E test waveform, exact transcript | yes | 1 character differs in some builds | no (4 frames change) | fails | +| Non-finite logits | 0 | 0 | 0 | none | + +Decision: do not ship FP8 for wav2vec2. + +- Speed: FP8 only touches the encoder GEMMs, which are a fifth to a quarter of the + fp16 engine time. The feature encoder's first normalization (over every time + step of the clip), the convolutions and the attention stay as they are, so the + end-to-end gain is 1% to 8%. On B200 the gain is within 2%; the remaining gain + is plan size, which alone is not a reason to ship. +- Quality: WER against the reference text does not get worse (5.58% against + 5.68%), but the FP8 engine changes 1.0% of the words of the reference model's + transcripts, and the exact transcript and CTC id check of the E2E case fails on + its test waveform. +- Against the fp32 default, FP8 in the fp32 graph is 1.3x to 1.4x faster, but the + fp16 graph is far faster with almost no transcript change, so FP8 in the fp32 + graph is not useful either. + +### Recipe that was measured + +No FP8 or NVFP4 wav2vec2 checkpoint exists on the Hub and no official +quantization config exists for it. The recipe follows the self-quantization flow +of the other encoder families: quantize the transformer projections, keep the +convolutional front end, the norms and the output head. + +| Choice | Decision | +|---|---| +| Quantized (72) | `q_proj`, `k_proj`, `v_proj`, `out_proj`, `intermediate_dense`, `output_dense` of the 12 encoder layers | +| Not quantized | the 7 feature-encoder convolutions, the feature projection, the positional convolution, every norm, the attention matmuls and softmax, the CTC head | +| Weight scale | per tensor, amax / 448 of the checkpoint weight, at build time | +| Activation scale | static, per tensor: max(abs(input)) / 448 recorded with forward pre-hooks on the Transformers fp32 model over 192 LibriSpeech validation clips (96 clean, 96 other). ModelOpt `FP8_DEFAULT_CFG` records the same statistic but cannot wrap the weight-normalized positional convolution. | +| Graph | `input -> Quantize -> Dequantize -> MatMul <- Dequantize <- FP8 weight`, both Dequantize layers in the type of the graph (fp16 in the fp16 graph, fp32 in the fp32 graph); the dual-GEMM build route is turned off when TensorRT lists it | +| Scale table | one key per checkpoint: model config (width, layers, vocabulary) and the first three CTC head biases | + +### Method + +- Checkpoint: `facebook/wav2vec2-base-960h` at the manifest revision + `22aad52d435eb6dbaf354bdad9b0da84ce7d6156`. +- Speed: the family build path (dynamic clip length, one optimization profile up + to 30 s), CUDA graph replay of one clip of 2, 5, 15 and 30 s, GPU time, median + of 7 interleaved rounds (5 on TensorRT 11.1); all variants built and timed in one + job per GPU. +- Quality: 400 LibriSpeech test clips (200 `clean`, 200 `other`, 46 min, at most + 29 s each, fixed random selection; 200 clips on TensorRT 11.1), disjoint from the + validation clips used for calibration. Input normalized like the runtime. WER + against the reference text, WER and character error against the transcript of + the Transformers fp32 model, logit cosine, per-frame argmax agreement and a scan + for non-finite logits. +- The E2E test waveform (`tests/data/speech_16k_mono.wav`) with the transcript and + CTC ids of the Transformers fp32 model, as the premerge case checks it. + +### Results + +Latency (ms) and plan size; the last columns are fp16 time over FP8 time: + +| GPU | TensorRT | Engine | 2 s | 5 s | 15 s | 30 s | plan MiB | vs fp16 (2 / 5 / 15 / 30 s) | +|---|---|---|---|---|---|---|---|---| +| B200 | 11.4 | fp32 | 5.169 | 5.605 | 10.228 | 20.861 | 362.5 | - | +| B200 | 11.4 | fp16 graph | 1.063 | 1.167 | 1.957 | 3.044 | 183.6 | - | +| B200 | 11.4 | FP8, fp16 graph | 1.051 | 1.150 | 1.923 | 2.973 | 103.4 | 1.01 / 1.02 / 1.02 / 1.02 | +| B200 | 11.4 | FP8, fp32 graph | 4.023 | 4.250 | 7.327 | 15.261 | 120.7 | 0.26 / 0.27 / 0.27 / 0.20 | +| B300 | 11.4 | fp32 | 4.970 | 5.358 | 9.913 | 20.418 | 362.6 | - | +| B300 | 11.4 | fp16 graph | 1.091 | 1.187 | 1.945 | 3.058 | 185.1 | - | +| B300 | 11.4 | FP8, fp16 graph | 1.007 | 1.100 | 1.850 | 2.873 | 103.4 | 1.08 / 1.08 / 1.05 / 1.06 | +| B300 | 11.4 | FP8, fp32 graph | 3.855 | 4.051 | 7.031 | 14.718 | 120.7 | 0.28 / 0.29 / 0.28 / 0.21 | +| B200 | 11.1 | fp32 | 5.181 | 5.595 | 10.302 | 21.075 | 362.5 | - | +| B200 | 11.1 | fp16 graph | 1.064 | 1.168 | 1.960 | 3.039 | 183.6 | - | +| B200 | 11.1 | FP8, fp16 graph | 1.031 | 1.126 | 1.892 | 2.946 | 103.8 | 1.03 / 1.04 / 1.04 / 1.03 | +| B200 | 11.1 | FP8, fp32 graph | 4.011 | 4.215 | 7.356 | 15.459 | 121.1 | 0.27 / 0.28 / 0.27 / 0.20 | + +Quality, TensorRT 11.4, 400 clips (B200 and B300 give the same FP8 numbers): + +| Engine | WER vs reference text | WER / CER vs Transformers text | identical transcripts | logit cosine mean / min | frame argmax agreement | +|---|---|---|---|---|---| +| Transformers fp32 | 5.68% | - | - | - | - | +| fp32 | 5.68% | 0.01% / 0.00% | 399 | 1.00000 / 0.99996 | 0.9997 | +| fp16 graph (B200 / B300) | 5.70% / 5.72% | 0.09% / 0.02% | 393 / 394 | 0.99995 / 0.99969 | 0.9982 | +| FP8, fp16 graph | 5.58% | 0.99% / 0.19% | 348 | 0.99731 / 0.99311 | 0.9778 | +| FP8, fp32 graph | 5.54% | 1.11% / 0.22% | 336 | 0.99728 / 0.99271 | 0.9774 | + +TensorRT 11.1 (B200, 200 clips): WER 5.70% (fp32), 5.70% (fp16 graph), 5.65% (FP8, +fp16 graph), 5.76% (FP8, fp32 graph); 199, 197, 169 and 169 of 200 transcripts +identical to the Transformers transcript. No non-finite logit in any engine. + +E2E test waveform: both FP8 engines change the argmax of 4 of 207 frames (3 on +TensorRT 11.1), logit cosine 0.9976, and the transcript and CTC ids change, so the +exact premerge check of this family fails for FP8. + +Kernel time per group, 30 s clip, TensorRT 11.4, B200 / B300 (CUDA kernel time in +ms, profiler): + +| Kernel group | fp16 graph | FP8, fp16 graph | +|---|---|---| +| norms with their fused pointwise work | 1.256 / 1.215 | 1.260 / 1.218 | +| convolutions | 0.705 / 0.678 | 0.713 / 0.676 | +| encoder and head GEMMs | 0.620 / 0.667 | 0.511 / 0.562 | +| fused attention | 0.479 / 0.454 | 0.455 / 0.429 | +| other | 0.109 / 0.033 | 0.108 / 0.038 | +| sum | 3.168 / 3.122 | 3.047 / 2.923 | + +The GEMMs are 20% of the fp16 time at 30 s (25% at 5 s) and FP8 removes 18% of +their time. The largest single kernel is the feature encoder's first +normalization (a group norm over every time step of the clip), 1.2 ms at 30 s. + +### Variants + +| Variant | Result | +|---|---| +| fp32 Dequantize in the fp16 graph, TensorRT 11.4 | wrong output on B200 and B300: WER 99.96% / 100%, logit cosine 0.91; no FP8 GEMM kernel in the plan. TensorRT 11.1 builds it correctly (1.6% WER against the Transformers transcript on 50 clips). A single FP8 GEMM with the same Dequantize types is correct on both versions, so the fault needs the full graph. fp16 Dequantize (the recipe) is correct on both. | +| FP8 also on the feature-encoder convolutions and the feature projection (TensorRT 11.1, B300, scratch graph with a normalization-layer form of the group norm) | 1.08x (30 s) to 1.10x (5 s) at batch 1 against the fp16 scratch graph, against 1.05x to 1.09x for the encoder projections alone; 95 instead of 102 of 120 cropped clips identical to the Transformers transcript | +| One projection kind at a time (TensorRT 11.1, B300, scratch graph, 120 clips cropped to 5 s) | `out_proj` 115, `q/k/v` 113, `output_dense` 111, `intermediate_dense` 109 of 120 identical; no subset keeps the transcripts and still gives a measurable gain | + +### NVFP4 + +NVFP4 of the same 72 projections was tried in a scratch graph (TensorRT 11.1, +B300, 120 clips cropped to 5 s): logit cosine 0.980, 5.1% WER against the +Transformers transcript, 71 of 120 transcripts identical. It is too lossy, and the +only measured speed gain (1.07x against the fp16 graph) needs batch 8, which this +family does not build. + +### When to revisit + +- When the family builds batches of clips: the GEMMs then take a larger share of + the time. +- When the feature encoder's normalization and the convolutions get faster, so + that the GEMMs are a larger share of the fp16 time (in the scratch graph, the + group norm written as a normalization layer was 11% faster at 30 s but 7% slower + at 5 s on 11.1). +- When a calibration keeps the transcripts within the fp16 drift (for example per + channel weight scales or skipping the most sensitive projections); none of the + measured subsets did.