You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs(timm_senet): record why FP8 is not shipped - #2148
families/timm_senet builds timm/senet154.gluon_in1k as one image classification engine (batch 1, 224 x 224, fp16 in the manifest, builder optimization level 3 since #2123) and has no quantized build. No FP8 or NVFP4 checkpoint of this model exists on the Hub, so FP8 self-quantization of the convolutions (build-time per-channel FP8 weights, static per-tensor activation scales from max calibration, the recipe of the shipped timm_resnet FP8 build) was built and measured on TensorRT 11.4 (B200, B300) and 11.1 (B200). FP8 does not pay for this family: it is slower than fp16 in every layout, 0.73x to 0.79x at batch 1 with the family's level 3, and 0.53x to 0.79x at level 1 (batch 1 and 32). Quality is fine (top-1 within 0.2 points of PyTorch fp32). This change adds a README to the family with the study, like the yolov10 (#2093) and timm_efficientnet (#2102) studies, so that it does not have to be repeated. The FP8 code is not part of this change. NVFP4 was not tried: TensorRT does not support NVFP4 on convolutions.
The study also records a TensorRT 11.4 behavior that is independent of FP8: each grouped 3x3 convolution (64 groups) runs as one kernel per group (3343 to 3859 kernels per image in fp16), so fp16 batch 1 takes 10.3 to 12.6 ms, while PyTorch fp16 takes 4.36 ms on B300. Builder optimization level 3 does not change the kernel count.
Exit Criteria
The README records the recipe, the method, speed, plan size and build time on B200 and B300 (TensorRT 11.4) and B200 (11.1) at builder optimization level 3 (family default) and level 1, top-1, agreement and KL divergence against PyTorch fp32, the kernel counts that explain the missing speedup, the fp16 grouped-convolution behavior, NVFP4, what was not tried and when to revisit.
No code, test, manifest or bundle change; builds are unchanged.
Non-goals: shipping FP8 or NVFP4 builds, an FP8 manifest or CI case, a fix for the per-group lowering.
Implementation
families/timm_senet/README.md (new): a short family description and the section "FP8 quantization study (not shipped)": summary and decision, recipe, method, results (level 3 and level 1 speed and size, quality, kernels per call), fp16 grouped convolutions, NVFP4, what was not tried, when to revisit.
Recipe that was measured (code kept out of the repository):
FP8 for every convolution after the 3x3 stem ("all", 254 of 257), or without the 100 squeeze-excitation convolutions ("no SE", 154), or also without stage layer1 ("no SE, no layer1", 144, the shipped timm_resnet layout). Stem and classifier stay fp16.
Weights: per-output-channel scale amax / 448 of the batch-norm-folded kernel at build time. Activations: static per-tensor scale max |input| / 448 over 500 ImageNet validation images. fp16 Dequantize.
Why no gain: the fp16 engine is already limited by the per-group kernels of the grouped 3x3 convolutions. In FP8, TensorRT keeps the per-group split and adds a concatenation kernel per group (about 6200 to 6630 kernels per image). At level 3, TensorRT runs most FP8 grouped convolutions with fp16 kernels, so the FP8 plan is not smaller (237 vs 230 MiB on B300) and the build takes 3.3x to 4.4x longer.
pytest families/timm_senet/tests (TensorRT 11.4 B200 and B300, 11.1 B200)
8 passed, 1 skipped
Premerge E2E senet154-gluon-in1k (TensorRT 11.4)
B200 pass, B300 pass
Markdown check of the README (table column counts)
pass
git diff --check
pass
python3 tools/pr_metadata.py validate on this description
pass
The family tests ran on the base commit; this change edits only the README.
The numbers in the README come from the local study (batch 1, best FP8 layout "no SE, no layer1"; x is fp16 time over FP8 time):
GPU
TensorRT
Level
fp16 ms
FP8 ms
x
plan MiB fp16 / FP8
B200
11.4
3
11.40
14.75
0.77
232.7 / 234.0
B300
11.4
3
10.84
13.66
0.79
229.7 / 232.2
B200
11.1
3
10.86
14.18
0.77
232.3 / 231.5
B200
11.4
1
12.53
15.84
0.79
227.8 / 134.8
B300
11.4
1
10.81
15.13
0.71
225.6 / 134.5
Top-1 on 1500 held-out images: PyTorch fp32 82.67%, fp16 engine 82.67%, FP8 engine 82.60% to 82.80% (98.67% to 99.00% agreement with PyTorch top-1, KL 0.0027 to 0.0028). No NaN or Inf on 2000 images and 29 stress inputs in any engine.
Hardware, Environment, and Revisions
Repository: this commit on top of upstream/main at f8800ce3.
Checkpoint: timm/senet154.gluon_in1k at 86f6d239ed30acbe3aca0cbb5769da2621a4e02c (manifest revision).
Data: 2000 images of clip-benchmark/wds_imagenet1k validation (fixed stride of 25): 500 for calibration, 1500 held out.
Study: B200 and B300, one GPU per job, all variants of a job built and timed together; TensorRT 11.4.0.106 (target) and 11.1.0.106, torch 2.14.0 (CUDA 13). Level-3 rows on upstream/main at f8800ce3; level-1 rows (batch 1 and 32) from an earlier run on upstream/main at f8800ce3, when the family built at level 1. Reference: the timm PyTorch model in fp32 with the family preprocessing.
Not Run / Remaining Gaps
The study cannot be rerun from this repository because the FP8 code was not merged.
Not measured: FP8 for the 1x1 convolutions only, batch 32 at level 3, entropy or percentile calibration, quantization-aware training, H100, the native runtime with an FP8 bundle.
Main-branch behavior found (recorded, not fixed): TensorRT 11.4 runs each fp16 grouped convolution of SENet-154 as one kernel per group, so batch 1 is 2.5x slower than PyTorch fp16 on B300 (10.8 ms vs 4.36 ms); at batch 32 both are about equal.
Contributor Self-Review
I have completed a self-review of this change.
Notes For Future Readers
Suggested review order: "Summary and decision", the kernels-per-call table, "fp16 grouped convolutions", "When to revisit".
Revisit FP8 when a TensorRT release runs a grouped convolution as one kernel instead of one per group; the fp16 baseline gets faster first, so FP8 has to be measured against it again.
Risk level
Low
Medium
High
Risk rationale: documentation only; one README added, no code, build, bundle or test change.
Add a README to the timm_senet family with the FP8 self-quantization
study of senet154 on TensorRT 11.4 (B200, B300) and 11.1: recipe
(per-channel FP8 weights, static per-tensor activation scales, the
timm_resnet FP8 recipe), measurement method, speed, plan size, build
time, ImageNet top-1 against PyTorch fp32 and the kernel counts that
explain the result. FP8 is slower than fp16 in every layout: 0.73x to
0.79x at batch 1 with the family's builder optimization level 3, and
0.53x to 0.79x at level 1 (batch 1 and 32). TensorRT 11.4 runs each
grouped 3x3 convolution (64 groups) as one kernel per group (3343 to
3859 kernels per image in fp16); FP8 keeps that split and adds a
concatenation kernel per group (about 6600 kernels). At level 3 most
FP8 grouped convolutions run with fp16 kernels and the FP8 plan is not
smaller. Top-1 stays within 0.2 points. The README also records that
fp16 batch 1 is 2.5x slower than PyTorch fp16 on B300 for the same
reason. FP8 is not shipped. NVFP4 was not tried (no NVFP4 convolution
in TensorRT).
Documentation only; no code change.
Signed-off-by: Zhenshan Xie <zhenshanx@nvidia.com>
No actionable comments were generated in the recent review.
Summary
Summary
Adds families/timm_senet/README.md with the SENet-154 TensorRT configuration and an FP8 self-quantization study. The README records recipes, measurement methods, results, and untested cases.
The documented results show FP8 was slower than FP16 in every measured layout, despite top-1 accuracy staying within 0.2 percentage points of PyTorch FP32. The README attributes the slowdown to grouped convolutions running as one kernel per group, with FP8 adding concatenation kernels. It also records the observed FP16 grouped-convolution behavior and states that NVFP4 was not tested because TensorRT does not support it on convolutions.
The README concludes that FP8 is not shipped. It identifies TensorRT grouped-convolution improvements as a condition for revisiting FP8.
Architecture impact
Family-owned files: Adds one file, families/timm_senet/README.md.
Changed shared surfaces: None shown in the diff.
New dependency directions: None. The change is documentation-only.
Affected consumers: No runtime or build consumers are changed. The README may inform future SENet model or TensorRT work.
Unresolved blast-radius questions: None indicated for implementation, since no code or shared surfaces changed. The reported measurements and their reproducibility were not independently verified here.
Outcome: HUMAN REVIEW REQUIRED. The available evidence does not resolve the measurement provenance question. This outcome is not a finding of a violation.
Review finding counts: not available. CI, build, and test results on the current head: not available.
The SENet-154 README documents the FP8 quantization study, its evaluation methods and measured results, and the configurations not tested. It also records limitations and conditions for revisiting FP8.
Changes: SENet-154 FP8 Study
Layer / File(s)
Summary
Study scope and method families/timm_senet/README.md
The README describes the engine configuration, FP8 recipe, tested layouts, hardware, software, datasets, and evaluation procedure.
Performance and quality measurements families/timm_senet/README.md
The README reports latency, plan size, build time, quality, kernel counts, and grouped-convolution timing for measured configurations.
Study limits and revisit conditions families/timm_senet/README.md
The README records untested configurations, TensorRT convolution-support limitations, and conditions for revisiting FP8.
The change documents the FP8 study without changing the build behavior. No actionable merge-blocking risk remains.
Architecture risk: Low · up to 00dbf
The change affects 1 system.
Changed systems:families
Architecture concerns
No architecture-level concerns identified.
Review details
Systems and components
observed — families (service) was modified; 1 changed file maps to changed impact.
Before / after behavior
observed — Modified behavior in families/timm_senet/README.md: The README describes the timm SENet-154 engine configuration and states that the family has no FP8 build; it records that measured FP8 layouts were slower than fp16 across the tested GPUs and batch sizes.
observed — Modified behavior in families/timm_senet/README.md: The README records the level-3 batch-1 comparison, the decision not to ship FP8, and the reported kernel counts, plan sizes, build times, and quality results.
observed — Modified behavior in families/timm_senet/README.md: The README documents the measured FP8 convolution quantization recipe, the three tested convolution layouts, which layers remain fp16, and the weight and activation scaling choices.
observed — Modified behavior in families/timm_senet/README.md: The README describes the GPUs, TensorRT versions, builds, datasets, quality and non-finite checks, and timing procedure used in the study.
Pre-merge checks: 8 · 1
Failed checks: 1 inconclusive
Check name
Status
Explanation
Resolution
Benchmark Validation Integrity
The new report compares TensorRT FP16, TensorRT FP8, and PyTorch FP16 latency. The README records CUDA graph replay but does not define the timed region for each path. The FP8 study code was not merge…
Provide the study timing code or equivalent evidence showing the timed operations, synchronization, transfers, and output handling for each compared path.
Passed checks: 8 passed
Check name
Status
Explanation
Title check
The title clearly summarizes the main change: documenting why FP8 is not shipped for timm SENet-154.
Description check
The description is mostly complete. It covers the background, exit criteria, implementation, change category, validation results, hardware and revisions, remaining gaps, self-review, and risk rational…
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary
The PR adds only families/timm_senet/README.md. It describes a recipe similar to timm_resnet, but adds no dependency on that family's implementation or artifacts. It makes no registry or other fam…
Shared Semantic Neutrality
The PR adds only families/timm_senet/README.md. It does not change shared code.
Shared Change Blast Radius
The pull request adds only families/timm_senet/README.md. The diff does not change a shared surface.
Full details: Description check
Explanation
The description is mostly complete. It covers the background, exit criteria, implementation, change category, validation results, hardware and revisions, remaining gaps, self-review, and risk rationale.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Full details: Family Ownership Boundary
Explanation
The PR adds only families/timm_senet/README.md. It describes a recipe similar to timm_resnet, but adds no dependency on that family's implementation or artifacts. It makes no registry or other family changes.
Full details: Benchmark Validation Integrity
Explanation
The new report compares TensorRT FP16, TensorRT FP8, and PyTorch FP16 latency. The README records CUDA graph replay but does not define the timed region for each path. The FP8 study code was not merged, so the comparison boundaries cannot be verified.
Resolution
Provide the study timing code or equivalent evidence showing the timed operations, synchronization, transfers, and output handling for each compared path.
Reviewing files that changed from the base of the PR and between f8800ce and 00dbf10.
Files selected for processing: 1
families/timm_senet/README.md
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
removed
the
run-internal-ci
Background
families/timm_senetbuildstimm/senet154.gluon_in1kas one image classification engine (batch 1, 224 x 224, fp16 in the manifest, builder optimization level 3 since #2123) and has no quantized build. No FP8 or NVFP4 checkpoint of this model exists on the Hub, so FP8 self-quantization of the convolutions (build-time per-channel FP8 weights, static per-tensor activation scales from max calibration, the recipe of the shippedtimm_resnetFP8 build) was built and measured on TensorRT 11.4 (B200, B300) and 11.1 (B200). FP8 does not pay for this family: it is slower than fp16 in every layout, 0.73x to 0.79x at batch 1 with the family's level 3, and 0.53x to 0.79x at level 1 (batch 1 and 32). Quality is fine (top-1 within 0.2 points of PyTorch fp32). This change adds a README to the family with the study, like the yolov10 (#2093) and timm_efficientnet (#2102) studies, so that it does not have to be repeated. The FP8 code is not part of this change. NVFP4 was not tried: TensorRT does not support NVFP4 on convolutions.The study also records a TensorRT 11.4 behavior that is independent of FP8: each grouped 3x3 convolution (64 groups) runs as one kernel per group (3343 to 3859 kernels per image in fp16), so fp16 batch 1 takes 10.3 to 12.6 ms, while PyTorch fp16 takes 4.36 ms on B300. Builder optimization level 3 does not change the kernel count.
Exit Criteria
Implementation
families/timm_senet/README.md(new): a short family description and the section "FP8 quantization study (not shipped)": summary and decision, recipe, method, results (level 3 and level 1 speed and size, quality, kernels per call), fp16 grouped convolutions, NVFP4, what was not tried, when to revisit.Recipe that was measured (code kept out of the repository):
layer1("no SE, no layer1", 144, the shippedtimm_resnetlayout). Stem and classifier stay fp16.Why no gain: the fp16 engine is already limited by the per-group kernels of the grouped 3x3 convolutions. In FP8, TensorRT keeps the per-group split and adds a concatenation kernel per group (about 6200 to 6630 kernels per image). At level 3, TensorRT runs most FP8 grouped convolutions with fp16 kernels, so the FP8 plan is not smaller (237 vs 230 MiB on B300) and the build takes 3.3x to 4.4x longer.
Change categories
Validation
Commands and Results
python3 -m tools.model_ci validatepytest tools/tests/test_architecture.py tools/tests/test_public_source_hygiene.py(CPU)pytest families/timm_senet/tests(TensorRT 11.4 B200 and B300, 11.1 B200)senet154-gluon-in1k(TensorRT 11.4)git diff --checkpython3 tools/pr_metadata.py validateon this descriptionThe family tests ran on the base commit; this change edits only the README.
The numbers in the README come from the local study (batch 1, best FP8 layout "no SE, no layer1"; x is fp16 time over FP8 time):
Top-1 on 1500 held-out images: PyTorch fp32 82.67%, fp16 engine 82.67%, FP8 engine 82.60% to 82.80% (98.67% to 99.00% agreement with PyTorch top-1, KL 0.0027 to 0.0028). No NaN or Inf on 2000 images and 29 stress inputs in any engine.
Hardware, Environment, and Revisions
upstream/mainatf8800ce3.timm/senet154.gluon_in1kat86f6d239ed30acbe3aca0cbb5769da2621a4e02c(manifest revision).clip-benchmark/wds_imagenet1kvalidation (fixed stride of 25): 500 for calibration, 1500 held out.upstream/mainatf8800ce3; level-1 rows (batch 1 and 32) from an earlier run onupstream/mainatf8800ce3, when the family built at level 1. Reference: the timm PyTorch model in fp32 with the family preprocessing.Not Run / Remaining Gaps
Contributor Self-Review
Notes For Future Readers
Risk level
Risk rationale: documentation only; one README added, no code, build, bundle or test change.
🤖 Generated with Claude Code