[zhenshanx-nv]

Background

families/bart builds BART encoder-decoder checkpoints as an encoder engine and a one-token decoder step engine, and rejects --quantization. No official FP8 checkpoint exists for BART, so FP8 self-quantization (build-time FP8 weights, ModelOpt-calibrated static activation scales) was built and measured for facebook/bart-base and facebook/bart-large-cnn on TensorRT 11.4. FP8 does not pay for this family: end-to-end generation is 0.93x to 1.04x of fp16 at batch 1, and bart-large-cnn drifts far from the reference (KL 0.20, top-1 77.6%, vs fp16 0.0005 and 98.2%). This change adds a family README that records the study, like the SAM (#1986) and Cosmos3-Edge (#2003) studies, so that it does not have to be repeated. The FP8 code is not part of this change. NVFP4 was not built: FP8 already gives no decode-step gain at these model configs (0.98x to 1.03x), and NVFP4 is lossier than FP8.

Exit Criteria

• The README records the recipe, the method, speed on B200 and B300 (TensorRT 11.4), quality against Transformers fp32, the projection sensitivity, NVFP4 and the reasons for not shipping.
• The README records three problems on main found during the study (not fixed here).
• No code, test, manifest or bundle change; builds are unchanged.
• Non-goals: shipping FP8 or NVFP4 builds, an FP8 manifest or CI case, fixing the problems found on main.

Implementation

• families/bart/README.md (new): short family overview and the section "FP8 quantization study (not shipped)": summary and decision, models, recipe, method, speed, quality, sensitivity, size and kernel evidence, NVFP4, what was not tried, found on the way, when to revisit.

Recipe that was measured (code kept out of the repository):

• FP8 GEMMs for encoder q, k, v, fc1, fc2 and decoder self-attention q, k, v, cross-attention q, k, v, fc1, fc2 of every layer (78 projections in bart-base, 156 in bart-large-cnn). Attention output projections in fp16: on TensorRT 11.4 an FP8 Quantize right after the fused attention does not build (it builds on 11.1).
• Weights: per-tensor scale amax / 448, quantized from the original checkpoint at build time. Activations: static per-tensor scales from ModelOpt FP8_DEFAULT_CFG (max) on PyTorch fp32 during greedy generation of 256 CNN/DailyMail training articles. fp16 Dequantize. Outlier rule (20x median) excluded nothing.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

Documentation only.

Check	Result
python -m tools.model_ci validate	pass
Markdown check of the new README (table column counts, code fences)	pass
python3 tools/pr_metadata.py validate on this description	pass
Measured FP8 code (not in this change): pytest families/bart/tests with its test_quantization.py, TensorRT 11.4 and 11.1	29 passed, 2 skipped on each

Speed (TensorRT 11.4, batch 1, median of 7 interleaved rounds; end to end = encoder plus 128 decode steps with a host argmax per step):

Model, max_sequence_length	GPU	Encoder	Decode step	End to end
bart-large-cnn, 1024	B200 / B300	1.10x / 1.06x	1.03x / 1.03x	1.04x / 0.99x
bart-large-cnn, 256	B200 / B300	1.03x / 1.04x	1.01x / 1.00x	0.93x / 0.96x
bart-base, 256	B200 / B300	1.04x / 1.08x	0.98x / 1.00x	0.93x / 0.94x

Size: bundle -34% (bart-large-cnn, B200; -31% on B300), -20% (bart-base).

Quality (TensorRT 11.4, max_sequence_length 256, against Transformers fp32 greedy output on CNN/DailyMail test articles; no NaN or Inf):

Model	Build	KL	Top-1	Greedy equal
bart-base (200 articles, B200)	fp16 / FP8	6e-8 / 0.0002	100% / 99.98%	100% / 98.5%
bart-large-cnn (300 articles, B200)	fp16 / FP8	0.0005 / 0.20	98.2% / 77.6%	95.7% / 32.7%

Sensitivity (bart-large-cnn, 60 articles, KL): attention only 0.016, no fc1 0.033, encoder only 0.12, decoder only 0.13, all 0.19, MLP only 0.36 (fp16 2e-5). No subset reaches the fp16 floor.

Hardware, Environment, and Revisions

• Repository: this commit on top of upstream/main at 1d82c9c8.
• Checkpoints: facebook/bart-base at aadd2ab0ae0c8268c7c9693540e9904811f36177 (manifest revision), facebook/bart-large-cnn at 37f520fa929c961707657b28798b30c003dd100b.
• Data: CNN/DailyMail 3.0.0 (abisee/cnn_dailymail): training split for calibration, test split for evaluation.
• GPUs: B200 and B300, one GPU per job. TensorRT 11.4.0.106 (target) and 11.1.0.106 (unit tests).

Not Run / Remaining Gaps

• The native C++ runtime was not run with an FP8 bundle; quality and speed used the engines from a Python harness.
• Speed was measured on TensorRT 11.4 only; the sensitivity subsets were not timed. bart-large and mBART were not measured.
• Found on the way, independent of FP8, recorded in the README and not fixed here:
  • The fp16 engine at max_sequence_length 1024 gives wrong logits (bart-large-cnn KL 4.78, top-1 49.4%; bart-base KL 3.02, top-1 42.3%), while 256 is correct and the encoder output matches. The manifest uses 256, so CI does not cover it.
  • The BART runtime does not apply forced_bos_token_id (0 in the bart-large-cnn generation config). Forcing it in the harness does not fix the 1024 result, so the two problems are separate.
  • The loader cannot load facebook/bart-large-cnn at the revision above: its shared token table is stored only as model.decoder.embed_tokens.weight.

Contributor Self-Review

• I have completed a self-review of this change.

Notes For Future Readers

• Revisit FP8 if a TensorRT release builds FP8 after the fused attention and gives a larger decode-step gain at batch 1, if the family supports batch above 1, or if a recipe keeps bart-large-cnn near the fp16 floor.
• The MLP projections (fc1 most of all) cause most of the drift; start there.

Risk level

• Low
• Medium
• High

Risk rationale: documentation only; no code, test or bundle change.

🤖 Generated with Claude Code

[coderabbitai]

Summary

Summary

Adds families/bart/README.md with BART and mBART engine configuration details and an unshipped FP8 quantization study.

The study reports no consistent end-to-end speed gain over FP16 and substantial output drift for bart-large-cnn. It documents the quantization recipe, measurement method, speed and quality results, projection sensitivity, bundle sizes, and why FP8 and NVFP4 are not shipped. It also records three issues reported on main: incorrect FP16 logits at max_sequence_length 1024, missing runtime handling of forced_bos_token_id, and a loader failure for facebook/bart-large-cnn.

Architecture impact

• Family-owned files: Adds only families/bart/README.md.
• Shared surfaces: None changed.
• Dependency direction: No code dependencies are added or changed.
• Affected consumers: README readers and users evaluating BART or mBART configurations. No runtime or build consumer is changed.
• Unresolved blast-radius questions: The README reports issues that affect BART behavior, but this change does not fix or test them. The native runtime was not tested with an FP8 bundle, and the documented study does not cover bart-large, mBART, or speed on TensorRT 11.1. The reported study measurements and validation results are not independently established by the available evidence.

Review outcome

HUMAN REVIEW REQUIRED — The change is documentation-only, but the available evidence does not independently verify the reported measurements or resolve the documented BART issues. This outcome is not a finding that the README is incorrect.

The BART README now describes BART and mBART engine settings and records an FP8 study. It reports study methods, performance and quality results, bundle-size measurements, untested configurations, and findings the study does not fix.

Changes: BART documentation

Layer / File(s)	Summary
Engine overview families/bart/README.md	Documents BART and mBART engine outputs, supported precision and sequence-length settings, and the facebook/bart-base manifest configuration.
FP8 study methods and results families/bart/README.md	Documents the quantization recipe and evaluation method, then reports speed, quality, projection-sensitivity, and bundle-size measurements.
Study limits and follow-up conditions families/bart/README.md	Lists untested configurations, three findings stated as not fixed by the study, and conditions for revisiting FP8.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Possibly related PRs

• NVIDIA/TensorRT-Model-Connect#1973: Changes the default task and bart-base manifest task to conditional_text_generation, while retaining text_generation support.

Merge Risk: Minimal · up to 8d738

This change documents the FP8 study and why it is not being shipped, without changing runtime behavior; no concrete product or operational risk is established.

Pre-merge checks: 8 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Benchmark Validation Integrity		The new report compares fp16 and FP8 using a common stated timing method (families/bart/README.md:102-111). The measurement harness and FP8 code are not in the PR or checkout. The available evidence…	Provide the measured fp16 and FP8 harness code, or equivalent detailed evidence of their timing and validation boundaries.

Passed checks: 8 passed

Check name	Status	Explanation
Title check		The title clearly describes the main documentation change.
Description check		The description is mostly complete. It does not identify an originating issue or discussion, give the exact Markdown check command, or specify the OS, CUDA, and dependency versions.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary		The PR adds only families/bart/README.md. It documents an unshipped BART study (lines 10–18) and does not introduce a cross-family dependency or a required registry change.
Shared Semantic Neutrality		The PR adds only families/bart/README.md. It does not change shared code.
Shared Change Blast Radius		The diff adds only families/bart/README.md. It documents a BART-specific study and does not change a shared surface. The check does not apply.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Benchmark Validation Integrity

Explanation

The new report compares fp16 and FP8 using a common stated timing method (families/bart/README.md:102-111). The measurement harness and FP8 code are not in the PR or checkout. The available evidence cannot verify that both paths include the same synchronization, transfers, and reductions.

Resolution

Provide the measured fp16 and FP8 harness code, or equivalent detailed evidence of their timing and validation boundaries.

[coderabbitai]

Additional comments: 1

families/bart/README.md-36-36: 1

36-36: Minor · Maintainability & Code Quality · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Hyphenate “end-to-end” in both places.

Line 14 already uses end-to-end. Match that spelling at Lines 36 and 104.

Also applies to: 104-104

Source: Linters/SAST tools

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/TensorRT-Model-Connect
• Branch: main
• File: .coderabbit.yaml
• Commit: 1d82c9c
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 89409747-4846-4a24-9e65-650b9c09d94e

Commits

Reviewing files that changed from the base of the PR and between 1d82c9c and 8d738a9.

Files selected for processing: 1

• families/bart/README.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.