[zhenshanx-nv]

Background

families/qwen3_omni builds the Thinker of Qwen/Qwen3-Omni-30B-A3B-Instruct (Qwen3 MoE, 48 layers, 128 routed experts, top 8) as one bf16 text-generation engine and has no quantized build. No ModelOpt FP8 or NVFP4 checkpoint of this model exists on the Hub, so FP8 self-quantization of the Thinker (FP8 attention q/k/v with calibrated static activation scales, FP8 routed-expert stacks with one scale per expert) was built and measured on TensorRT 11.1 (B200). FP8 does not work for this family: the FP8 expert path gives wrong output (logit cosine about -0.09 vs the bf16 engine) and is not faster even so (decode 0.93x, 64-token prefill 0.66x). FP8 attention alone is correct but covers under 2% of the weights (decode 1.05x, perplexity +1.8%). A tiny-model check on TensorRT 11.4 (B200, B300) shows the FP8 Dequantize + Gather expert graph gives NaN there. This change adds a README for the family that records the study, like the yolov10 (#2093), bart (#2004) and timesfm (#2005) studies, so that it does not have to be repeated. The FP8 code is not part of this change.

Exit Criteria

• The README records the recipe, the method, speed, plan size and quality (vs the bf16 engine and Hugging Face) of the measured variants, the tiny-model check of the expert path on TensorRT 11.1 and 11.4, NVFP4, what was not tried and when to revisit.
• No code, test, manifest or bundle change; builds are unchanged.
• Non-goals: shipping FP8 or NVFP4 builds, an FP8 manifest or CI case.

Implementation

• families/qwen3_omni/README.md (new): a short family description and the section "FP8 quantization study (not shipped)": summary and decision, recipe, method, results, tiny-model check, NVFP4, what was not tried, when to revisit.

Recipe that was measured (code kept out of the repository):

• Attention q, k, v: FP8 weights (per-tensor amax / 448) and static per-tensor FP8 activation scales from max calibration on 192 nvidia/Nemotron-Post-Training-Dataset-v1 prompts; fp32 Dequantize and matmul.
• Routed experts: one FP8 [experts, in, out] constant per projection and layer with one scale per expert, Dequantize on the stack, then the existing top-8 Gather and bf16 batched matmuls. The scale is multiplied by a run-time 1 so that TensorRT does not fold the stack back to bf16.
• Kept bf16: attention output projection, attention, router, embedding, lm_head, norms.

Why no gain: the routed experts (95% of the Thinker weights) run as a Gather of the selected experts from a stacked constant, not as a plain GEMM. With an FP8 stack, the run-time Dequantize + Gather gives wrong results (NaN on TensorRT 11.4 in the tiny check), prefill gets 1.5x slower, and a second execution context asks for 19.3 GB of scratch memory, so the dequantized stacks seem to be written to memory instead of being read as FP8 by the matmul.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

python3 -m tools.model_ci validate
python3 -m pytest tools/tests/test_architecture.py tools/tests/test_public_source_hygiene.py -q
python3 tools/pr_metadata.py validate --event <event.json>   # Pull-request metadata is complete.
git diff --check

Documentation only.

Check	Result
python3 -m tools.model_ci validate	pass
pytest tools/tests/test_architecture.py tools/tests/test_public_source_hygiene.py (CPU)	62 passed
Markdown check of the README (table column counts, code fences)	pass
git diff --check	pass
python3 tools/pr_metadata.py validate on this description	pass

The numbers in the README come from the local study and are not re-run by this change (full model, B200, TensorRT 11.1, max sequence length 256; ratio is bf16 time over FP8 time):

Engine	Plan	Decode step	Prefill 64 tokens	Perplexity (HF 14.945)	Cosine / top-1 vs bf16
bf16	61.1 GB	7.81 ms	157.2 ms	14.889	1 / 100%
FP8 attention only	60.6 GB	7.41 ms (1.05x)	157.5 ms (1.00x)	15.150	0.9964 / 92.6%
FP8 experts only	32.1 GB	8.39 ms (0.93x)	236.7 ms (0.66x)	545,097	-0.084 / 0%
FP8 both	31.6 GB	7.99 ms (0.98x)	236.5 ms (0.66x)	653,873	-0.097 / 0%

Tiny random Thinker (2 layers, 8 experts), FP8 experts only, logit cosine vs bf16:

Expert graph	TensorRT 11.1 (B200, B300)	TensorRT 11.4 (B200, B300)
FP8 stack, run-time Dequantize, Gather (the recipe)	0.9921	NaN
FP8-rounded weights as a bf16 constant	0.99997	0.99997

Hardware, Environment, and Revisions

• Repository: this commit on top of upstream/main at 5801406c.
• Checkpoint: Qwen/Qwen3-Omni-30B-A3B-Instruct at 26291f793822fb6be9555850f06dfe95f2d7e695 (manifest revision).
• Data: Salesforce/wikitext wikitext-2 test split (12 texts x 128 tokens, held out); nvidia/Nemotron-Post-Training-Dataset-v1 (192 calibration prompts).
• Study: full model on B200, TensorRT 11.1.0.106, family code at 4f9ab925 with the FP8 change; tiny-model check on B200 and B300, TensorRT 11.4.0.106 and 11.1.0.106. Reference: Hugging Face Qwen3OmniMoeForConditionalGeneration Thinker in bf16.

Not Run / Remaining Gaps

• No code was run for this change; the study cannot be rerun from this repository because the FP8 code was not merged.
• Not measured: the full model on B300 or TensorRT 11.4 (only the tiny-model check ran there), interleaved timing rounds, longer sequences, FP8 attention output projection or lm_head, NVFP4.
• No problem on main was found: the bf16 engine matches Hugging Face (cosine 0.9983, perplexity 14.889 vs 14.945).

Contributor Self-Review

• I have completed a self-review of this change.

Notes For Future Readers

• Suggested review order: "Summary and decision", "Tiny-model check of the expert path", "When to revisit".
• Revisit FP8 when TensorRT gives correct results for Dequantize + Gather of an FP8 constant and reads FP8 weights inside the expert matmul, or when this family gets an FP8 grouped-GEMM expert path.

Risk level

• Low
• Medium
• High

Risk rationale: documentation only; one README added, no code, build, bundle or test change.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review.

Summary

Summary

Adds a family-owned README for Qwen3-Omni. It describes the bf16 TensorRT Thinker build and records an FP8 self-quantization study, including its recipes, measurements, correctness results, and limitations.

The README reports that FP8 routed experts produced incorrect output and did not improve speed in the measured setup. It documents that FP8 attention alone was correct but covered under 2% of the weights, and that NVFP4 was not tried. The document states that FP8 is not shipped and that the family rejects quantization options other than none. No implementation change accompanies these statements.

Architecture impact

Review outcome: PASS. The reviewed change adds only families/qwen3_omni/README.md. It adds no shared surface or dependency direction. The documentation describes behavior for the Qwen3-Omni family; the documented family build remains the relevant consumer. No code or validation artifact changed.

The README documents the Qwen3-Omni Thinker’s bf16 build configuration and an FP8 study. It reports quantization recipes, quality and performance measurements, expert-path results, unmeasured cases, and conditions for revisiting FP8.

Changes: Qwen3-Omni build and quantization documentation

Layer / File(s)	Summary
Build scope and FP8 decision families/qwen3_omni/README.md	The README describes the single-engine bf16 build, excluded components, and rejection of quantization values other than none. It summarizes the FP8 study results and states that FP8 is not shipped.
FP8 recipe and measurements families/qwen3_omni/README.md	The README records the measured FP8 recipe, component choices, measurement methods, quality and build results, and bf16 baseline comparison.
Expert-path results and study limits families/qwen3_omni/README.md	The README reports tiny-model expert-path results across TensorRT versions, notes unmeasured or unavailable configurations, and lists conditions for revisiting FP8.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: Minimal · up to 023a6

No actionable issue is established that would prevent merging this documentation change.

Architecture risk: Low · up to 023a6

The change affects 1 system.

Changed systems: families

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

• observed — families (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

• observed — Modified behavior in families/qwen3_omni/README.md: The README documents the Thinker’s single-engine text_generation build configuration, excluded model components, bf16 batch-1 setup, and rejection of quantization values other than none.
• observed — Modified behavior in families/qwen3_omni/README.md: The README adds an FP8 study summary and decision, including B200/TensorRT 11.1 measurements, quality and speed comparisons, the incorrect expert-path outputs, and attention-only results.
• observed — Modified behavior in families/qwen3_omni/README.md: The README records the measured FP8 recipe, parameter split, per-component quantization choices, and the TensorRT constant-folding and Gather constraints described for the expert path.
• observed — Modified behavior in families/qwen3_omni/README.md: The README documents the quality and speed measurement methods and reports quality and build results for bf16, FP8 attention-only, FP8 experts-only, and combined FP8 variants.

Pre-merge checks: 8 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Benchmark Validation Integrity		The new report compares bf16 and FP8 timing and quality, but the study code is not available. README lines 76–91 describe a common method, but do not establish synchronization or transfer boundaries f…	Provide the study code or measurement records for both variants, including timing boundaries, synchronization, transfers, and the reference used for each quality metric.

Passed checks: 8 passed

Check name	Status	Explanation
Title check		The title clearly states that the change documents why FP8 is not shipped for Qwen3-Omni, which matches the README’s main purpose.
Description check		The description covers the required template sections, including background, exit criteria, implementation, change category, validation, environment, remaining gaps, self-review, and risk rationale.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary		The PR adds only families/qwen3_omni/README.md. The diff introduces no cross-family dependency or central registry change.
Shared Semantic Neutrality		The PR adds only families/qwen3_omni/README.md. It does not change shared code.
Shared Change Blast Radius		The pull request adds only families/qwen3_omni/README.md. It does not change a shared surface, so this check does not apply.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Benchmark Validation Integrity

Explanation

The new report compares bf16 and FP8 timing and quality, but the study code is not available. README lines 76–91 describe a common method, but do not establish synchronization or transfer boundaries for each variant. The report also gives nearly identical comparison metrics against the bf16 engine and Hugging Face; the underlying results are unavailable to verify their attribution.

Resolution

Provide the study code or measurement records for both variants, including timing boundaries, synchronization, transfers, and the reference used for each quality metric.

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/TensorRT-Model-Connect
• Branch: main
• File: .coderabbit.yaml
• Commit: 5801406
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: c841681f-349f-47e4-b23e-811824c6bd71

Commits

Reviewing files that changed from the base of the PR and between 5801406 and 023a602.

Files selected for processing: 1

• families/qwen3_omni/README.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 4 remain after this review.