Bug: bart fp16 engine is wrong at max_sequence_length 1024; bart-large-cnn cannot be loaded and forced_bos_token_id is ignored #2046

Affected area: Inference output or task execution

TensorRT-Model-Connect version: main @ 5a2ce7e

Installation method: Development container

Model and revision: facebook/bart-base @ aadd2ab0ae0c8268c7c9693540e9904811f36177 (manifest model); facebook/bart-large-cnn @ 37f520fa929c961707657b28798b30c003dd100b

Environment: Linux, NVIDIA B200 and B300, TensorRT 11.4.0.106, fp16 engines; reference Transformers fp32 greedy output.

Description

Three findings in families/bart, recorded in the FP8 study section of families/bart/README.md (#2004, section 8). They are independent of FP8.

1. fp16 at max_sequence_length 1024 gives wrong logits. The encoder output matches Transformers (cosine at least 0.9998); the decoder output does not. At 256 the same models are correct (bart-large-cnn KL 0.0005, top-1 98.2%; bart-base KL 6e-8). CNN/DailyMail 3.0.0 test articles, truncated to 1024 tokens, teacher-forced KL and top-1 agreement against Transformers fp32:

   Model, fp16, 1024	Articles	KL	Top-1	Greedy equal	ROUGE-1 (Transformers fp32)
   bart-large-cnn, forced BOS, B200	300	4.78	49.4%	11.0%	10.7 (27.5)
   bart-large-cnn, forced BOS, B300	300	4.79	49.4%	11.0%	10.8 (27.5)
   bart-base, B200	20	3.02	42.3%	0%	-

   512 and an fp32 build at 1024 were not measured. The manifests use 256, so CI does not cover 1024.

2. forced_bos_token_id is not applied. bart-large-cnn sets forced_bos_token_id: 0 in its generation config; the bundle runtime config takes only eos_token_id from it. (With the token forced in a test harness, item 1 is still wrong at 1024, so the two problems are separate.)

3. facebook/bart-large-cnn cannot be loaded. Its safetensors store the shared token table only as model.decoder.embed_tokens.weight, and the loader stops with:

   BART: cannot find shared embedding weight
   

Minimal reproducer

1. Build facebook/bart-base (revision above) in fp16 with max_sequence_length 1024 and with 256; run the decoder teacher-forced on CNN/DailyMail test articles and compare the logits with Transformers fp32: 1024 gives KL about 3, 256 matches.
2. Build facebook/bart-large-cnn (revision above): BART: cannot find shared embedding weight.
3. Inspect the runtime.json of any bart bundle: no forced BOS token, although the checkpoint's generation config has forced_bos_token_id.

Expected behavior

1. fp16 engines match Transformers at every supported max_sequence_length, including 1024 (and a test case covers it).
2. The runtime applies forced_bos_token_id from the generation config.
3. The loader falls back to model.decoder.embed_tokens.weight when neither shared.weight nor model.shared.weight is present.

Actual behavior

1. Wrong decoder output at 1024 (KL 3 to 4.8, top-1 42% to 49%).
2. The first generated token is not forced.
3. bart-large-cnn fails to load.