Bug: m2m_100 frames the NLLB source as text </s> src_lang instead of src_lang text </s>, lowering translation quality #2042

Affected area: Inference output or task execution

TensorRT-Model-Connect version: main @ 5a2ce7e

Installation method: Development container

Model and revision: facebook/nllb-200-distilled-600M @ f8d333a098d19b4fd9a8b18f94170487ad3f821d (the nllb-200 manifest)

Environment: Linux, NVIDIA B200 and B300, TensorRT 11.4.0.106, fp16 engines; reference Transformers fp32, sacrebleu 2.6.0.

Description

families/m2m_100/runtime/request_tokens.h (m2m_100_apply_source_language_token) appends the source-language token after EOS, so the encoder input is

text </s> src_lang      (runtime)

The Transformers NLLB tokenizer frames the source as

src_lang text </s>      (default since legacy_behaviour=False)

The runtime framing is the legacy NLLB one. The model was trained and is evaluated upstream with the current framing.

Minimal reproducer

Run Transformers fp32 facebook/nllb-200-distilled-600M greedy translation (forced target-language token, up to 128 new tokens) on WMT16 newstest2016, 6 language pairs, both directions, 25 sentences per direction (300 sentences), once with the Transformers tokenizer framing and once with the runtime framing (text </s> src_lang), and score both against the human references with sacrebleu:

84 of the 300 outputs differ; the worst direction is en-tr (BLEU 20.12 -> 18.49). The same framing is what the native runtime feeds the engine, so the engine output follows the second row.

Expected behavior

The runtime frames the source as the current Transformers tokenizer does: src_lang text </s> (or follows the checkpoint tokenizer's legacy_behaviour setting).

Actual behavior

The runtime frames it as text </s> src_lang, which lowers chrF2 by 0.5 and BLEU by 0.7 on the set above.

Notes

Found during the FP8 study recorded in families/m2m_100/README.md (#2038). The same README lists three inefficiencies, independent of FP8: the 256206 x 1024 embedding is stored three times (about 1.5 GiB of the 2.2 GiB 600M bundle); cross-attention K and V are projected from the encoder output on every decode step (24 GEMMs of 128 x 1024 x 1024 per step for 600M) instead of once per sentence; and each decode step costs about 0.63 ms of host enqueue against 0.85 ms of GPU time because the native runtime does not replay the step as a CUDA graph.