Bug: marian cannot build OPUS-MT tc-big checkpoints, and runs the encoder and cross-attention K/V over all 512 positions on every step #2041

Affected area: Model resolution or TensorRT engine build

TensorRT-Model-Connect version: main @ 3db3162

Installation method: Development container

Model and revision: Helsinki-NLP/opus-mt-en-ru @ bb09c99d180016eac6819df3dae68edb1690fdee (manifest model); Helsinki-NLP/opus-mt-tc-big-en-fr @ 6e062862 (tc-big)

Environment: Linux, NVIDIA B200 and B300, TensorRT 11.4.0.106, fp16 engines (as in the marian-en-ru manifest).

Description

Three findings in families/marian, found while measuring FP8 (see the FP8 study in families/marian/README.md, #2035). They are independent of FP8.

1. tc-big checkpoints cannot be built. The loader reads model.encoder.embed_positions.weight and model.decoder.embed_positions.weight from the checkpoint. OPUS-MT "tc-big" checkpoints (model_type: marian) do not store these tensors (Transformers computes the sinusoidal table). In addition, the MLP activation is hard-coded to SiLU (graph_ops.add_activation(..., "silu", ...) in model.py), while these checkpoints use activation_function: relu, so adding only the position table would still give wrong output.
2. Encoder and cross-attention K/V always use 512 positions. The encoder runs at max_position_embeddings (512) positions for every source, and each decode step projects the cross-attention K and V of every decoder layer from the full 512-row encoder output (12 GEMMs of 512 x 512 x 512 plus 12 casts of 512 x 512 per step), instead of once per sentence for the real source length. Real sources in WMT16 test are about 30 tokens.
3. The shared 62518 x 512 embedding is stored three times (encoder gather table, decoder gather table, decoder lm_head): about 183 MiB of the 278 MiB fp16 bundle (encoder plan 99 MiB, decoder plan 172 MiB).

Minimal reproducer

1. Build Helsinki-NLP/opus-mt-tc-big-en-fr at the revision above with the marian family build:

   KeyError: 'Tensor not found: model.encoder.embed_positions.weight'
   

2. Build marian-en-ru (fp16) and inspect the plans: the encoder input is fixed at 512 positions, and the decoder step plan contains the cross-attention K/V projections over 512 rows in every layer. The plan sizes give the embedding copies (item 3).

Expected behavior

1. tc-big checkpoints either build correctly (compute the sinusoidal position table when the checkpoint has none; honor activation_function) or are refused up front with a clear error.
2. Cross-attention K/V are computed once per sentence from the encoder output of the real source length (or at least not re-projected on every step).
3. The embedding table is shared, or stored once per engine where needed.

Actual behavior

1. KeyError at load time for tc-big checkpoints; SiLU is used for every checkpoint.
2. Per-step cost includes 12 projections over 512 rows regardless of source length (not timed separately).
3. About 66% of the fp16 bundle is three copies of one table.