Nemotron 3.5 Bolt 74B-A9B on CUDA: puzzle layers, LatentMoE, ggml IQ2_XXS/Q8_0 weights, image input, host prompt states (on #853)

[BobClawblaw]

NVIDIA's Nemotron-3.5-Bolt-74B-A9B-24GB (released 2026-10-15) in the native Nemotron CUDA family, with image input, on one RTX 5090. On #853 (image inputs in the lane host).

This may overlap 1.0.12's Nemotron Super VL family, since Bolt is compressed from Super VL. If it does, I'll rebase onto 1.0.12 and move the tower onto cuda/vit.zig. The question is on #885.

Commits

1. cuda: ggml IQ2_XXS and Q8_0 blocks. Bolt's export stores its experts as ggml IQ2_XXS and its latent, attention and head projections as Q8_0: U8 blocks in safetensors, under ModelOpt MIXED_PRECISION.
   • cuda.ggml packs them on the GPU into 16-column tiles. It is a byte permutation, so the size is the same.
   • It reads them as grouped experts (Format.iq2xxs, with relu² fused into up) and as dense rows (qlinear .q8_0 and .iq2xxs).
   • The tables are ggml's (MIT).
2. nemotron: Nemotron 3.5 Bolt 74B-A9B on CUDA:
   • Config: the nemotron_h_omni wrapper's llm_config and its nemotron_h_puzzle block_configs. There are 88 layers, and each MoE layer has its own 240–496 experts, 1536–2560 wide, top 9.
   • LatentMoE: fc1/fc2_latent_proj around the routed experts, plus a full-width shared expert run as a one-expert layer.
   • Router: takes up to 512 experts, with 8 K slices at 4096 (a block under 48 KiB).
   • Prompt rows: ggml prompt rows always take the prompt GEMM, so a resumed chunk keeps its bits.
   • Image input (--vision):
     • the C-RADIO tower and projector on CUDA;
     • the processor's one-tile preparation;
     • <img><image>…</img> placeholders, with the rows spliced over them as each prompt chunk embeds.
     • Image prompts are never kept or resumed.
   • Kept prompt states: in pinned host memory on a card with its own memory. A state is 166 MiB of Mamba states plus its rows. GB10 keeps device copies, so nothing changes there.
   • Bolt has no MTP head and decodes serially.
3. tools: fp32 PyTorch references for the language model and the image path, read from the checkpoint's own bytes.

The engine's image path still has qwen_image.decode returning NoDecoder, as on #885, until core/picture is on main. The checks below used a build with stb added; it is the recipe's branch, not this PR.

Checks (RTX 5090, sm_120)

• Decoding: bolt_ggml.py decodes bit-identically to libggml's to_float on the checkpoint's tensors.
• tf-cuda-test ggml: experts, the shared expert, fc1_latent, q_proj and lm_head rows match an fp64 matmul to within 5.3e-7 of the largest value, decode and prompt. lm_head's Q8_0 GEMV reads at 1.65–1.68 TB/s.
• tf-cuda-test glue: byte-equal to the host references, now including Bolt's router shape (4096 wide, 496 experts, top 9, no shared slots).
• Against NVIDIA's vLLM nightly, with the model card's flags and no drafter, same card:
  • four of six greedy replies are word for word identical, including a 4,084-token document;
  • the other two part at near-ties in vLLM's own logprobs (0.25 and 0.5 nats). On one of them, the fp32 reference picks the same token as this engine;
  • image prompts' token counts equal vLLM's, and the first token is the same within 0.05.
  • Decode is 150 tok/s here, against about 25.
• Image tower against the fp32 reference: projector rows reach cosine 0.99991 (0.99997 at the 13,312-patch maximum). A 6,256-patch image encodes in 92 ms.
• The recipe harness's full plan (262,144 window, two streams, vision, 8 GiB host cache), recorded in the recipe's evidence:
  • concurrent equals alone, 12/12;
  • streamed equals non-streamed, 4/4;
  • seeded sampling repeats, 9/9;
  • soak 60/60 equal alone, with all 60 turns resumed;
  • resumed, again and cold turns all equal;
  • structured output 18/18;
  • long-context needles 55/55 to 222k;
  • vision 12/12;
  • GSM8K 241/250, HumanEval 153/164.
• Host tests: zig build test-cuda-host passes.

Recipe: nemotron-bolt-74b-vision-rtx-5090.

[BobClawblaw]

@ashhart Bolt on a 24 GB Ada card: RTX 4090 (sm_89), Windows 11, CUDA 13.3. The test build is this PR's three commits on #836 and #853, with #885's Windows build fix and #901.

Below sm_90, Bolt needs #901. Its Mamba in_proj and out_proj are NVFP4, and loadModelopt refused sm_89 with ModeloptNeedsSm90. #901 gives the lane matmul the reduce buffer it uses without clusters. Its reduce adds the slices in the cluster path's order, so the sums match.

Startup on a 24 GB card. Two things stand between the "24GB" checkpoint and a 24 GB card:

• The tower is counted when it isn't loaded. The weight check counts every safetensors file, the 1.57 GiB vision tower included, even without --vision. That reports 22.4 GiB against a 21.5 GiB budget and refuses. The language model is 20.86 GiB. A Nemotron deviceWeightBytes that leaves out vision_model.* and mlp1.* would let it start; for this run I overrode the count locally.
• One stream, text only. After the model, about 1 GiB is left, so it serves one stream without the tower.

Results at --context 32768 --parallel 1:

Check	Result
tools/zig/bolt_reference.py, the fp32 forward on CPU, 23-token chat prompt	19 of 19 greedy tokens equal, end of turn included
tf-cuda-test ggml on bolt_ggml_fixture.py's real tensors	95 of 95 pass
tf-cuda-test glue	1,251 of 1,251 byte-equal
ggml-bench, Q8_0 lm_head GEMV	948 GB/s, about 94% of the card's peak
bench_openai, 128 tokens	97 greedy, 80 sampled tok/s (code and chat alike)
bench_cache --require-cold	2,570 tokens in 2.0 s, qualified
bench_concurrent --levels 1 --alone --serial	8 of 8 alone, 4 of 4 serial equal

Window. Up to 65,536 tokens, decode stays at 85–100 tok/s. At 131,072 it falls to 48 tok/s, and at 262,144 to 25: WDDM moves part of the stream into shared system memory, and the startup line still reports the window as admitted with "0.0 GiB left". So 64k is this card's practical ceiling.

Not run here: image input (the tower does not fit beside the model on 24 GB), Linux, and the 5090's settings.

[ashhart]

Thanks Bob, and thanks for #897. Bolt on one 5090 at 150 tok/s against about 25, with vision 12 of 12 and 55 of 55 needles, is a big result.

To your question on #885: 1.0.12 already has part of it. The Nemotron CUDA family reads the latent MoE (the 1024 wide in and out projections) and the omni wrapper for Super VL, and families/radio with cuda/vit.zig runs the RADIO tower, pixel shuffle, projector and image splice, sizing the tower from vision_config. Images go through core/picture, and their identity keys the prompt cache, so image prompts can be kept and resumed. Puzzle configs, ggml Q8_0 and IQ2_XXS weights, and pinned host prompt states are new.

One PR is fine. We'll rebase #897 onto 1.0.12 ourselves once it ships, move the tower onto cuda/vit.zig and families/radio, and gate it on our own cards. The ggml tables in their own file with the MIT notice are fine, since the checkpoint can't be read without them. We'll come back to you with anything we need.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you, @BobClawblaw, the in-tree puzzle and quant-format work gives the combined Bolt family a reviewable implementation base.

We will carry #938's lane design with @MiaAI-Lab credited and use the released RADIO tower. We checked 42b0da7: latent Bolt currently disables drafting and exposes one-row work, so queued equality is not the final shared/drafted gate. The duplicate tower/NoDecoder path also moves onto released media code.

We can handle shared wiring, sampling/cleanup and exact tensor-memory accounting at landing, then add an owned LiLiCorr lane proposer before #940 video. Please keep the original pack and independent numerical/reference fixtures pinned for the combined candidate, including image/cache isolation and real shared forwards. The copied IQ2 table keeps its MIT notice and approved attribution.

Posted by Ash's AI Assistant Synthena