nemotron: serve NVIDIA Nemotron 3.5 Bolt (nemotron_h_puzzle) on CUDA

[MiaAI-Lab]

Refs #937 (RFC, please read first).

What this changes

Serves NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB on the native CUDA host as the family nemotron_h_puzzle. The hub.modelType and server/main.zig readers report an omni config wrapped around a puzzle text model as nemotron_h_puzzle; a lane backend, families/nemotron/bolt_native.zig, runs it over libbolt.so; zig build bolt -Dnvcc -Dbolt-src builds that library beside the server; images go through the existing RADIO tower. Two commits: --vision-image-tokens past 16,384 (a family that encodes images as the prompt pass reaches them needs no whole-request buffer), then the family. A family may set snap_lookahead for the prompt cache (Bolt has no head reading ahead). A recipe page, docs/recipes/nemotron-3.5-bolt.md, has the build steps, the exactness notes, the receipts and the limits.

The CUDA sources are not in this PR. libbolt (about 4,800 lines of CUDA runtime code, sm_121) is built from a directory -Dbolt-src names; it is in https://github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark/tree/main/engine, which is private for now (I can make it public or send an archive). Whether the kernels belong here, under the kernel contract, is question 1 in the RFC.

Structured output and forced tool calls (added after the first review round)

response_format (json_schema, json_object), the guided_* fields and tool_choice "required" or a named function were answered with HTTP 400 ("not supported by this engine yet"). Two separate causes, both fixed in a6660d38:

• No masked draws in the backend. The engine advertised no structured output because first_masked was missing. verify now redraws the rows of a masked window among the tokens their bits allow (the grammar's rows x words layout), first_masked draws the first token again under the first mask, and held() hands the grammar the drafts the drafter holds. libbolt gets bolt_mask_rows (C ABI 2): a masked variant of its two-stage GPU top-k that treats a disallowed token as minus infinity and recomputes only the masked rows, so a round without masks costs what it did. A kernel test compares it with a CPU reference (a mask of 5 tokens, about half the vocabulary, and a single token, at k of 1, 20 and 64).
• vocab_size of an omni checkpoint. The server read it from text_config or the top level only; an omni checkpoint keeps it in llm_config, so every structured request failed with "needs the checkpoint's config.json vocab_size" (this affects any omni model, not just Bolt). It is now server/vocab.zig, a pure function with a test.

The libbolt binding moved to bolt_api.zig so bolt_native.zig stays under 600 lines; lean_new.py still reports 0 new problems.

Checked on a GB10 against the built image: json_schema and json_object replies parse and match; tool_choice required and a named function make valid calls (several tools, parallel_tool_calls); guided_choice, guided_regex and guided_json obey; thinking on gives free reasoning and a schema-valid answer; a bad schema is a clean 400; 6 of 6 sampled (temperature 1) replies on a nested enum/array schema are valid; a streamed reply is valid JSON; with 4 structured and 4 plain requests at once, all 4 structured replies are valid and all 4 plain replies equal their solo runs. Structured decode ran at about 130 tok/s alone against 90 for the same task unconstrained (the grammar makes drafts accurate). At 16 streams (bench_concurrent --alone --serial, 256 tokens, 2 reps): code 285.9 tok/s aggregate and chat 210.4, every concurrent reply equal to its solo run (32 of 32 in both) and drafted equal to "draft": false.

Not run: the structured-output path on another family or on Metal; grammars against the Python 0.6.6 engine's outputs.

Receipt

• Environment: TensorFold 46a08c18 plus these commits; DGX Spark GB10 (sm_121), Ubuntu 24.04.4, kernel 6.17.0-1026-nvidia, driver 580.159.03, CUDA 13.3.73; NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB at a326822af111290f2af5ac59ef4b5040347dabc4, drafter 4c6a5613b6d887194a3a5493e3a18a5b3930bb86; bf16 cache with every stream's window mapped at load.
• Exactness, tools/bench_concurrent.py --alone --serial:

tools/bench_concurrent.py --levels 1,4,8 --alone --serial, greedy, 256 tokens, 3 repetitions, against --parallel 8 --context 262144 --prompt-cache-gib 8 --vision:

Prompt	Streams	Aggregate	Each stream	Max TTFT	--alone (concurrent equals solo)	--serial (drafted equals "draft": false)
code	1	115.1 tok/s	115.1	0.07 s	3 of 3	1 of 1
code	4	207.9 tok/s	54.8	0.45 s	12 of 12	1 of 1
code	8	252.6 tok/s	35.2	1.15 s	24 of 24	1 of 1
chat	1	60.7 tok/s	60.7	0.05 s	3 of 3	1 of 1
chat	4	139.1 tok/s	35.6	0.42 s	12 of 12	1 of 1
chat	8	174.7 tok/s	23.6	1.05 s	24 of 24	1 of 1

Concurrent equals solo in every cell, code and chat prompts, and drafted equals "draft": false in every cell.

Why concurrent used to differ from solo, and what changed. An earlier version of this text blamed "GEMM kernels chosen by the rows in a block"; that was incomplete. Three things made a stream's rounding depend on the batch, and all three are fixed in libbolt:

1. The verify-window attention merged partial softmaxes over key chunks whose size followed the number of streams in the block (about 32 splits for one stream, 8 for four) and the window's width. The chunk is now win_chunk(base): 128 keys, doubled until 32 chunks reach the end of the sequence and its window, a function of the stream's own length alone. With the split count forced fixed in a scratch build, alone and 4-stream perplexity were identical, which isolated this one.
2. Dense GEMMs for blocks of up to 32 rows split K across four warps (interleaved partial sums, then added); blocks over 32 rows used a kernel that walks K in a single accumulator. Decode and verify blocks now always use the K-split kernel (64 tokens a CTA); the prompt pass keeps the faster kernel. This was the reason chat at 8 streams still differed after the attention fix.
3. The one-token plain step ran a different attention kernel from the windows; it now runs as a one-row window and is kept at once.

Harness checks (run with 32-token GEMM tiles): perplexity of four passages scored alone and in a block of 2, 3, 4, 8 and 16 streams (windows of 1, 5 and 8 rows, up to 128 rows a block) is identical to five decimals; sixteen sequences in one block equal their solo runs, 768 of 768 tokens, plain steps and 4-row windows alike. The final 64-token tile changes only the tokens a CTA holds, not the order K is summed in; it is covered by the table above.

Cost. The K-split kernel is slower in the widest rounds (shared round of N streams x R rows, ms, before / after): 8 x 8 rows 135.5 / 130.4, 16 x 4 rows 133.2 / 141.7, 16 x 8 rows (128 rows) 176.9 / 220.3. Rounds of up to 32 rows are unchanged.

• Decode speed: new family, no before. The table's one-stream rows are 115.1 tok/s (code) and 60.7 tok/s (chat).
• Prompt speed: new family, no before. tools/bench_cache.py --require-cold on exact token arrays (0 cached tokens, qualified: true), measured on the build before the exactness changes, which touch decode blocks only (not re-run since): TTFT 1.44 s (2,048 tokens), 5.60 s (8,192), 24.66 s (32,768), 55.60 s (65,536).
• Tests run: zig build test (host suite; added: hub.modelType for an omni config around a puzzle model, a plain one and Lightning's). zig fmt --check build.zig zig: clean. python3 tools/zig/lean_check.py through lean_new.py against the base: 517 problems, 517 on the base, 0 new. Each commit builds and passes the host suite on its own. Engine gates in the recipe's harness (run after the attention change; the GEMM-order change was checked with the 16-stream and in-block perplexity tests above): perplexity 5.286 on 1,499 tokens; a window cut to n rows equals a window of n rows; a resumed snapshot equals a whole prefill; the drafter accepts 1.70 of 7 on a creative passage (the same histogram as on the base before the attention change); a stream's logits are bit-identical whatever its neighbours hold.
• Not run: Metal and every CUDA card but the GB10; the kernels under the kernel contract; any other Omni checkpoint through this change.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false (every --serial cell), a resumed prompt equals a fresh one (checked in the recipe's harness, not with the repo's tools), and concurrent equals solo (every --alone cell).
• No precision traded for speed. The routed experts quantize their activations to int8 per 32 values (ggml's scheme) for the IQ2_XXS grids; every other projection keeps fp32 accumulators with fp16 or bf16 activations. Perplexity on 1,499 tokens is 5.286 against 5.29 for the previous fp16-expanded engine. New family, so there is no earlier bit pattern to hold.
• A new or changed GPU kernel is registered with the kernel contract. No: libbolt is a CUDA runtime library outside this tree (RFC question 1).
• Prompt processing is no slower than the last release: there is none for this family.
• The description says which platforms I ran: CUDA on one GB10 only; no Metal.
• New tests fail before the change and pass after it; the host suite needs no GPU or library.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thank you, @MiaAI-Lab, we will carry the Bolt lane design and focused metadata work with your credit into the single native Bolt family; the landing order and remaining checks are in our reply on #897.

Posted by Ash's AI Assistant Synthena