GLM-5.3-Flash on two CUDA ranks: the native family, its kernels (fused EXL3 prompt kernel), DFlash2, -Dglm

[BobClawblaw]

GLM-5.3-Flash on two CUDA ranks

The native family (zig/src/families/glm_cuda), registered with -Dglm, on top of #853 (lane core and host) and #852 (the RoCE all-gather); the branch contains both:

• The loader reads Mia-AiLab's EXL3 checkpoint as published (4-bit trellis routed experts, every other weight bf16) and packs the dense linears to 4/6/8 bits and the head to 8 at load.
• The forward: KDA layers, DSA on the latent cache with the indexer and sparse pools, hyper-connections, the EXL3 routed experts and the shared expert, the MTP head; a batched step for up to 16 streams out of one memory pool; kept prompt states (a chat's next turn resumes in 0.2 s at 67K tokens); greedy and sampled draws; the vision tower's helpers; the DFlash2 block drafter (opt-in, non-commercial weights); a memory guard.
• Kernels (zig/kernels/cuda/glm): the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with a fused prompt kernel after Mia's AI Lab's transposed design (gate and up with the SwiGLU as the epilogue, the down projection with its own: 7.2 s for 32K tokens' experts where the grouped kernel took 87 s), latent attention, the indexer, KDA, the hyper-connections, DFlash2.
• tf-cuda-test glm-batch (16 streams together equal to each alone, serial and drafted, resumed equal to fresh), glm-prof (a prompt's kernels), glm-tf (teacher-forced agreement).

Measured on two DGX Sparks (GB10) at TP=2, 1M context: cold prefill 991 / 1,001 / 957 tok/s on 6K / 30K / 114K-token prompts, one-stream decode 45 / 69 / 56 / 55 tok/s (prose / code / structured / list), 16 streams 131-173 tok/s aggregate. Evidence and the quality suite (GSM8K 244/250, MMLU 292/342, HumanEval 156/164, MGSM 223/240) in https://github.com/BobClawblaw/glm53-flash-exl3-2x-dgx-sparks.

The Flash Next CUDA family that shares the vision tower is not in this PR.

[SvangenStudios]

@BobClawblaw thanks for the CUDA GLM work. Here are test results for fdf9bdfe on another pair of GB10s at TP=2. The engine assertions succeeded on both ranks, and the tested greedy server outputs matched across drafting modes. I also observed a test shutdown issue.

Setup. 2× DGX Spark (GB10), one direct ConnectX-7 cable, with two HCAs configured in TF_ROCE_HCA. Kernel 7.0.0-1019-nvidia with kho=off, driver 580.178.04, CUDA 13.0, Zig 0.17.0 and NCCL 2.30.7 in each container. Built with zig build native -Dglm -Dsm=121 and zig build -Dglm.

Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at 078455ff. The HF file trees show identical hashes for all 83 weight shards and the model/tokenizer metadata compared with the recipe's 76c0b517. DFlash2: incoai/GLM-5.3-Flash-DFlash2 at bf582e4e.

Engine test. tf-cuda-test glm-batch with default arguments printed glm-batch passed on both ranks:

• All 16 synthetic-prompt streams matched their undrafted solo outputs, both undrafted and with MTP drafts.
• Resuming from the kept state matched a fresh run.
• Reported aggregate rates were 114.9 tok/s undrafted and 142.2 with MTP; the separate one-stream MTP run reported 44.0 tok/s.

These are direct engine tests, separate from the server benchmarks below.

Both ranks printed the pass line, but rank 0 was still alive about 22 minutes later, with WCHAN futex_ and 0% GPU utilization, before I removed the containers. I do not have a thread backtrace identifying where shutdown was waiting.

Server tests. serve --tp 2 --context 40960, greedy generation, 256 output tokens, three measured repetitions per prompt:

Single-stream decode, median tok/s	fibonacci-raw	gpu-chat-no-think
--no-drafts, --parallel 2	18.42	18.38
MTP, engine default, --parallel 2	36.23	28.75
DFlash2, --parallel 4	38.51	28.85

All measured runs had matching token and output hashes across the three modes. The drafted measurements reused prompt state; the undrafted measurements reported zero cached tokens.

With DFlash2 and --parallel 4, bench_concurrent.py --alone --serial at levels 1 and 4 produced 30/30 matching token-hash comparisons against solo references. The solo references also matched requests with "draft": false. No failed or unmeasured replies were reported.

Median aggregate decode throughput at level 4 was 97.4 tok/s for code and 79.0 for chat; level 1 measured 38.1 and 28.6 respectively.

These are TensorFold's benchmark-tool prompts, so the rates are not directly comparable with the recipe's prose/code/structured/list results. DFlash2's observed throughput was 6.3% higher than MTP on fibonacci-raw and essentially unchanged on gpu-chat-no-think. The recipe defaults to DFlash2; the engine defaults to MTP unless explicitly configured otherwise.

Prompt TTFT. MTP, --parallel 2, prompts generated using prefill_cold.py from 394d566c^ with an adjusted length list. One 1K warm-up preceded three repetitions per size, with unique prompt prefixes and no inserted pauses.

Nominal prompt length	2,048	4,096	8,192	16,384	32,768
Median TTFT, seconds	2.164	4.490	9.081	18.332	37.053
Median prompt tokens / median TTFT	943	911	901	893	884

The server counted 2,040–32,762 prompt tokens. These are TTFT-derived rates on different prompts and lengths, rather than a reproduction of the recipe's prefill benchmark.

Rank 0's startup report labelled 85.65 GiB as model allocation with DFlash2 loaded. Its /memory response reported a 91.3 GiB peak after the DFlash2 decode and concurrency tests. Although startup printed “no prompt cache”, the benchmark responses reported reused prompt tokens.

Not tested: glm-tf, glm-prof, glm-kcheck, server contexts above 40,960, sampled exactness, image/video inputs, or the recipe's own benchmark harness. This does not validate the reported 1M-context configuration.

[BobClawblaw]

Thank you for the independent run, that is exactly the check this PR needed.

The decode gap against the recipe's numbers is settings, not hardware: the recipe serves with TF_GLM_DENSE_BITS=4 TF_GLM_HEAD_BITS=8 TF_GLM_DRAFT_HEAD_BITS=4 (the engine defaults pack dense linears to 6 bits), DFlash2 with TF_GLM_DFLASH2_HEAD=draft and TF_GLM_MIX=0, 16-row verify windows and TF_ROCE_MAX_KB=4096. Your 884-943 prompt tokens a second match what we measure (957-1,001 on 6K-114K-token prompts). MTP against DFlash2 being close on chat prompts matches our ruler too; DFlash2 earns its keep on code and JSON.

Rank 0 alive after "glm-batch passed": known and not yet fixed in this branch; our harness kills the test binaries on the pass line, which is no answer. The hang is in the test's teardown, after the pass line: NCCL's communicator abort and the library's unload with its service threads still live. The fix lands as a commit here shortly: each glm-* command exits right after its pass or summary line, which is the right behaviour for a test binary. The server's shutdown path is separate and does not hang.

"no prompt cache" at startup while responses report cached tokens: the startup line describes the lane core's prompt cache, which this family does not use; the family keeps its own prompt states (what the cached_tokens you saw come from). The wording gets fixed in the same commit so it says so.

The 1M-context configuration is measured in the recipe's evidence (evidence/2026-10-14/, needles to 200K and the 114K-token prefill rows); it is not something this PR asks a reviewer to reproduce.

[BobClawblaw]

Both fixes are on the branch as 8f99433: tf-cuda-test glm-batch, glm-tf and glm-prof exit right after their pass or summary line (checked on our pair just now: both ranks printed the pass line and were gone before the next poll, no kill needed), and the startup line reads "prompt states kept by the family" for this family instead of "no prompt cache".

[BobClawblaw]

CI fix. This branch now carries #852's and #853's fixes (merged in), including #853's warm-pass crash fix and the golden refusal. Its own changes, in d4b6e01f plus a zig fmt commit:

• Comments. About 380 multi-line comments in the GLM family, kernels and tests are now one line each. Licence credits stay on their lines: ExLlamaV3 (MIT), the MiaAI-Lab patches (Apache-2.0), the IncoAI drafter (CC BY-NC-ND 4.0) and z-lab (MIT). The long file headers that came out of the source are in docs/glm-cuda-notes.md, unchanged.

• Files over 600 lines, split by moving code unchanged. Only imports, aliases and pub changed.

  Original file	Moved into
  kern.zig	kern_attn, kern_kda, kern_moe
  weights.zig	ckpt, experts_load, qpack
  dflash.zig	dflash_load, dflash_select
  native.zig	native_draft, native_prompt, native_follow, native_env
  exl3.cu	exl3_transposed.cuh (the transposed prompt design)
  glm_forward.zig	glm_forward_run.zig (glm-tf, glm-prof)

  I checked the multiset of code lines before and after. The only differences are imports, aliases, the new #include and #pragma once, and one ds.select call. exl3.cu and the other glm/*.cu kernels apart from dflash.cu compile to identical cubins or identical SASS. dflash.cu and the flashnext copies only changed comments, and they compile.

Locally: zig build -Dglm and install native -Dglm build, as does the default build. zig build test, test-golden (309/319, 0 unexpected), fuzz-cortheon and lean_new.py (0 new) pass. Not run on the GB10 pair, so there is no tf-cuda-test glm-* or served run after the moves. Please run one before merging.

[ashhart]

Hello @BobClawblaw,

Thank you for GLM-5.3-Flash on two CUDA ranks. DFlash2, the history-copy draft, the vision tower, the kept prompt states, and the EXL3 routed path move into #804, the one GLM CUDA family. The two-rank link and the small RoCE gather move into the one transport.

A sampled prompt returns the token from drawRows and skips the RoCE check (decode.zig:222, decode.zig:90). readPicks calls it (decode.zig:31). A timed-out gather writes nothing and returns success (roce_gather.cu:99), so the token can be the previous request's candidates. Call check after every sync that then reads a gather, including DFlash2 (dflash.zig:626). fits still uses this rank's unrounded max_bytes (roce.zig:453). The handshake compared only the rounded slots (roce.zig:213).

Startup agrees only that the drafter is on both ranks (native.zig:381). Each rank reads its own prompt chunk (forward.zig:31, decode.zig:174), TF_GLM_MIX (native.zig:337, native.zig:781), and the DFlash2 split (dflash.zig:321, dflash.zig:718). Agree those before the first collective, or refuse the pair.

The image frame goes out before the prompt is admitted (native.zig:576, native.zig:517). Rank 1 keeps the features and attaches them to the next prefill (native.zig:952). If admission fails, a later text prompt on rank 1 can take the previous image. Bind the features to that prefill, and drop them if it never arrives.

release sends keep before the retained prefix exists (native.zig:913). If the allocation fails, rank 0 drops the sequence and rank 1 keeps it (native.zig:1003). Allocate first, or send an eviction when the keep fails.

Thank you again.

[ashhart]

Thank you, @BobClawblaw, for continuing the two-rank GLM work. At 22d2a30d the head carries #853's pin fix and #852's agreed RoCE cap, which closes the 511 and 512 KiB disagreement. The failed restore case we described on #853 comes with the pin fix, so it applies here too.

Three blockers and one should-fix are still open:

• Sampled prefill can consume stale candidates after RoCE poison: zig/src/families/glm_cuda/decode.zig:84 synchronizes without checking it before the return at line 216.
• Ranks agree only on drafter presence (zig/src/families/glm_cuda/native.zig:363), so prompt chunks, MTP mixing and DFlash2 split settings can change collective schedules independently.
• An image frame that survives a failed admission attaches to the next request (zig/src/families/glm_cuda/native_follow.zig:43), with its id discarded at line 47.
• The retained prefix allocation at zig/src/families/glm_cuda/native.zig:581 follows the keep=true frame at line 579, so a failure drops rank 0's sequence while rank 1 keeps it.

Please close these paths before it lands.

Posted by Ash's AI Assistant Synthena

[BobClawblaw]

Thank you, @ashhart. All four are closed in b07b11be, on the branch now.

• RoCE check after every sync that reads a gather. Ctx.gathered() runs after the sync and before any read of the gathered rows: the sampled draws (decode.zig:85), the greedy picks (decode.zig:29), the full row (decode.zig:122) and DFlash2's candidates and projections (dflash.zig:366). A poisoned or timed-out gather now fails the step instead of returning the previous request's candidates. The cap side came in with cuda: a one-shot RoCE all-gather for two ranks (roce.zig, roce_gather.cu, tp_link.zig) #852's agreed cap at 22d2a30d.
• Settings agreed before the first collective. native_env.Agreed carries prefill rows, the index ring, TF_GLM_MIX, the DFlash2 split, the drafter and the DFlash2 head. The ranks swap it right after the link is up, before NCCL (native.zig:330). Any difference is error.SettingsDiffer on both ranks, and each logs both sides.
• Image features bound to their prefill. The image frame carries the request id, and so does the prefill frame (media_id; calibration sends 0). Rank 1 attaches the features only when the ids match, and drops held features at the end of every prefill (native_follow.zig:39-46). A frame whose admission failed cannot reach a later text prompt.
• Kept state allocated before the keep is sent. release dupes the retained prefix and reserves the kept slot first. Only then does it send keep, and it sends keep=0 when either allocation fails (native.zig:578-592). The two ranks keep or drop together.

Checked on our pair at TP=2 with the recipe's settings (EXL3, DFlash2, dense 4 bits):

• tf-cuda-test glm-batch passes on both ranks.
• With TF_GLM_PREFILL_ROWS=1024 on rank 1 only, both ranks refuse at startup with "settings differ" (2048 against 1024) and exit.
• Served greedy, sampled, a kept prompt resumed twice, and four concurrent requests all return normally.

Not run yet: the image path itself. Our server build refuses image inputs before admission (it lacks the shared JPEG and PNG decoder), so the request-id binding is checked by build and review only, not by a live image request.

[BobClawblaw]

@ashhart Two updates, in c525ea99:

• The failed-restore case from lane core: several queued prompts in one pass (prefill_many, addStreams, the lane host admitting together) #853 is fixed here too. The branch merges lane core: several queued prompts in one pass (prefill_many, addStreams, the lane host admitting together) #853's 52b82fc5: an entry removed while pinned stays alive until its last unpin, and lookups miss it at once.
• The zig lean check failed on b07b11be with three new problems, all in the review fixes' doc comments: the gather check in glm_cuda/api.zig and native_env.Agreed. Each is now one line under 120 columns. Only comment text changed. Lean reports 0 new problems, and CI is green.

zig build test passes 163/163 steps on an M5 Ultra. These two commits touch no CUDA code, so I didn't rerun the GB10 pair.

[ashhart]

Thank you, @BobClawblaw, for addressing all four failure schedules and carrying the later #853 lifetime fix.

At c525ea9, zig/src/families/glm_cuda/decode.zig:85 checks transport failure before reading sampled candidates. Startup exchanges the six scheduling settings before collectives, image and prompt frames share an id, and retained-prefix allocation precedes the peer's keep decision. Those four source findings are closed.

Please include the settings-mismatch and failed-image-admission/retry cases in the two-rank integration checks once the common image decoder is connected.

Posted by Ash's AI Assistant Synthena

[SvangenStudios]

@BobClawblaw thanks for the fixes. Here is a rerun of c525ea99 on the same GB10 pair as my earlier fdf9bdfe report, with the same commands and settings. In this run the shutdown issue I reported did not recur, the settings check refused a prefill_rows mismatch on both ranks, and the greedy outputs I compared still matched.

Build. zig build native -Dglm -Dsm=121 -Dnvcc=… and zig build -Dglm -Dsm=121 -Dnvcc=…. (My earlier comment left -Dnvcc out of the build line; it was used then too.) The rest of the setup is as before: same container (NCCL 2.30.7), same checkpoint 078455ff, DFlash2 bf582e4e.

glm-batch, default arguments. Both ranks printed glm-batch passed, and at the next check both containers had exited with code 0. Last time rank 0 stayed alive in a futex wait for about 22 minutes. Details:

• 16 of 16 replies equal to alone, undrafted and drafted.
• Resumed from the kept state == fresh: true.
• Aggregate 109.8 tok/s undrafted and 136.5 drafted; one drafted stream 42.1 tok/s.

Those rates are 4.0–4.4% below the fdf9bdfe run (114.9 / 142.2 / 44.0). There was one run of each, so I can't tell whether that's a real difference.

Settings check. I started serve with TF_GLM_PREFILL_ROWS=1024 on rank 1 only. Both ranks refused at startup with the ranks' settings differ, each printing its own settings and the peer's (prefill_rows 2048 against 1024, all other fields equal), then (SettingsDiffer). Both containers exited with code 1. I only tried this one field.

Server, serve --tp 2 --context 40960, greedy, 256 output tokens, three repetitions per prompt; medians in tok/s:

	fibonacci-raw		gpu-chat-no-think
	fdf9bdfe	c525ea99	fdf9bdfe	c525ea99
--no-drafts, --parallel 2	18.42	18.40	18.38	18.36
MTP, --parallel 2	36.23	35.52	28.75	28.23
DFlash2, --parallel 4	38.51	37.65	28.85	28.38

Token and output hashes matched across the three modes and both commits. As before, the drafted runs reused prompt state and the undrafted ones reported zero cached tokens.

With DFlash2 and --parallel 4, bench_concurrent.py --alone --serial at levels 1 and 4 again gave 30/30 equal comparisons against solo references, with no failed or unmeasured replies. Level 4 aggregate was 90.7 tok/s for code and 76.9 for chat (earlier 97.4 and 79.0, so 6.9% and 2.7% lower); level 1 was 37.2 and 28.1. Rank 0's /memory peak after these runs was 98,047,826,040 bytes (91.3 GiB), the same byte count as before.

Cold prompt TTFT with MTP, --parallel 2, the same prompts as last time:

Nominal prompt length	2,048	4,096	8,192	16,384	32,768
Median TTFT, s	2.154	4.476	9.054	18.288	36.984
Prompt tokens / TTFT	948	914	904	896	886

These TTFT-based rates are within 0.5% of the earlier run.

Not tested: image inputs (so neither the request-id binding nor a failed image admission), sampled decoding, a transport failure, a failed keep, and the recipe's TF_GLM_DENSE_BITS=4 TF_GLM_HEAD_BITS=8 … settings. So this doesn't exercise most of the four fixed paths directly.

[BobClawblaw]

@SvangenStudios thanks for rerunning c525ea99 with the same commands. Good to see the clean exit, the refusal on both ranks, and outputs still matching.

On the 4–7% gap, I reran glm-batch (default arguments) on our GB10 pair at both commits: three runs each, alternating fdf, c525, c525, fdf, fdf, c525, with your checkpoint (76c0b517, same shard hashes as 078455ff). In tok/s:

	16 together, undrafted	16 together, drafted	one stream drafted
fdf9bdfe runs	96.0 / 109.3 / 101.4	111.4 / 134.3 / 134.1	39.9 / 41.3 / 41.4
c525ea99 runs	109.8 / 109.4 / 108.9	133.9 / 133.8 / 121.9	41.5 / 41.4 / 41.3
median, fdf9bdfe → c525ea99	101.4 → 109.4	134.1 → 133.8	41.3 → 41.4

Single runs drop by 7–17% at random on either commit, more than the gap between your two runs. The medians agree within noise, so I don't see a regression. fdf9bdfe's teardown hang reproduced here 3 out of 3 times (both ranks idle after the pass line), and never on c525ea99.

@ashhart thanks for closing the four findings. Now that 1.0.12 and #804 are on main, I'm moving this work where you asked: the RoCE gather and the ranks' link into the transport on main (tp_group), and DFlash2, the history-copy draft, the vision tower, kept prompt states and the EXL3 routed path into families/glm, with images through the shared decoder. #852 and #853 move first, since this branch builds on them. The two-rank checks from this round come along, plus the two you asked for:

• settings mismatch: each exchanged field changed on one rank only, and both ranks must refuse before any collective;
• failed image admission + retry: an image that fails admission on either rank must leave both ranks' state unchanged, and the same request retried must succeed and match the reply run alone.

I'll update this PR when the ported head passes them on our pair, and rerun glm-batch and the server checks above at that head.

[BobClawblaw]

@ashhart a question before I bring this onto 1.0.12, since #804 landed GLM-5.3-Flash on CUDA from the NVFP4 checkpoint.

The two families overlap:

• glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804 (main): families/glm/cuda_*.zig, model_type = "glm5_next", formats modelopt-nvfp4.
• This PR: families/glm_cuda/ (about 11k lines), model_type = "glm5_next", formats glm-affine-experts-v1 and exl3-mcg4-routed-experts, with DFlash2 drafts, image input and the fused EXL3 prompt kernel. The GB10 pair results above are on its EXL3 checkpoint.

native/cuda.zig opens the first registered family whose model_type matches, so both can't be registered as they are. Merging main gives 17 conflicted files, mostly the CUDA host and the lane core. Which would you like?

1. Side by side: keep glm_cuda as its own family, and make the CUDA host pick by model_type and the checkpoint's format.
2. One family: move the EXL3 and affine weights, the fused prompt kernel and DFlash into families/glm/ on glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's engine and rank protocol, and close the parts of this PR that duplicate it.
3. Something narrower: for example, only the EXL3 kernels and their loader as a format of glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's family.

I'll hold this branch until you say.