Repository navigation
GLM-5.3-Flash and full GLM-5.3 on the native CUDA engine (tracker) #593
Description
This issue collects all the GLM work on the native CUDA engine in one place. Eight threads asked overlapping questions about the same port: #509, #552, #402, #310, #308, #214, #339 and #566. Each is closed with a link here, and its proposals and receipts stay readable there. Please post claims, plans and receipts here, so two people don't build the same piece.
Where things stand
- Native CUDA serves Nemotron 3.5 Lightning only. The native GLM family serves GLM-5.3-Flash on Metal.
- Several working GLM-5.3-Flash CUDA ports exist outside the tree: @MiaAI-Lab's EXL3 port at TP=2 on two GB10s (their recipe, greedy replies with drafts off equal to the Python engine token for token), @Bizuayeu's TP=2 port over NCCL (GLM-5.3-Flash on CUDA: let a long prefill wait between chunks while any rank runs hot #339, Zig CUDA ports: shared layers below
lanes.Backend, and one place to coordinate them #548), and our own MLX affine 4-bit port, in progress. - Shared CUDA layers are coordinated in Zig CUDA ports: shared layers below
lanes.Backend, and one place to coordinate them #548. cuda: one quantized-weight interface (qlinear.zig); Nemotron's 4-bit projections route through it #587 (one quantized-weight interface,qlinear.zig) is in review, and the NVFP4 entry follows it.
The gate
Every PR here passes the same gate as the rest of the engine:
- drafted equal to
"draft": false, resumed equal to fresh, concurrent equal to solo, on the device; - an FP64 reference for every projection class whose reduction order changes, with worst-column errors;
- prefill and decode measured against the current release and against vLLM on the same hardware;
- bf16 KV, and no arithmetic below today's precision;
- drafts and concurrency through the lane core, with no serial serving path;
- files under 600 lines, one-line comments, and no Python in the engine or its tools.
Bit equality with the Python engine is not required.
The pieces, in order
- Shared layers (Zig CUDA ports: shared layers below
lanes.Backend, and one place to coordinate them #548): the quantized-weight interface (cuda: one quantized-weight interface (qlinear.zig); Nemotron's 4-bit projections route through it #587), then NVFP4 behind it, including grouped NVFP4 experts on prompt rows. - Multi-rank communication:
zig/src/cuda/nccl.zighas AllReduce, AllGather and Broadcast. Send/recv and an exact reduce-scatter in rank order (GLM-5.3-Flash on CUDA over three ranks (--tp 3): how it should sit beside #159, the communicator interface and #243 #310's) come as one small PR. - The GLM-5.3-Flash family on one GPU, then TP=2 and TP=3 with GLM-5.3-Flash on CUDA over three ranks (
--tp 3): how it should sit beside #159, the communicator interface and #243 #310's unequal shares. Any of the ports above can be the first PR. Whichever passes the gate first lands, and the others build on it. Say here before you open one. - Checkpoint formats on that family: MLX affine 4-bit, EXL3 on the reference decoder from core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496, and NVIDIA's NVFP4 checkpoint read as stored, W4A16 (GLM-5.3-Flash on CUDA: read nvidia/GLM-5.3-Flash-NVFP4 (ModelOpt NVFP4) as stored, W4A16 #308).
- Full GLM-5.3 (
glm_moe_dsa) as its own family that reuses the Flash family's CUDA pieces rather than copying them: TP=3 (Full GLM-5.3 (glm_moe_dsa) on three DGX Sparks, as layers of the GLM-5.3-Flash CUDA engine: how should it sit beside #159? #402) and TP=4 (Zig CUDA family for full GLM-5.3 (glm_moe_dsa) at TP4: plan and questions #552). - Later, on top: the heat wait on GLM prefill under the engine's
TF_HEAT_HIGHandTF_HEAT_LOW(GLM-5.3-Flash on CUDA: let a long prefill wait between chunks while any rank runs hot #339), DFlash2 drafts (Native CUDA: GLM-5.3-Flash EXL3 with DFlash2 and TP/EP/DCP #509), attention on a desktop GPU with routed experts on the Sparks (GLM-5.3-Flash on CUDA: attention on one GPU, routed experts on the Sparks (proposal) #214), and Living Weights for GLM-5.3-Flash on CUDA (Living Weights for GLM-5.3-Flash on NVIDIA GB10 (DGX Spark) CUDA — port request + test hardware offer #566, after Living Weights learns on CUDA (GB10): the training step, its kernels checked against float64 #591).
Answers to #552's questions
- Yes, a
glm_moe_dsafamily is wanted, sharing the Flash family's pieces as in step 5. Who is doing what is listed above. - NCCL, which the CUDA engine already uses. The cluster layer's links are built for the Macs' Thunderbolt RDMA. A
TF_NCCL_LIBoverride for hosts without a system NCCL is fine. - The gate above. Bit equality with the Python engine is not required.
- Hand-written
.cu. 1.0.3 moved Nemotron to our own CUDA kernels, and new families follow. - Yes, bf16 KV. An FP8 KV cache, such as Native CUDA: GLM-5.3-Flash EXL3 with DFlash2 and TP/EP/DCP #509's
fp8_ds_mlalayout, would trade precision, so it isn't on the default path. - Yes, a tiny synthetic checkpoint for host tests is welcome, made by a Zig tool in the tree.
Activity
glm_moe_dsa) on three DGX Sparks, as layers of the GLM-5.3-Flash CUDA engine: how should it sit beside #159? #402 as a duplicate of this issue--tp 3): how it should sit beside #159, the communicator interface and #243 #310 as a duplicate of this issuedocs: the CUDA recipe links the GLM-on-CUDA tracker (ashhart#593)
Bizuayeu commentedon Oct 10, 2026
Claiming step 2: NCCL send/recv and an exact reduce-scatter in rank order, as one small PR.
It doesn't depend on the weight format. It sums the fp32 partial rows that each layer's attention and MLP (or experts) leave on every rank, so it serves any of the GLM-5.3-Flash ports above (MLX affine, EXL3, NVFP4) and the full GLM-5.3 family alike.
zig/src/cuda/nccl.zig:ncclSend/ncclRecvbeside the AllReduce, AllGather and Broadcast already there.--tp 3): how it should sit beside #159, the communicator interface and #243 #310's whole units: the firstunits % worldranks take one more).Where we use it: our GLM-5.3-Flash port reads
nvidia/GLM-5.3-Flash-NVFP4as stored (the NVFP4 family claimed in #548). At TP=2, each prompt chunk's partial rows go through this exchange: each rank glues only its own rows, and the bits are equal across ranks. I'll say here before opening it.Bizuayeu commentedon Oct 10, 2026
Step 2 is ready: NCCL send/recv and a reduce-scatter in rank order, one commit on main, checked byte-equal on one GB10, two (TP=2) and three (TP=3, unequal slices). Opening the PR now.
GustavBlack commentedon Oct 11, 2026
Hello. Following the direction from #545/#546 (own kernels, MLX affine checkpoint, own reference),
I have been building the GLM-5.3-Flash CUDA family in a fork and want to line up with this tracker
before opening anything, as the issue asks.
What exists now (fork branch, rebased onto
v1.0.5; the kernel byte-gates have run green onour two GB10s — see below):
matmul's tiled layout), per-sequence state, the forward (embed → per layer [hc_pre → kda | sparse
MLA → hc_post] → MoE → head), and the engine (init/bind/commit/prefill/graphs), following the
Nemotron CUDA files' structure.
glm_glue.cu/glm_hc.cu(7+2 kernels, fp32 fixed-order reductions derived from a recorded kernel set, device-side
Sinkhorn at the captured order),
glm_route.cu(sigmoid + bias top-k, raw-score weights), andglm_kda.cu(the Python engine'skda.cuvendored verbatim per theqmm_grouppattern).tf-cuda-test glm-glue | glm-kda | glm-mlacells are staged and byte-compare each kernelagainst its host reference (the
glm-mlacell is thesparse.tokensstep, which is host mathbetween async copies).
nativeopen/admission, multi-family dispatch.glm-kda66/66,glm-mla4/4, andglm-glue121/121 — the first run failed 36/121 and found tworeal bugs (
warp_combinegave the cross-warp total only to lanes < wpc, sormsnorm/hc_finishran
inv_rms(0) ≈ 316×;swiglureadGUat stridewwhere the fused rows are2w), bothfixed byte-exactly against the host references. On the way the engine's CUDA admission proved
itself: the 169.2 GiB MLX checkpoint is refused cleanly against a 103.9 GiB single-card budget,
so TP2 is mandatory for this model and the two-rank path is the next piece here.
on the two-rank path — no
--tp/--rank/--masteron the CUDA native side yet, the forward'sall-gather is still a stub, nothing sets
Engine.setComm(the split rules, per-rank slicing andcuda/nccl.zigare in-tree); sampled mode is refused by design (greedy first, as the gateexpects); MTP drafting and sparse-bucket graphs are later rungs; EXL3 is deliberately not loaded
(MLX affine is the direction, and core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496's trellis decoder is the later path we would build on).
What I can contribute that the tracker's gate currently lacks: a two-machine GB10 pair on a
direct CX7 link — the gate asks for prefill/decode and the exactness suite on the device, and as
far as I can tell no one has two-machine CUDA GLM receipts yet. Already run and postable from the
first window:
tf-cuda-test grouped-planPASS (all cases byte-equal toexperts.route, #558)and
fp8-lanePASS (all row counts plus theqlineardecode/prompt check, #482/#587) on ourbox;
carveout(#487) refuses withDumbBufferRefusedunder privilege — the GB10's displaycontroller won't hand out a dumb buffer, so that experiment is not usable here as-is. The Python
engine's TP2 greedy reference on the MLX checkpoint (≈19.2 tok/s, ≈118.5 s startup) is captured and
will be the token target the moment our two-rank path lands. I can also run these gates on someone
else's port on request.
Two questions before I open anything:
qlinear.zig(MLX affine 4-bit behind oneLinear.decode/promptinterface) andthat "our own MLX affine 4-bit port" is in progress here — should my MLX-affine projections move
behind
qlinear.zigbefore a PR, or is a parallel packing implementation (same tiled layout)acceptable to review and align afterward? I would rather land receipts first and refactor against
your interface than race two implementations of the same layer.
native/cuda.zig's prompt-cache wiringassumes the family provides
loaded.targetand uses Nemotron's snapshot vtable. I guarded itper family at compile time (a family without kept prompt states gets no store, and its budget
gates nothing on a cache). Is a family-owned
snapsdeclaration the shape you would wantinstead? Happy to send that as its own small PR either way.
No PR yet, per the issue's "say here first". Thanks — the tracker's ordering (shared layers →
collectives → family → formats) matches what the port ended up needing, and the published gate is
exactly what our staged tests were written against.
— Gustav (@GustavBlack)
(The port is AI-assisted work, recorded honestly in the project notes; authorship is mine per the repo conventions.)
Bizuayeu commentedon Oct 11, 2026
@GustavBlack welcome, and thanks for laying this out. A few facts from our side that might be useful, since we're working on the same model:
nvidia/GLM-5.3-Flash-NVFP4(claimed in Zig CUDA ports: shared layers belowlanes.Backend, and one place to coordinate them #548). It's a different checkpoint format from your MLX affine one.a6aa4782, with the NVFP4 linears and experts behindqlinearandexperts.Grouped.native/cuda.zig. Our branch builds the store from the opened family's ownsnap_*declarations, and a family that declares none gets no prompt cache. That looks like the same shape as your guard.GustavBlack commentedon Oct 11, 2026
Thanks — yes, this contributes in three places, and thank you for the #642 pointer.
snap_*family shape is better than what I had. My port carries a compile-time guard(
if (comptime F == nemotron.native)) so GLM gets no prompt cache and its budget gates nothingon one; the family-declaring-its-own version is what I hoped for when I raised it, and I will
take your shape into the port.
forward.gather: an all-gather of each rank's fp32 partial, then a rank-order sum (rank 0first). So cuda: a reduce-scatter in rank order over NCCL send and receive #642 is not a dependency — the collectives are in and host-gated — but a
rank-order reduce-scatter with the same bits and only each rank's own rows is exactly the
upgrade that step wants. I would like to take it when it lands, and if a second-machine
receipt is useful I will run it on our pair (TP=2) and post the byte-equality beside yours.
qlinear/experts.Groupedare the homes I was aiming at. The MLX-affine projectionspack into the shared lane-matmul layout already; the remaining slice is the dense and expert
wiring. If your NVFP4 extensions are landing behind those same two homes, I would rather rebase
onto your branch's shape than invent a parallel one — is that branch going to become a PR soon,
or should I keep targeting v1.0.5's shape and reconcile after?
The one answer that would change my immediate plans: is the MLX-affine CUDA path still wanted
now that NVFP4 is the line you are porting? The v1.0.5 GLM recipe documents
TensorFold/GLM-5.3-Flash-MLX-4bit-MTPwith--tp 2for the native binary, which is what I havebeen building toward — weights loader done, the affine-4 kernels and the two-rank path in, kernel
byte-gates green on our pair (
glm-glue121/121,glm-kda66/66,glm-mla4/4), and a Pythonengine reference on the same checkpoint captured (TP=2 greedy, ≈19.2 tok/s). If MLX stays a
documented CUDA checkpoint, I will finish that slice and boot it against the reference. If NVFP4
is the CUDA direction and MLX was ever only the Metal format, say so and I will redirect — the
format-independent parts (the engine, the TP2 comm, the kernel slices) carry either way, and our
pair's receipts are yours to use.
From our side, on the same two GB10s, already posted above:
tf-cuda-test grouped-planandfp8-laneare byte-equal passes;carveoutrefuses withDumbBufferRefusedunder privilege (theGB10's display controller will not hand out a dumb buffer, so that experiment is not usable on
this box as-is).
— Gustav (@GustavBlack)
(AI-assisted work, recorded honestly in the project notes; authorship is mine per the repo
conventions.)
Bizuayeu commentedon Oct 11, 2026
@GustavBlack thanks. Yes, please run #642 on your pair when it's useful; a byte-equality receipt from a second pair next to ours would help.
On our branch: we're pushing to get it to a PR, but a few items of the gate above are still open on our side (the FP64 references, the release and vLLM comparisons, and tests that don't lean on fixtures from our Python engine), so we can't give a date yet. It runs at TP=2 on two GB10s with greedy decode equal to our Python engine. We'll say here before opening it.
If you build on its shape before then, a few things to know:
a6aa4782and will be rebased as main moves.qlinearand the prompt GEMM, askipon the staged prompt kernel, andpackGateUppacking gate and up in place.RankScatteronce that lands.On MLX affine: #545 (comment)
25 remaining items
glm: EXL3 checkpoints on the CUDA family (attention, shared expert, d…
BHCC2025 commentedon Oct 13, 2026
@Bizuayeu, two updates.
Family glue on
53309a72.glm-exl3-wipon our fork now sits on53309a72(9a099a7, force-pushed as announced). It carries #740, #741 and #806, then the family glue, including @jschmied'smtp.safetensorsfix. Our vocabulary commit is gone because your branch has it. On two and four GB10s (sm_121, CUDA 13, turboderpGLM-5.3-Flash-exl34.05 bpw, prompts of 2.3k–3.1k tokens), all of these PASS:glm-shared-check, at TP=2 and TP=4: 108 rows of 5 streams over 12 rounds equal their lone twins' logits, tokens and drafts byte for byte.glm-lanes-concurrent, at TP=2 and TP=4: a sampled stream equals its solo run over 128 ids, with prompts a chunk at a time, and 338 and 342 forwards respectively held several streams' windows.Four tiles in flight (0185c10). We timed it as you suggested, and it goes the same way as your NVFP4 result, so it stays out. A is #806's head (ff00ed3); B is the same plus 0185c10. Each figure is one GB10 decode
qlinearcall (rotation included, 16-row windows): the median of 7 runs of 20, then the median of 3 interleaved A/B rounds. The table shows B's time against A's, so + means slower.The long matrices gain 1–4%. The routed-expert matrices, which take most of a decode round, lose 3–7% at 3 and 40 rows.
Bizuayeu commentedon Oct 14, 2026
@BHCC2025, thanks for both. Your TP=2 and TP=4 runs of
glm-shared-checkandglm-lanes-concurrenton53309a72arethe family's first at TP=4 (ours are TP=2 and TP=3); I'll point to them from #804's description.
On the four tiles: your table splits the way our NVFP4 run did, and by shape as much as by rows. The long matrices
(
q_b_proj,o_proj,layers.0.mlp.down_proj) gain at every row count, while the routed experts lose from 3 rowson. If you come back to it, choosing the depth per matrix shape rather than per row count might keep the 1-4% without
the experts' loss.
BHCC2025 commentedon Oct 14, 2026
@Bizuayeu, thanks, and we tried the per-shape depth. A second decode kernel keeps four tiles in flight; the host picks it when a column's part runs at least 16 128-blocks, which covers
o_proj, the dense down and the head. Everything else keeps the one-ahead loop, unchanged. Two things came out of it:o_proj,layers.0.mlp.down_projandlm_headmoved between −1.8% and +2.7%.q_b_proj, which used the unchanged kernel in this build, moved as much as 6%. The 1–4% in the earlier table doesn't hold up against that.So the branch stays as it is, and we'll come back to it if a profile of a served round shows those matrices waiting on loads.
jschmied commentedon Oct 14, 2026
@BHCC2025, a TP=1 decode profile in case it helps decide where to look next. One GB10, turboderp
GLM-5.3-Flash-exl32.05 bpw. The prompt is 2,998 tokens, followed by greedy decode. The per-token figures are (128-token run − 8-token run) / 120, from nsys. exllamav3 v1.6.1 ran on the same checkpoint and the same prompt ids, profiled with--cuda-graph-trace=node.The build is
5da1e007: your family glue plus0185c103(four tiles), on an older base, throughglm-decode-check. That test is gone from9a099a7c. As far as I can tell, the only GLM decode kernel that differs from9a099a7cis that four-tile loop inexl3.cu.tf_exl3_tc_2+ 259rotate_f)exl3_gemv_int8_sq)With MTP drafts on the same build, decode is 26.8 / 29.3 / 30.6 tok/s on three prompts. The ids equal the plain run's on all three. Prefill is 5.5–6.6 s against exllamav3's 6.5–7.0 s for the same prompts.
Bizuayeu commentedon Oct 14, 2026
@jschmied, on the graph point: the NVFP4 path at TP=2 measured the other way. There the GPU idled 0.84 ms of a decode step (1.4%), and capturing the verify forward per slot left plain decode level and drafted decode 0.9–1.5% slower (#804's tried table). Your 4.7 ms a token at TP=1 is about 10% of the step, so graphs may pay off there where they didn't for us; #635's graph cache is where we would take them.
glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP…
glm: the vocabulary's TP cut in the checkpoint format's unit (128 for…
BHCC2025 commentedon Oct 14, 2026
@jschmied, thank you, that's the comparison we didn't have. Two notes on the build, then where it points.
5da1e007carries0185c103, the four-tile decode loop, which we dropped after timing it (above): the routed-expert shapes lost 3–7% at 3 and 40 rows, and the per-shape version gained nothing outside the noise.9a099a7cdecodes with the one-ahead loop. Your experts go throughexperts.Grouped, not that loop, so the 13.9 ms row shouldn't move; the 20.4 ms projection row may, slightly.glm-decode-checkwent with the move onto53309a72, glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's branch; there the decode receipts areglm-shared-checkandglm-lanes-concurrent.Your table puts the projections at 3.6 ms over exllamav3 with twice the calls, and each of our 259 is a rotation launch plus a matmul launch. Two things to try, in order:
rotate_flaunch. That takes 259 launches a token off, and the rotation's write and read of the rotated rows.q_a/kv_a, gate/up) in one launch. We haven't read exllamav3's kernel either, but about half the calls suggests it pairs them this way. Each matrix keeps its ownsuh, so they share the input, not the rotation.Graphs (#635) on top of that, as @Bizuayeu says, since at TP=1 the gaps are about 10% of the step. We'll time both against your numbers on the same 2.05 bpw checkpoint and prompt, and post them here.
BHCC2025 commentedon Oct 14, 2026
@jschmied, an update on the first of the two items above. The input rotation now runs inside the decode kernel (one launch per projection instead of a
rotate_flaunch plus the matmul), on a local branch over9a099a7c, not pushed yet:tf-cuda-test exl3,glm-shared-checkandglm-lanes-concurrent(TP=2) pass, and the served greedy hashes match the baseline.rotate_fgone), 39.39 → 37.88 ms a token.The second item, projections that read the same input in one launch, we measured before building: in this family that's
q_a+kv_ain the DSA layers and gate/up in the dense layers, 14-20 launches of ~1,540 a token, so about 0.2%, under the noise. We're leaving it.Your TP=1 comparison on the 2.05 bpw pack is on hold: our Sparks are on another model this week. If you have a moment to run your table on the branch once it's up, that would be the cleaner comparison anyway.
ashhart commentedon Oct 14, 2026
Hello @BHCC2025,
Thank you for dropping the four-tile loop after you timed it, and for the two build notes.
CUDA on this main registers Nemotron only (zig/src/native/cuda.zig:13). The GLM recipe says the native binary has no GLM CUDA backend yet (docs/recipes/glm-5.3-flash.md:9). Qualified GLM-5.3-Flash is Metal on two M5 Ultras (README.md:72).
#804 is the NVFP4 GLM pull, reviewed. It waits on the two-GPU gate: rank 0 admits by the smallest room, checked after both loads. The measurements that would move the tracker are the ones you named. Do the input rotation inside the decode kernel, then pair q_a with kv_a and gate with up, timed on the same 2.05 bpw checkpoint. #635 is a DeepSeek graph cache and waits on the maintainer. It is separate from this GLM pull.
Thank you.
BHCC2025 commentedon Oct 14, 2026
@ashhart, thank you. Understood: the measurements that move the tracker are the two we named, timed on @jschmied's 2.05 bpw checkpoint.
The first is done on our side: the input rotation runs inside the decode kernel, byte for byte equal to the two-launch path, with TP=2 numbers in our comment above. We'll push that branch and add the second (
q_awithkv_a, gate with up in one launch, each keeping its ownsuh), then time both against jschmied's TP=1 table on the same 2.05 bpw pack and prompt, on one GB10. Our Sparks are on another model's runs today, so the timing follows when one frees up; we'll post it here.ashhart commentedon Oct 16, 2026
Thank you, @BHCC2025, for keeping the input rotation byte-equal and for retaining each projection's own suh in the paired-launch work.
#804 is in the 1.0.12 candidate for NVFP4 GLM. Its checks include a four-layer checkpoint on two GB10s; the full checkpoint still needs its served gate. EXL3 continues through the shared module choice in #832 and the same GLM family.
Please post the promised one-GB10 comparison on the pinned 2.05 bpw pack and prompt, with the rotation and paired-projection changes measured separately against the same baseline. That will give the next speed change a comparable receipt.
Posted by Ash's AI Assistant Synthena
glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP…
glm: the vocabulary's TP cut in the checkpoint format's unit (128 for…