Repository navigation
glm: GLM-5.3-Flash on CUDA, prompts 267.6 → 447.9 tok/s and decode 19.9 → 25.1 tok/s, every reply unchanged (#593) - #839
justincasey wants to merge 22 commits into
Conversation
…h, a width per expert Hand-written ports of the Python engine's EXL3 kernels: - the trellis decode into mma.m16n8k16 B fragments; - a row-invariant linear for 1 to 128 rows (mul1 half steps included), with prompts in 128-row blocks and qlinear's .exl3 arm; - grouped routed experts with each expert's own width; - the checkpoint's trellis groups from its safetensors headers, and their upload. tf-cuda-test exl3 checks them against float64 references, row and chunk invariance, and, with fixtures, the Python engine's bytes. tf-cuda-test exl3-tp2 checks the GLM split rules on two ranks over NCCL.
…run by tf-cuda-test glm-model Hand-written kernels ported from the Python engine's Triton and CUDA in their order of operations: - the bf16 matmul and glue; - KDA's decode chain and chunked prompts; - the DSA indexer and top-pool selection; - the latent cache, absorb/expand and latent attention; - hyper-connection mixing, the router and EXL3 experts, the dense MLP of layers 0-2, and the head's greedy pick across ranks. cuda_load reads each rank's shard; cuda_model runs prompt chunks of 2,048 rows and serial greedy steps. tf-cuda-test glm checks the kernels and whole layers against float64 and, with fixtures, the Python engine's bytes. tf-cuda-test glm-model checks greedy replies against recorded ones.
…(six times the blocks, the same sums)
…s a lane and K step
…red slot in place, no copies
…ll (each launch leaves them zero)
… (sparse kernels read the device position)
…ible key (that tile and later ones change nothing)
…ead trellis words as staged 16-byte no-allocate loads, two steps in flight
… same split, the same sums)
…ocate loads, two steps in flight
… fragment order too
…its own, so the same bits)
… once, not from global every K step
… in order to the first empty one, not a block a tile
…es, not every expert scanning every pick
…s before its FMA chains
…nce, with x staged in shared memory
|
Thank you, @justincasey. Prompts from 267.6 to 447.9 tok/s and decode from 19.87 to 25.10 tok/s on two GB10s, with all eight replies unchanged, one technique a commit and every commit building and passing on its own: this is how speed work should arrive. Those wins should land wherever GLM-5.3-Flash on CUDA ends up. As the reply on #838 says, we would like them in #804's family and the shared EXL3 module, in particular the decode graphs, the weights in fragment order, the latent attention changes and the 64-row EXL3 prompt linear. Each carries your name. Thank you for the care in the receipts too: both heads in one session, token ids against the Python engine. |
|
Thank you, @justincasey, for this speed work: prompts from 267.6 to 447.9 tok/s and decode from 19.9 to 25.1 tok/s with every reply unchanged, in separately checked commits measured in the same session. We are closing this branch because #804 is the one GLM CUDA family. The decode graphs, fragment order weights, latent attention changes and the 64 row EXL3 prompt linear are the pieces to bring into #804 and the shared module chosen in #832, under your name. Please send them as focused picks on that base, with the same unchanged token comparisons. Posted by Ash's AI Assistant Synthena |
2 participants
What this changes
Depends on #838: its two commits come first in this diff, and this PR's own are the last 20.
Speed work on PR 2's GLM-5.3-Flash layers, one technique a commit, with every reply unchanged.
"draft": falseDecode:
hc_partiala block per four mixing dots.Prompts:
cuMemcpy2DAsync, through a sharedDeviceBuffer.copyRows.Every commit builds and passes
test-cuda-hoston its own.Receipt
c126fe16; two DGX Sparks (GB10, sm_121), CUDA 13.0, RoCE; checkpointvcruz305/RED-SNOW-5.3-FLASH-EXL3-SAGE-4.91bpw.glm-model12/12; all eight replies equal the Python engine's greedy"draft": falseids.tf-cuda-test glm-modelon this head and on glm: GLM-5.3-Flash's layers on CUDA at TP=2 from EXL3, checked end to end by tf-cuda-test glm-model (#593) #838's head in one session.exl3: 28/28.glm: 25/25.glm-model: 12/12.test-cuda-host: pass at every commit.zig fmt: clean.c126fe16.bench_openai.py,prefill_cold.pyand vLLM (not served); Metal (no change).Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The runner is serial and greedy.)