Repository navigation
glm: GLM-5.3-Flash's layers on CUDA at TP=2 from EXL3, checked end to end by tf-cuda-test glm-model (#593) - #838
justincasey wants to merge 2 commits into
Conversation
…h, a width per expert Hand-written ports of the Python engine's EXL3 kernels: - the trellis decode into mma.m16n8k16 B fragments; - a row-invariant linear for 1 to 128 rows (mul1 half steps included), with prompts in 128-row blocks and qlinear's .exl3 arm; - grouped routed experts with each expert's own width; - the checkpoint's trellis groups from its safetensors headers, and their upload. tf-cuda-test exl3 checks them against float64 references, row and chunk invariance, and, with fixtures, the Python engine's bytes. tf-cuda-test exl3-tp2 checks the GLM split rules on two ranks over NCCL.
…run by tf-cuda-test glm-model Hand-written kernels ported from the Python engine's Triton and CUDA in their order of operations: - the bf16 matmul and glue; - KDA's decode chain and chunked prompts; - the DSA indexer and top-pool selection; - the latent cache, absorb/expand and latent attention; - hyper-connection mixing, the router and EXL3 experts, the dense MLP of layers 0-2, and the head's greedy pick across ranks. cuda_load reads each rank's shard; cuda_model runs prompt chunks of 2,048 rows and serial greedy steps. tf-cuda-test glm checks the kernels and whole layers against float64 and, with fixtures, the Python engine's bytes. tf-cuda-test glm-model checks greedy replies against recorded ones.
|
Thank you, @justincasey. GLM-5.3-Flash's layers end to end on two GB10s from the unsplit EXL3 checkpoint, every kernel against float64 and the Python engine's bytes, and greedy replies equal to the Python engine's on eight prompts up to 16,416 tokens: a big piece of work, done with real care. To keep one GLM family on CUDA, we plan to build on @Bizuayeu's #804, which already has any number of ranks, the prompt cache and the server. The best home for this work is EXL3 weights through #804's family: your EXL3 experts, the glue in the Python engine's order of operations, and the harness that checks replies against it. One difference to note: #838 keeps the committed latent cache in fp8 with per-row scales, while #804 keeps it in bf16. Our quality rule makes a lower-precision cache an opt-in flag, so it would come along as an option, off by default. Would you be up for carrying the EXL3 path into #804's family? We are coordinating in #832 and #593. Your authorship stays on everything that carries your work, and the landing credits the authors of the Python kernels it follows, as docs/credits.md does. |
|
@ashhart Do you want me to carry the EXL3 into 804? Or is someone else already working on that? Whats the best way to coordinate? |
|
Thank you, @justincasey, for the GLM layer kernels and the TP=2 reference checks against complete replies. We chose #804 as the one GLM CUDA family, so we are closing this separate family. Your layers and reference fixtures are the work to carry into #804, and the shared EXL3 module is being chosen in #832 with every author's credit kept. Please bring the reusable pieces onto #804, including the model's image path in our own kernels. Your end to end comparisons will make good gates there. Posted by Ash's AI Assistant Synthena |
2 participants
What this changes
Depends on #837: its commit comes first in this diff, and this PR's own commit is the last.
GLM-5.3-Flash's layers on the native CUDA engine at TP=2, from the unsplit EXL3 checkpoint
vcruz305/RED-SNOW-5.3-FLASH-EXL3-SAGE-4.91bpw, run end to end bytf-cuda-test glm-model(#593).cuda/glm_layer.zigis a layer's forward.families/glm/cuda_load.zigreads each rank's shard;cuda_model.zigruns 2,048-row prompt chunks and serial greedy steps.It adds no serving path: the server does not load it. The runner is a test harness for the layers and a source of kernels for a lane-core family. One commit, with no conflicts on
main(c126fe16).Receipt
c126fe16; two DGX Sparks (GB10, sm_121), CUDA 13.0, RoCE; checkpointvcruz305/RED-SNOW-5.3-FLASH-EXL3-SAGE-4.91bpw.tf-cuda-test glm DIR: every kernel against float64 references and, withDIR, the Python engine's bytes; whole KDA and DSA layers at TP=2.tf-cuda-test glm-model: greedy replies to eight prompts (221 to 16,416 tokens) equal the Python engine's"draft": falsereplies, token for token.glm-model).glm-model).glm: 25 pass, 0 fail.glm-model: 12/12.exl3: 28/28.test-cuda-host: pass.zig fmt: clean.c126fe16.bench_openai.py,prefill_cold.py, vLLM: the server doesn't load this runner.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (The runner is serial and greedy.)