Repository navigation
Conversation
…ecodeF32 and promptF32) A tensor-parallel family sums its ranks' partial rows before rounding, so it needs a projection's fp32 sums as the kernel holds them. qmmf.cu and prompt16.cu already take F32 as a template argument; this instantiates FP4's fp32 tiles (and reduce_kernel<true> for the split path), loads them beside the bf16 ones, and adds Lane.matmulOut, Prompt.matmulOut and qlinear's decodeF32 and promptF32. Other formats refuse fp32 out with EpilogueNotBuilt, as the experts do. matmul, decode and prompt keep their signatures and bf16 bits.
kernels/cuda/exl3.cu decodes EXL3 trellis tiles as checkpoints store them ([K/16, N/16, 16 bits] int16), with core/exl3_format.zig's codebooks (3inst, mcg, mul1) and whole widths 1-8. Rows are rotated by suh and the 128-point Hadamard transform, scaled by 2^-4 and rounded once to fp16; each tile decodes once straight into the B fragments of two mma.m16n8k16 (the tile's value order is theirs), fp32 sums, then the output transform, svh and 2^-3 (docs/recipes/exl3.md, Prompts). Decode rows go 1-16 a launch, read as A fragments from rows written in fragment order; matrices with fewer than 96 column blocks split their input blocks into a shape-fixed number of parts that the last part sums in order. Prompt rows go 64 a block from shared memory. Every row is independent, so a row's bits are the same at every row count on its path. qlinear takes it as Weight.exl3, refused with FormatNotLoaded when the module is absent. tf-cuda-test exl3 [DIR]: every codebook state and random tiles at every width bit-equal to exl3_format; FP64 references from exl3_format.dequantize with a bound for the kernels' rounding order, worst column and aggregate error; decode windows of 1-16 rows and prompts of 2-300 rows byte-equal to each row alone; DIR adds real matrices of a checkpoint and times them.
…ght, and fails when none is EXL3
A tensor-parallel family sums its ranks' partial rows before rounding, so an EXL3 projection must hand out its fp32 sums as ashhart#757 does for NVFP4. The decode kernel already takes an fp32 output flag; the prompt kernel gains the same flag (tf_exl3_store), and qlinear's decodeF32 and promptF32 route .exl3 to them. decode and prompt keep their bf16 bits: the bf16 out is the fp32 out rounded. The new exl3 test checks both entries within FP64 bounds, the same bits at every row count, and the bf16 out as their rounding; it fails with EpilogueNotBuilt before this change.
|
GPU checks on this head (ff00ed3, on #757's 9a14da6): GB10 sm_121, driver 580.178.04, CUDA 13,
|
|
Hello @BHCC2025, Thank you for EXL3 projections behind Your body reports Metal was not run, and the server benches were not run. No served family calls The body says the branch is stacked on #740, and that the first two commits belong to #757 and #740. The parent of your commits is #757 ( The head includes #757. The change sits on v1.0.9, and the trial merge onto that base is clean. This pull request is not in 1.0.10. We will wrap the lines past 120 columns at landing, including Thank you again. |
|
@ashhart, thank you. The GPU re-run on On the stacking, the body was loose and I've corrected it. The parent is #757's head Agreed on the gate. The first family to call |
|
@BHCC2025, thank you for the GPU re-run on @justincasey's #837 builds the same EXL3 projection and grouped experts from the Python engine's kernels, so there are now two implementations of one piece. We want one module behind If you and Justin want to compare notes first, #832 is the place. |
|
@ashhart, thank you, that plan works for us: one module behind One note for the family side: the GLM EXL3 glue we've carried is already on #804's branch ( @justincasey, glad to compare notes in #832. |
3 participants
Based on #757's head
9a14da6a(qlineardecodeF32/promptF32). The first two commits are #740's two (EXL3 projections behindqlinear) rebased onto it:a35cc231is the same patch as #740's76ac2867, and241ac1bbdiffers from7b6dc024only in the import lists that met #757's. The last commit,ff00ed30, is this PR. For #593: the GLM-5.3-Flash family sums its ranks' partial rows before rounding, so an EXL3 projection has to hand out its fp32 sums as #757 does for NVFP4.Change. The EXL3 decode kernel already took an fp32 output flag; the prompt kernel gains the same flag (
tf_exl3_store), andqlinear.decodeF32/promptF32route.exl3to them instead ofEpilogueNotBuilt.decodeandpromptkeep their bf16 bits: the bf16 out is the fp32 out rounded. No other format changes.Test.
tf-cuda-test exl3gains a check of both entries throughqlinearon a 2048x256 4-bitmcgmatrix: every fp32 value within its FP64 bound, the same bits at every row count (decode 1-16 rows, prompt 1-300), and the bf16 out equal to the fp32 out rounded. With only theqlinearrouting reverted it fails withEpilogueNotBuilt.Receipt. GB10 (sm_121), CUDA 13, Zig 0.17.0,
-Dsm=121; checkpoint turboderpGLM-5.3-Flash-exl34.05 bpw.tf-cuda-test exl3 <ckpt>: all PASS. New check: 80,896 FP64 values, max 0.0031 of the bound. The 7 real matrices unchanged (max 0.0308 of the bound).tf-cuda-test quant-ref: 4/4 PASS (cuda: the NVFP4 lane matmul and prompt GEMM write fp32 too (qlinear decodeF32 and promptF32) #757's NVFP4 fp32 checks).zig build test test-cuda-host,zig fmt --check zig/: clean.Those GPU runs were on #757's previous head (2f3988f). After rebasing onto its current head (9a14da6, on 1.0.9) the build, host tests and
zig fmt --checkpass, and the GPU checks were re-run on this head (ff00ed30) with the same results: #806 (comment)Not run: Metal (no Mac change), server benches (no served family uses EXL3 fp32 out yet; the family PR will).