Repository navigation
Conversation
…unchanged Two fills a layer that a one-rank pass over ModelOpt weights never needed: - the MoE output y [rows, 11, dims] fp32 was cleared before the experts, though the down kernel writes every routed slot and sharedScatter the shared one; a rank that skips the shared slot (tensor parallel) still clears it; - the native out projections (DeltaNet and attention) reduce their own K slices, then cleared the out_sk - 1 slices the writeback sums after the first. The writeback now adds one zero slice kept at part's cap-row stride instead: x + 0 repeated is x + 0 once (-0 to +0 the same), so its bits are the same. The zero slice is filled at allocation; a legacy lane that writes slices into part, or a native one past cap rows, marks it spent and the old path runs. Kept branches (training) and collectives keep the old path. At 1,536 rows that is 173 MB plus 110 MB of fills and 94 MB of reads a layer: the 8k prompt takes 5.77-6.74 s (main 6.09-6.96), 30,922 tokens 25.04 s (25.74), every reply byte-identical.
The lean gate's new findings on this branch: the three multi-line comments folded to one line within 120 columns, which also brings cuda_batch.zig to 599 lines. Comments only; lean_new.py: 0 new.
|
Thanks @jschmied for finding these fills with per-op timing, and for the clear write-up of why adding one zero slice keeps every bit, - Synthema, Ash's AI Assistant. |
… with an invariance test, cuda_batch.zig under 600 lines - cuda_zero_slice.zig holds the zero slice's rules unchanged: which lane writes spend it, the lane call that skips the clears, when a layer may use it, the writeback's slices, and which ranks clear the MoE output. - The K-slice sum, window shift and column copy launches move to cuda_batch_ops.zig, the Pass keeping its methods, so the ported cuda_batch.zig goes from 612 lines to 586. - cuda_zero_slice_test.zig runs Pass.lane over an emulated, poisoned part buffer reused across passes and checks the zero slice's writeback sums equal the old clears' bit for bit (-0, NaN and infinities among the values) at part and full capacity, with an MTP head pass or a legacy lane between native passes, a kept branch and two ranks, and that the fills are skipped; a planted control undoes the spending calls' invalidation and must see the sums differ.
|
Landed on main in 93a215b..35e46c0, with your records under your authorship plus the maintainer fix-ups. Thank you. Measured with the whole group (#1139, #1140, #1143, #1147, #1164, #1169, #1171 and #1170's Flash Next slice) against main on GB10s:
- Synthema, Ash's AI Assistant. |
What this changes
Two fills a layer that a one-rank pass over ModelOpt weights (
nvidia/Qwen3.8-Flash-Next-FP4-Local, local-inference-lab'sNVFP4 exports) never needed, found with per-op-class timing of the batched pass:
y([rows, 11, dims]fp32) was cleared before the experts. The down kernel writes every routedslot and
sharedScatterthe shared one, so at one rank nothing reads a cleared value. A rank that skips the sharedslot (tensor parallel) still clears it.
out_proj, attentiono_proj) reduce their own K slices into the first fp32slice and then cleared the
out_sk - 1 = 7slices the slice writeback sums after it. The writeback now adds onezero slice kept at
part's cap-row stride:x + 0repeated isx + 0once (-0becomes+0the same way, NaNstays NaN), so every output bit is the same. The zero slice is filled when the scratch is allocated; a legacy lane
that writes slices into
part, or a native one pastcaprows, marks it spent and the old path runs. Kept branches(training's
mix_branch) and collectives keep the old path.At 1,536 rows that is 173 MB plus 110 MB of fills and 94 MB of reads a layer, 18 GB a chunk.
Receipt
Environment: TensorFold
maind1018b31plus this branch; one DGX Spark (GB10, sm_121, CUDA 13.0, nvcc 13.0.88);nvidia/Qwen3.8-Flash-Next-FP4-Local@b9ce18a6(33/33 files against the publisher's hashes).Exactness: served, greedy, against
mainon the same checkpoint and flags: every reply byte-identical (threechats, three ~8k agent contexts, one 30,922-token context; 64-96 tokens each).
Prompt speed (cold,
--no-drafts --parallel 4 --context 40960,--prompt-cache-gib 0, time to first token):Decode speed: not measured (the fills a one-row pass skips are 112 KB and 70 KB a layer).
Kernel contract: no kernel changes.
Tests run:
zig build test --summary all(built-Dnvcc): 211/211 steps, 2,959 of 2,973 tests passed, 14 skipped.zig fmt --checkclean.Not run, and why: two ranks (the change keeps their path); an MLX affine Flash Next checkpoint (none on this box;
their out projections are legacy lanes, which take the old path and mark the zero slice spent).
Gotchas
cuda_batch.zig's ModelOpt expert path. The fix is one hunk: keep Flash Next CUDA: NVFP4 experts on prompt tiles with the decode form's sums, bit for bit #1143's routing andrunFact, and wrap itscuMemsetD8Asyncin this PR'sif (native.world != 1 or native.rank != 0). Whichever lands second takes that join.Checklist
"draft": false(8/8), concurrent equals solo (8/8), and a resumed two-turn prompt equals a fresh one (7,147 of 7,184 tokens restored), each equal tomain's output (MTP drafts on,--prompt-cache-gib 4; measured at97b782d3, before the later lean fix, which only folds comments and moves code).main).