Repository navigation
Conversation
jschmied
force-pushed
the
cuda-attn-shared
branch
from
October 13, 2026 16:25
c5936c7 to
4d596fe
Compare
This was referenced Oct 13, 2026
…e_attention.cu, block_attention.cu), checked against core/attention_ref.zig
… merge combines them in order; a slice takes its own one-warp block where four-warp blocks would leave SMs idle (TreeAttention.sms), the same bits
…lves again, only one-warp blocks leave them to the merge (a small fixed slice area), so the scratch stays at one partial a group
…async stages), the same arithmetic in the same order
…w), partials sized to the query heads, one-warp blocks with 3 tile stages where they fit one wave of the SMs, else 2
…ts) before folding them, the same arithmetic in the same order
jschmied
force-pushed
the
cuda-attn-shared
branch
from
October 14, 2026 09:18
c817f13 to
654b48b
Compare
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
CUDA has no shared attention for drafted rows. A family that drafts a tree on CUDA writes its own attention over the
committed keys plus each row's path, and a DFlash-style drafter writes its own block attention. This adds both as
shared modules, operator-only (no model calls them yet), next to
cuda/tree_round(#734), which does the verify andcommit step of the same round:
zig/kernels/cuda/tree_attention.cu,zig/src/cuda/tree_attention.zig: drafted rows of any number of streams inone call. Each row sees its stream's committed keys and then its root-to-row path (
paths()builds the paths fromtree-local parents, depth up to 128). Keys run in 512-key groups on tensor cores, merged in group order, at head
dims 128 and 256 with grouped KV heads. A stream's fully committed groups are computed once for all its rows (16
(row, head) pairs as one mma tile, the same tiles and merge as a row's own block), so every partial keeps its bits
whether a row runs alone or beside others. The grid is fixed: at most 16 group slots per (row, key head), each
walking its groups, so a captured graph does not grow with the context. A group's key tiles go to 4 slices; a
four-warp block combines its slices in order; where four-warp blocks would leave SMs idle (
TreeAttention.sms),each slice takes its own one-warp block and the merge combines the 4 in the same order (a small fixed slice area
in the scratch), so the bits do not depend on the device's size. A one-warp block fetches its tiles 2 ahead (3
cp.async stages where the one-warp grid fits one wave of the SMs, else 1 ahead in 2, two blocks an SM; stages
never change bits) while it computes one. Partials hold the query heads only. The merge runs a block per (row, key
head, head).
zig/kernels/cuda/block_attention.cu,zig/src/cuda/block_attention.zig: a DFlash-style drafter's block rowsover their stream's context (each stream's K and V cache through a pointer table, within a sliding window) and then the block itself,
causal or not. One warp per head on tensor cores.
zig/src/core/attention_ref.zig: one query head's attention in float64 with a per-element tolerance (2^-7 ofsum p|v| / sum p, plus a bf16 half-ulp), the reference both tests check against.
Users we know of: the Qwen3.8-27B family in #608 (target tree attention at dim 256 and its DFlash2 drafter), and any
CUDA family that drafts trees with #734.
Receipt
c36b536d(1.0.9) plus these commits; one DGX Spark (GB10, sm_121, CUDA 13). Nocheckpoint for the tests.
tf-cuda-test tree: dim 128 (48 query / 4 KV heads) and 256 (24 / 4), several streams with different committedlengths and trees, every element within the float64 tolerance (worst 0.47 of it). A stream's rows run alone
equal their bits in the batch; one shared block walking every (row, group), a grid sized for a 262k context,
and one-warp blocks equal the default grid.
tf-cuda-test block: windowed contexts then the block, causal and not, block widths 7 and 16, withintolerance (worst 0.38). A stream run alone equals its bits in the batch.
off by one, the causal mask off by one, the merge leaving out a group's fourth slice (19x the tolerance, and the chunked merge's fourth slice), a tile fetched into the stage being read.
attention, drafted long prompt: attention 46.1 ms against Triton's 49.0 ms; 16 streams at 32k context, code and
chat at temperature 0 and 1: every
bench_concurrent --alone --serialcell exact, throughput within -4..+2 % ofTriton. Replies equal 0.6.6's except two at bf16 near-ties (an exact top-2 tie, and one ulp), where drafted still
equals serial. Teacher-forced on 1,023 rows of real text: NLL 0.8058 vs 0.8029, argmax agreement 99.12 %.
The slices, one-warp blocks and stages (
dc54a0e5,1adfc970,b7d1bdc0) change no bits: 4,092 teacher-forcedlogit rows byte-identical to the build before, drafted and 8-stream shared rounds with the same shas and round
counts. 512-key groups (
01969f6d) give the 27B new bits once, on every device: drafted == serial on all fourprompts in both block forms, and against the Triton set on 4,092 teacher-forced rows dNLL +0.0009, argmax
agreement 99.32 %, mean KL 0.00066 (2,048-key groups: +0.0011, 99.12 %, 0.00071). Serial decode after a 4,722-token
prompt, 128 tokens, GB10: attention 275.8 ms with the first version, 209.1 ms now (one-warp blocks; four-warp
217.5 ms; Triton 244.3 ms). On an RTX 5090 (Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608): attention 34.6 µs a call against Triton's 59.7,
long2.92 vs2.94 s. The merge issuing a chunk's loads first (
c817f13c) changes no bits (the same 4,092 logit rows, shas andround counts) and takes the merge from 10.0 to 4.6 µs a call on GB10, attention 199.2 ms in all.
zig build testandzig build test-cuda-host(27/27 steps, 79/79 tests).zig fmt --checkclean.lean_check.py: 538 findings, as onmain.mainuses it.Registration lines in
build/cuda.zig,cuda/kernels.zig,cuda/root.zigandtests/cuda/main.zigsit where#734, #736 and #740 add theirs; whichever lands second takes both.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Noserving path changes.)