Repository navigation
Conversation
8 tasks done
jschmied
force-pushed
the
cuda-tree-round
branch
2 times, most recently
from
October 13, 2026 16:23
07674c3 to
d162ac9
Compare
… family The Metal engine's tree round (picks, each tree's verified path, the commit steps that read it) as shared CUDA kernels: tf_round_bf16_argmax, tf_tree_round_match over a batch of trees (tree-local parents, per-tree budget and EOS ids), and keep rows / gather rows / keep head for the commit. Results use core/tree_round's Result layout, so decode() reads them as-is. tf-cuda-test tree-round checks argmax against rowArgmax (ties, NaN, +-inf, padding past vocab), 4096 batched random trees against oracle() and decode() (every stop and status, full 16-row paths), trees alone against the batch, and the commit kernels against the host; four kernel mutations are caught.
jschmied
force-pushed
the
cuda-tree-round
branch
from
October 14, 2026 09:18
d162ac9 to
8d48cf9
Compare
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Metal runs a drafted tree round on the GPU through
core/tree_round.zig's contract: one pick per row, each tree's verified path and emitted tokens, then the commit steps that read that result. CUDA has tree attention and the GDN tree kernel, but no verify or commit step, so every CUDA family that drafts a tree has to write its own.This adds that step as a shared module,
zig/kernels/cuda/tree_round.cuwithzig/src/cuda/tree_round.zig:tf_round_bf16_argmax: each row's first largest finite bf16 logit, at any row count and stride. A row with NaN or ±inf gets no token and the maskcore/tree_rounduses (1, 2, 4).tf_tree_round_match:oracle()for a batch of trees, one thread per tree. EachTree{first, rows, budget, eos_first, eos_count}names its rows, its budget and its EOS ids. Parents are tree-local. Results usecore/tree_round.Result's layout, sodecode()reads them unchanged.tf_round_keep_rows(each tree's kept round rows and their count),tf_round_gather_rows(those rows of any bf16 buffer, in path order, for hidden rows or drafter taps) andtf_round_keep_head(the last kept row's logits, which the next round's first pick reads). Each one ignores a tree whose result is notok.No family calls it yet. Nemotron's chain path is unchanged. The 27B in #608 and any other tree drafter can take it as it is.
core/root.zignow exportstree_roundso the CUDA test runner can check against it.Receipt
0efbd447(1.0.8) plus this commit; one DGX Spark (GB10, sm_121, driver 580.173.02, CUDA 13). No checkpoint needed.tf-cuda-test tree-round:rowArgmaxbit for bit. That covers ties, NaN, +inf, -inf and all -inf rows. Each row's padding pastvocabholds a large finite decoy that must never be picked.oracle()byte for byte, anddecode()accepts every result. The trees hit every stop (bonus, length, EOS, invalid) and every status, and include full 16-row paths, budget 0, and malformed trees and picks. 64 trees run alone equal their batched results.zig build testandzig build test-cuda-host(25/25 steps, 208/208 tests), all passing.tf-cuda-test samplestill matches 2,880 of 2,880.lean_check.py: 539 findings, as onmain.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No serving path changes.)