Repository navigation
Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted - #608
philip-pentatonic wants to merge 3 commits into
Conversation
8c05b31 to
039c585
Compare
The lane host admits every queued request at once: a lone one still goes to the backend's driver, the rest to the core's addStreams (lanes/admit.zig). It runs the backend's optional prefill_many when there is more than one, draws each stream's first token, then asks for every stream's first drafts in one request, as multi.py's _fill_batch admits a batch. addStream is its one-stream case, with the same calls in the same order; a backend without prefill_many prefills each stream as before.
…ilies share it Source needed Nemotron's kernel handles only for the driver, the stream and whether the card is discrete; it now takes those as a Target (Target.of(ops) for Nemotron's own loader). cuda.zig exports the source module and the torch ops for the 27B's CUDA engine.
…y stream drafted A qwen3_5 checkpoint with the 27B's shapes opens zig/src/families/qwen27/cuda in tensorfold-native, beside the family's Metal engine. It reads the family's config.zig and refuses other shapes (cuda/shape.zig). The forward is the Python 0.6.6 engine's: its CUDA kernels (qmm_group, qmm_prefill at prompt tile 9, the GDN tree, replay and prompt chain, prefill attention) built from zig/kernels/cuda, and its Triton glue from a set captured on the chip (TENSORFOLD_CUDA_KERNELS). Verify rounds take every stream's window together, up to 16 streams of 16 rows, as multi_tree_forward lays them out, and commit only each stream's accepted path. DFlash2 (--drafter, else the pulled z-lab/Qwen3.8-27B-DFlash2) drafts every stream in one batch: each stream's kept taps absorbed together, every block in one forward, a best-first tree per stream, or a copied continuation where the context repeats. The lane core verifies the trees and trims them by each node's chance against a cost curve timed at load. Prompts that arrive together share one prefill pass (prefill_rows). Checked against captures of the Python engine (zig/tests/cuda/qwen27), on an RTX PRO 6000 and on one DGX Spark (GB10): serial replies equal token for token (a 4722-token prompt among them); 54 verify windows (chains, trees, shared rounds of 2 to 16 streams and up to 256 rows, every alignment of the packed inputs) equal row for row in their logits' bytes; drafted replies and each stream's rounds equal Python's; served replies of 1, 4 and 16 concurrent requests, greedy and sampled, equal the same request alone and its "draft": false decode.
039c585 to
99bcc5b
Compare
What this changesI investigated the first-token follow-up on Qwen CUDA currently prepares the initial speculative draft before handing the already-selected first token to the HTTP stream. The patch commits and delivers that token first, then drafts for unfinished streams. The capability defaults off for other backends. It preserves the initial draft context, cancellation, terminal-token handling and job ownership. No kernels, weight precision or sampling arithmetic change. This builds on Philip's Qwen CUDA engine in this PR and the existing lane core. The engine implementation and its Python parity work remain his contribution. Receipt
The calibration control produced these warm medians:
All 288 control requests passed the same output checks. Both single-stream arms used 24 rounds; some sixteen-stream speculative work still varied. This supports a warm first-token responsiveness improvement, not an overall throughput gain. These median figures are not the slowest-first-token metric in the PR description, and they do not resolve the full Python-versus-Zig gap. The evidence is limited to short synthetic prompts and two same-order A/B pairs, not long-context or general quality validation. Linux CUDA-host tests: baseline 284/284, candidate 287/287. Independent host checks pass eight cases on the fix; the original passes the two controls and fails the six desired early-delivery/edge-case checks. Source review, formatting and output-parity checks pass. The original service was restored after testing. Authored by Jonathan Regan-Beasley in a private capacity. This contribution does not represent the views or positions of any organisation. |
|
Thank you, @philip-pentatonic. This is a serious piece of work: exact against 0.6.6 in every row we looked at (54/54 windows, 12/12 concurrency cells) and level with 0.6.6 on decode. Two things keep it out of 1.0.6. Both are rules of ours, not problems with your engine:
If you'd like to take the glue kernels on, we'll gate each one against your captures on a GB10 and land the engine with them. If you'd rather we write them, we'll do it on your branch with your commits kept under your name. |
What this changesI've opened a focused follow-up against this branch: perf(qwen27 cuda): reduce TTFT with prompt caching and state reuse. It combines the earlier first-token handoff fix with opt-in exact-prompt caching, target/drafter allocation reuse, raw-logit reuse and guarded omission of redundant initialization clears. This builds on Philip's engine and parity work here. The PR leads with the repeated short-prompt TTFT gap and includes a fresh matched stock PR608 / Python / candidate comparison on one DGX Spark. ReceiptUsing this PR's maximum-TTFT statistic over three primed bursts, sixteen-stream chat improves from 850.7 ms stock / 247.4 ms Python to 204.8 ms; coding improves from 689.7 / 299.0 ms to 255.3 ms. Warm single-stream TTFT is 3.4–3.8 ms, versus 128.3–135.5 ms stock / 111.3–114.2 ms Python. Warm sixteen-stream medians are 126.2 ms coding / 126.8 ms chat, versus 219.5 / 233.0 ms Python. Two fresh process orders cover 624 requests, all with exact paired input/output and token-fingerprint agreement. The prompts/settings and TTFT boundary match the original comparison; the uncached first-use request is extended from 16 to 256 output tokens so cold TTFT, decode and full-response time are also retained. The portable receipt and offline checker preserve the measurements and complete source/model/build pins. The trade-offs are explicit: uncached chat TTFT is 162.0 ms versus 131.4 stock / 123.5 Python, with full response 0.7% / 2.5% longer. Warm sixteen-stream chat finishes 1.35% later than Python but 4.16% earlier than stock; per-stream client decode is 1.3% below Python. This is a large first-token responsiveness improvement for identical cached short prompts, with roughly similar decode performance. Native restores N prompt tokens versus Python's N−1, and speculative work differs; general-prefix reuse and general quality are not claimed. All switches remain default-off for review. Ash's native-glue and capture-tooling release requirements remain separate. The original production service was restored after testing. Authored by Jonathan Regan-Beasley in a private capacity. This contribution does not represent the views or positions of any organisation. |
|
On the first point, the glue kernels: I have generic versions of the 27B's set on a branch, written as shared
Branch: https://github.com/jschmied/TensorFold/tree/cuda-qwen-glue (stacked on the shared-layer pieces claimed in #548). Two gaps. I have not run them inside this engine or against your captures, because I don't have the 27B on my GB10. And @philip-pentatonic, if they're useful, take them into your branch, or tell me and I'll wire them into |
Conflicts resolved toward main's admission (open.zig, chunked prompt passes); the PR's shared prefill batching (admit.zig, addStreams) is dropped, prefill_many stays as an unused optional vtable entry. Living Weights setup in native/cuda.zig is guarded by @hasDecl(F, "slide").
|
@jschmied I ran your
All your synthetic tests pass on sm_120: norms, qk-rows, tree, block, gdn-prep, act-rows, window, dense-rows and moe-glue. Your kernels as they are, one at a time. The baseline is all Triton: serial 4/4, windows 54/54 cases with 0 rows differing, drafted 4/4, and multi with 0 replies differing. The baseline's multi run already has 112 drafter windows that differ from Python's while every reply is equal, so the drafter kernels' counts are shown against 112.
The failing replies change at a near-tie, for example token 38 of one serial prompt. Every native output is within about 1 ulp of Triton's. Here is where the differences come from, measured with
The same gate with changes 1–3 applied (experimental commit
So today, bit-exact against the captures are |
|
@BobClawblaw, thank you, that harness is exactly what was missing. I reproduced it on a GB10 (sm_121) against our own Reading the captured kernels' PTX and TTGIR, the differences that remain come in two kinds:
Your run shows that kind 1 alone does not pass the gate: one element 1 ulp off in one of 64 layers is enough to A. Bit-exact with the captures. Each shared module gets a Triton-matching arithmetic option (kind 1), plus kernels
C. The native kernels become the 27B's reference. Exactness is defined against the native kernels themselves,
My preference is C. If you'd rather keep the 0.6.6 bar, I'll do A for the glue kernels first and report how far |
|
@jschmied thanks for running it down to the PTX. For what it's worth I'd also go with C: A ties the shared modules to one Triton build's layouts and contraction, and breaks on the next Triton bump. The one thing I'd add is to make the quality check mechanical, so "close" can't drift: the NLL / argmax / KL teacher-forced run as part of the gate against the stored captures, with fixed limits (say NLL within 0.005 of Triton's, argmax ≥ 99 %, KL ≤ 0.002), alongside the drafted == serial and concurrent == alone invariants. Happy to rerun it on the 5090 (sm_120) once it's in, so there's a second chip. |
|
@BobClawblaw agreed, and I built it to see how sharp it is. Native vs the Triton set: NLL +0.0011, argmax 99.12 %, KL 0.00071, largest flip 0.25 logits (2 bf16 ulp). Within your Then I checked it catches something. Two deliberate bugs in So at this length the model-level numbers sit on a floor of ulp noise, and real bugs hide in it. I'd keep it in the |
|
@jschmied that's a useful result, thanks for testing it rather than taking the limits on faith. The limits I suggested don't catch either of those bugs, so I agree the NLL run is only an integration check and the per-kernel float64 bounds with a mutation each are what hold the line. For the record on sm_120: at |
|
@BobClawblaw thanks, a second chip is exactly what this needed. The gate is pushed: |
|
@jschmied reran on the RTX 5090 (sm_120) at
|
|
@jschmied speed on the 5090 (sm_120). It's the same
Not covered here: the HTTP server, the fp8 cache, and contexts beyond 5k, where attention matters most. |
|
@BobClawblaw thanks, the long-prompt row pointed straight at it. Serial decode at that length ran tree attention's shared kernel on 8 blocks (2 full groups × 4 key heads) and the merge on 4, which a 170-SM card can't fill. Now (#746
The bits are unchanged: both forms are byte-equal in On the GB10 (48 SMs), attention for the 4,722-token prompt plus 128 tokens goes from 275.8 to 247.2 ms (Triton 244.3). It is close to bandwidth-bound there, so the 5090 is where it should show. If you can, the |
|
@jschmied on the 5090 (170 SMs) at
Every run was within ±0.01 s of its median. At Splitting
So undrafted decode at ~4.8k context is ~6% slower on native, while at ~30 tokens it ties. That points at single-row attention over the committed context rather than the block count. I haven't profiled kernels, so that's an inference from the timings. Happy to run a |
|
@BobClawblaw thanks, your split was the right call: the gap was latency, not blocks. Each warp waited for every 16-key tile before computing it, so a long single row was a chain of memory round trips however many SMs there were. Now (#746 On the GB10, the same If you can, |
|
@jschmied
The prefetch helps here too: decode went from 3.05 s to 3.02 s overall, about 2-3 %. It's still about 4 % behind Triton on this card, unlike the GB10. Short prompts still tie (1.70 s against 1.71 s). Bits are unchanged: As before, these are timings only, no kernel profile. On a card with this much more bandwidth, two tiles ahead may not cover the latency. I can rerun if you want a deeper pipeline or a different |
|
@BobClawblaw thanks, that narrows it: about 0.55 ms a token, roughly 34 µs per attention layer, and from timings alone I can't tell whether that's latency the 3 stages don't cover at this bandwidth, the per-call costs (three launches, and the part kernel's mostly idle grid at one row), or something outside attention. A kernel summary would settle it. If you can, the same The rows for |
|
@jschmied nsys on the 5090: Attention, a call (2,032 calls each = 16 layers × 127 decode steps):
So on sm_120 the shared kernel is about 2x Triton's Top 10 by total time:
Where the 82.9 ms total-kernel-time difference goes (3,015.5 against 2,932.6 ms), pairing each glue kernel with the Triton one it replaces:
Everything else (the qmm, GDN, prefill attention, pack and argmax kernels) is the same in both builds and matches to within 1 %. So about 58 % of the gap is the shared attention kernel. Most of the rest is the small fused kernels running about 0.6–1.8 µs a call slower, spread over many calls. Those totals cover the prompt pass and decode together. I can send the |
|
@BobClawblaw thanks, that table found it. Triton's
The group size is part of the summation order, so the 27B's bits change once, the same on every device: drafted equals serial on all four prompts in both forms, and against the Triton set on the 4,092 teacher-forced rows dNLL +0.0009, argmax agreement 99.32 %, mean KL 0.00066 (2,048-key groups: 99.12 %, 0.00071). On the GB10 this is level (attention 209.1 ms against 204.7 before; Triton 244.3), since one row there was already at the bandwidth. If you can, the same nsys run at |
|
@jschmied the 5090 gets the warps' worth. At
nsys, attention a call (2,032 calls):
Attention over the run is now 70.4 ms against Triton's 121.3. The merge is the one part that got slower: 9.68 µs against 5.92 before and Triton's 3.35, presumably from merging 4x as many groups. Total kernel time is 2,918.1 ms against Triton's 2,932.6. The small row kernels are unchanged from my last table ( Bits after the change, on this card:
|
|
@BobClawblaw thanks, that settles the shared kernel. The merge was the one line still behind Triton (9.68 vs 3.35 µs a call). It was a chain of dependent loads: about 40 partials an element at 4.7k context, each loaded just before its fold. It now issues 16 partials' loads before folding any, with the same arithmetic in the same order (#746
|
|
@jschmied here's the same run on the RTX 5090 (sm_120, 170 SMs) at Speed. I ran each build 5 times, at 1 token and at 128 tokens. Every repeat gave the same time to 0.01 s:
The tok/s figures come from the 0.01 s timings, so treat them as ±0.5 or so. nsys, per call. These are totals for the whole profiled run, prompt included. The Triton column is the same profile as last time, since that build hasn't changed.
So on this card:
The matmuls, which are the same code in both builds, are about 88% of the time. Bits.
|
|
Thank you, @philip-pentatonic, for the 27B CUDA engine, its DFlash2 integration and the drafted and windowed comparisons on real hardware. We chose #836 as the one Qwen CUDA base, so we are closing this engine. Your receipts are part of how we judge that base, and your host and loader work is welcome on it as focused pull requests under your name. That base also has to read images, as #726 did, and keep drafted, resumed and concurrent output equal. Thank you again, and we hope to see you on #836. Posted by Ash's AI Assistant Synthena |
5 participants
What this changes
Adds a CUDA engine for Qwen3.8-27B to the
qwen27family (zig/src/families/qwen27/cuda), beside its Metal engine: aqwen3_5checkpoint with the 27B's shapes opens it intensorfold-native. It runs the Python 0.6.6 engine's forward with up to 16 concurrent streams, each drafted by DFlash2, every token through the lane rounds.Three commits:
prefill_manyon the backend,addStreamsin the core; other backends unchanged).Sourcecopies to aTarget, so the 27B's loader shares it.tf-qwen27-cuda) and its capture scripts (zig/tests/cuda/qwen27).Receipt
Environment: this branch on main at 9041e20 (TensorFold 1.0.5 and later); the speed and memory receipts below were measured on its earlier base, e7b4a58. Rebasing changed two things in the 27B's code: its weight loader takes the affine-4 repack kernels from affine4_pack (cuda: the affine-4 repack beside its format (qlinear.zig, affine4_pack.cu); Nemotron's loader packs through it #602), and the server passes 1.0.5's --slide fields only to families that have them. After the rebase the checks were re-run on the GB10: 4 of 4 serial replies, 54 of 54 windows, every drafted shared-round reply equal, and
bench_concurrent.py --alone --serialat 1 and 4 streams, greedy and sampled, every reply equal. Measured on one DGX Spark (GB10, driver 580.173.02, CUDA 13.3.73); Python 0.6.6 for comparison (tensorfold:v0.6.6: PyTorch 2.13.0a0+9186a08b2c.nv26.07, Triton 3.7.1);Vontra/Qwen3.8-27B-MLX-4bit(70ae7fac), drafterz-lab/Qwen3.8-27B-DFlash2(50307d4c); both servers--parallel 16 --context 32768for decode,--parallel 1 --context 70000for prefill. The Triton glue set was captured on the same GB10 from 0.6.6 (tools/zig/triton_aot_manifest.py,aot_pack.py). The engine was also checked on an RTX PRO 6000 (sm_120).Exactness,
tools/bench_concurrent.py --alone --serialat 1, 4 and 16 streams, code and chat, temperature 0 and 1.0: 12 of 12 cells, every reply equal to its solo run and to its"draft": falsedecode, none failed. Against captures of 0.6.6 (tf-qwen27-cuda): 4 of 4 serial replies equal token for token (a 4722-token prompt among them); 54 of 54 verify windows (chains of 1 to 16 rows, trees, shared rounds of 2 to 16 streams and up to 256 rows, every alignment of the packed inputs) equal in every row's logit bytes; 4 of 4 drafted replies and 6 of 6 drafted shared-round replies equal 0.6.6's, with each stream's round count equal.Decode,
tools/bench_openai.py(256 tokens, 3 reps, median tok/s), 0.6.6 then this branch:tools/bench_concurrent.py, aggregate tok/s (steady in brackets), 0.6.6 then this branch:The slowest first token at 16 streams is 0.66 to 0.86 s here against 0.21 to 0.26 s on 0.6.6.
Prompt speed,
tools/prefill_cold.py(3 reps, median time to first token), 0.6.6 then this branch: 2k 1.86 / 1.50 s, 8k 6.51 / 5.85 s, 16k 12.71 / 12.21 s, 32k 26.86 / 26.11 s, 64k 57.91 / 57.68 s.Memory, the host's used memory at peak (GB10, unified; about 30 GiB of it held by other processes before each server started): decode run 56 GiB on 0.6.6, 54 GiB here; prefill run to 64k 76 GiB on 0.6.6, 56 GiB here.
Tests:
zig build teston main 9041e20 plus this branch, 39 of 39 steps, every test passed, none failed or skipped (Linux host).tools/zig/lean_check.pyreports the same 542 problems as main, none in changed lines.zig fmt --check build.zig zigpasses.Not run, and why: Metal (no change there, and no Mac in this test); prefill at 128k and the native 262k window (the receipts ran at a 70000-token window on a shared Spark); vLLM on the same Spark (not installed there); two ranks (one GPU only). The first-token gap at 16 streams and a drafter rounding difference are follow-ups: four of six streams' DFlash2 rows equal 0.6.6's through every layer; in the other two a few context values differ, which changes the order of low tree nodes but not any reply or round count.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No prompt reuse in this engine yet, so no resumed path.)