Repository navigation
perf(qwen27 cuda): reduce repeated-prompt TTFT - #747
Closed
jregan-beasley-smc wants to merge 24 commits into
Closed
jregan-beasley-smc wants to merge 24 commits into
jregan-beasley-smc wants to merge 24 commits into
Conversation
The lane host admits every queued request at once: a lone one still goes to the backend's driver, the rest to the core's addStreams (lanes/admit.zig). It runs the backend's optional prefill_many when there is more than one, draws each stream's first token, then asks for every stream's first drafts in one request, as multi.py's _fill_batch admits a batch. addStream is its one-stream case, with the same calls in the same order; a backend without prefill_many prefills each stream as before.
…ilies share it Source needed Nemotron's kernel handles only for the driver, the stream and whether the card is discrete; it now takes those as a Target (Target.of(ops) for Nemotron's own loader). cuda.zig exports the source module and the torch ops for the 27B's CUDA engine.
…y stream drafted A qwen3_5 checkpoint with the 27B's shapes opens zig/src/families/qwen27/cuda in tensorfold-native, beside the family's Metal engine. It reads the family's config.zig and refuses other shapes (cuda/shape.zig). The forward is the Python 0.6.6 engine's: its CUDA kernels (qmm_group, qmm_prefill at prompt tile 9, the GDN tree, replay and prompt chain, prefill attention) built from zig/kernels/cuda, and its Triton glue from a set captured on the chip (TENSORFOLD_CUDA_KERNELS). Verify rounds take every stream's window together, up to 16 streams of 16 rows, as multi_tree_forward lays them out, and commit only each stream's accepted path. DFlash2 (--drafter, else the pulled z-lab/Qwen3.8-27B-DFlash2) drafts every stream in one batch: each stream's kept taps absorbed together, every block in one forward, a best-first tree per stream, or a copied continuation where the context repeats. The lane core verifies the trees and trims them by each node's chance against a cost curve timed at load. Prompts that arrive together share one prefill pass (prefill_rows). Checked against captures of the Python engine (zig/tests/cuda/qwen27), on an RTX PRO 6000 and on one DGX Spark (GB10): serial replies equal token for token (a 4722-token prompt among them); 54 verify windows (chains, trees, shared rounds of 2 to 16 streams and up to 256 rows, every alignment of the packed inputs) equal row for row in their logits' bytes; drafted replies and each stream's rounds equal Python's; served replies of 1, 4 and 16 concurrent requests, greedy and sampled, equal the same request alone and its "draft": false decode.
…ore admission; GLM-5.3-Flash reads them on Metal Server (engine-independent): - `Info.vision` is an engine's offer: a `prepare` hook (an image's bytes to its tower's patches, grid, tokens and a content hash, decoded by the platform's own decoder), the image token, the wrapper's tokens, and a request's visual-token budget and image limit. `--vision`, `--vision-max-images` and `--vision-image-tokens` reach native engines. - User messages keep their image_url parts (data URLs) beside their text for the template. Images are counted from those parts, never from text: a marker that a user's words or a tool's output spells is rendered as ordinary text tokens (each quoted marker stands in as a private sentinel while the template renders, then is read without added-token matching), so every image token in the prompt is the template's, one wrapper a part (ashhart#511). A prompt that quotes no marker gets exactly the ids it got before. - Each image's placeholder is expanded to its tokens before the window, the chunks and the cache see the prompt; `Request.images` carries the prepared patches and positions, and `Request.cache_key` replaces each image's rows by ids derived from its content, so a kept state of one image never resumes another under the same placeholders. - 0.6.6's limits and refusals: 4 images, 10 MB each and 20 MB a request, 8192 px a side and 16 MP, a single frame, a shared visual-token budget, detail low at 256 tokens; PNG and JPEG (images.py's formats), WebP refused by name until a decoder exists; remote URLs refused. GLM-5.3-Flash on Metal: the image tower (24 pre-norm blocks with 2-D rotary attention, the 2x2 merge and the merger, its projections on gemm_nax), its rows written over the placeholders' embeddings in the prompt pass and one position on for the MTP head; preprocessing as 0.6.6's vision/images.py and mlx-vlm's Glm5NextImageProcessor (EXIF orientation, alpha over white, PIL's 8-bit bicubic resampler). tf-glm-vision compares a reference dump's stages. tools/zig/glm_vision_fixtures.py draws the fixed image set and dumps the Python frontend's tensors; tools/zig/vision_served.py checks a running server.
…0.6.6 did media_fetch.zig follows images_http.py: an https URL on port 443 without credentials, a fragment, a backslash or controls; every address the host resolves to public (Python's is_global less 192.0.0.0/24 and 168.63.129.16; IPv6 global unicast only, no 6to4 or Teredo); TLS verified for the host on the address that was checked; GET with identity encoding; up to 3 redirects, each checked again; the declared media type JPEG, PNG or WebP (WebP is then refused by its bytes, as with data URLs); the bytes within what the request has left of 10 MB an image and 20 MB in all; 10 s a download and 30 s for a request's downloads. std's connect has no timeout yet: a nonblocking connect and poll bound it, the socket's send and receive timeouts bound the rest. The HTTP exchange reads from any reader, so the response rules (redirects joined and rechecked, status, types, lengths, chunking, encodings, the deadline) are tested against canned responses; the URL and address rules are tested directly. Without --vision-urls an https part is refused with the flag's name, other schemes as 0.6.6 refused them; --vision-urls needs --vision. tools/zig/vision_urls_served.py checks a served model against public hosts.
Conflicts resolved toward main's admission (open.zig, chunked prompt passes); the PR's shared prefill batching (admit.zig, addStreams) is dropped, prefill_many stays as an unused optional vtable entry. Living Weights setup in native/cuda.zig is guarded by @hasDecl(F, "slide").
… URLs) onto qwen27-cuda-1.0.7
…its check against 0.6.6 Row cosine against 0.6.6's flash-attention tower on nine fixture images: mean 0.9997-0.99998, min 0.981-0.99998; 0.6.6's own math-vs-flash attention gives min 0.9895 on the same shapes image.
…rt_resize, PIL bicubic, patches PNG decode and patches equal 0.6.6's bit for bit on seven fixtures; JPEGs differ in 1-6% of bytes (stb's IDCT against libjpeg-turbo's).
…he prompt pass, M-RoPE positions - Media (vision.zig): the request's images encoded, each prompt row's feature row, and get_rope_index's three positions; rope_delta kept on the stream for decode and verify positions (the drafter's stay plain, as 0.6.6's do) - Forward.chunk: image rows take the tower's rows; a chunk holding them runs attn_prep with three position rows (qwen_vision.cu), others the captured kernel at their shifted 1-D positions - native/cuda.zig: --vision for families that serve it (Info.vision from the family's prepareImage and markers), the tower's call scratch kept out of the streams' room; other CUDA families refuse --vision - JPEGs through libturbojpeg when it loads (PIL's decoder): patches equal 0.6.6's on every fixture, PNG and JPEG - aot: a launch with no exact captured variant takes one that assumes less of its arguments; the 27B's attention plan segments start on 16-byte boundaries, as the Python engine's tensors do Checked on an RTX 5090 against 0.6.6: prompt token counts equal on five image requests, three replies word for word, text exactness unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 6/6), concurrent text replies equal solo beside an image request.
…itted streams --kv-cache fp8 keeps the 27B's attention caches as e4m3 rows with one fp32 scale a (position, key head) row (amax / 448), half the bytes of bf16: 256K tokens with the image tower in one stream on a 32 GB card. bf16 stays the default and its bits are unchanged (0.6.6 exactness: serial 4/4, windows 54/54, drafted 4/4, shared 0 replies differ). - tf_kv_quant (qwen_ops.cu) quantizes commits' and prompts' rows into the caches - pattn8_kernel: prefill attention staging e4m3 rows widened to bf16, a step ahead in registers (within 5% of bf16 at 100K) - _shared8/_tail8 (tests/cuda/qwen27/attention_kv8.py, captured like the other glue): the decode kernels over e4m3; _tail8 quantizes the window's own rows as their commit will, so a key's bits never depend on windows, drafts or batching (drafted equals serial, concurrent equals solo); attention.py's _merge folds the partials unchanged - attention partials leave the scratch arena: sized after admission for the streams admitted (Engine.fit, native fit hook) and charged to each stream, not for 16 streams at full context (1.4 GiB back at 122,880, 2.9 GiB at 262,144) - tf-qwen27-cuda kvcompare: fp8 against bf16 teacher-forced, top-1 agreement and KL
…tate The 27B serves the shared prompt cache (core/prompt_cache.zig) as Nemotron does, its copies in pinned host memory: a long window's caches leave the device no room for them. - reuse.zig: a kept state is a stream's DeltaNet recurrences and conv windows whole and its attention rows (and fp8 scales) through the mark, about 275 MiB at 4,096 tokens under fp8 - with a cache, prompts cut the zero-anchored 4,096-row grid (engine.Chunks), so a pass resumed or kept at a grid point computes what a fresh pass of the same prompt does; without one (--prompt-cache-gib 0, and tf-qwen27-cuda) the even splits stay, bit-exact against 0.6.6 - prefillFrom (engine, lone): a restored stream's pass from its grid point, stopping at the lane host's marks; a resumed drafting stream's context holds only the rows it read (drafts change no token) - image prompts neither resume nor keep (their rows' positions are the tower's) - native/cuda.zig: each family's snapshot vtable, and a host-memory budget (8 GiB or --prompt-cache-gib, at most half of MemAvailable) for a family whose copies live there - zig fmt over native/cuda.zig and qwen27/cuda/triton.zig (the fmt check) The qwen suite's soak check: 60/60 replies equal to the same request alone and 60/60 turns resumed (0/60 resumed before), in 117 s against 244 s; drafts, concurrent and behaviour unchanged.
Conflicts resolved toward both sides: - build/cuda.zig: nativeServer takes main's core module and the branch's qwen27 - engine_api, lanes/stream, lane_host: main's untaught and prompt_logprobs beside the branch's images and cache key; the lookup skips untaught and prompt_logprobs requests (main) and matches by cacheKey (branch) - native/cuda.zig: Nemotron's store as main makes it (keeps at any row, snap_ready's charge); the 27B's on its grid in host memory; prompt_logprobs and vision both in Info - glm_host: main's serves window with the branch's vision; growSnap and prepareImage both kept - metal.zig: the fp8 refusal before main's refusal(), the vision refusal before glm_moe_dsa - lane_host_reuse_test: the event switch handles prompt_logprobs zig build test 274/274; zig fmt clean; tf-qwen27-cuda against 0.6.6 unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 0 replies differ).
attention_kv8.py (which records the fp8 attention launches) and qwen_vision_fixtures.py (which records the vision fixtures) import tensorfold, so they move out of the tree; their recordings stay fixture files, made once outside it.
Port early first-token delivery and opt-in exact-prompt state reuse to PR726, preserving its host-prefix cache and admission flow. Keep the experimental caches restricted to supported BF16 text requests.
jregan-beasley-smc
marked this pull request as ready for review
October 12, 2026 22:48
Conflicts resolved toward both sides: - build/cuda.zig: the CUDA test list keeps qwen27 beside main's tokenizer tests - lane_host: main's spans helper and its warm-pass lookahead rule - tokenizer: the branch's encodePlain beside main's memoized encodeSegment - cuda/kernels.zig: the branch's torch_norm_rotary beside main's norms and embed_rows - server/cli.zig: --cortheon beside --vision and --kv-cache - native/cuda.zig: main's shared lane binder (cuda_bound.zig); the startup line takes main's "passes of up to N rows" with the branch's fp8 note and host-memory cache; Nemotron's cache rules take main's .warm = .waits; warm turns stay off for the 27B, whose cache keeps states on its prompt grid Follow-ups the merge needs: - cuda_bound.zig forwards prefill_many, which the 27B's backend gives - qwen27's LoneRun takes main's resumed argument (the 27B has no lone driver) - qwen27 declares takes_drafter: main's new --drafter refusal would otherwise refuse its DFlash2 drafter Checked on an RTX 5090: zig build test 48/48; bf16 against 0.6.6 serial 4/4, windows 54/54, drafted 4/4, shared rounds no reply differs; fp8 serial, drafted and shared rounds unchanged; the server with the prompt cache on matches 0.6.6 in 20/20 (12 resumed); images, prompt-cache resumes and decode speed unchanged.
Integrate the exact tested 5cbd313 production tree, preserving the original PR747 history and receipts. Add the accepted single-order startup, request and decode result summary with remaining regressions. Shared native CUDA operator work is by jschmied (PR796/PR797). The Qwen27 integration builds on PR726 by BobClawblaw and PR608 by philip-pentatonic. No source adaptation or new tests in this commit. Authored by Jonathan Regan-Beasley in a private capacity; this does not represent the views or positions of any organisation.
Owner
|
Thank you, @jregan-beasley-smc, for isolating the first token handoff and exact prompt reuse, and for separating your gains from the engine work underneath. We chose #836 as the one Qwen CUDA base, so we are closing this branch, which is built on #608 and #726. Your time to first token comparisons and the handoff approach are a good guide for that engine. Please bring focused patches onto #836, with request ownership, cancellation, and drafted and concurrent equality in the gates. We would welcome them there under your name. Posted by Ash's AI Assistant Synthena |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Addresses the repeated-prompt first-token delay discussed in #608 while preserving #726's longer-history prompt cache. Against the same updated native CUDA baseline on one DGX Spark, the validated candidate reduces repeated-prompt TTFT by 85.7–90.9% and median complete-response time by 1.6–17.1% across coding/chat at 1, 4 and 16 streams. Startup and uncached results are mixed; the remaining trade-offs and single-order limitation are documented below.
The branch incorporates #726 at
e3b72103and the pinned native CUDA integration, including the shared #796/#797 operator sources atbf3f16a2. Inherited engine and kernel work is not this contribution. Review only our 37-file change against the tested baseline.This continues my earlier TTFT contribution to the #608 author branch. Credit to @philip-pentatonic for #608's Qwen27 CUDA engine and DFlash integration, @BobClawblaw for #726's integration and host-prefix implementation, and @jschmied for the shared native CUDA kernels and #796/#797 updates. Existing history and authorship are retained.
Receipt — updated native CUDA, 14 October 2026
A separate follow-up experiment retains the TTFT approach and includes the operator updates from #796 and #797 in both the baseline and candidate. Credit for those shared kernel updates belongs to their authors. This isolates the candidate against the same updated operators; it does not attribute the upstream kernel improvements to this PR.
The PR now contains the validated candidate source, including post-first capture and startup K/V quantization reuse. All 1,839 production files are byte-identical to tested candidate
5cbd313e; publication only preserves historical receipts and adds a result summary. No new tests were run for this publication. Updated public result summary.Receipt
One DGX Spark, native CUDA BF16, the same pinned Qwen3.8-27B MLX-4bit target and DFlash2 drafter listed below, context 32768, parallel 16, 1 GiB host prompt cache, temperature 0 and seed 1234. Baseline source
394dcbc2047cac2349ad392b0768792d04c17190; candidate source5cbd313e7c8e0014a88f1037c2ecfd3fcd6c0fa2. All 46 CUDA assets are byte-identical between the two builds. The seven candidate switches are disabled in the baseline and enabled in the candidate; the buffer-reserve experiment is excluded.Six fresh-server startup pairs cover coding/chat at 1, 4 and 16 streams. Two full panels cover fresh and repeated prompts plus a 4096-token history follow-up. Short responses generate 256 tokens; long-history controls generate 32. 372 measured responses form 186 exact-matching baseline/candidate pairs. Both sampled-output and cancellation/recovery checks passed. All 16 jobs completed, and the production configuration and voice services were restored.
One sequential forward-order run only: baseline precedes candidate within each startup pair and in the full panels. No reverse-order repetition was performed. These are bounded observations, not a statistically established speedup or a claim of no regression.
Repeated prompts
Values below are full-panel medians. Decode is per stream, not aggregate throughput. Complete response is median request-to-final-response time, unlike the whole-burst wall time used in the original table below.
Repeated-prompt TTFT is 85.7–90.9% lower. Complete-response time is shorter in all six repeated-prompt cells. Decode improves at four and sixteen streams, while single-stream coding and chat show small measured declines of 0.2% and 0.4%, respectively.
Startup, uncached requests and remaining trade-offs
Saved run:
native-operators-total-forward-20261014-034455. Independent audit SHA-256:912179c6e3449eef20a955e74fb9944082663833f269f8167ec725f04496d11a. The full raw receipts remain local. The linked updated summary publishes the source/build identities, aggregate results and limitations; it is not a self-contained reproducer.Historical results for the original PR747 revision
These measurements and checks apply to
10e7fe07, not the updated source. Use the historical receipt's source verifier with that original checkout.Original PR-specific receipt
Measured on one DGX Spark, GB10/sm_121, Linux aarch64, Zig 0.17.0 native CUDA. Target:
Vontra/Qwen3.8-27B-MLX-4bitat70ae7fac63274ff2eac54152031433374cb80f2f; drafter:z-lab/Qwen3.8-27B-DFlash2at50307d4c4cde6860d4eee73e2547cd786fe8e8a4. BF16 KV, context 32768, parallel 16, greedy temperature 0, seed 1234, thinking disabled. Both builds use the same qualified SM121/AOT assets. Portable receipt, source identities and verification.Four fresh-process configurations: stock #726 with host cache off/on, and the patch with host cache off/on. Each runs 38 requests; 152/152 responses passed literal-request, output-hash and token-fingerprint checks. Code has 14 prompt tokens and chat 31. Each has a first-use request, one exact repeat, and a burst of sixteen repeated requests, all generating 256 tokens. Two real long-chat requests generate 32 tokens each. Live
/tokenizechecks bind the rendered chat history to the frozen fixture.These are preliminary observations from one sequential panel, ordered stock/cache-on → patch/cache-on → stock/cache-off → patch/cache-off. They are not a balanced performance qualification. The following comparisons keep #726's host cache enabled at 1 GiB on both builds; all five experimental switches are enabled only on the patched build.
Repeated-prompt time to first token
Decode and complete response
For sixteen streams, decode is the median per-stream rate; complete time is the whole burst's wall time. Single-stream complete time is request-to-final-response, not just the decode interval.
Within this repeated-prompt panel, complete-response time falls by 0.8% for single-stream chat, 14.4% for sixteen-stream code and 12.0% for sixteen-stream chat. Single-stream code takes 3.1% longer overall. Decode alone falls by 6.0% for single-stream code and 1.4% for single-stream chat, while increasing by 7.8% and 7.0% for the sixteen-stream code and chat workloads. The chat TTFT saving therefore more than offsets its small decode-rate decline in this observation.
First-use requests are uncached prompts in an already-loaded server, not model startup. Code TTFT was 140.04 → 140.93 ms and complete time 3.810 → 4.037 s; chat TTFT was 131.17 → 132.35 ms and complete time 5.252 → 5.320 s. Chat follows code.
Compatibility and trade-offs
There are concrete opportunities for further optimisation: profile the decode/round-count difference, assess whether allocation reuse can safely coexist with the host-cache hook, and repeat alternating-order measurements before making a stable performance claim. The present receipt establishes bounded compatibility and a promising repeated-prompt TTFT result; it does not claim universal speed improvement or no regression.
Checks and coverage
git diff --checkpasses. The runtime lean check reports 1207 inherited problems on both base and candidate, zero added findings; the nonzero exit is not reported as a clean baseline.draft:falseequivalence gate. These limits remain explicit for review.Authored by Jonathan Regan-Beasley in a private capacity. This contribution does not represent the views or positions of any organisation.