perf(qwen27 cuda): reduce TTFT with prompt caching and state reuse

[jregan-beasley-smc]

What this changes

The repeated short-prompt benchmark in ashhart/TensorFold#608 exposes a substantial time-to-first-token (TTFT) gap between its new Zig/CUDA Qwen3.8-27B engine and Python. This PR closes that gap on the tested workload and brings primed short-prompt TTFT below both unmodified PR608 and Python on the same DGX Spark.

Using PR608's maximum-TTFT statistic, sixteen-stream chat falls from 851 ms stock / 247 ms Python to 205 ms; coding falls from 690 / 299 ms to 255 ms. Once the short prompt and reusable request storage are warm, single-stream TTFT is 3.4–3.8 ms, versus 128–135 ms stock and 111–114 ms Python. Warm sixteen-stream medians improve by 81–85% against stock and 43–46% against Python. The receipt below compares all three implementations in fresh, matched runs.

The gains come from returning the first token earlier and avoiding repeated prompt computation, allocation and initialization:

• Deliver the first token before constructing the initial draft, retaining the earlier handoff fix as a separate commit.
• Cache the complete state after an identical short prompt and restore independently owned mutable target and drafter state for each request.
• Reuse target and drafter allocations within bounded pools.
• Cache immutable raw first-token logits while preserving per-request sampling and token selection.
• Omit initialization clears only where a proven full-state restore overwrites the affected storage. A stale hint resets before computation; a partial restore aborts admission.

The cache/reuse implementation remains opt-in for review: all five switches default off. Exact-prompt snapshots are limited to 512 prompt tokens and 256 MiB; allocation reuse and reset omission also require normal requested capacity at most 512. Target retained storage is capped at 3 GiB and drafter storage at 1.5 GiB. The configuration and fallback rules document the controls. A growing conversation changes the prompt and does not automatically receive this exact-prompt cache hit.

This follow-up targets see-4347-qwen27-cuda, the branch behind PR608, so it can be reviewed separately from the engine itself.

Receipt

Matched comparison: unmodified PR608 at 99bcc5b2, Python 0.6.6 at cb2ebf05, and the complete candidate (7acdc544 + 474bd89b). One DGX Spark / GB10 (sm_121), Linux aarch64, Zig 0.17.0, pinned NVIDIA 26.07 image; target Vontra/Qwen3.8-27B-MLX-4bit at 70ae7fac, drafter z-lab/Qwen3.8-27B-DFlash2 at 50307d4c. Both native implementations use the same captured CUDA/Triton assets. The portable three-way receipt records complete source, binary, model and client hashes, request-level measurements and an offline checker.

Protocol: the original coding/chat prompts from PR608 (14/31 tokens), temperature 0, seed 1234, thinking off, BF16 KV/prefill, 256 output tokens, --parallel 16 --context 32768. Two process orders: Python → stock → candidate, then candidate → stock → Python, with a fresh server process for every arm. Each prompt has one uncached single-stream request, then three bursts each at one and sixteen streams. All three implementations receive the same requests.

TTFT is client send to first nonempty SSE text/reasoning. This retains PR608's prompts, settings and maximum-TTFT comparison basis, with one disclosed protocol change: the uncached first-use request now generates 256 tokens instead of the original 16-token warmup, so cold response and decode measurements can also be retained. All sixteen-stream bursts are primed. Python restores N−1 prompt tokens; the candidate restores N; stock reports no cached tokens. That added cache capability is part of the improvement.

Maximum TTFT, matching PR608's reported statistic: maximum individual TTFT across all three primed bursts in each order, then the mean of the two order maxima. This includes the first burst's request-storage setup costs.

Workload	Stock PR608	Python 0.6.6	This PR
Coding, 1 stream	129.4 ms	116.8 ms	8.8 ms
Chat, 1 stream	139.8 ms	116.8 ms	9.8 ms
Coding, 16 streams	689.7 ms	299.0 ms	255.3 ms
Chat, 16 streams	850.7 ms	247.4 ms	204.8 ms

At sixteen streams, these maximum-TTFT reductions are 63.0% coding / 75.9% chat against stock, and 14.8% / 17.2% against Python.

Warm repeated TTFT: means of the within-burst medians for repetitions 1–2 across both orders, excluding the first primed burst. The approximately 3–4 ms single-stream figures measure first-token delivery for an identical cached short prompt, not the full response or every subsequently generated token.

Workload	Stock PR608	Python 0.6.6	This PR
Coding, 1 stream	128.3 ms	111.3 ms	3.4 ms
Chat, 1 stream	135.5 ms	114.2 ms	3.8 ms
Coding, 16 streams	681.8 ms	219.5 ms	126.2 ms
Chat, 16 streams	833.5 ms	233.0 ms	126.8 ms

Decode and full response: the same warm repetitions, with per-stream client decode measured as (output_tokens − 1) / (last_nonempty_SSE − first_nonempty_SSE). Full-response time runs from send to DONE; at sixteen streams the table reports the whole burst from earliest send to latest DONE.

Workload	Decode tok/s per stream, stock / Python / this PR	Full response or burst time, stock / Python / this PR
Coding, 1 stream	68.80 / 68.11 / 69.45	3.836 / 3.856 / 3.675 s
Chat, 1 stream	49.49 / 50.37 / 49.49	5.288 / 5.178 / 5.157 s
Coding, 16 streams	25.13 / 25.08 / 25.51	10.833 / 10.408 / 10.124 s
Chat, 16 streams	18.78 / 19.06 / 18.81	14.418 / 13.634 / 13.819 s

Decode is broadly similar, with small measured trade-offs: warm chat per-stream decode is 1.7% lower at one stream and 1.3% lower at sixteen streams than Python; sixteen-stream chat completes 1.35% later than Python, while still finishing 4.16% earlier than stock. Whole-burst aggregate chat decode is 285.8 / 304.8 / 295.4 tok/s, or 3.09% below Python. Earlier first-token delivery also changes the measured decode interval. Speculative work differs despite exact output agreement, so these are client-visible results, not isolated kernel-speed measurements.

Uncached first use, one stream: each prompt's first request has zero cached tokens and generates the same 256 output tokens. These are ready-server timings, not model loading; the coding prompt is first in the process and the chat prompt follows the coding panel.

Workload and metric	Stock PR608	Python 0.6.6	This PR
Coding TTFT	145.8 ms	205.9 ms	148.8 ms
Chat TTFT	131.4 ms	123.5 ms	162.0 ms
Coding full response	3.826 s	4.024 s	3.821 s
Chat full response	5.289 s	5.199 s	5.327 s

The cold chat cost is 31 ms TTFT versus stock and 38 ms versus Python (23.3% / 31.1%). Its full response is 0.7% / 2.5% longer, respectively. Cold coding is approximately unchanged versus stock and faster than Python. The core result is the large repeated-prompt TTFT gain; uncached requests do not all improve.

Validation: all 624 requests completed with exact paired input/output and token-fingerprint agreement. Both benchmark orders restored the original production and voice services. Percentages geometrically balance the two within-order ratios; displayed times are arithmetic means. These are two orders on one machine, not confidence intervals, and stock is the middle arm in both orders.

The unchanged runtime also retains the earlier implementation qualification: 334/334 Linux CUDA-host tests, native and qualifier builds, and actual CUDA checks of poisoned fresh/pooled destinations, independent ownership, target/draft state, logits and continuation equality, plus forced hint-miss reset. The new three-way runs add performance evidence without changing the runtime.

Scope is two fixed short greedy prompts. Long contexts, heterogeneous/growing conversations, general-prefix reuse, sampled HTTP and other devices/models are not qualified by these timings. Output agreement is evidence for these tested requests, not a general quality claim. Runtime defaults and the production model configuration remain unchanged.

Related work and attribution

This builds on Philip's Qwen27 CUDA engine, shared lane core and Python-parity work in #608, and TensorFold's existing Python reference and benchmark clients. Those remain their authors' contributions. The earlier first-token handoff fix is retained as its own commit, followed by cache/reuse implementation and validation commits.

The overlap review identified #651 as adjacent lane-admission work and #625 as core paged-cache infrastructure; neither supplies this Qwen27 snapshot/reuse path. Ash's native-glue and Python-dependent capture-tooling requirements remain separate release work. This validation uses PR608's captured Triton assets.

Authored by Jonathan Regan-Beasley in a private capacity. This contribution does not represent the views or positions of any organisation.