Repository navigation
Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA - #836
BobClawblaw wants to merge 26 commits into
Conversation
|
Linux results for this branch, on an RTX 5090 (sm_120, 32 GB), Ubuntu with kernel 7.0, driver CUDA 13.4, Host and kernel suites (
|
| code, T=1 | chat, T=1 | code, greedy | chat, greedy | |
|---|---|---|---|---|
main e7a73dba |
782 | 591 | 699 | 646 |
| this branch | 777 | 591 | 699 | 646 |
Bug 1: prompt chunks of more than 32 rows
rowanc is a u32 of a window's rows, but prompt chunks run up to 128 rows. From row 32 on, 1 << r overflows. Those rows then rotate at wrong positions in tf_q35_rope and attend the wrong keys in tf_q35_attn_partial.
tf-qwen35-cuda prompt on the first N ids of a 4,722-token prompt, last row against step's row N−1:
| ids | 2B KL before → after | 27B KL before → after | 27B argmax before / after |
|---|---|---|---|
| 31 | 0.0019 → 0.0019 | 0.0008 → 0.0008 | same / same |
| 33 | 0.0032 → 0.0002 | 0.102 → 0.000002 | same / same |
| 40 | 0.067 → 0.0003 | 2.31 → 0.0023 | differs / same |
| 64 | 0.030 → 0.0001 | 0.025 → 0.000003 | same / same |
| 128 | 0.145 → 0.0001 | 1.35 → 0.00006 | differs / same |
After the fix, the last row at 16, 31, 32, 33, 40, 64, 128, 129, 200 and 300 ids has KL at most 0.003 against step, with the same argmax in 20 of 20. Prompt chunks aren't expected to be bit-equal to steps, because they use the prompt GEMM.
The receipt's checks couldn't catch this: drafted and "draft": false replies prefill through the same pass, so they agree with each other. The replies still read as plausible. On a 2,272-token prompt at T=0, the 27B and the 2B each give a coherent answer with and without the fix, but the two answers differ.
Fix: a chain row (no parents) carries its index in its window under the top bit. Tree windows, at most 16 rows, keep their bits.
Bug 2: tree windows change bits across a 128-key chunk boundary
tf_q35_attn_partial cut its 128-key chunks at a key's cache row. In a tree window, a kept row's ancestors sit at rows len + slot, not len + depth, so a path that crosses a 128 boundary sums in different chunks than one-row steps do.
tf-qwen35-cuda tree over 300 ids differed from step:
- from row 126 on the 2B (window at 120, 14 rows; the kept row's key at cache row 132);
- from row 254 on the 27B (window at 251; key at cache row 257).
Fix: chunks are cut at the key's logical position (the stream's length plus the ancestors before it), with each window key's row found from the ancestor bits. With it, on both models over 300 ids:
stepis unchanged bit for bit from before;windowequalsstepbyte for byte;treeequalsstepbyte for byte.
Servers with the fix (3c8a2716), --parallel 4
- Qwen3.5-2B (
--context 32768) and Qwen3.8-27B with--drafter Qwen3.8-27B-DFlash2(--context 8192):bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serialgives every alone and serial check equal, with 0 failed. - One stream,
bench_openai.py --tokens 128 --reps 3, median tok/s:- 2B: 419 / 418 / 433 / 419 (code T=1, chat T=1, code greedy, chat greedy);
- 27B with the drafter: 191 / 136 / 192 / 137.
- 4-stream aggregate: 1,183 to 1,365 tok/s on the 2B, 234 to 362 on the 27B with the drafter.
These are one card and short replies. Not measured: the 27B without the drafter, prompt speed, and long contexts.
Not run here
- Windows with the fix. The kernel change is the same code on both platforms, but it needs the 4090's
tf-qwen35-cudastep/window/tree and receipt runs again. - Any check of the Qwen3.5 CUDA engine against the Python engine or Metal. Everything above compares the engine with its own one-row steps.
- NVFP4 Nemotron, GB10, Ampere.
- zig/src/platform: `sys`, imported by every module (zig/build/sys.zig wires it). On POSIX it is std.c, std.posix and std.DynLib unchanged; on Windows win_c.zig gives libc's surface over the UCRT and Win32 (integer descriptors, directory descriptors, readdir, pread/pwrite, high-resolution sleep, monotonic clock, /tmp and /proc/self/exe mapped), win_posix.zig the server's sockets over Winsock, console events as SIGINT/SIGTERM, and LoadLibraryW for DynLib. - std.c and std.DynLib calls go through `platform` (mechanical). - build.zig routes Windows targets to the CUDA build. - Server: socket closes and writes are explicit, the key file's mode check is skipped on NTFS (ACLs), descriptor limits are a no-op. - shard_edit: the shard's map is released before the rename, since Windows refuses to replace a mapped file. - Tests that need symlink privilege or mode bits skip on Windows. zig build test-cuda-host passes on x86_64-windows.
- Kernel fatbins build with nvcc and cl.exe: MSVC's conforming preprocessor is passed on Windows hosts (CUDA 13's headers need it), and explicit kernel instantiations carry the __restrict__ qualifiers their templates have (MSVC counts them in the function type; the generated code is unchanged). - Nemotron's checkpoint source keeps io_uring to Linux; elsewhere its existing thread reads run. - pull_exlr8 raises no descriptor limit on Windows. - modelopt test: the lane's slice partials are allocated. GPUs without clusters (before sm_90) reduce split K through them; the test passed none, so it wrote through a null pointer on Ampere and Ada. zig build native and install build with -Dnvcc=... -Dsm=89, and tf-cuda-test's synthetic suites pass on an RTX 4090.
- pull: where Windows refuses the snapshot's symlink, the snapshot path
becomes a hard link to its blob (CreateHardLinkW through `sys.c.link`):
no extra space, and learning already refuses linked shards. Link
failures are logged with their reason.
- HOME falls back to USERPROFILE (Windows sets no HOME) for the hub cache
and every libc getenv("HOME").
- Tests read optional fixture variables through `platform.getenv`;
std.process.Environ.getPosix does not build on Windows.
- The learned-state symlink tests skip on Windows (symlinks need
privilege); `sys.c.symlink` asks for an unprivileged one.
zig build test passes on x86_64-windows.
- docs/WINDOWS.md: requirements, build, run, how the port works and how it differs from Linux; the README points to it. - -Dstrip reaches tensorfold-native before its install step is made, so the step no longer expects a .pdb the stripped build does not write. - zig fmt on the platform files.
8080 and 8000 are taken by too many other servers; serve listens on 8888 unless --port says otherwise, and the docs and bench scripts follow.
The MLX affine 4-bit checkpoint on the native CUDA server, beside Nemotron: qwen35.cu holds the Metal engine's glue at its bf16 rounding points (gated delta rule, causal convolution, gated attention), cuda.qlinear's affine-4 kernels do the projections, sample.cu draws on the device. Windows of up to 16 rows record each row's recurrence and convolution tail, so the lane core shares rounds across streams and keeps partial copies; a window's rows equal one-row steps bit for bit. draw.zig is the host draw both backends now share. On an RTX 4090: 210 tok/s for one stream, 1250 tok/s across eight; logits agree with an fp32 reference at every position of a 36-token check.
…mory The 2B's CUDA engine now reads its widths from config.json (cuda_geometry.zig): linear value heads over fewer key heads, more query heads a key/value head, an untied head, an A_log stored as bf16. The 27B's logits agree with an fp32 reference at every position of a 36-token check, and its windows equal one-row steps bit for bit; the 2B's output is unchanged. Two Windows driver costs shaped the rest. Each kernel launch grew with the process's allocation count (1.2 ms a launch over 1,900 weight buffers), so a model's weights and scratch now live in one arena (cuda_weights.zig packs each projection through a reused staging buffer: the 27B loads in 25 s, not 200). And a process's first forwards through a buffer stall for seconds, so the engine warms every request path at load and the backend keeps a pool of stream caches that `reserve` allocates and runs a row through once the server knows its stream count. On an RTX 4090: the 27B decodes at 37 tok/s in the runner, 31 to 35 tok/s served, 210 tok/s across eight streams; the 2B stays at 210 and 1,250.
A forward's launches (the embedding to the kept rows' logits) are captured the first time a shape (rows, windows, recording) is seen and replayed after; the attention chunk count moves into the per-forward table so a graph stays valid as caches grow. Prompt chunks stay uncaptured: their GEMMs dwarf the launches, and each length would be a graph of its own. A new graph's first run pays the Windows driver's stall (about a second on a wide model), so `reserve` also captures the shared rounds' shapes, a row of each of 1 to n streams, at load. Bit for bit the same logits; on an RTX 4090 the 2B steps in 2.8 ms instead of 5.3 (served: 324 tok/s, 1,560 across eight streams) and the 27B in 23.8 ms instead of 26.9 (served: 40 tok/s, 243 across eight).
--drafter loads z-lab's DFlash2 draft model behind the lane core's Drafted wrapper: the target writes each row's residual after the drafter's five tapped layers into a ring a stream, the drafter folds those rows into a context of keys and values, and after every verify it drafts a block of seven tokens past the pending one (the target's own embedding and head at its ends, a codebook selector tracing one path through each row's best sixteen). Its projections are quantized to affine-4 on the host at load and run through the target's kernels; its glue is qwen35_draft.cu. The Windows driver stalls a buffer's first kernel and a new graph's first run for seconds, so the drafter pools its contexts and a captured forward now serves every verify shape: its launches cover a bucket of rows and every window, and the kernels read the true counts from the device. On an RTX 4090 the 27B serves 52 to 58 tok/s greedy with drafts against 40 plain, about 100 tok/s on long replies, the same tokens either way; plain decoding keeps its 24 ms step. The 2B is unchanged.
A verify window's rows may form a tree (each row's parent given): the convolution taps, the delta rule and attention follow each row's own branch, and the DFlash2 drafter fills the lane core's planned lanes (each lane its parent's rank-r choice by the codebook selector), so a lone stream verifies up to 12 lanes a round and the planner learns which shapes land. A verify no longer snapshots every row's state (1.3 GB shared, which capped the 27B at 8 rows): each stream logs its rows' decay, write and key, and keep replays the kept path onto the recurrence in one pass (and moves a tree's kept keys down), so any window width fits. One-row steps and prompts update in place as before. The lane host times the target's windows at load, which the depth rule had lacked (it had been capping chains at 3 drafts). The runner's new tree mode verifies tree windows with junk branches against step logits (bit-exact on both models).
What the port serves beyond Nemotron: Qwen3.5-2B and Qwen3.8-27B with its DFlash2 drafter, the speeds measured on an RTX 4090, the load time and why, the timing and chain-only switches, and what is not there.
The lean rule (tools/zig/lean_check.py) counts a comment that runs on to a second line; the Windows port's new files had a hundred such runs. Each is one line of at most 120 columns now, and the checker reports the same 539 problems as main.
direct_io opened files with O_DIRECT on Linux alone. On Windows the same open now asks for FILE_FLAG_NO_BUFFERING (which win_c maps O_DIRECT to), and the 4 KiB-aligned spans it already reads satisfy the sector alignment that needs; a file system that refuses it still falls back to buffered reads. The direct-read test runs on Windows too, checking the executable's bytes against buffered reads of it.
SIG.IGN is a function pointer made from 1, as std.posix has it. aarch64 requires function pointers to be aligned unless the type says align(1), so the cross-build for aarch64-windows refused it. With align(1), as std's own handler type has, tensorfold-native cross-compiles for aarch64-windows with the GB10 (sm_121) fatbins; no hardware has run it.
The Windows page opens with what the build is, what it is tested on and that it cross-compiles for aarch64-windows; the README's model table and the CUDA paragraph say Windows where they said unofficial port.
The no-GPU workflow runs zig build test on windows-latest beside ubuntu-latest, so the platform layer is built and tested on every pull request.
Every other module calls libc through `sys`; the drafter's four getenv calls were the last direct std.c calls in a file the Windows build compiles.
After a prompt the lane core hands the drafter every prompt row's features; the CUDA backend refused any request wider than its tap ring (2,176 rows), so a 27B request with --drafter and a prompt past that ended with RowsNotHeld, where the same prompt without the drafter ran. The ring keeps the newest tap_rows rows, which hold the drafter's whole context window, and absorb already reads only that window of what it is handed; the backend now refuses only rows not yet written or all gone. On an RTX 4090 the 27B with its drafter takes a 6,041-token prompt in 7.4 s, and its reply equals the one with "draft": false.
… attention cuts chunks at logical positions - rowanc held a window's rows as bits of a u32, but prompt chunks run up to 128 rows: rows from 32 on rotated at the wrong positions and attended the wrong keys. A chain row (no parents) now carries its index in its window under the top bit; tree windows (at most 16 rows) keep their bits. - tf_q35_attn_partial cut its 128-key chunks at a key's cache row, so a kept tree path whose window crossed a chunk boundary summed in other chunks than one-row steps and changed bits. Chunks are now cut at the key's logical position (the stream's length plus the ancestors before it). Checked on an RTX 5090 (Linux, sm_120), tf-qwen35-cuda over 300 ids on Qwen3.5-2B and Qwen3.8-27B: step bits unchanged, window and tree equal step byte for byte (tree differed from row 126 on the 2B and 254 on the 27B before). Prompt mode's last row against step's at 16 to 300 ids: KL at most 0.003 and the same argmax in 20 of 20 (before: 27B at 40 ids KL 2.3 with a different argmax, at 128 ids KL 1.35).
zig build test-golden and zig build fuzz-cortheon exist on every non-macOS host since the golden step moved into zig/build/golden.zig, so the golden check compiles against the platform layer on Windows: - the platform's posix gains sockaddr.in, timeval, connect and kill for a child's handle (TerminateProcess; HUP has no equivalent), setsockopt turns a timeval for SO_RCVTIMEO or SO_SNDTIMEO into Winsock's milliseconds, and a socket read that times out is WouldBlock, as std.posix reports it. - the check carries SystemRoot into the spawned server's environment, which Winsock needs to start, and drains the server's stdout after the port line on every host: the fake server logs each request there, and a full pipe (4 KiB on Windows) blocked it after sixteen cases. - on Windows the check cannot send SIGTERM or SIGHUP, so the sigterm-exit-0 outcome and the cases after a key rewrite are skipped and printed as such. On Windows the goldens give 306 of 315 equal with the same nine known differences as Linux, and the fuzz step passes its eight tests; the no-GPU workflow's Windows job runs both.
The loader tried only libcublasLt.so.13 and libcublasLt.so, so the cublaslt suite of tf-cuda-test reported LibraryUnavailable on Windows; there it opens cublasLt64_13.dll from the toolkit's bin folder.
3c8a271 to
ccd06b9
Compare
|
Windows results for the fix, on the RTX 4090 (driver 616.56, CUDA 13.3, Zig 0.17.0), with the branch rebased onto main
The fix's checks,
|
| model | one stream | 4 streams |
|---|---|---|
| Nemotron 3.5 Lightning | 567 / 461 / 564 / 469 | 696 to 1,087 |
Qwen3.5-2B (--context 32768) |
344 / 348 / 405 / 355 | 985 to 1,239 |
Qwen3.8-27B + DFlash2 (--context 8192) |
136 / 83 / 135 / 104 | 167 to 257 |
Prompt speed as the template now asks, tools/bench_cache.py --require-cold (the fixture's 2,571 tokens), cold ttft_s, qualified on each: Nemotron 0.173 s, the 2B 0.416 s, the 27B with the drafter 3.01 s.
The PR body is updated to this tip. Not run here: a GB10 or Ampere card, NVFP4 Nemotron, an ARM Windows machine, Metal serving of the two Qwen models.
Test run on an RTX 4080 SUPER (16 GB, sm_89), Windows 11Head
Qwen3.5-2B decode (tok/s):
Notes:
|
The platform handed O_APPEND to the C runtime as _O_APPEND, whose write seeks to the end and then writes, so two threads appending at once could overwrite each other's bytes: the request log lost lines under concurrent requests, which the server goldens' request-log case caught on a slow runner. A descriptor opened with O_APPEND now writes through WriteFile at the end-of-file offset, which the kernel places whole, as POSIX does; the file keeps its write access, so ftruncate on such a descriptor still works (prompt_imprint repairs a torn index that way). A request_log test appends 200 lines from each of 8 threads and expects every line whole; it lost lines on Windows before this change.
The lifecycle/request-log outcome said only that the lines differ; it now gives both logs' line counts and the first line that differs, cut to 100 bytes each, so a mismatch seen on a CI runner can be read from its log.
A checkout with core.autocrlf, as GitHub's Windows runners make, gives the frozen request log CRLF line endings while the server writes LF, so the lifecycle/request-log case differed at its first line's end on windows-latest and nowhere else. Both logs compare with CRLF read as LF; the other goldens are parsed as JSON and never saw the endings.
|
Three more commits from running the new Windows CI job, whose goldens failed
At this tip: Windows |
|
Quality on this branch, the qwen quality suite (the one the published Spark and 5090 recipes report) against the Windows server on the RTX 4090, thinking off, drafts on,
The 27B on Windows scores as it does on Linux, within a question or two on every check. The 2B is the model it is: it loops on 2 of 8 repetition prompts and reasons through 2 of 10 word problems. The suite's long-context documents are all larger than an 8k window, so that check did not run here. Reports are kept with the receipts. |
|
The suite's long-context check, which needs more than an 8k window, at
Reports beside the others. |
|
Two more measurements on this branch, from the recipe harness (github.com/BobClawblaw/recipe-harness, its Long context at a 64k window, one stream, the suite's ladder up to what the card holds beside the weights (no fp8 KV cache on this branch, so 128k and 256k do not fit the 27B on 24 GB; the 2B runs the whole ladder at 262k):
The harness's quick plan, one stream, greedy, ruler phases prose / code / structured / list:
The soak and turns checks also report that no turn resumed a kept state ( |
Test run on a GB10 (sm_121, aarch64 Linux)The platform this PR lists as not run. Head Setup. One DGX Spark: NVIDIA GB10 (sm_121, aarch64, 128 GB unified memory), Ubuntu kernel 7.0.0-1019-nvidia, driver 580.178.04, Build and host tests. The default install,
Servers,
The GB10's memory bandwidth (273 GB/s) is about a sixth of a 5090's, so the decode rates are lower by about that factor. Startup reports the unified memory correctly ("memory shared with the host"). |
|
Hello @BobClawblaw, Thank you for the Qwen3.5 CUDA family, the RTX 5090 Linux and RTX 4090 Windows runs, and the two kernel fixes you found on the way: chain rows past 32 keep their key index, and tree windows now sum across a 128-key boundary the way one-row steps do. The stdout drain in the golden check helps Linux too. Windows: it builds from source and its host tests run in CI. A Windows release archive waits for a qualified run on a Windows card. The engine: we want one Qwen3.5/3.8 engine on CUDA, and yours is the more native of the two. It uses source kernels, reads its geometry from config.json (so the 2B and the 27B), and drafts through the core drafter. Ours still loads captured Triton kernels for the 27B. So we plan to make yours the base, after a side-by-side on a GB10: tokens under the exactness contract, and decode with DFlash2. Our kept-prompt store and #726's vision tower then move onto it. Thank you again. |
|
Thanks, @BobClawblaw. We want this to become our Qwen CUDA base after the hardware side-by-side, and these are the fixes to prioritize first at ccd06b9. The blocker is Two config checks also need tightening in The earlier chain-row and logical tree-key fixes look right. These findings concern cases beyond the short receipt range; this review did not run the GPU checks. |
…'s widest stream The chunk width came from the longest stream or window in the forward, so a 201-key row cut its keys into 128-key chunks alone and 256-key chunks beside a stream past 8192, and a window crossing 8192 recut its earlier rows; either changed the fp32 sums. Each row now takes chunk_keys(its logical keys), the width its one-row step uses, in both the partial and the merge kernels, and the partials sit at a fixed max_chunks stride. tf-qwen35-cuda takes TF_Q35_PREFILL=N (start after N cycled ids) and TF_Q35_BESIDE=N (step an N-token stream in each forward), which cover both cases: before this change, windows from 8151 keys kept 36 of 120 rows and trees 35, and a 151-key stream beside one from 8189 kept 4 to 11 of 120; after it all of them equal one-row steps on the 2B and 27B.
…fits its selector rank The integer reader accepted zero, so a conv_group_size or num_key_value_heads of 0 reached a remainder at admission; sizes and divisors are now positive (mask_token_id alone may be 0). The host staging area held 16 rows at rank 256, so a larger selector_rank overran it; it is now sized for the admitted rank. Config.parse takes the bytes, so a host test covers both.
|
Thank you. All three are fixed, in two commits on top of
For the checks you asked for,
The "before" column is the old kernels with the new runner, on both models (the same counts on each). Also on the new tip: the old 36-id step references are 36/36 byte-equal on both models (steps under 8192 keys keep their bits), and windows and trees equal steps over 300 ids. Served at #856 is rebased onto this tip; the same threshold checks pass on Qwen3.6-35B-A3B there. |
|
Thank you, @BobClawblaw. We checked Next we run #836 beside our current Qwen CUDA engine on our own GB10s, for tokens, speed and memory. When that passes, #836 becomes the base for Qwen on CUDA, with #856 after it. Posted by Ash's AI Assistant Synthena |
|
@ashhart, before I start image input on this engine (you asked on #856 and when closing #726), I'd like to check the split so neither of us builds the same thing twice. What I'd bring, as one focused change stacked on #856 and #598, for Qwen3.8-27B and Qwen3.6-35B-A3B. Both checkpoints carry the same 27-block bf16 tower; only
Image prompts stay out of prompt reuse. Both drafters keep plain positions, as in 0.6.6. I'd leave the decoder to you. The family would take decoded RGB pixels from your shared libjpeg-turbo decoder once it lands, with resize and patchify after it. Until then I'd test with PNG through stb only. Two questions:
I'll hold until you reply. |
The ashhart#853 merge removed learnBytes (ashhart#836 reads learn_bytes in place); its doc line stayed above toGib. Claude-Session: https://claude.ai/code/session_019t9cKWEuJaAitqMuN9rYFE
…d a trailing blank line - vision_inputs opens libtfvideo through sys.DynLib (LoadLibraryW on Windows, as tfvideo.dll): std.DynLib has no Windows. media_fetch's connect is Linux socket calls; a Windows server refuses image URLs by name. ashhart#836 builds the server's tests on Windows and ashhart#853 brought both, so they met here. - tf-qwen35-cuda's header and two comments within one line and 120 columns. - docs/vision.md ends in one newline (git diff --check). Claude-Session: https://claude.ai/code/session_019t9cKWEuJaAitqMuN9rYFE
|
@ashhart Following up on my question above: #885 now answers both parts. It moves #726's tower onto this engine, and the tower takes its sizes from |
|
Thank you, @BobClawblaw, for extending #885 through vision_config to cover the smaller tower as well. The original three #836 findings remain closed at 5e1c3fe. For the image follow-up, Please reuse Posted by Ash's AI Assistant Synthena |
2 participants
What this changes
TensorFold builds and serves natively on Windows with an NVIDIA GPU, through the same CUDA engine the Linux build uses:
zig build native install -Dsm=89 -Dnvcc=...from a Visual Studio prompt givestensorfold-native.exe,zig build testruns the host tests, andtf-cuda-test.exeruns the GPU suites. The port is a platform layer (zig/src/platform, imported assysby every module) that gives libc's and std.posix's names over the UCRT, Win32 and Winsock, so the engine's code is unchanged apart from callingplatform.cwhere it calledstd.c. On top of it, the Qwen3.5 hybrids (Qwen3.5-2B and Qwen3.8-27B, MLX affine 4-bit) get a CUDA engine of their own with the Metal kernels' arithmetic and rounding points, one CUDA graph per decode shape, and the 27B's DFlash2 drafter drafting trees through the lane core's own planner. docs/WINDOWS.md has the build, the run and how the port works; the README names Windows in the model table.Two things to know when reviewing:
windows-latestjob running the host tests, the server goldens and the Cortheon fuzz step. It found one Windows-only test gap on the way (std's Io has no hard links on Windows; the shard_edit test now uses the platform's link there, which is whatpulluses), and the goldens found that nobody drainedfake_serve's stdout, which a 4 KiB pipe turns into a blocked server.Groundwork beyond the RTX 4090: the host code cross-compiles for
aarch64-windowswith the GB10 (sm_121) fatbins (one function-pointer type neededalign(1)for it, as std's own handler type has). No ARM Windows machine has run it.Commits, in order: the platform layer and host tests; the kernels, native server and GPU suites on sm_89;
pullinto the hub cache (hard links where symlinks need privilege); build notes; the default port; Qwen3.5-2B on CUDA; Qwen3.8-27B with one device arena; CUDA graphs per decode shape; DFlash2 drafts; tree drafts with the kept path replayed from a log; docs; the lean rule's one-line comments; direct (unbuffered) checkpoint reads on Windows; the aarch64 handler type; README and Windows page wording; the Windows CI job; the drafter reading TF_TIMING through the platform layer; the drafter taking a prompt longer than its tap ring; the prompt-chunk and tree-attention fix found on a Linux RTX 5090 (the comments below have its receipt); the server goldens and the fuzz step on Windows hosts; cuBLASLt by its Windows DLL name; writes on an O_APPEND descriptor landing at the end whole (the goldens' request-log case caught concurrent appends overwriting each other on a slow runner; a request_log test now appends from 8 threads); the golden check naming a request log's first differing line, and comparing the logs with LF endings, since an autocrlf checkout gives the frozen one CRLF.If you would rather take this in two pull requests (the platform port, then the Qwen3.5 CUDA engine), say so and I will split it.
Receipt
Environment: this branch on main 2984ea0, Zig 0.17.0, CUDA 13.3
nvccwith Visual Studio 2026 Build Tools, Windows 11 Pro build 26300, driver 616.56, RTX 4090 (24 GB, sm_89, WDDM). Checkpoints:TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit,mlx-community/Qwen3.5-2B-MLX-4bit,Qwen3.8-27B-MLX-4bitwith theQwen3.8-27B-DFlash2drafter (4-bit). Mac: M5 Ultra, macOS 27. Linux: RTX 5090 (sm_120), Ubuntu, CUDA 13.4 driver, in the comments below.Exactness,
tools/bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --alone --serialon each model (code and chat prompts, temperature 1.0 seeded and 0): every reply'stoken_shaequal alone and at 4 streams, and equal to"draft": false; 0 unequal, 0 failed, for Nemotron, the 2B and the 27B with its drafter.tf-qwen35-cudaover 300 ids on both Qwen models:windowandtreeequalstepbyte for byte;prompt's last row at 16 to 300 ids againststep's has KL at most 0.002 with the same argmax in 20 of 20. On Linux, all 40 Nemotrontoken_shaequal main's.Decode speed,
tools/bench_openai.py --tokens 128 --reps 3(median tok/s, one stream; no "before" exists on Windows, and on Linux Nemotron's numbers equal main's within noise):The 27B plain (no drafter) decodes at about 40 tok/s, the card's bandwidth for 16 GiB of weights.
Prompt speed:
tools/bench_cache.py --require-cold(its 2,571-token fixture), the cold first request'sttft_s, qualified on each model: Nemotron 0.173 s, the 2B 0.416 s, the 27B with the drafter 3.01 s. No "before" exists on Windows; no shared prompt path changes.Tests run:
zig build teston Windows with a fresh cache: 733 tests, 719 passed, 14 skipped, 0 failed (the skips are symlink-privilege, mode-bit and Linux-only cases).zig build test-goldenon Windows: 306 of 315 equal, 0 unexpected differences, the same nine known ones as Linux, the 4 signal-driven cases skipped and printed as such.zig build fuzz-cortheonon Windows: 8 of 8.tf-cuda-testinfo, smoke, graph, launch-ex, ptx, symbols, cublaslt, norms, norm-rows, sample, glue (1,242 of 1,242 byte-equal), pocket, quant-ref, modelopt, exlr8 and exlr8-dense on the RTX 4090: all pass. On the M5 Ultra:zig build test1,774 tests, 1,766 passed, 8 skipped, 0 failed;test-golden309 of 319 equal, 0 unexpected;zig build tensorfoldbuilds;tf-metal-tests6 of 6.tools/zig/lean_check.py: 528 problems on this branch and on main, none added.zig fmt --check build.zig zig: clean. The aarch64-windows cross-build of the native server links.Not run, and why: the Qwen3.5 CUDA engine on a GB10 or an Ampere card (sm_89 and sm_120 only). NVFP4 Nemotron (needs sm_90+; the 5090 run was the MLX 4-bit checkpoint). An ARM Windows machine. Metal serving of the 2B and 27B on the Mac (the checkpoints are not on it), so the shared
draw.zigsplit and the drafter contract's new optional fields are covered there by the host tests and the Metal build alone.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Receipt above; Nemotron's prompt cache on Windows is the Linux code, and the Qwen3.5 engine has no prompt cache yet, so resumed equals fresh by construction.)tf-qwen35-cudaneeds a checkpoint and a GPU.)