Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836)

.github/workflows/no-gpu.yml

@@ -38,3 +38,18 @@ jobs:
           (cd ../lean-base && python3 tools/zig/lean_check.py > ../lean-base.txt || true)
           python3 tools/zig/lean_check.py > ../lean-head.txt || true
           python3 tools/zig/lean_new.py ../lean-base.txt ../lean-head.txt
+  zig-windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: mlugg/setup-zig@v2
+        with:
+          version: 0.17.0
+      # The same host unit tests on Windows: the platform layer (zig/src/platform) over the UCRT, Win32 and Winsock.
+      - name: zig build test
+        run: zig build test
+      # The server goldens and the Cortheon fuzz harness on Windows too; the goldens skip the signal-driven cases.
+      - name: zig build test-golden
+        run: zig build test-golden
+      - name: zig build fuzz-cortheon
+        run: zig build fuzz-cortheon

CONTRIBUTING.md

@@ -54,9 +54,9 @@ Record pass, fail and skip counts and explain any unrun platform or feature.
 For served decode, the existing standard-library client needs Python 3.11 or newer:
 
 ```sh
-python3 tools/bench_openai.py http://127.0.0.1:8080 local-model \
+python3 tools/bench_openai.py http://127.0.0.1:8888 local-model \
   --tokens 256 --reps 3 --temperatures 0 --label candidate --output decode.json
-python3 tools/bench_concurrent.py http://127.0.0.1:8080 local-model \
+python3 tools/bench_concurrent.py http://127.0.0.1:8888 local-model \
   --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial --output concurrent.json
 ```
 

README.md

@@ -23,15 +23,17 @@ tensorfold serve "$HOME/models/nemotron-lightning" \
 
 Use a complete local model directory or an existing Hugging Face cache.
 The binary in a release archive is `bin/tensorfold-native`; the [runbook](RUNBOOK.md) covers archive installation and CUDA runtime files.
+
+On Windows with an NVIDIA GPU, build `tensorfold-native.exe` from source as [docs/WINDOWS.md](docs/WINDOWS.md) describes.
 Use `tensorfold-native --help` or `-h`, and `tensorfold-native capabilities --json`, to inspect the installed binary.
 Model weights have separate downloads and licenses.
 
-The API listens at `http://127.0.0.1:8080/v1` by default.
+The API listens at `http://127.0.0.1:8888/v1` by default.
 It serves OpenAI chat completions, completions and Responses, plus Anthropic Messages.
 Use the model ID returned by `/v1/models`, or set one with `--name`.
 
 ```sh
-curl -fsS http://127.0.0.1:8080/v1/chat/completions \
+curl -fsS http://127.0.0.1:8888/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
 ```
@@ -47,11 +49,11 @@ tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
 src=$(ls -d ~/.cache/huggingface/hub/models--TensorFold--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit/snapshots/* | head -1)
 mkdir -p ~/models && cp -cRL "$src" ~/models/nemotron-living   # always teach a copy
 tensorfold serve ~/models/nemotron-living --slide
-curl -N http://127.0.0.1:8080/v1/slide/learn -H 'Content-Type: application/json' -d '{"text": "I like blue."}'
+curl -N http://127.0.0.1:8888/v1/slide/learn -H 'Content-Type: application/json' -d '{"text": "I like blue."}'
 ```
 
 Send it a document instead and it finds the facts in it and learns them one by one. Open
-`http://127.0.0.1:8080/slide` to watch in 3D: the facts, the neurons each one took in the model's layers, the shard
+`http://127.0.0.1:8888/slide` to watch in 3D: the facts, the neurons each one took in the model's layers, the shard
 files that hold them, and the lesson running now.
 
 Learning runs on Apple-silicon Macs and on an NVIDIA GB10 (DGX Spark), and a folder that learned on one serves on the
@@ -71,8 +73,9 @@ The platform column names the hardware tested for each model.
 | Qwen3.8 Flash Next | MLX affine 6-bit, group 32 | Metal on M5 Ultra |
 | GLM-5.3-Flash | MLX affine 4-bit, group 64 | Metal on two M5 Ultras |
 | GLM-5.3 | EXLR8 `home` profile (trellis K2/K3/K4 experts) | Metal on two M5 Ultras |
-| Qwen3.5-2B | Pinned MLX affine 4-bit, group 64, tied embeddings | Metal on M5 Max |
-| Qwen3.8-27B | MLX affine 4-bit, group 64, own output head; DFlash2 drafter | Metal on M5 Max and M3 Ultra |
+| Qwen3.5-2B | Pinned MLX affine 4-bit, group 64, tied embeddings | Metal on M5 Max; CUDA on RTX 4090 (Windows) |
+| Qwen3.8-27B | MLX affine 4-bit, group 64, own output head; DFlash2 drafter | Metal on M5 Max and M3 Ultra; CUDA on RTX 4090 (Windows) |
+| Qwen3.6-35B-A3B | MLX affine 4-bit, group 64, 8-bit routers; its MTP layer in `mtp-4bit.safetensors` | CUDA on RTX 4090 (Windows) |
 
 Nemotron's named checkpoint is `TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit`.
 NVIDIA's NVFP4 checkpoint, `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4`, serves on CUDA as stored; its bf16 MTP head is quantized to 4-bit at load for drafts.
@@ -88,6 +91,7 @@ Its prompt processing trails mlx_lm for now, and we are fixing it. An M3 Ultra r
 Bonsai, Gemma 4, Qwen3.6 and DeepSeek-V4 are still under qualification for 1.0.x.
 The Python 0.6.6 engine remains available at the `v0.6.6` tag for those models and other backends.
 On CUDA, 1.0.11 serves Nemotron on a GB10, greedy and sampled, with concurrent requests sharing each round.
+On Windows ([docs/WINDOWS.md](docs/WINDOWS.md)), the native build also serves the 2B and the 27B on CUDA, the 27B with its DFlash2 drafter drafting trees, and Qwen3.6-35B-A3B with its MTP layer drafting chains.
 The Linux x86_64 archive adds NVIDIA Ampere cards with compute capability 8.6 (the RTX 30 series, RTX A6000, A10 and A40), tested on an RTX 3090, and Blackwell cards with compute capability 12.0 (the RTX 50 series and RTX PRO 6000 Blackwell), tested on an RTX PRO 6000; the A100 (8.0) is not supported yet.
 
 ## Exact decoding
@@ -150,7 +154,7 @@ The binary's `capabilities --json` response lists its supported flags and platfo
 
 | Flag | Meaning |
 | --- | --- |
-| `--host HOST`, `--port PORT` | Listen address and port, default `127.0.0.1:8080`. |
+| `--host HOST`, `--port PORT` | Listen address and port, default `127.0.0.1:8888`. |
 | `--name NAME` | Model ID advertised to clients. |
 | `--alias NAME` | Another accepted model ID; repeat for multiple aliases. |
 | `--api-key KEY` | Require a bearer key; repeat for multiple keys. |

RUNBOOK.md

@@ -104,14 +104,14 @@ bin/tensorfold-native serve "$HOME/models/nemotron-lightning" \
 Leave the server running and check it from another terminal:
 
 ```sh
-curl -fsS http://127.0.0.1:8080/health
-curl -fsS http://127.0.0.1:8080/v1/models
-curl -fsS http://127.0.0.1:8080/v1/chat/completions \
+curl -fsS http://127.0.0.1:8888/health
+curl -fsS http://127.0.0.1:8888/v1/models
+curl -fsS http://127.0.0.1:8888/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
 ```
 
-Use `http://127.0.0.1:8080/v1` as the OpenAI client base URL.
+Use `http://127.0.0.1:8888/v1` as the OpenAI client base URL.
 For streaming, add `"stream": true` to the request.
 The native server also has `/v1/responses` and Anthropic `/v1/messages` routes.
 Serve flags and request overrides are described in [README.md](README.md#serve-flags).

build.zig

@@ -12,6 +12,7 @@ const counters_build = @import("zig/build/metal_counters.zig");
 const exl3_build = @import("zig/build/exl3.zig");
 const exlr8_build = @import("zig/build/exlr8.zig");
 const golden_build = @import("zig/build/golden.zig");
+const sys_build = @import("zig/build/sys.zig");
 
 comptime {
     const required = std.mem.trim(u8, @embedFile(".zig-version"), "\r\n");
@@ -153,13 +154,13 @@ pub fn build(b: *std.Build) void {
     test_step.dependOn(&cost_run.step);
     switch (target.result.os.tag) {
         .macos => metalTargets(b, target, optimize, draft_ids, build_options, test_step),
-        .linux => cuda_build.targets(b, target, optimize, draft_ids, build_options),
+        .linux, .windows => cuda_build.targets(b, target, optimize, draft_ids, build_options),
         else => {},
     }
     const xgrammar = grammar_build.option(b);
     dist_build.targets(b, draft_ids, build_options, release_version, xgrammar);
     cuda_build.hostTests(b, draft_ids, build_options, test_step, target.result.os.tag != .macos);
-    if (target.result.os.tag == .linux) {
+    if (target.result.os.tag == .linux or target.result.os.tag == .windows) {
         // the engine seam's tests (lane_host.zig and its siblings), which the macOS branch runs from its own api module
         const lanes_mod = b.createModule(.{ .root_source_file = b.path("zig/src/core/lanes/lanes.zig"), .target = target, .optimize = optimize, .link_libc = true });
         const api_mod = b.createModule(.{ .root_source_file = b.path("zig/src/core/engine_api.zig"), .target = target, .optimize = optimize, .link_libc = true, .imports = &.{.{ .name = "lanes", .module = lanes_mod }} });
@@ -189,6 +190,7 @@ pub fn build(b: *std.Build) void {
     test_step.dependOn(&warning_run.step);
     b.step("test-learn-warning", "CPU production-log refusal, backoff and recovery test").dependOn(&warning_run.step);
     grammar_build.steps(b, target, xgrammar);
+    sys_build.wire(b);
 }
 
 /// `zig build native -Dcpu=apple_m1`: tensorfold-native with the Metal engines for the Python package's bundle (a native M5 build traps on M1-M4).

docs/WINDOWS.md

@@ -0,0 +1,90 @@
+# TensorFold on Windows (native)
+
+`tensorfold-native.exe` runs on Windows with an NVIDIA GPU through the same CUDA engine the Linux build uses. Only the
+CUDA backend is built on Windows; Metal is macOS's and HIP is not built. It is tested on an RTX 4090 (x86_64); the
+host code also cross-compiles for `aarch64-windows` with the GB10 (sm_121) kernels, untested on hardware so far.
+
+## What you need
+
+- An NVIDIA GPU and a current display driver (it ships `nvcuda.dll`; no CUDA toolkit is needed to run).
+- To build: [Zig 0.17.0](https://ziglang.org/download/) (the version in `.zig-version`), the CUDA Toolkit
+  (13.x tested) for `nvcc`, and Visual Studio Build Tools with the C++ workload (`cl.exe`, nvcc's host compiler).
+
+## Build
+
+From a "x64 Native Tools" prompt (or after running `vcvars64.bat`), so `cl.exe` is on `PATH`:
+
+```bat
+zig build native install -Dsm=89 -Dstrip=true ^
+  -Dnvcc="%CUDA_PATH%\bin\nvcc.exe"
+```
+
+`-Dsm` takes the GPU's compute capability without the dot, comma separated for several: 86 (RTX 30), 89 (RTX 40),
+120 (RTX 50). The server is `zig-out\native\bin\tensorfold-native.exe`; `zig-out\bin\tf-cuda-test.exe` runs the GPU
+suites (`tf-cuda-test info`, `smoke`, `glue`, `sample`, `quant-ref`, `modelopt`, `exlr8`).
+
+`zig build test` runs the host tests without a GPU.
+
+## Run
+
+```bat
+tensorfold-native pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
+tensorfold-native models
+tensorfold-native serve <model path> --name local-model --parallel 1 --context 8192 --temperature 0 --no-thinking
+```
+
+The server listens on `http://127.0.0.1:8888/v1`. Ctrl+C (or closing the console) stops it cleanly.
+
+## Models on CUDA
+
+Nemotron 3.5 Lightning serves as on Linux. The Qwen3.5 hybrids have a CUDA engine of their own in this port
+(`zig/src/families/qwen3_5/cuda*.zig`, kernels `zig/kernels/cuda/qwen35*.cu`), the Metal kernels' arithmetic with
+the same rounding points, so a window of up to 16 rows reproduces one-token steps bit for bit:
+
+```bat
+tensorfold-native serve <Qwen3.5-2B-MLX-4bit> --name qwen2b --context 32768
+tensorfold-native serve <Qwen3.8-27B-MLX-4bit> --name qwen27b --context 8192 --parallel 4 ^
+  --drafter <Qwen3.8-27B-DFlash2>
+tensorfold-native serve <Qwen3.6-35B-A3B-MLX-4bit-MTP> --name qwen36 --context 8192 --parallel 2 ^
+  --drafter <Qwen3.6-35B-A3B-MLX-4bit-MTP>
+```
+
+- Qwen3.5-2B: about 340 tokens/s for one request on an RTX 4090, 1,500 across eight.
+- Qwen3.8-27B (16 GiB): 40 tokens/s plain, at the card's memory bandwidth; 240 across eight requests.
+- `--drafter` (4-bit only) loads z-lab's DFlash2 draft model: the target writes five layers' residuals into a ring
+  for it, and it drafts a tree of up to 12 lanes a round that the round loop's planner shapes from what the target
+  took. A lone greedy request decodes at 80 to 160 tokens/s by how predictable the text is (prose at the low end,
+  code at the high), sampled ones at 50 to 70, and four at once reach about 160 aggregate; replies stay identical to
+  plain decoding. Loading takes about 80 s: the drafter's weights are quantized to 4 bits on the host and every
+  shape is run once, as the driver stalls a process's first use of each kernel and buffer.
+- Qwen3.6-35B-A3B (`qwen3_5_moe`, 18.6 GiB): the same layers with 256 routed experts, 8 a token, in place of the
+  dense MLP. Routing follows the frozen engine's `cuda/moe.py` rule, and the experts run through the grouped affine-4
+  kernel's SwiGLU form, the shared expert last in the same tables. About 100 tokens/s plain for one request. With
+  `--drafter` naming the checkpoint's own directory, its MTP layer (`mtp-4bit.safetensors`) drafts chains of three
+  from each kept row's final normed state: 230 to 260 tokens/s greedy, 180 to 250 sampled, 420 aggregate across two
+  requests; replies stay identical to plain decoding. `TF_MTP_DRAFTS=N` sets the chain's length, and `TF_MTP_TREES=1`
+  drafts the planner's trees instead, which measured no faster than chains on this head.
+- A verify window's rows are not snapshotted: each stream logs its rows' recurrence updates and a keep replays the
+  accepted path, so the 27B records 32 rows a forward where 8 fit before.
+- `TF_TIMING=1` prints each verify, keep, absorb and hold's milliseconds and the drafted shape (chain or lanes);
+  `TF_NO_TREES=1` drafts chains only.
+- No prompt cache (`--prompt-cache-gib`), `--learn` or `--slide` for these two models.
+
+## How the port works
+
+- `zig/src/platform/sys.zig` (`@import("sys")`, wired into every module by `zig/build/sys.zig`) is the operating
+  system as the engine calls it. On Linux and macOS it is `std.c`, `std.posix` and `std.DynLib` unchanged; on Windows
+  `win_c.zig` and `win_posix.zig` give the same names over the UCRT, Win32 and Winsock.
+- Files are opened with every share mode, so rename-over and unlink-while-open behave as on POSIX. Checkpoint reads
+  skip the page cache (`FILE_FLAG_NO_BUFFERING`, as `O_DIRECT` on Linux). `/tmp` is the user's temporary folder and
+  `HOME` falls back to `USERPROFILE`.
+- The Hugging Face cache links snapshot files to blobs. Windows allows symlinks only with Developer Mode or elevation,
+  so `pull` falls back to hard links (no extra space).
+- Checkpoint reads use threads (io_uring is Linux's).
+- The API key file is checked for type and size; Windows keeps access in ACLs, not mode bits, so the "unreadable by
+  other users" check is the folder's ACL rather than `chmod 600`.
+
+## Differences from Linux
+
+- No io_uring, `ulimit` or GB10 carveout, and no `SIGHUP` (the key file is still reread when it changes).
+- `TF_HEAT_*` thermal pacing reads Linux's `/sys/class/thermal` and does nothing on Windows.

docs/api.md

@@ -1,6 +1,6 @@
 # Compatible APIs
 
-The base URL is `http://127.0.0.1:8080/v1` with the default server settings.
+The base URL is `http://127.0.0.1:8888/v1` with the default server settings.
 
 ## Native chat template override
 
@@ -480,5 +480,5 @@ redacted thinking and other context edits return HTTP 400.
 Connect Claude Code using the ID from `/v1/models`:
 
 ```bash
-ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_API_KEY=local claude --model local-model
+ANTHROPIC_BASE_URL=http://127.0.0.1:8888 ANTHROPIC_API_KEY=local claude --model local-model
 ```

docs/cache-benchmarks.md

@@ -23,7 +23,7 @@ The standard-library-only cache client runs one first request, identical repeate
 concurrent group using the same prompt, greedy temperature, seed and reply limit. There is no warm-up:
 
 ```sh
-python3 tools/bench_cache.py http://127.0.0.1:8080 MODEL \
+python3 tools/bench_cache.py http://127.0.0.1:8888 MODEL \
   --tokens 64 --repeats 2 --streams 4 --require-cold --output cache.json
 ```
 
@@ -36,7 +36,7 @@ prefix retention (`--prompt-cache-gib 0`, where supported) for independent cold-
 For exact 2K/8K/32K/64K raw prefill, supply a JSON array of IDs from the checkpoint's own tokenizer:
 
 ```sh
-python3 tools/bench_cache.py http://127.0.0.1:8080 MODEL \
+python3 tools/bench_cache.py http://127.0.0.1:8888 MODEL \
   --prompt-file public-8192-tokens.json --expected-prompt-tokens 8192 \
   --tokens 64 --require-cold --output cache-8192.json
 ```

docs/control.md

@@ -118,7 +118,7 @@ The error from `tensorfold tui` prints that command with the venv's Python. `ric
 ```bash
 tensorfold tui
 tensorfold tui --profile code
-tensorfold tui --url http://127.0.0.1:8080/v1
+tensorfold tui --url http://127.0.0.1:8888/v1
 tensorfold tui --url https://spark.example.invalid --token-env TENSORFOLD_API_KEY
 tensorfold tui --demo
 tensorfold tui --demo --snapshot preview.svg

docs/cortheon.md

@@ -100,8 +100,8 @@ tensorfold serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --cor
 ```
 
 ```bash
-curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
+curl -s http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
   -d '{"messages": [{"role": "user", "content": "What changed in the latest Zig release?"}]}'
 ```
 
-Point any OpenAI-compatible client at `http://127.0.0.1:8080/v1` as before.
+Point any OpenAI-compatible client at `http://127.0.0.1:8888/v1` as before.