qwen3_5 cuda: mixed-width MLX checkpoints, 2 to 8 bits in groups of 64

.github/workflows/no-gpu.yml

@@ -41,3 +41,18 @@ jobs:
           (cd ../lean-base && python3 tools/zig/lean_check.py > ../lean-base.txt || true)
           python3 tools/zig/lean_check.py > ../lean-head.txt || true
           python3 tools/zig/lean_new.py ../lean-base.txt ../lean-head.txt
+  zig-windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: mlugg/setup-zig@v2
+        with:
+          version: 0.17.0
+      # The same host unit tests on Windows: the platform layer (zig/src/platform) over the UCRT, Win32 and Winsock.
+      - name: zig build test
+        run: zig build test
+      # The server goldens and the Cortheon fuzz harness on Windows too; the goldens skip the signal-driven cases.
+      - name: zig build test-golden
+        run: zig build test-golden
+      - name: zig build fuzz-cortheon
+        run: zig build fuzz-cortheon

CONTRIBUTING.md

@@ -55,9 +55,9 @@ Record pass, fail and skip counts and explain any unrun platform or feature.
 For served decode, the existing standard-library client needs Python 3.11 or newer:
 
 ```sh
-python3 tools/bench_openai.py http://127.0.0.1:8080 local-model \
+python3 tools/bench_openai.py http://127.0.0.1:8888 local-model \
   --tokens 256 --reps 3 --temperatures 0 --label candidate --output decode.json
-python3 tools/bench_concurrent.py http://127.0.0.1:8080 local-model \
+python3 tools/bench_concurrent.py http://127.0.0.1:8888 local-model \
   --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial --output concurrent.json
 ```
 

README.md

@@ -23,15 +23,17 @@ tensorfold serve "$HOME/models/nemotron-lightning" \
 
 Use a complete local model directory or an existing Hugging Face cache.
 The binary in a release archive is `bin/tensorfold-native`; the [runbook](RUNBOOK.md) covers archive installation and CUDA runtime files.
+
+On Windows with an NVIDIA GPU, build `tensorfold-native.exe` from source as [docs/WINDOWS.md](docs/WINDOWS.md) describes.
 Use `tensorfold-native --help` or `-h`, and `tensorfold-native capabilities --json`, to inspect the installed binary.
 Model weights have separate downloads and licenses.
 
-The API listens at `http://127.0.0.1:8080/v1` by default.
+The API listens at `http://127.0.0.1:8888/v1` by default.
 It serves OpenAI chat completions, completions and Responses, plus Anthropic Messages.
 Use the model ID returned by `/v1/models`, or set one with `--name`.
 
 ```sh
-curl -fsS http://127.0.0.1:8080/v1/chat/completions \
+curl -fsS http://127.0.0.1:8888/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
 ```
@@ -48,12 +50,12 @@ src=$(ls -d ~/.cache/huggingface/hub/models--TensorFold--NVIDIA-Nemotron-3.5-Lig
 mkdir -p ~/models && cp -cRL "$src" ~/models/nemotron-living   # always teach a copy
 (umask 077; openssl rand -hex 24 > ~/models/slide-key)          # --slide starts only with a key
 tensorfold serve ~/models/nemotron-living --slide --api-key-file ~/models/slide-key
-curl -N http://127.0.0.1:8080/v1/slide/learn -H "Authorization: Bearer $(cat ~/models/slide-key)" \
+curl -N http://127.0.0.1:8888/v1/slide/learn -H "Authorization: Bearer $(cat ~/models/slide-key)" \
   -H 'Content-Type: application/json' -d '{"text": "I like blue."}'
 ```
 
 Send it a document instead and it finds the facts in it and learns them one by one. Open
-`http://127.0.0.1:8080/slide` to watch in 3D: the facts, the neurons each one took in the model's layers, the shard
+`http://127.0.0.1:8888/slide` to watch in 3D: the facts, the neurons each one took in the model's layers, the shard
 files that hold them, and the lesson running now.
 
 Learning runs on Apple-silicon Macs and on an NVIDIA GB10 (DGX Spark), and a folder that learned on one serves on the
@@ -76,8 +78,8 @@ The platform column names the hardware tested for each model.
 | Qwen3.8 Flash Next | MLX affine 6-bit, group 32 | Metal on M5 Ultra |
 | GLM-5.3-Flash | MLX affine 4-bit, group 64 | Metal on two M5 Ultras |
 | GLM-5.3 | EXLR8 `home` profile (trellis K2/K3/K4 experts) | Metal on two M5 Ultras |
-| Qwen3.5-2B | Pinned MLX affine 4-bit, group 64, tied embeddings | Metal on M5 Max |
-| Qwen3.8-27B | MLX affine 4-bit, group 64, own output head; DFlash2 drafter; images with `--vision` ([image input](docs/vision.md#qwen38-27b)) | Metal on M5 Max and M3 Ultra |
+| Qwen3.5-2B | Pinned MLX affine 4-bit, group 64, tied embeddings | Metal on M5 Max; CUDA on RTX 4090 (Windows) |
+| Qwen3.8-27B | MLX affine 4-bit, group 64, own output head; DFlash2 drafter; images with `--vision` ([image input](docs/vision.md#qwen38-27b)) | Metal on M5 Max and M3 Ultra; CUDA on RTX 4090 (Windows) |
 | LFM2.5-24B-A2B | MLX affine 4-, 6- or 8-bit, group 64, tied embeddings | Metal on M4 Pro |
 
 Nemotron's named checkpoint is `TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit`.
@@ -110,6 +112,7 @@ Bonsai, Gemma 4, Qwen3.6 and DeepSeek-V4 are still under qualification for 1.0.x
 The Python 0.6.6 engine remains available at the `v0.6.6` tag for those models and other backends.
 On CUDA the native server runs Nemotron 3.5 Lightning on a GB10, an RTX 3090 and an RTX PRO 6000, and Nemotron 3.5 Super VL with images on one GB10 or two, greedy and sampled, with concurrent requests sharing each round.
 GLM-5.3-Flash also serves across two CUDA ranks, and Liquid AI's d1-3B answers `/v1/systemone` and `/v1/decisions`.
+On Windows ([docs/WINDOWS.md](docs/WINDOWS.md)), the native build also serves the 2B and the 27B on CUDA, the 27B with its DFlash2 drafter drafting trees.
 The Linux x86_64 archive adds NVIDIA Ampere cards with compute capability 8.0 (the A100) and 8.6 (the RTX 30 series, RTX A6000, A10 and A40; tested on an RTX 3090), and Blackwell cards with compute capability 12.0 (the RTX 50 series and RTX PRO 6000 Blackwell), tested on an RTX PRO 6000; FP8 and NVFP4 checkpoints need compute capability 9.0 or newer.
 
 ## Exact decoding
@@ -177,7 +180,7 @@ The binary's `capabilities --json` response lists its supported flags and platfo
 
 | Flag | Meaning |
 | --- | --- |
-| `--host HOST`, `--port PORT` | Listen address and port, default `127.0.0.1:8080`. |
+| `--host HOST`, `--port PORT` | Listen address and port, default `127.0.0.1:8888`. |
 | `--name NAME` | Model ID advertised to clients. |
 | `--alias NAME` | Another accepted model ID; repeat for multiple aliases. |
 | `--api-key KEY` | Require a bearer key; repeat for multiple keys. |

RUNBOOK.md

@@ -104,14 +104,14 @@ bin/tensorfold-native serve "$HOME/models/nemotron-lightning" \
 Leave the server running and check it from another terminal:
 
 ```sh
-curl -fsS http://127.0.0.1:8080/health
-curl -fsS http://127.0.0.1:8080/v1/models
-curl -fsS http://127.0.0.1:8080/v1/chat/completions \
+curl -fsS http://127.0.0.1:8888/health
+curl -fsS http://127.0.0.1:8888/v1/models
+curl -fsS http://127.0.0.1:8888/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
 ```
 
-Use `http://127.0.0.1:8080/v1` as the OpenAI client base URL.
+Use `http://127.0.0.1:8888/v1` as the OpenAI client base URL.
 For streaming, add `"stream": true` to the request.
 The native server also has `/v1/responses` and Anthropic `/v1/messages` routes.
 Serve flags and request overrides are described in [README.md](README.md#serve-flags).

build.zig

@@ -18,6 +18,7 @@ const server_build = @import("zig/build/server.zig");
 const golden_build = @import("zig/build/golden.zig");
 const contract_build = @import("zig/build/kernel_contract.zig");
 const video_build = @import("zig/build/video.zig");
+const sys_build = @import("zig/build/sys.zig");
 
 comptime {
     const required = std.mem.trim(u8, @embedFile(".zig-version"), "\r\n");
@@ -163,14 +164,14 @@ pub fn build(b: *std.Build) void {
     merge_build.targets(b, target, optimize); // tf-merge-check and test-merge, host only on every OS
     switch (target.result.os.tag) {
         .macos => metalTargets(b, target, optimize, draft_ids, build_options, test_step),
-        .linux => cuda_build.targets(b, target, optimize, draft_ids, build_options),
+        .linux, .windows => cuda_build.targets(b, target, optimize, draft_ids, build_options),
         else => {},
     }
     const xgrammar = grammar_build.option(b);
     dist_build.targets(b, draft_ids, build_options, release_version, xgrammar);
     cuda_build.hostTests(b, draft_ids, build_options, test_step, target.result.os.tag != .macos);
     xpu_build.hostTests(b, test_step);
-    if (target.result.os.tag == .linux) {
+    if (target.result.os.tag == .linux or target.result.os.tag == .windows) {
         // the engine seam's tests (lane_host.zig and its siblings), which the macOS branch runs from its own api module
         const lanes_mod = b.createModule(.{ .root_source_file = b.path("zig/src/core/lanes/lanes.zig"), .target = target, .optimize = optimize, .link_libc = true });
         const api_mod = b.createModule(.{ .root_source_file = b.path("zig/src/core/engine_api.zig"), .target = target, .optimize = optimize, .link_libc = true, .imports = &.{.{ .name = "lanes", .module = lanes_mod }} });
@@ -204,6 +205,7 @@ pub fn build(b: *std.Build) void {
     b.step("test-learn-warning", "CPU production-log refusal, backoff and recovery test").dependOn(&warning_run.step);
     grammar_build.steps(b, target, xgrammar);
     contract_build.hostTests(b);
+    sys_build.wire(b);
 }
 
 /// `zig build native -Dcpu=apple_m1`: tensorfold-native with the Metal engines for the Python package's bundle (a native M5 build traps on M1-M4).
@@ -557,7 +559,8 @@ fn metalTargets(b: *std.Build, target: std.Build.ResolvedTarget, optimize: std.b
     test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = helpers })).step);
     test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = engine })).step);
     test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = lanes })).step);
-    test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = fabric })).step);
+    // the fake fabric maps its windows with mmap: the Macs' RDMA, not Windows
+    if (target.result.os.tag != .windows) test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = fabric })).step);
     const dashboard = b.createModule(.{ .root_source_file = b.path("zig/src/server/dashboard_test.zig"), .target = target, .optimize = optimize, .link_libc = true });
     test_step.dependOn(&b.addRunArtifact(b.addTest(.{ .root_module = dashboard })).step);
     const kimi_test = b.step("test-k3", "Kimi K3's host-side unit tests (no GPU work)");

docs/WINDOWS.md

@@ -0,0 +1,99 @@
+# TensorFold on Windows (native)
+
+`tensorfold-native.exe` runs on Windows with an NVIDIA GPU through the same CUDA engine the Linux build uses. Only the
+CUDA backend is built on Windows; Metal is macOS's and HIP is not built. It is tested on an RTX 4090 (x86_64); the
+host code also cross-compiles for `aarch64-windows` with the GB10 (sm_121) kernels, untested on hardware so far.
+
+## What you need
+
+- An NVIDIA GPU and a current display driver (it ships `nvcuda.dll`; no CUDA toolkit is needed to run).
+- To build: [Zig 0.17.0](https://ziglang.org/download/) (the version in `.zig-version`), the CUDA Toolkit
+  (13.x tested) for `nvcc`, and Visual Studio Build Tools with the C++ workload (`cl.exe`, nvcc's host compiler).
+
+## Build
+
+From a "x64 Native Tools" prompt (or after running `vcvars64.bat`), so `cl.exe` is on `PATH`:
+
+```bat
+zig build native install -Dsm=89 -Dstrip=true ^
+  -Dnvcc="%CUDA_PATH%\bin\nvcc.exe"
+```
+
+`-Dsm` takes the GPU's compute capability without the dot, comma separated for several: 86 (RTX 30), 89 (RTX 40),
+120 (RTX 50). The server is `zig-out\native\bin\tensorfold-native.exe`; `zig-out\bin\tf-cuda-test.exe` runs the GPU
+suites (`tf-cuda-test info`, `smoke`, `glue`, `sample`, `quant-ref`, `affine8`, `modelopt`, `exlr8`).
+
+`zig build test` runs the host tests without a GPU.
+
+## Run
+
+```bat
+tensorfold-native pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
+tensorfold-native models
+tensorfold-native serve <model path> --name local-model --parallel 1 --context 8192 --temperature 0 --no-thinking
+```
+
+The server listens on `http://127.0.0.1:8888/v1`. Ctrl+C (or closing the console) stops it cleanly.
+
+## Models on CUDA
+
+Nemotron 3.5 Lightning serves as on Linux. The Qwen3.5 hybrids have a CUDA engine of their own in this port
+(`zig/src/families/qwen3_5/cuda*.zig`, kernels `zig/kernels/cuda/qwen35*.cu`), the Metal kernels' arithmetic with
+the same rounding points, so a window of up to 16 rows reproduces one-token steps bit for bit:
+
+```bat
+tensorfold-native serve <Qwen3.5-2B-MLX-4bit> --name qwen2b --context 32768
+tensorfold-native serve <Qwen3.8-27B-MLX-4bit> --name qwen27b --context 8192 --parallel 4 ^
+  --drafter <Qwen3.8-27B-DFlash2>
+```
+
+- Qwen3.5-2B: about 340 tokens/s for one request on an RTX 4090, 1,500 across eight.
+- Qwen3.8-27B (16 GiB): 40 tokens/s plain, at the card's memory bandwidth; 240 across eight requests.
+- `--drafter` (4-bit only) loads z-lab's DFlash2 draft model: the target writes five layers' residuals into a ring
+  for it, and it drafts a tree of up to 12 lanes a round that the round loop's planner shapes from what the target
+  took. A lone greedy request decodes at 80 to 160 tokens/s by how predictable the text is (prose at the low end,
+  code at the high), sampled ones at 50 to 70, and four at once reach about 160 aggregate; replies stay identical to
+  plain decoding. Loading takes about 80 s: the drafter's weights are quantized to 4 bits on the host and every
+  shape is run once, as the driver stalls a process's first use of each kernel and buffer.
+- A verify window's rows are not snapshotted: each stream logs its rows' recurrence updates and a keep replays the
+  accepted path, so the 27B records 32 rows a forward where 8 fit before.
+- `TF_TIMING=1` prints each verify, keep, absorb and hold's milliseconds and the drafted shape (chain or lanes);
+  `TF_NO_TREES=1` drafts chains only.
+- No prompt cache (`--prompt-cache-gib`), `--learn` or `--slide` for these two models.
+
+### Mixed-width checkpoints
+
+The Qwen3.5 engine also loads MLX checkpoints whose modules carry their own widths (the config's `quantization`
+table names a width per module, as mlx-lm's mixed recipes and the OptiQ and oQ conversions write), 2 to 8 bits in
+groups of 64. Nothing is rounded: a module of 2 or 3 bits is packed onto the 4-bit tiles and one of 5 or 6 bits onto
+8-bit tiles, each code keeping its value, scale and bias, and the 8-bit tiles run `affine8.cu`'s lane GEMV, the
+4-bit one's arithmetic a byte wide. An embedding of an odd width is widened the same way at load. Windows still
+reproduce one-token steps bit for bit, and a uniform 4-bit checkpoint loads and runs exactly as before.
+
+- Tested on an RTX 4080 SUPER (16 GB): `mlx-community/Qwen3.5-4B-OptiQ-4bit` (4 and 8 bits, a tied 8-bit
+  embedding) at about 130 tokens/s for one request and 400 across four, and `mlx-community/Qwen3.5-9B-OptiQ-4bit`
+  (an untied 8-bit head) at about 78 and 260; replies alone, at four streams and with `"draft": false` equal.
+- Widened modules take their served width's memory: a 2-bit module the 4-bit size, a 5-bit one the 8-bit size. So a
+  mostly 2-bit checkpoint such as `TensorFold/Qwen3.8-27B-oQ2` needs about as much memory as the 4-bit 27B.
+- 8-bit modules have no prompt GEMM yet: prompt chunks run them on the decode tiles, 16 rows at a time.
+- `tf-cuda-test affine8` checks every width's tiles against its codes widened on the host, the 8-bit products
+  against float64, and every row count and split-K path byte for byte.
+
+## How the port works
+
+- `zig/src/platform/sys.zig` (`@import("sys")`, wired into every module by `zig/build/sys.zig`) is the operating
+  system as the engine calls it. On Linux and macOS it is `std.c`, `std.posix` and `std.DynLib` unchanged; on Windows
+  `win_c.zig` and `win_posix.zig` give the same names over the UCRT, Win32 and Winsock.
+- Files are opened with every share mode, so rename-over and unlink-while-open behave as on POSIX. Checkpoint reads
+  skip the page cache (`FILE_FLAG_NO_BUFFERING`, as `O_DIRECT` on Linux). `/tmp` is the user's temporary folder and
+  `HOME` falls back to `USERPROFILE`.
+- The Hugging Face cache links snapshot files to blobs. Windows allows symlinks only with Developer Mode or elevation,
+  so `pull` falls back to hard links (no extra space).
+- Checkpoint reads use threads (io_uring is Linux's).
+- The API key file is checked for type and size; Windows keeps access in ACLs, not mode bits, so the "unreadable by
+  other users" check is the folder's ACL rather than `chmod 600`.
+
+## Differences from Linux
+
+- No io_uring, `ulimit` or GB10 carveout, and no `SIGHUP` (the key file is still reread when it changes).
+- `TF_HEAT_*` thermal pacing reads Linux's `/sys/class/thermal` and does nothing on Windows.

docs/api.md

@@ -1,6 +1,6 @@
 # Compatible APIs
 
-The base URL is `http://127.0.0.1:8080/v1` with the default server settings.
+The base URL is `http://127.0.0.1:8888/v1` with the default server settings.
 
 ## Native chat template override
 
@@ -586,5 +586,5 @@ redacted thinking and other context edits return HTTP 400.
 Connect Claude Code using the ID from `/v1/models`:
 
 ```bash
-ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_API_KEY=local claude --model local-model
+ANTHROPIC_BASE_URL=http://127.0.0.1:8888 ANTHROPIC_API_KEY=local claude --model local-model
 ```

docs/cache-benchmarks.md

@@ -23,7 +23,7 @@ The standard-library-only cache client runs one first request, identical repeate
 concurrent group using the same prompt, greedy temperature, seed and reply limit. There is no warm-up:
 
 ```sh
-python3 tools/bench_cache.py http://127.0.0.1:8080 MODEL \
+python3 tools/bench_cache.py http://127.0.0.1:8888 MODEL \
   --tokens 64 --repeats 2 --streams 4 --require-cold --output cache.json
 ```
 
@@ -36,7 +36,7 @@ prefix retention (`--prompt-cache-gib 0`, where supported) for independent cold-
 For exact 2K/8K/32K/64K raw prefill, supply a JSON array of IDs from the checkpoint's own tokenizer:
 
 ```sh
-python3 tools/bench_cache.py http://127.0.0.1:8080 MODEL \
+python3 tools/bench_cache.py http://127.0.0.1:8888 MODEL \
   --prompt-file public-8192-tokens.json --expected-prompt-tokens 8192 \
   --tokens 64 --require-cold --output cache-8192.json
 ```

docs/community-benchmarks.md

@@ -30,7 +30,7 @@ tensorfold benchmark TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit -
 # Attach to a server you started first, in another terminal. Give the public checkpoint ID it serves,
 # and its --name as --model-id.
 tensorfold serve example-org/public-checkpoint --name local-model
-tensorfold benchmark example-org/public-checkpoint --server http://127.0.0.1:8080 --model-id local-model --backend mlx
+tensorfold benchmark example-org/public-checkpoint --server http://127.0.0.1:8888 --model-id local-model --backend mlx
 ```
 
 Attached-server results do not claim the client's hardware is the server's hardware. They carry unknown hardware, versions and checkpoint revisions and remain unranked. An attached run records the backend, model and context you declare, unchecked against the server. The runner does not clear or restart an attached server's caches.

docs/control.md

@@ -118,7 +118,7 @@ The error from `tensorfold tui` prints that command with the venv's Python. `ric
 ```bash
 tensorfold tui
 tensorfold tui --profile code
-tensorfold tui --url http://127.0.0.1:8080/v1
+tensorfold tui --url http://127.0.0.1:8888/v1
 tensorfold tui --url https://spark.example.invalid --token-env TENSORFOLD_API_KEY
 tensorfold tui --demo
 tensorfold tui --demo --snapshot preview.svg

docs/cortheon.md

@@ -100,8 +100,8 @@ tensorfold serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --cor
 ```
 
 ```bash
-curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
+curl -s http://127.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
   -d '{"messages": [{"role": "user", "content": "What changed in the latest Zig release?"}]}'
 ```
 
-Point any OpenAI-compatible client at `http://127.0.0.1:8080/v1` as before.
+Point any OpenAI-compatible client at `http://127.0.0.1:8888/v1` as before.