Qwen3.8-Flash-Next support for oQ in Zig engine

[grantoverton]

Rebased onto main c36b536 (1.0.9) on 2026-10-13: no conflicts in this change's code (1.0.9's pleRef fix from #783 sits beside it); the tip passes the gate again on the new base, tf-flashnext-widths included.

Part of #716; stacks on #755. This branch carries #755's six commits until that one merges, so please review the last four commits here (from "Flash Next: the depth rule and copy drafts move to drafts.zig" on). Flash Next admits only 6-bit/group-32 on main, and its packs refuse group 128 and fused members of mixed widths. With this the native engine loads, drafts and serves TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP @0a1bf6f, -oQ3-MTP @993186e and Jundot/Qwen3.8-Flash-Next-oQ4e-mtp @2615fc0 exactly on an M3 Ultra, on GPUs with and without tensor units. The uniform 6/32 checkpoints keep their recorded kernels and launches, and their packs keep every tensor byte for byte (only the pack's identity line changes).

What this changes

Four commits on the shared change:

1. Flash Next: the depth rule and copy drafts move to drafts.zig, the rounds buffers to engine_rounds.zig. A pure move, to make room under replay.zig's and engine.zig's lengths for what follows (3321 → 3265 and 899 → 857 lines at this commit). No behaviour changes.
2. Flash Next admits and packs each matrix at its own MLX width; the engine still runs 6-bit group 32 only.
   • Widths (widths.zig): each module's width through core/quant; disagreeing spellings and half-declared overrides refused. Admission reads MLX affine 2, 3, 4, 5, 6 or 8 bits in groups of 32, 64 or 128 for every text and MTP matrix, refuses the first other one under its checkpoint tensor name, and refuses an n-gram table whose group does not divide its 160-wide rows.
   • Lane packs follow 0.6.6's _stacked: a fused projection's members grouped by bits in first-seen order, adjacent or not, each group one run at its smallest group, regrouped exactly (core widen; group 128 read as 64), with .runs/.members tables. A run whose column count is not a multiple of 32 keeps MLX's layout and runs untiled, as decode.py's nt = 0; the one public case is oQ4e's layer-21 GDN input (4/64, 5/128, 5/128, 6/64), a 5-bit run of 6192 columns and a 6-bit run of 48. A projection of one width keeps its single run.
   • Hyper-connections: a down and inject pair of two widths widens exactly to the wider (QWeights.of); the up projection packs at its stored width. Prompt projections of several widths keep each member's bytes in order with a .parts table. Experts, embedding and n-gram tables, which the engine reads straight from the checkpoint, are shape-checked at their declared widths when the pack is built.
   • oMLX's layout: an MTP head kept at top-level mtp.* is read where the index keeps it, and the n-gram tables under either shard spelling (shard_N, shards.N). An n-gram table's weight_scale (oQ conversions of the FP8 checkpoint keep it, as oQ4e does) must be one finite, nonzero value; the next commit applies it.
   • Pack format 3, whose identity also hashes config.json, so a config-only change rebuilds the pack. Pack building frees every buffer when an allocation fails.
3. Flash Next runs every MLX width on the GPU, with and without tensor units, in decode and in prompt.
   • Lanes (decode, the MTP head, lane rounds; lane_w.zig): the recorded lane kernel's text compiled per (N, K, bits, group): codes staged as bytes, group sums over the group, K slices as lane_qmm.split_k; one launch a run, columns placed through the .members map (FZ_MAP in the kernel's store), untiled runs as MLX lays them out. A 6/32 lane keeps its recorded kernels and launches; a lane at other widths holds no 6/32 pair, so a 6-bit kernel can't read one of its runs (LaneNot6Bit32). With FZ_LANE=1, 6/32 lanes take core's lane projection as before and other widths take lane_w.
   • Hyper-connections, experts, embedding and n-gram tables (variants.zig): 0.6.6's any-width qa_ kernels, recorded at 6/32, launched at the tensor's bits and group. The 6-bit fast paths (fz_xgu/fz_xdown, fz_dense, fz_hc_up) run only on 6/32 tensors; an expert whose gate and up differ in width, and n-gram shards of different widths, are refused (one kernel reads both).
   • Prompt products (prompt_w.zig, prefill/qmmw_nax.metal): qmm6_nax's tiles and fp32 sums with codes read at any width: dense, split-K (the hyper-connection down), sorted-expert gathers and the fused gate/up; a fused projection of several widths runs one product a member into its columns. Every prompt weight is registered with its width at load, and a launch on an unregistered one fails the pass by name (PromptWeightWidthUnknown).
   • The n-gram table's weight_scale (ple_scale.zig): oQ conversions of the FP8 checkpoint store the unscaled rows, and oMLX's runtime multiplies each looked-up row by the scale (bf16 row times bf16 scale, one rounding). One in-place kernel after the lookup, in decode and in prompt, compiled at load only when the scale is not 1.
   • Fail closed: every width kernel compiles at load, so a compile failure refuses the load, not a request. Width tables are read on the host, never through device buffers. Speed-up mode (two Macs) splits the 6/32 kernels only and refuses any other checkpoint by name (SpeedUpNeeds6Bit32); the recipe's --tp 2 lines for other widths are the CUDA engine's. The replay harness refuses other widths (HarnessNeeds6Bit32).
4. tf-flashnext-widths: Flash Next's products at every MLX width against float64, each row's bits in every window.

On upgrade, every 6/32 user's pack is rebuilt once (the new identity); the old cache directory stays on disk.

Exactness

TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP @0a1bf6f, TensorFold/Qwen3.8-Flash-Next-MLX-oQ3-MTP @993186e, Jundot/Qwen3.8-Flash-Next-oQ4e-mtp @2615fc0 and a uniform 6/32 (below), greedy, MTP drafts on unless "draft": false. Same checkpoint, binary and settings within each comparison.

• Before: main refuses oQ2, oQ3 and oQ4e at load (6-bit/group-32 only).
• Uniform 6/32, main against this branch. No published 6/32 Flash Next checkpoint was on the machine, so I built a full-size one from oQ3: every affine module dequantized at its declared width and requantized min/max at 6/32, same layout and names (158 GB). Main and this branch alternated twice (main, branch, branch, main): every bench_openai and bench_concurrent token hash is identical across the four runs, and the earlier exactness runs gave greedy tokens identical on 6 prompts × plain and drafted × 2 repetitions, drafted equal to plain on both, and the same first token of a 5,267- and a 12,467-token prompt; packs: all 1160 + 291 + 9 tensors identical, and each file's data section byte-identical; only the __metadata__ identity differs.
• oQ2 @0a1bf6f (loads in 12.7 s with its pack): drafted equals "draft": false 4/4 (MTP acceptance 55 to 59%, 2.8 to 3.8 tokens a round); resumed equals fresh (a 5.3k-token prompt at 0 and 5,262 cached tokens); the second turn resumed at 5,262 equals a server with --prompt-cache-gib 0; concurrent equals solo (bench_concurrent's alone and serial checks at 1 and at 4 requests over 2 streams, both prompts: 34/34 equal); greedy text sane (RYB and RGB primaries, a Fibonacci function with doctests, "5:45 PM").
• oQ3 @993186e (loads in 15.9 s): drafted equals plain 4/4 (acceptance 51 to 72%, 2.6 to 4.6 tokens a round); resumed equals fresh at 5,262 cached tokens; the second turn equals a cache-off server; concurrent equals solo (34/34); text sane ("Red, blue, and yellow."; "The train arrives at 5:45 PM").
• oQ4e @2615fc0 (loads in 17.7 s): the checkpoint with the layer-21 ragged GDN runs, the 4/64 down with a 5/64 inject, n-gram shards at 4/32 under a 4/64 default, and an n-gram weight_scale of 1.993e-4. Concurrent equals solo (34/34); drafted equals plain and the cancellation steps below pass (MTP acceptance 47 to 59%); text sane ("Here is a short Python function to compute the Fibonacci sequence…"). Without the scale, every reply is garbage while every equality still holds, which is why fidelity is checked against a reference too.
• Cancellation, reset and partial acceptance (--parallel 2, on oQ2, oQ3 and oQ4e): a fresh "draft": false probe (each bench prompt, greedy, 128 tokens) gives the reference hash and the drafted probe equals it, with partial acceptance seen (accepted < drafted) on both prompts; then after a stream dropped mid-decode, a long prompt dropped mid-prompt, a stream dropped beside a running probe (whose hash is checked too) and a request past the window (refused, the reset path), the drafted and plain probes equal the fresh hashes. 25 steps, pass, on each checkpoint (hashes oQ2 17516a9b267a / 9d1e003353b7, oQ3 a5c773efe57b / 150f131d4962, oQ4e dc4869db94b0 / bf13c8e44c84).
• Seams (tf-flashnext-seams, a test program on this branch, not in the PR): the exact 2,048-token public prefill array, in the engine's one 8,192-row chunk against chunks of 128 and of 37, comparing every DeltaNet layer's conv and recurrent state, every attention layer's and the MTP head's key, value and indexer rows, the pooled indexer blocks, the n-gram tail and history, and the last row's logits at seams 256, 512, …, 1,792, the prompt end, and after 64 one-row decode windows; the served replies equal the windows' too.
  • At the prompt end and after decode every part is byte-equal against both, on oQ2, oQ3, oQ4e and the uniform 6/32.
  • At the seams before the end, every part is byte-equal except the MTP head's key rows. Main does the same on the uniform 6/32: I ran the same test program on main 0efbd44 (the program only, no engine change) and it gives the same result, part for part.
• Rows. Every new product gives a row the same bits alone and in every window (1 to 16 rows in decode, 1 to 65 in prompt), and the generic texts at 6/32 give the recorded kernels' bits.
• Precision. Activations stay bf16, group sums fp32, at every width; the 6/32 kernels are the recorded ones. The n-gram scale is one bf16 product a looked-up value, as oMLX computes it.

FP64 per class

tf-flashnext-widths on the M3 Ultra (no tensor units): seeded synthetic codes, scales, biases and inputs packed as the engine packs them; every projection class at bits {2,3,4,5,6,8} × group {32,64,128}, decode and prompt separately on identical input rows, plus each public checkpoint's mixed stacks and oQ4e's ragged one. Each check reports its worst column against a float64 bound, and a reference with scales 1% off must exceed it.

Path	Class	Checks	Worst column, of the bound	Worst |err| / max|y|	Scales 1% off, least
decode	lanes (attention, GDN, PLE, head, MTP; K slices 1, 2, 4, 8)	47	0.9619	3.89e-3	3.195×
decode	hyper-connection down (partials)	24	0.0005	1.08e-5	3.536×
decode	hyper-connection up	24	0.9683	7.67e-3	9.543×
decode	experts gate/up (through silu)	22	0.6958	1.24e-2	4.571×
decode	experts down	22	0.9947	3.89e-3	511.75×
decode	embedding and n-gram tables	21	0.4980	3.89e-3	5.865×
prompt	dense	48	0.9256	3.84e-3	1.012×
prompt	split-K (hyper-connection down)	18	0.0002	4.10e-6	1.816×
prompt	expert gathers, gate/up	18	0.3310	1.02e-2	1.163×
prompt	expert gathers, down	18	0.5571	3.79e-3	1.532×

2722 checks pass, worst 0.9947 of the bound (bf16 output rounding dominates); all 262 negative controls exceed the bound, the least by 1.012× (prompt dense, K 6144, 3/64). The relative errors above 3.9e-3 are the classes whose outputs pass through a nonlinearity or mix (silu, the hyper-connection up's sigmoid). 8-bit codes are checked over their full range for the hyper-connections, experts, embedding and tables. 2- to 5-bit experts read 16 codes a lane step as 0.6.6's lane_values does, so they are not bit-equal to a 6/32 recode of the same values; 6- and 8-bit experts are. The n-gram scale kernel equals the host's bf16 product on 4,099 values at oQ4e's scale (ple_scale.zig's test).

Performance

Mac Studio, M3 Ultra (80-core GPU), 256 GB, macOS 26.4.1 (25E253), Zig 0.17.0. Fresh builds from clean detached worktrees, zig build native -Dcpu=apple_m1; tensorfold-native sha256 92dca1896a14c982… (this branch's tip), 18adc658fa963516… (the branch before the n-gram scale, which no checkpoint without a weight_scale reaches: oQ2 gives identical hashes on both) and 1c8c96881568cc5f… (main 0efbd44). Every run starts from a fresh server, with other checkpoints' file cache evicted first.

Decode and concurrency. serve <checkpoint> --name flashnext --parallel 2 --context 16384 --prompt-cache-gib 0, MTP drafts on; bench_openai.py … --tokens 256 --reps 3 --temperatures 0 (medians; fibonacci-raw, code, and gpu-chat-no-think, chat) and bench_concurrent.py --levels 1,4 --alone --serial (aggregate tok/s; 4 requests share the 2 streams):

Run	Weights	bench_openai code	chat	concurrent code ×1	code ×4	chat ×1	chat ×4	Peak footprint
uniform 6/32, main 1	158 GB	111.64	89.30	110.6	104.0	89.1	83.8	184.30 GB
uniform 6/32, this branch 1	same	111.57	89.18	111.6	103.9	89.1	83.9	184.31 GB
uniform 6/32, this branch 2	same	111.87	89.18	111.6	103.9	89.1	83.8	184.30 GB
uniform 6/32, main 2	same	111.57	89.19	111.6	103.9	89.2	84.0	184.28 GB
oQ4e, this branch	106 GB	112.92	90.17	112.9	105.0	90.1	84.6	132.71 GB
oQ3, this branch	93 GB	104.40	99.10	104.2	97.3	99.1	92.5	118.93 GB
oQ2, this branch	71 GB	99.90	85.93	99.9	93.6	85.9	80.9	97.10 GB
uniform 6/32, oMLX 0.7.1.dev1	same	68.35	68.23	68.5	153.6	68.2	151.5	158.63 GB

The uniform path is unchanged: every cell of the branch's runs is within 0.9% of main's (the largest, concurrent code ×1, is main's first run below the other three), and every token hash is identical. The oQ checkpoints have no before-run: main refuses them at load. So their speed is reported as served decode beside oMLX, the standard server for Flash Next; stock mlx-vlm 0.7.6 refuses the checkpoint's MTP head parameters at its strict load. oQ4e decodes at 1.01× the uniform checkpoint's code and chat, oQ3 at 0.93× code and 1.11× chat, and oQ2 at 0.89× and 0.96×. oMLX batches four requests (151 to 154 tok/s at ×4); this engine serves two streams. Their non-6/32 products run on the generic kernels; a follow-up PR series works on that speed (#716). The oQ rows ran at the branch's tip (oQ3 before the n-gram scale, as above).

Cold prefill (uniform 6/32; main and this branch). Exact token arrays: upstream's public prefill paragraph ("A lighthouse keeper writes in the log as the boats cross the bay.", tools/zig/qwen27_gate.py) repeated and tokenized once without special tokens; /v1/completions with the ids, max_tokens 1, --parallel 1 --context 66560 --prompt-cache-gib 0, every request cold (cached_tokens 0), a 2k warmup, three reps of each size, medians:

Tokens	main tok/s (s)	this branch tok/s (s)
2,048	1289.9 (1.59)	1289.9 (1.59)
8,192	1382.4 (5.93)	1382.7 (5.92)
32,768	1466.6 (22.34)	1467.0 (22.34)
65,536	1460.5 (44.87)	1460.3 (44.88)
131,072, once (--context 132096)	1444.3 (90.75)	1444.9 (90.71)

Native-window prefill not run; happy to add it if wanted.

oQ prefill (the earlier exactness servers, cache off): not measured against the uniform checkpoint's arrays; the oQ checkpoints have no before-run.

Memory

Weights as stored: oQ2 71 GB, oQ3 93 GB, oQ4e 106 GB, the uniform 6/32 158 GB; oQ2's pack is 4.5 GB. Physical footprint (darwin_rusage_info_v4, the server process) at --parallel 2 --context 16384 after load and at the end: uniform 184.25 / 183.90 GB (peak 184.31 GB, main and this branch equal to 0.03 GB), oQ4e 131.97 / 132.23 GB (peak 132.71 GB), oQ3 118.19 / 118.46 GB (peak 118.93 GB), oQ2 96.31 / 96.63 GB (peak 97.10 GB). The uniform 6/32 peaks at 184.58 GB at --parallel 1 --context 66560 and --context 132096 on both.

Fidelity

Teacher-forced against oMLX 0.7.1.dev1 on the same checkpoint: oMLX's own load path (its pre-load patches, with mlx-vlm 0.7.7's qwen4_exp and oMLX's compat patch, and each per-module width's runtime aliases), which applies the n-gram weight_scale. oMLX renders upstream's four public teacher prompts (copy, arithmetic, story, code; chat template, thinking off), prefills each in one pass and decodes 256 greedy tokens with EOS ignored, no drafts, keeping its top-20 logits at every step; that continuation is the fixed history. This branch's tf-flashnext-teacher (a test program, not in the PR) then feeds every history token and records its top two at all 1,024 positions. Logits are bf16.

Checkpoint	Positions	Top-1 equal	Differences	Beyond the tie policy
uniform 6/32, main	1,024	1,011	13	10
uniform 6/32, this branch	1,024	1,011	13 (the same positions and tokens)	10
oQ4e	1,024	1,014	10	4
oQ3	1,024	1,017	7	7
oQ2	1,024	994	30	23

On the uniform checkpoint main and this branch give the same 13 differences at the same positions, so that gap to oMLX is there before this change. oQ4e and oQ3 sit under it; oQ2 has more, and at 2 bits the two engines' different products (this branch's fp32 group sums against MLX's 2-bit kernels) move more small margins past one spacing. In every difference this branch's pick is in oMLX's top four (second in 51 of 60), and no position has a pick below another token's logit.

Every difference (oMLX's pick and its margin over the runner-up; this branch's pick and its margin; this branch's pick's rank in oMLX's top 20):

uniform 6/32 (main and this branch, identical): 13 differences, 10 beyond the tie policy

Prompt	Position	oMLX pick	oMLX margin	this branch's pick	its margin	its rank in oMLX	Verdict
copy	14	248045	0.375	846	0.125	2	beyond tie policy
copy	18	846	3.9375	248046	0.0625	3	beyond tie policy
copy	19	198	0.875	248046	2.875	2	beyond tie policy
copy	29	760	0.875	2064	0.25	2	beyond tie policy
arithmetic	59	56491	0.125	27567	0.0	2	near tie
arithmetic	205	846	0.875	248045	3.75	2	beyond tie policy
arithmetic	215	2064	1.0	760	0.375	2	beyond tie policy
story	17	16433	0.0	9047	0.0	3	beyond tie policy
story	38	8831	0.125	6316	0.125	2	near tie
story	76	279	0.125	18717	0.125	2	near tie
story	139	17415	0.25	9117	0.0	2	beyond tie policy
code	143	248046	1.125	248045	0.875	2	beyond tie policy
code	156	71093	0.125	760	0.5	2	beyond tie policy

oQ4e: 10 differences, 4 beyond the tie policy

Prompt	Position	oMLX pick	oMLX margin	this branch's pick	its margin	its rank in oMLX	Verdict
story	47	10184	0.125	1801	0.125	2	near tie
story	57	49264	0.125	9770	0.0	2	near tie
story	111	279	0.0	424	0.125	2	near tie
story	120	22317	0.125	20618	0.0	2	near tie
story	158	86532	0.25	24435	0.0	2	beyond tie policy
story	171	16002	0.125	12701	0.125	2	near tie
story	212	324	0.0	70676	0.125	2	near tie
code	142	248045	0.0625	1787	0.375	4	beyond tie policy
code	154	248046	0.125	248045	0.875	2	beyond tie policy
code	173	2493	0.125	557	0.375	2	beyond tie policy

oQ3: 7 differences, 7 beyond the tie policy

Prompt	Position	oMLX pick	oMLX margin	this branch's pick	its margin	its rank in oMLX	Verdict
copy	15	248046	0.5	198	2.0	2	beyond tie policy
arithmetic	37	220	0.0	279	0.25	2	beyond tie policy
arithmetic	228	248046	0.0625	248045	0.5625	2	beyond tie policy
story	88	11	0.125	13	0.25	2	beyond tie policy
story	207	14656	0.0	34909	0.25	2	beyond tie policy
code	33	5447	0.25	561	0.0	2	beyond tie policy
code	139	248045	0.625	198	1.0	3	beyond tie policy

oQ2: 30 differences, 23 beyond the tie policy

Prompt	Position	oMLX pick	oMLX margin	this branch's pick	its margin	its rank in oMLX	Verdict
copy	77	4021	0.25	2688	0.25	2	beyond tie policy
copy	84	369	0.125	7701	0.3125	3	beyond tie policy
copy	95	846	0.875	74455	0.625	2	beyond tie policy
copy	106	760	0.125	40	0.75	2	beyond tie policy
copy	148	846	0.25	74455	2.25	2	beyond tie policy
arithmetic	21	5089	0.0	72677	0.125	2	near tie
arithmetic	78	220	0.125	18	0.125	2	near tie
arithmetic	122	4613	0.25	303	0.0	2	beyond tie policy
arithmetic	208	60439	0.25	283	0.125	2	beyond tie policy
arithmetic	237	25273	0.125	283	0.125	2	near tie
story	4	15218	0.0	22725	0.25	2	beyond tie policy
story	27	27583	0.125	2919	0.0	3	beyond tie policy
story	34	2919	0.0	9117	0.1875	2	beyond tie policy
story	44	27797	0.25	2356	0.0	3	beyond tie policy
story	45	6183	0.1875	4559	0.0625	3	beyond tie policy
story	136	725	0.25	28552	0.5	2	beyond tie policy
story	157	59035	0.25	9101	0.0	2	beyond tie policy
story	170	3184	0.5	13087	0.0625	2	beyond tie policy
story	180	2064	0.125	1465	0.25	2	beyond tie policy
story	183	5983	0.8125	2682	0.0	2	beyond tie policy
story	205	1274	0.0	11840	0.0625	2	near tie
story	211	16002	0.0	41217	0.4375	2	beyond tie policy
story	226	18473	0.3125	279	0.0	2	beyond tie policy
story	228	264	0.0	34830	0.125	2	near tie
story	233	1248	0.125	41423	0.0625	2	near tie
story	244	1049	0.125	561	0.0	2	near tie
code	36	220	0.0	321	0.375	2	beyond tie policy
code	198	248045	1.375	248046	0.75	2	beyond tie policy
code	202	248044	0.0	248046	0.9375	3	beyond tie policy
code	204	248045	0.875	248046	0.625	2	beyond tie policy

Host suite and lean check

• zig build test test-golden -Dcpu=apple_m1 at each of the four commits, from a clean detached worktree: 158 of 160 steps; goldens 309/319 equal, the 10 declared differences, 0 unexpected; native and tf-flashnext-run (with flashnext_bench), -reuse, -pack and -think build; tf-flashnext-widths passes from the commit that adds it on. The one failing step is main's own: core.checkpoint_metal's "a split checkpoint keeps later tensors in the file, packs the rest, and stages the same bytes" logs an error (get calls std.log.err before returning the error.DeferredTensor the test expects, and the runner counts logged errors). It fails the same way on main 0efbd447. zig build test-native at the tip: 16/16, the n-gram scale test included.
• Tests that fail before: widths_test.zig; load_admission_test.zig's overrides at other widths, global groups 64/128, refusal under the tensor's name and oMLX's mtp.*; pack_lane_test.zig (runs by bits, a pack per run, ragged runs untiled at oQ4e's real rows 10240/6144/48/48, allocation failures free everything); pack_hc_test.zig (oQ4e's 4/64 down with a 5/64 inject as one 5/64 matrix; the up projection at its stored width; mixed prompt members); pack_widen_test.zig; pack_test.zig's half-declared and contradicting raw tensors; pack_source_test.zig's config-only change; pack_cache_test.zig's format-less identities; lane_w_test.zig, engine_widths_test.zig, prompt_w_test.zig and ple_scale.zig's test (new with their files). The uniform packs' digest is pinned.
• python3 tools/zig/lean_check.py: 531 problems against main's 539 at every commit, no new findings. Gone: four multi-line comments in replay.zig and one in engine.zig (the moved code's comments are one line now), and Core support for oQ in Zig engine #755's three. Inherited and left: engine.zig (863 lines here, 899 on main), replay.zig (3298, 3321 on main) and tests/flashnext_run.zig (1689, as on main) are over 600 lines on main and do not grow; their multi-line and over-long comments; three in tests/flashnext_bench.zig; a multi-line comment in kernels/metal/sources.zig; and Core support for oQ in Zig engine #755's. Every new file is under 600 lines; zig fmt clean.

Platforms

• Run: Metal on a Mac Studio M3 Ultra, 256 GB (80-core GPU, no tensor units), macOS 26.4.1, Xcode 26.6, Zig 0.17.0.
• Also run, thanks to @sethforprivacy on an M5 Ultra (tensor units): tf-flashnext-widths PASS, and oQ2 serves exactly (drafted, cached and concurrent).
• And to @mrv777 on an M1 Max: tf-flashnext-widths PASS.
• Not run: M1, M2 and M4 (no hardware; the M3 runs the same non-tensor-unit kernels); CUDA (no CUDA code changes).
• Not run: M5. The tensor-unit lane text is patched at every width, and the prompt, hyper-connection and expert kernels compile their tensor-unit variants, but none of it has run. Commands:

zig build tf-flashnext-widths native -Dcpu=apple_m1
./zig-out/bin/tf-flashnext-widths
./zig-out/native/bin/tensorfold-native serve <oQ2, oQ3 or oQ4e dir> --name flashnext --port 8080 --context 16384 --parallel 2

tf-flashnext-widths lane|hc|moe|prompt runs one class. Then, greedy, per prompt: a request and the same with "draft": false (same tokens); a ~5k-token prompt sent twice (cached 0, then cached) and a second turn compared with a server started with --prompt-cache-gib 0; two requests at once compared with each alone. I have no M5; @sethforprivacy has since run these (above).

Reproduction settings

tensorfold-native serve <checkpoint> --name flashnext --parallel 2, greedy; cache-off runs add --prompt-cache-gib 0. Each server builds or reuses its own pack. One model process on the machine at a time, every load and check program behind one GPU lock, and the file cache of other checkpoints evicted before each load. The machine's iogpu.wired_limit_mb is 240000, which I set as its owner; nothing else was raised.

Where the arithmetic differs from 0.6.6

Bit equality with 0.6.6's kernels is not the bar (#716); the float64 bound per product, row invariance and the exactness contract are.

• Before M5, 0.6.6 runs every width but 4/32 through qmv_rows, row by row (decode.py:174-187). This branch runs the engine's own recorded lane kernel (its twin for GPUs without tensor units) at every width instead.
• On M5, 0.6.6's lane uses the nibble lowbit kernel for 2- and 3-bit and coop (64 columns) for 4-bit (lane_qmm.py:315-327). This branch stages every width as bytes there too.
• The split rule (_stacked), the exact regroup, group 128 read as 64 in lane packs, the untiled ragged runs and the hyper-connection widen are 0.6.6's.

The open question in #716: the column map

#716 asks whether the lane kernel should take a per-member output offset table, or run the grouped product into scratch and copy. This branch ships the table in Flash Next's own lane kernel (FZ_MAP, lane_w.zig), because 0.6.6 groups members that aren't adjacent (q6 k6 v8 idx6 is one 6-bit product of three members and one 8-bit product), and the table writes each member's columns in place with no copy. Core's lane_projection is unchanged in this respect: it has only the output row stride. I'd like your answer on this before it lands; if you'd rather have scratch and a copy, or the table in core, it's a small change.

Follow-ups

• The MTP head's key rows at a prompt seam depend on the chunk rows, on main and here (above).
• oQ decode speed on Macs without tensor units, and oMLX's 4-request batching: a separate PR series.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (oQ2, oQ3 and oQ4e served; the uniform 6/32 tokens equal main's).
• No precision traded for speed. Every new product keeps fp32 group sums; the differences from 0.6.6 are listed above.
• Prompt processing is no slower than the last release: the uniform 6/32 prefills 2k to 128k at main's rate (above). The oQ checkpoints have no before-run (main refuses them).
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Receipts: https://gist.github.com/grantoverton/668032ee274e8559aaf628ef05f55964

[sethforprivacy]

Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra.

@grantoverton the M5 checks for this PR at b29d032 all pass, the live ones on TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP @ 0a1bf6f. Mac Studio M5 Ultra (80-core GPU, tensor units), 256 GB, macOS 27.0 (26A428), Xcode 27.0, Zig 0.17.0. Each branch was built clean from its PR head, zig build ... -Dcpu=apple_m1.

tf-flashnext-widths   device Apple M5 Ultra, tensor units true
                      PASS 2722 checks: every product within 0.9947 of its float64 bound, every row's bits equal in every window

Live. serve <oQ2> --name flashnext --context 16384 --parallel 2; it loaded in 26.4 s, with 89.6 GiB in use. Greedy throughout.

• tools/bench_concurrent.py ... --tokens 256 --reps 3 --temperatures 0 --levels 1,2 --alone --serial:

Prompt	Streams	Alone check	Serial ("draft": false)	token_sha	Aggregate tok/s
code	1	3/3 equal	equal	601b6c36c7f7	179.3
code	2	6/6 equal	equal	601b6c36c7f7	171.9
chat	1	3/3 equal	equal	bee38ee68498	159.0
chat	2	6/6 equal	equal	bee38ee68498	152.7

• Cached against fresh: a 4,425-token prompt, sent cold (0 cached) and again (4,418 cached). The replies are equal (7d62a21be9d9).
• Second turn against --prompt-cache-gib 0: resumed at 4,418 of 4,495 tokens, it equals a server started without a prompt cache (0d54783a99d3).

[mrv777]

M1 Max check at b29d0321: 32 GPU cores, 32 GB RAM, macOS 26.7, Zig 0.17.0 (-Dcpu=apple_m1), no tensor units. Fresh native/check-program builds; Metal API validation enabled.

tf-flashnext-widths PASS 2,722 checks: every product within 0.9947 of its float64 bound; every row's bits equal in every window

GPU Shader Validation reports a 45,056-byte threadgroup-memory requirement against M1's 32,768-byte limit; the complete unmodified check passes with API validation alone.

Live drafted/plain, cached/fresh and concurrent/solo checks were not run: the smallest listed oQ2 checkpoint is about 71 GB, exceeding this Mac's 32 GB capacity.

test test-golden: 1,729/1,738 passed, 8 skipped; the existing GLM sparse-attention failure and checkpoint logged-error diagnostic remain. Goldens have zero unexpected differences. Raw logs kept local.

[ashhart]

Thank you, @grantoverton. Qwen3.8-Flash-Next on a Mac serves mixed-width checkpoints such as TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP and -oQ3-MTP. A uniform 6-bit group-32 checkpoint keeps its recorded kernels and launches. Its pack is rebuilt once, because the pack's identity now covers config.json. Speed-up mode across two Macs still takes 6-bit group-32 checkpoints only.

Your commits are in 1.0.10 under your name (25d4dd00c, d81384dec, 9cf7bfdbf, 33f4fb459). The CHANGELOG line is 65256e6b5. Your head a42b54137 is merged at 3c5764ada with no change to the tree. The checks below ran at b29d03216. build.zig keeps #773's GLM check step and your Flash Next check step. 027645132 reads module widths through core/quant's keys. Same prefixes, same keys.

The oQ checkpoints rest on your runs on an M3 Ultra and @sethforprivacy's on an M5 Ultra, both at b29d03216. @mrv777 ran the width checks on an M1 Max. Thank you both. The uniform 6/32 checkpoint was not served for this change. Its recorded kernels are unchanged, its 6/32 lane layouts give 1.0.9's output bytes on an M5 Max and an M3 Ultra, and tf-flashnext-widths passes on the M3 Ultra.