Repository navigation
Qwen3.8-27B support for oQ in Zig engine - #756
Merged
Merged
Conversation
7 tasks
grantoverton
force-pushed
the
oqxe/27b
branch
from
October 13, 2026 01:49
4191469 to
8c05ac9
Compare
8 tasks done
Contributor
|
Posted by Claude (Anthropic's AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work on a Mac Studio M5 Ultra. @grantoverton the M5 checks for this PR at What fails on M5 (
The fix: such runs take the row kernels on every chip, as they do on the M3. Mixed stacks already handle row runs ( --- a/zig/src/families/qwen27/projection.zig
+++ b/zig/src/families/qwen27/projection.zig
- if (!device.tensorUnits()) return prepareRows(device, allocator, owned, parts, n, k, tile, slices);
- if (n % tile != 0) return error.BadProjectionShape;
+ if (!device.tensorUnits() or n % tile != 0) return prepareRows(device, allocator, owned, parts, n, k, tile, slices);
--- a/zig/src/families/qwen27/widths.zig
+++ b/zig/src/families/qwen27/widths.zig
- if (chip.tensor_units) return (lane.Layout{ .n = n, .k = k, ... }).validate();
+ if (chip.tensor_units and n % 32 == 0) return (lane.Layout{ .n = n, .k = k, ... }).validate();
--- a/zig/src/families/qwen27/gpu_weights.zig
+++ b/zig/src/families/qwen27/gpu_weights.zig
- if (p.runs.len == 0 and p.bits == 4 and p.group == 64) try jobs.append(...);
+ if (p.runs.len == 0 and p.bits == 4 and p.group == 64 and p.n % 32 == 0) try jobs.append(...);The last line keeps the 48-row 4/64 run, now a row matrix, out of Live, with the fix,
|
Contributor
|
M1 Max check at Lanes/DFlash pass Metal API validation. GPU Shader Validation instead hits the attention-tail's instrumented threadgroup-memory limit (36,864 > 32,768 bytes). Live oQ2
|
oQ checkpoints write per-module widths as language_model.model.layers.N..., while loaders ask for model.layers.N... or the bare path, so an exact-key lookup missed every override and read the tensor at the global width. The table keeps each entry under the config's own key, as before, and also indexes it under its canonical module key (no .weight suffix, no known prefix, the rule the 27B and Flash Next families use); `width` takes the key as given, then its module. Two spellings of one module that declare different widths refuse the table as unsupported, the error `detect` already returns, so its callers' error sets are unchanged; the same width twice is fine. Test that fails before: the fixture reads a 5-bit down_proj declared under the oQ prefix by the loader spelling; on main it fails the packed shape check (read at the global width). Also: every prefix form and a trailing .weight name one module; aliases must agree; an override in any spelling answers every spelling and stays under its own key. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
… tables repeated for a smaller group A projection of mixed-width members (the Flash Next hyper-connection down and inject pair, where 0.6.6's QWeights.of widens) reads as one matrix at their common width, the most bits and the smallest group. Narrowing and a group that does not divide the source group are refused. Used only where 0.6.6 widens; no caller on this commit. Tests (widen.zig, new): every source width and every wider target dequantize to the same fp64 values, bit for bit; a stack of experts widens row for row; the same width comes back as it is; narrowing bits, growing groups and widths the kernels do not run are refused; the common width takes the most bits and the smallest group; fused members keep each one's values in order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
…wider output Chips without tensor units now decode every MLX affine width (2, 3, 4, 5, 6, 8 bits) in the row kernel: 3- and 5-bit rows unpack eight codes from three or five bytes, low code first, as MLX packs them. The 2-, 4-, 6- and 8-bit paths are unchanged. A 3- or 5-bit row must end on a word, as the checkpoint stores it. callStrided launches the row kernel with an output row stride, so one member of a fused stack (one run of equal width) writes its columns straight into the stack output. The default call, its dims and its kernels are unchanged. tf-row-widths-check (zig/build/core_checks.zig) runs it on the GPU: each width at groups of 32, 64 and 128 against an fp64 product from the MLX reference dequantization, within the fp32-sum bound; one row alone against the same row in 2-, 3-, 8- and 16-row windows, relu2, indexed slots and a strided output, bit for bit; a fused stack of 48-column members at mixed widths (4-bit among them, group 64) written through the stride at windows up to 48 rows, each member as its own call; and, given the unchanged kernel source, the widths it already served byte for byte against it. Tests that fail before: "shapes the row kernels serve" now validates 3 and 5 bits (main refuses them), "3- and 5-bit rows take whole words of codes", "every width has its own pipeline slot" and "a strided call writes rows at least a matrix wide" (no callStrided on main). tf-row-widths-check on an M3 Ultra (no tensor units), given main's row_projection.metal: PASS 64512 values within the fp32-sum bound (worst err/bound 0.9400), 408192 bit-equal pairs, 301056 outputs byte-equal to main's kernels at the widths main serves (3 and 5 bits have no main kernel to compare). Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…d 16-row members Format accepts 2-, 3- and 5-bit codes beside 4, 6 and 8, in groups of 32, 64 or 128. The non-4-bit path widens 2- and 3-bit codes to nibbles for the bf16 x uint4 op and 5-, 6- and 8-bit codes to bytes for the uint8 op, the staging the Python lane matmul uses, so each width keeps that arithmetic. 3- and 5-bit groups of 32 are an odd number of words a lane; they are fetched a word at a time instead of in pairs, which would drop the last word. Layout.y_stride (default N) sets the elements between output rows in every kernel (tf_lane, tf_lane_pair, tf_lane_reg), so a member of a fused stack writes its columns of the fused rows; the output bound checks follow it. The store of the simdgroup-matrix layout (Macs without tensor units, ashhart#679) takes the same stride and keeps its column guard. N may be a multiple of 16 on the plain kernel: the last 32-column tile is packed with zero codes past N, its metadata reads and stores are already guarded by N, and weightWords counts the padded tiles (sourceWords is MLX's W). Cooperative and reg layouts still need N % 32 == 0. encodeSums checks its sums buffer by the group count, as ashhart#698 sizes the dispatch. Without tensor units the admitted layout is unchanged: 6-bit group 32 on the plain kernel (ashhart#679); every other width still answers TensorUnitsRequired there, now checked at 2, 3 and 5 bits too. With the defaults the 4-, 6- and 8-bit kernels preprocess (metal -E) to the same text as main's for every group, schedule, output, prepared-sums/reg layout and both fragment layouts tried (148 variants), so the uniform paths keep their kernels. Host tests that fail before: 2, 3 and 5 bits validate; y_stride reaches the source and below N is refused; a 48-row member validates (cooperative and reg refuse it); every width x group x {64, 48} rows packs MLX codes bit for bit with zero padding. The device program covers every width, 48-row members and strided output on tensor units; without them (M3 Ultra): PASS 14 layouts, 1904 bit-equal row/window pairs, 208 windows equal to fn_lane's recorded bytes, 7424 CPU values (max roundoff-bound fraction 0.0144), 10 refusals, and a 48-row member and two strided layouts at 6/32 equal the unstrided rows with the gaps untouched. fn_lane's recorded windows are compared only on the layouts they were recorded at. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
affine_mm took 4- and 8-bit weights only, so a 2-bit matrix (oQ2's routed experts) had no tiled prompt product. At 2 bits a thread loads its 32 codes of a step as 8 bytes and writes each code into the bf16 tile as itself, with no offset (as at 8 bits; 0..3 is exact in bf16), so the group sums are the 4- and 8-bit path's. A first try with the 4-bit "+128" offset stayed within the float64 bound, but a reference with scales 1% off no longer exceeded it at K = 4096, so the codes enter exactly. The 4- and 8-bit sources preprocess as before. Test that fails before: "dense and gathered, each format, against exact sums on the host" adds 2/64 and 2/32, dense and gathered; source() refused 2 bits. The 4- and 8-bit sources preprocess (metal -E) to main's text at both groups, both outputs and both fragment layouts (16 variants). tf-affine-ref (next commit) holds the 2-bit products to float64. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…y row's bits in every window The device side of ashhart#633's quant-ref for the shared prompt products, item 5 of ashhart#716. tf-affine-ref takes no arguments and needs no model: seeded synthetic codes, scales, biases and inputs, packed as MLX affine. The engine module now names core/affine_mm so a check program can reach it. - affine_mm at 2, 4 and 8 bits, groups 32 and 64, bf16 and fp32 out, dense and as a one-expert gather, at [192, 1024] and [256, 4096]. - simd_qmm (4-bit, group 64) as tiles and as the scalar twin at 1 and 2 rows, at [40, 512], [256, 4096] and [6400, 256] (K chunks of 32, 16 and 8). The reference dequantizes the same codes in float64 and multiplies in float64. Each product prints its worst column (max abs error, and that error over the column's largest |y|) and its largest error as a fraction of the bound: (group + K / group + 8) fp32 roundings of the terms' magnitudes as the kernels sum them (4-bit affine_mm codes plus 128), plus one bf16 ulp for bf16 outputs. A reference with its scales 1% off must exceed the bound. Then every row of a 129-row call must equal, byte for byte, the same row computed in windows of 1, 2, 3, 8, 16, 17, 33, 64 and 65 rows (every full window over the 129 rows, so each row is seen at every position), which crosses the 16-row fragments, the 32- and 64-row tiles and the gather's switch between them. The twin must equal the tiles wherever twinExact says it does, since only then do callers take it. Receipts, M3 Ultra (no tensor units: nax.h's simdgroup-matrix layout), on main 0efbd44: - every product within 0.4935 of its bound; every row's bits equal at every window, both kernels, every format; the twin equals the tiles at 1 and 2 rows on all three shapes, as twinExact says. - worst columns, bf16 out (dense and gather equal), at [192, 1024] and [256, 4096]: affine_mm q2 g32 2.64e-2 (rel 3.23e-3) and 6.22e-2 (3.78e-3), q2 g64 2.87e-2 (3.51e-3) and 3.12e-2 (2.47e-3), q4 g32 2.45e-2 (2.65e-3) and 3.12e-2 (3.01e-3), q4 g64 1.56e-2 (2.89e-3) and 3.12e-2 (2.53e-3), q8 g32 0.496 (3.41e-3) and 0.968 (3.63e-3), q8 g64 0.453 (2.87e-3) and 0.956 (3.38e-3); simd_qmm 1.19e-2 (2.57e-3), 3.13e-2 (2.87e-3) and 1.54e-2 (3.74e-3). All under 2^-8 of the column's largest |y|; 0.20 to 0.49 of the bound. - fp32 out: at most 0.0081 of the bound (q2 1.5e-6 to 7.7e-6, rel 1.9e-7 to 6.0e-7). - scales 1% off: 2.3 to 29 times the bound for bf16 outputs, 2.7 to 391 for fp32, 44 to 273 for simd_qmm. Not run: an M5, where affine_mm takes the tensor units; `zig build tf-affine-ref && ./zig-out/bin/tf-affine-ref`. The fp32 bound is the worst case for any summation order, so a changed order still passes; the measured errors sit two orders below it, and the bit checks are what pin the order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…nst the kernels this chip runs Only the modules the engine loads are resolved, so draft-head and vision overrides never refuse a checkpoint. A width or group the projection kernels cannot run yet refuses naming the module, layer and width; fused members of different widths are admitted as runs of one width.
… width, mixed fused groups run per width Uniform 4-bit g64 keeps its bytes, kernels and launches. Other widths take the row kernel before tensor units and the non-reg lane kernel on them; fused members of different widths become runs, each its own product into the fused rows, one row a launch until the core output stride lands. The embedding takes the checkpoint's width. The Neural Engine share declines unless every MLP matrix is 4-bit g64, and the DFlash2 draft head refuses a non-4-bit embedding or head for now. Synthetic graphs load at uniform 4-bit g64.
…eckpoint widths The draft vocabulary copies whole rows (or 32-row tiles) of k * bits / 8 bytes and k / group metadata, and the borrowed embedding is validated at its resolved width. The draft-head check covers 2-, 4- and 8-bit heads.
…nce does The reference stacks fused members only when all share one width and group; otherwise each member is its own call at its own K split. Members of one width beside a member of another no longer share a product.
…before tensor units Each member is still its own product, now one row-kernel launch over all rows writing its columns at the stack width, in place of one launch a row. The row kernel gives a row the same bits in any window, so every row count still matches.
…ainst float64 without tensor units tf-qwen27-widths runs the family's own products (quant_gpu.Kernels.quant on projection.prepare's matrices: the row kernels, or simd_qmm at 4/64, as a chip without tensor units picks them) on synthetic MLX affine weights at the shape of every projection class (GDN qkv, z, b, a and out; attention q, k, v and o; MLP gate_up and down; lm_head) at 2, 3, 4, 5, 6 and 8 bits, group 64, which covers every width oQ4 and oQ2 use: - a prompt call (128 rows) and decoded rows (one row a call) against core.quant.mlx.reference in float64, the worst column within the fp32-sum bound; scales 1% off exceed the bound in every class and width; - every row's bits equal in calls of 1, 2, 3, 8, 16, 17 and 128 rows; - the mixed stacks [z4 | b5 a5], [z4 b4 | a5], [z2 b2 | a3] and [k5 | v6], built as the loader builds them: every row's bits equal in every call, each member's columns within the float64 bound and equal bit for bit to its own product (the row kernel at its width, unstrided), and to the member loaded as a matrix of its own, except a 4-bit z, which alone takes simd_qmm (no output stride). The engine module exports core.quant for the reference. The program refuses a chip with tensor units, whose lane kernels it does not check. M3 Ultra (no tensor units): PASS 10,575,872 values within 0.482 of the float64 bound (bf16 output rounding; worst-column |err| / max|y| 2.5e-3 to 3.9e-3), scales 1% off at least 2.1 times the bound in every class; 1,515,200,224 bit-equal pairs; 31 s. A 4-bit z inside a mixed stack differs from the same z as a uniform layer's simd_qmm product in 51 and 72 of 128 rows, by up to 3.1e-2; both are within the bound. zig build test test-golden -Dcpu=apple_m1: 158/160 steps, 1566/1574 tests (8 skipped), goldens 309/319 with 0 unexpected differences; the one failure is main's core.checkpoint_metal split-checkpoint test. tools/zig/lean_check.py: 536 findings, 0 new against main.
…oat64 tf-qwen27-widths now also runs the family's embedding gather (glue_gpu.Kernels.embedding, q27_embed) on a synthetic MLX affine table at the 27B's shape [248320, 5120], group 64, at 8 bits (oQ2's embedding) and 3 bits, for 64 tokens including both ends of the vocabulary and a repeat: - every gathered value equals core.quant.mlx.reference in float64 rounded once to bf16, bit for bit; - one-row calls give the same bits as one 64-row call; - the reference with scales 1% off differs in a large share of values, so the comparison sees a wrong decode. M3 Ultra (no tensor units): 8-bit 0 of 327,680 values differ, 3-bit 0 of 327,680; one-row calls equal; scales 1% off differ in 160,115 and 142,528 values. The projection checks are unchanged (PASS 10,575,872 values within 0.482 of the bound). zig build test test-golden -Dcpu=apple_m1: 158/160 steps, goldens 309/319 with 0 unexpected differences; the one failure is main's core.checkpoint_metal split-checkpoint test. tools/zig/lean_check.py: 536 findings, 0 new.
…pt chunk seams and after decode --compare-chunk compared Session.fingerprint() (every GDN layer's recurrent and conv state, the visible KV rows of every attention layer, the last row's bf16 logits) at the prompt end only, and decode by tokens. With --compare-chunk C it now also compares the fingerprint after the decoded reply in both arms (the prefill to decode seam and every decode step's state), failing with DecodeStateOrLogitsDiffer. --seams N adds, for every N-th prompt row, a prefill of the prompt up to that row at --chunk and at C, fingerprints compared byte for byte; the receipt lists each seam and seams_byte_equal, and any difference fails with SeamStateOrLogitsDiffer. Test program only; nothing in the engine. M3 Ultra (no tensor units), the exact 2048-token public prefill array, chunk 128 against 16 and against 37, seams every 256 rows, 64 decoded tokens: every seam, the prompt end and the decoded state byte-equal on the uniform 4-bit checkpoint and on oQ4 and oQ2.
…kpoint never drafts on another's costs Window costs are kept per build, chip and model shape, and every 27B checkpoint has one shape: whichever checkpoint a build served first wrote the table every later one read. An oQ2 server then chose its draft depth from the uniform 4-bit costs (wide windows nearly free) and drafted deeper than its row kernels pay for: same tokens, 28% more drafted rows, 15% slower code decode. A mixed-width checkpoint now adds a tag of its widths to the shape; a uniform 4-bit g64 checkpoint keeps its shape string, so its key and costs are unchanged.
…he row kernels, on every chip With tensor units, projection.prepare refused any run with n % tile != 0 (BadProjectionShape) and widths.check gave every 4/64 run the reg lane layout, which needs n % 32 == 0. A lone 48-row GDN in_proj_a or in_proj_b is such a run: TensorFold/Qwen3.8-27B-oQ4's layer 37 (in_proj_a at 5/64, in_proj_b a 48-row 4/64 run) was refused at admission and -oQ2 failed at load on M5. Those runs now take the row kernels, as they do before tensor units; mixed stacks already send a row run to rowStrided. A 4/64 run off the 32-row lanes stays out of regOrder. Before tensor units nothing changes (those runs already took prepareRows), and a uniform 4/64 checkpoint's matrices are multiples of 32 rows. Test: widths.zig: a 48-row 4/64 run admits on tensor units through the row kernels, a 64-row one through the lanes (fails before: the lane layout refused n % 32 != 0). Reported with the fix by Seth For Privacy (@sethforprivacy) on an M5 Ultra (ashhart#756), with oQ4 and oQ2 served exactly there.
grantoverton
force-pushed
the
oqxe/27b
branch
from
October 13, 2026 20:21
8c05ac9 to
a2420e7
Compare
Contributor
Author
|
@sethforprivacy thank you, that was exactly it. Your three lines are now their own commit on this PR (a2420e7, credited to you), with a host test that a lone 48-row 4/64 run admits on tensor units through the row kernels. It changes nothing before tensor units: those runs already took the row kernels there. The series is also rebased onto 1.0.9. |
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
The contributor's nine commits landed above, two of them carried onto the candidate's Neural Engine fit check. Our two commits after them load projection runs that are not whole 32-row tiles on the row kernels on every chip (the fix posted on the PR from an M5 Ultra run) and read module widths through core/quant's keys. This keeps the PR's head in the history with no change to the tree.
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
The contributor's four commits landed above with the same code; the quant export one of them adds was already in from ashhart#756. This keeps the PR's head in the history with no change to the tree.
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
…units takes the row kernels (ashhart#756) From ashhart#756's a2420e7, whose code change is 52932f4 here. Co-authored-by: Grant Overton <65690763+grantoverton@users.noreply.github.com>
Owner
|
Thank you, @grantoverton, and @sethforprivacy. Qwen3.8-27B on a Mac serves mixed-width checkpoints such as TensorFold/Qwen3.8-27B-oQ4 and -oQ2. A uniform 4-bit checkpoint keeps 1.0.9's kernels, launches and window costs. Your commits are in 1.0.10 under your name ( Commits of ours follow yours. The oQ checkpoints rest on your runs on an M3 Ultra and @sethforprivacy's on an M5 Ultra at |
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto main c36b536 (1.0.9) on 2026-10-13: two conflicts in
model.zigwith 1.0.9's Neural Engine share, merged by hand (the share still loads only for a uniform MLP); every commit passes the gate again. One commit added at the end: the M5 fix @sethforprivacy reported below (a run a tensor-unit tile doesn't divide takes the row kernels on every chip), with a test. On the rebased series 27B oQ2 serves exactly on the M3 (concurrent equals solo, the 25 cancellation steps pass with the same reference hashes).Part of #716; stacks on #755. This branch carries #755's six commits until that one merges, so please review the last ten commits here (from "The 27B resolves each loaded module's affine width" on). The Qwen3.8-27B admits only 4-bit/group-64 on main (
UnqualifiedQwenFormat). With this it loads, drafts and serves TensorFold/Qwen3.8-27B-oQ4 @fb60862 (4/5/6-bit) and TensorFold/Qwen3.8-27B-oQ2 @086dcb0 (2/3/5/6/8-bit), and any checkpoint whose loaded modules are 2, 3, 4, 5, 6 or 8 bits in groups of 32, 64 or 128. The uniform 4-bit/group-64 checkpoint keeps its packed bytes, pipeline keys, kernels and launches.What this changes
Nine commits on the shared change:
qwen27/widths.zig, throughcore/quant). Only the modules the engine loads are resolved, so MTP and vision overrides never refuse a checkpoint. A width or group the projection kernels cannot run refuses the load, naming the module, layer and width.projection.zig,gpu_weights.zigon main's staged load,quant_gpu.zig). Every member is staged and read at its own width (k * bits / 32words,k / groupmetadata). Kernels by width: before tensor unitssimd_qmmfor 4/64 matrices and the row kernel for every other width; on tensor units the non-reg lane kernel. The pipeline key carries bits, group and stride. The embedding takes the checkpoint's width (oQ2's is 8-bit). The Neural Engine prompt share, which decodes 4-bit codes only, is created only when every MLP matrix is 4/64. Synthetic graphs load at uniform 4/64.k * bits / 8bytes andk / groupmetadata, and the borrowed embedding is validated at its resolved width. The draft-head check covers 2-, 4- and 8-bit heads (oQ2's head is 2-bit).lane_fuse.py:88-91,row_matmul.py:110-117). Members of one width beside a member of another no longer share a product.tf-qwen27-widths: the family's own products for every 27B projection class at every oQ width against float64, without tensor units.tf-qwen27-widthsgathers the embedding at 8 and 3 bits against float64.tf-qwen27-run --seams N(test program only): with--compare-chunk Cit also comparesSession.fingerprint()(every GDN layer's recurrent and conv state, the visible KV rows of every attention layer, the last row's bf16 logits) after the decoded reply in both arms, and for every N-th prompt row prefills the prompt up to that row at--chunkand at C and compares the fingerprints byte for byte. Any difference fails the run. So the seam receipts below can be reproduced.Exactness
TensorFold/Qwen3.8-27B-oQ4 @fb60862, TensorFold/Qwen3.8-27B-oQ2 @086dcb0 and the uniform Vontra/Qwen3.8-27B-MLX-4bit @70ae7fa, each with z-lab/Qwen3.8-27B-DFlash2 (
model.safetensorssha25667fc76d68dc5a941…), greedy. Every check below ran on this branch's tip (4191469d, a fresh build:tensorfold-nativesha256d640b9501b70d250…,tf-qwen27-runbedbe3942f0e7b74…).UnqualifiedQwenFormat.--parallel 2 --context 65536): on oQ4, oQ2 and the uniform 4-bit each, drafted equals"draft": false9/9 and resumed equals fresh 2/2 (a 26,130-token prompt resumed at 26,123 cached tokens, same token hash);bench_concurrent.py --levels 1,2 --alone --serial: every cell equal, 0 unequal, 0 failed. At--parallel 4 --context 65536 --prompt-cache-gib 0,bench_concurrent.py --levels 1,4 --alone --serial: every alone cell (3 at one stream, 12 at four) and serial cell equal, 0 failed, on all three checkpoints; on the uniform checkpoint the code and chat token hashes (2b28aa2108de,cbdd31cd4736) equal main's in every run.tf-qwen27-run --seams 256, commit 8): the exact 2,048-token public prefill array, chunk 128 against 16 and against 37, 64 decoded tokens. On oQ4, oQ2 and the uniform 4-bit, the state/KV/logits fingerprint is byte-equal at every 256th prompt row (7 seams, each prompt ending there), at the prompt end and after the decoded reply, for both chunk pairs.tf-qwen27-synthetic,tf-qwen27-lanes-check(lane core equals plain),tf-qwen27-dflash-synthetic,tf-qwen27-served-checkandtf-qwen27-dflash-served-checkpass at every commit.tf-qwen27-widthschecks every class's rows in calls of 1, 2, 3, 8, 16, 17 and 128 rows, so prompt chunks and decode rows are the same arithmetic.--parallel 4, on oQ4, oQ2 and the uniform 4-bit): a fresh"draft": falseprobe (each bench prompt, greedy, 128 tokens) gives the reference hash and the drafted probe equals it, with partial acceptance seen (accepted < drafted) on both prompts; then after a stream dropped mid-decode, a long prompt dropped mid-prompt, a stream dropped beside a running probe (whose hash is checked too) and a request past the window (refused, the reset path), the drafted and plain probes equal the fresh hashes. 25 steps, pass, on each checkpoint (hashes oQ4c201c615aba6/d8c31284ff02, oQ22ec414650f51/70d57ef6b44f, uniformf5c9409cfe40/73e9fd21b09e, as main).Accuracy against float64, per projection class
tf-qwen27-widthson the M3 Ultra (no tensor units): the family's own products (quant_gpu.Kernels.quantonprojection.prepare's matrices: the row kernel, orsimd_qmmat 4/64, as a chip without tensor units picks them) on seeded synthetic MLX affine weights at each class's real shape, at 2, 3, 4, 5, 6 and 8 bits, group 64, which covers every width oQ4 and oQ2 use. The reference iscore.quant.mlx.referencein float64; every column up to 1,024 columns, else 512 (the first and last 128 and 256 strided). Prompt is one 128-row call, decode one row a call, on identical input rows.Worst column per class over the six widths, as a fraction of the fp32-sum bound (the width it occurs at), and the worst column's |err| over its largest |y|:
10,575,872 values within 0.482 of the bound (bf16 output rounding dominates), 1,515,200,224 bit-equal pairs. Prompt and decode agree because the row kernel's bits don't depend on the call's row count.
q27_embed) at [248320, 5120], group 64, 64 tokens including both ends of the vocabulary and a repeat: every gathered value equals the float64 reference rounded once to bf16, 0 of 327,680 differ at 8 bits and at 3 bits; one-row calls equal one 64-row call. Scales 1% off: 160,115 and 142,528 values differ.Performance
Mac Studio, M3 Ultra (80-core GPU), 256 GB, macOS 26.4.1 (25E253), Zig 0.17.0. Fresh builds from clean detached worktrees,
zig build native tf-qwen27-run -Dcpu=apple_m1;tensorfold-nativesha2566a42d48dc1b08a1d…(main0efbd447),d640b9501b70d250…(this branch,4191469d). DFlash2 on. Clients are main'stools/at0efbd447on Python 3.12.14; one model process on the machine, the file cache of other checkpoints evicted before each load, every run behind one GPU lock. Every receipt (manifests with binary hashes,decode.json,concurrent.json, prefill records with the exact token arrays and request bodies,/memorysnapshots,time -l, load and cleanup timestamps, the fidelity and seam records) is in https://gist.github.com/grantoverton/dbbc25f85cecacca369b9b7c447294ac (index in00-README.md; the host suite summary istest-summary.txt).Decode and concurrency.
serve <checkpoint> --drafter <DFlash2> --parallel 4 --context 65536 --prompt-cache-gib 0;bench_openai.py … --tokens 256 --reps 3 --temperatures 0(medians;fibonacci-raw, code, andgpu-chat-no-think, prose, greedy) andbench_concurrent.py … --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial(aggregate tok/s). The uniform rows alternate branch, main, main, branch.bench_openaicodemlx_lmmlx_lmmlx_lmThe uniform path is unchanged: the branch is 1.0% below main on
bench_openaicode (93.33 against 94.23) and equal on chat (68.34 against 68.27), and everybench_concurrentcell is within 0.6%, with identical token hashes. The two oQ checkpoints decode below the uniform 4-bit checkpoint on this M3, oQ4 at 0.92× (code) and 0.76× (chat), oQ2 at 0.65× and 0.60×, although both read fewer bytes a token: every non-4/64 product runs on the generic row kernel, and a mixed stack is one launch per member.mlx_lm0.32.0 (no drafts; it serves this model's streams one after another) is slower than this branch on the uniform checkpoint and on oQ4, and faster on oQ2: 71.52 / 47.80 tok/s against 60.76 / 41.34. This PR is about loading and serving these checkpoints exactly; a follow-up PR series will work on their speed.The oQ checkpoints have no before-run: main refuses them (
UnqualifiedQwenFormat). So their speed is reported as served decode (above, besidemlx_lm), and the before/after receipts (decode, concurrency and cold prefill) are on the uniform checkpoint, which both binaries load.Commit 9's effect, on the same oQ2 checkpoint, prompts and tokens (hashes
1396ae566f52/af21ef189db8in both): an earlier build of this branch without it, which had served the uniform checkpoint first, drafted oQ2 on the uniform checkpoint's window costs (wide windows nearly free there) and decoded 44.42 / 30.48 tok/s, drafting 475 to 477 rows in 38 rounds for the 256-token code reply; with oQ2's own costs, 246 to 264 rows in 54 to 60 rounds, and 60.76 / 41.34 tok/s.Cold prefill (uniform 4-bit; main and this branch). Exact token arrays: upstream's public prefill paragraph ("A lighthouse keeper writes in the log as the boats cross the bay.",
tools/zig/qwen27_gate.py) repeated and tokenized once without special tokens by the checkpoint's owntokenizer.json, each array the first N ids (every array decodes and re-encodes to itself).POST /v1/completionswith the ids asprompt,max_tokens1, temperature 0,ignore_eos; server--parallel 1 --context 262144 --prompt-cache-gib 0, one request at a time, a fresh KV every measurement (every record hascached_tokens0 andprompt_tokensequal to N). 2k to 64k: one 2,048-token warm-up, then 3 rounds over the sizes, wall-clock medians. 128k: once per binary, in a server of its own (main's record isq27-u4.pfl-main-f1.prefill-128k.json: that server was stopped during a later request, so itsprefill.jsonwas not written). The arrays, the exact request bodies (with sha256), every request's timings and cached-token counts and the server's/memoryafter each are in the gist.mlx_lmtok/s (s)Native-window prefill (262,143 tokens) not run: about 1.6 h per run on this machine; happy to add it if wanted.
Main's three reps of each size are within 0.1% of each other. The branch's spread more: 2k 286.8, 282.1 and 268.5; 8k 259.2, 258.0 and 247.0; 32k 183.4, 183.4 and 181.0; 64k 132.0, 131.7 and 130.7, the third round low at every size. Its 2k median is 1.5% under main's; from 8k up the medians are within 0.3%. Nothing in this branch changes a uniform kernel or launch, and its decode and concurrency on the uniform checkpoint match main's (above). Peak physical footprint after the 64k prompts: 40.17 GB (main) and 40.18 GB (branch); after the branch's 128k prompt, 40.19 GB.
mlx_lm(2k to 64k only) takes text prompts: the same arrays decoded, each record checkingprompt_tokensequals N andcached_tokensis 0; it prefills faster than the native engine at every size measured, 1.39× at 2k and 2.28× at 64k.Memory
Weights as stored: oQ4 16 GB, oQ2 11 GB, uniform 4-bit about 15 GB. At
--parallel 4 --context 65536 --prompt-cache-gib 0with DFlash2: physical footprint after load 23.7 to 24.8 GB (uniform), 25.4 GB (oQ4), 19.9 GB (oQ2); lifetime peak physical footprint after the bench and cancel steps 38.16 GB (uniform, main and this branch alike), 38.71 GB (oQ4), 33.23 GB (oQ2) (/memoryandtime -lagree). The qwen27 Metal backend reports no device allocations in/memory. RAM class: M3 Ultra, 256 GB. Load and cleanup boundaries are in each run'sevents.tsv; prefill memory at the 262,144-token context is in the prefill receipts.Fidelity
Teacher-forced against
mlx_lm0.32.0 on the same checkpoint.mlx_lmrenders upstream's four public teacher prompts (tools/zig/qwen27_gate.py: copy, arithmetic, story, code; chat template, thinking off), prefills each in one pass and decodes 256 greedy tokens with EOS ignored, keeping its top-20 logits at every step; that continuation is the fixed history. This branch'stf-qwen27-run --run --teacher --force-length --chunk 128 --compare-chunk 16then feeds every history token and records its top two at all 1,024 positions (prompt state byte-equal between chunk 128 and 16 on every prompt). Every top-1 difference is listed below with both engines' top two and margins (logits in bf16). Upstream'sqwen27_teacher_judge.pyreads the same files; its tie policy accepts a difference when the two engines name the same two contenders and both margins are within one bf16 spacing at that logit (0.0625 between 8 and 16, 0.125 between 16 and 32).On the uniform checkpoint main's binary gives the same 6 differences at the same positions.
Every difference (
mlx_lmpick and runner-up, logits, margin; this branch's the same):mlx_lmtop 2On oQ2, 7 of the 11 differences fall outside the tie policy: six have a margin above one bf16 spacing in at least one engine (up to 0.75 and 1.0 at copy position 33), and at copy position 31 the engines name different runner-ups. So upstream's judge fails on oQ2. Where the two engines agree on the top token, their top-1 logits differ by about as much on every checkpoint (largest: uniform 3.375, oQ4 3.0, oQ2 3.125), so oQ2 does not show a larger arithmetic gap. It has more positions whose margins are small enough to flip, and at 2 bits the two engines' different products (MLX's 2-bit
quantized_matmulagainst this branch's row kernel with fp32 group sums) move some of them past one spacing.Host suite and lean check
zig build test test-golden -Dcpu=apple_m1at every commit, last at the tip4191469d: 158 of 160 steps; at the float64 commit 1566/1574 tests passed, 8 skipped; goldens 309/319 equal, the 10 declared differences, 0 unexpected. The one failing step is main's own:core.checkpoint_metal's "a split checkpoint keeps later tensors in the file, packs the rest, and stages the same bytes" logs an error (getcallsstd.log.errbefore returning theerror.DeferredTensorthe test expects, and the runner counts logged errors). It fails the same way on main0efbd447: 158/160 steps, 1505/1513 tests passed, 8 skipped.language_model.prefix; the per-member run plan; lane tile packing at 4/6/8 × 32/64/128; 2-, 4- and 8-bit draft heads; the window-cost key (the uniform checkpoint keeps main's shape string, oQ4 and oQ2 get distinct tags). The failure they fix is main's: it refuses oQ4 and oQ2 withUnqualifiedQwenFormat.python3 tools/zig/lean_check.py: 536 problems against main's 539, no new findings; the 27B's files add none. The inherited findings left in touched files are the shared change's (affine_mm.metal,affine_mm.zig,row_projection.metal,row_projection.zig; listed there). Every touched file is under 600 lines (widths.zig300,gpu_weights.zig247,quant_gpu.zig278,native/qwen27_lanes.zig580,tests/qwen27/widths.zig496,tests/qwen27/run.zig214).Platforms
tf-qwen27-widths,tf-qwen27-lanes-checkandtf-qwen27-dflash-syntheticPASS.tf-qwen27-widthsrefuses a chip with tensor units, since it doesn't check the lane kernels. Commands:then, greedy, each prompt as sent and with
"draft": false(same tokens), and a long prompt sent twice (cached 0, then cached). I have no M5; @sethforprivacy has since run these (above).Reproduction settings
tensorfold-native serve <checkpoint> --drafter <Qwen3.8-27B-DFlash2>, greedy, at--parallel 2 --context 65536for the exactness checks,--parallel 4 --context 65536 --prompt-cache-gib 0for the standard clients and cancellation, and--parallel 1 --context 262144 --prompt-cache-gib 0for cold prefill. One model process on the machine at a time, every load and check program behind one GPU lock, and the file cache of other checkpoints evicted before each load. The machine'siogpu.wired_limit_mbis 240000, which I set as its owner; the 27B checkpoints load far below it, and nothing else was raised.Where the arithmetic differs from 0.6.6
Bit equality with 0.6.6's kernels is not the bar (#716); the float64 bound per product, row invariance and the exactness contract are.
simd_qmmat 4 bits,simd_qmm_bitsat 5, 6 and 8 bits in groups of 64 or 128, andaffine_rowsotherwise (row_matmul.py). This branch runssimd_qmmfor 4/64 matrices and core's row kernel, with fp32 group sums, for every other width.simd_qmmhas no output stride; 0.6.6 runs it onsimd_qmm. Its bits differ from the same z as a uniform layer'ssimd_qmmproduct in 51 and 72 of 128 rows, by up to 3.1e-2; both are within the float64 bound. An output stride forsimd_qmmwould remove the difference; that belongs with the speed follow-ups.The open question in #716
The 27B follows 0.6.6's 27B rule: once a fused group's members differ in width, every member is its own product. So no product spans non-adjacent members, and each member writes its own contiguous columns through core's output row stride. The per-member column map #716 asks about isn't needed here (Flash Next is the family that uses one).
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (oQ4, oQ2 and the uniform 4-bit, served, at this branch's tip).0efbd447; the 2k median is 1.5% under, with the branch's third round low at every size (above). The oQ checkpoints have no before-run (main refuses them).