Repository navigation
Qwen3.8-27B faster oQ kernels in Zig engine - #784
Merged
Merged
Conversation
7 tasks
oQ checkpoints write per-module widths as language_model.model.layers.N..., while loaders ask for model.layers.N... or the bare path, so an exact-key lookup missed every override and read the tensor at the global width. The table keeps each entry under the config's own key, as before, and also indexes it under its canonical module key (no .weight suffix, no known prefix, the rule the 27B and Flash Next families use); `width` takes the key as given, then its module. Two spellings of one module that declare different widths refuse the table as unsupported, the error `detect` already returns, so its callers' error sets are unchanged; the same width twice is fine. Test that fails before: the fixture reads a 5-bit down_proj declared under the oQ prefix by the loader spelling; on main it fails the packed shape check (read at the global width). Also: every prefix form and a trailing .weight name one module; aliases must agree; an override in any spelling answers every spelling and stays under its own key. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
… tables repeated for a smaller group A projection of mixed-width members (the Flash Next hyper-connection down and inject pair, where 0.6.6's QWeights.of widens) reads as one matrix at their common width, the most bits and the smallest group. Narrowing and a group that does not divide the source group are refused. Used only where 0.6.6 widens; no caller on this commit. Tests (widen.zig, new): every source width and every wider target dequantize to the same fp64 values, bit for bit; a stack of experts widens row for row; the same width comes back as it is; narrowing bits, growing groups and widths the kernels do not run are refused; the common width takes the most bits and the smallest group; fused members keep each one's values in order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 539 problems (main 539), no new findings.
…wider output Chips without tensor units now decode every MLX affine width (2, 3, 4, 5, 6, 8 bits) in the row kernel: 3- and 5-bit rows unpack eight codes from three or five bytes, low code first, as MLX packs them. The 2-, 4-, 6- and 8-bit paths are unchanged. A 3- or 5-bit row must end on a word, as the checkpoint stores it. callStrided launches the row kernel with an output row stride, so one member of a fused stack (one run of equal width) writes its columns straight into the stack output. The default call, its dims and its kernels are unchanged. tf-row-widths-check (zig/build/core_checks.zig) runs it on the GPU: each width at groups of 32, 64 and 128 against an fp64 product from the MLX reference dequantization, within the fp32-sum bound; one row alone against the same row in 2-, 3-, 8- and 16-row windows, relu2, indexed slots and a strided output, bit for bit; a fused stack of 48-column members at mixed widths (4-bit among them, group 64) written through the stride at windows up to 48 rows, each member as its own call; and, given the unchanged kernel source, the widths it already served byte for byte against it. Tests that fail before: "shapes the row kernels serve" now validates 3 and 5 bits (main refuses them), "3- and 5-bit rows take whole words of codes", "every width has its own pipeline slot" and "a strided call writes rows at least a matrix wide" (no callStrided on main). tf-row-widths-check on an M3 Ultra (no tensor units), given main's row_projection.metal: PASS 64512 values within the fp32-sum bound (worst err/bound 0.9400), 408192 bit-equal pairs, 301056 outputs byte-equal to main's kernels at the widths main serves (3 and 5 bits have no main kernel to compare). Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…d 16-row members Format accepts 2-, 3- and 5-bit codes beside 4, 6 and 8, in groups of 32, 64 or 128. The non-4-bit path widens 2- and 3-bit codes to nibbles for the bf16 x uint4 op and 5-, 6- and 8-bit codes to bytes for the uint8 op, the staging the Python lane matmul uses, so each width keeps that arithmetic. 3- and 5-bit groups of 32 are an odd number of words a lane; they are fetched a word at a time instead of in pairs, which would drop the last word. Layout.y_stride (default N) sets the elements between output rows in every kernel (tf_lane, tf_lane_pair, tf_lane_reg), so a member of a fused stack writes its columns of the fused rows; the output bound checks follow it. The store of the simdgroup-matrix layout (Macs without tensor units, ashhart#679) takes the same stride and keeps its column guard. N may be a multiple of 16 on the plain kernel: the last 32-column tile is packed with zero codes past N, its metadata reads and stores are already guarded by N, and weightWords counts the padded tiles (sourceWords is MLX's W). Cooperative and reg layouts still need N % 32 == 0. encodeSums checks its sums buffer by the group count, as ashhart#698 sizes the dispatch. Without tensor units the admitted layout is unchanged: 6-bit group 32 on the plain kernel (ashhart#679); every other width still answers TensorUnitsRequired there, now checked at 2, 3 and 5 bits too. With the defaults the 4-, 6- and 8-bit kernels preprocess (metal -E) to the same text as main's for every group, schedule, output, prepared-sums/reg layout and both fragment layouts tried (148 variants), so the uniform paths keep their kernels. Host tests that fail before: 2, 3 and 5 bits validate; y_stride reaches the source and below N is refused; a 48-row member validates (cooperative and reg refuse it); every width x group x {64, 48} rows packs MLX codes bit for bit with zero padding. The device program covers every width, 48-row members and strided output on tensor units; without them (M3 Ultra): PASS 14 layouts, 1904 bit-equal row/window pairs, 208 windows equal to fn_lane's recorded bytes, 7424 CPU values (max roundoff-bound fraction 0.0144), 10 refusals, and a 48-row member and two strided layouts at 6/32 equal the unstrided rows with the gaps untouched. fn_lane's recorded windows are compared only on the layouts they were recorded at. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
affine_mm took 4- and 8-bit weights only, so a 2-bit matrix (oQ2's routed experts) had no tiled prompt product. At 2 bits a thread loads its 32 codes of a step as 8 bytes and writes each code into the bf16 tile as itself, with no offset (as at 8 bits; 0..3 is exact in bf16), so the group sums are the 4- and 8-bit path's. A first try with the 4-bit "+128" offset stayed within the float64 bound, but a reference with scales 1% off no longer exceeded it at K = 4096, so the codes enter exactly. The 4- and 8-bit sources preprocess as before. Test that fails before: "dense and gathered, each format, against exact sums on the host" adds 2/64 and 2/32, dense and gathered; source() refused 2 bits. The 4- and 8-bit sources preprocess (metal -E) to main's text at both groups, both outputs and both fragment layouts (16 variants). tf-affine-ref (next commit) holds the 2-bit products to float64. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…y row's bits in every window The device side of ashhart#633's quant-ref for the shared prompt products, item 5 of ashhart#716. tf-affine-ref takes no arguments and needs no model: seeded synthetic codes, scales, biases and inputs, packed as MLX affine. The engine module now names core/affine_mm so a check program can reach it. - affine_mm at 2, 4 and 8 bits, groups 32 and 64, bf16 and fp32 out, dense and as a one-expert gather, at [192, 1024] and [256, 4096]. - simd_qmm (4-bit, group 64) as tiles and as the scalar twin at 1 and 2 rows, at [40, 512], [256, 4096] and [6400, 256] (K chunks of 32, 16 and 8). The reference dequantizes the same codes in float64 and multiplies in float64. Each product prints its worst column (max abs error, and that error over the column's largest |y|) and its largest error as a fraction of the bound: (group + K / group + 8) fp32 roundings of the terms' magnitudes as the kernels sum them (4-bit affine_mm codes plus 128), plus one bf16 ulp for bf16 outputs. A reference with its scales 1% off must exceed the bound. Then every row of a 129-row call must equal, byte for byte, the same row computed in windows of 1, 2, 3, 8, 16, 17, 33, 64 and 65 rows (every full window over the 129 rows, so each row is seen at every position), which crosses the 16-row fragments, the 32- and 64-row tiles and the gather's switch between them. The twin must equal the tiles wherever twinExact says it does, since only then do callers take it. Receipts, M3 Ultra (no tensor units: nax.h's simdgroup-matrix layout), on main 0efbd44: - every product within 0.4935 of its bound; every row's bits equal at every window, both kernels, every format; the twin equals the tiles at 1 and 2 rows on all three shapes, as twinExact says. - worst columns, bf16 out (dense and gather equal), at [192, 1024] and [256, 4096]: affine_mm q2 g32 2.64e-2 (rel 3.23e-3) and 6.22e-2 (3.78e-3), q2 g64 2.87e-2 (3.51e-3) and 3.12e-2 (2.47e-3), q4 g32 2.45e-2 (2.65e-3) and 3.12e-2 (3.01e-3), q4 g64 1.56e-2 (2.89e-3) and 3.12e-2 (2.53e-3), q8 g32 0.496 (3.41e-3) and 0.968 (3.63e-3), q8 g64 0.453 (2.87e-3) and 0.956 (3.38e-3); simd_qmm 1.19e-2 (2.57e-3), 3.13e-2 (2.87e-3) and 1.54e-2 (3.74e-3). All under 2^-8 of the column's largest |y|; 0.20 to 0.49 of the bound. - fp32 out: at most 0.0081 of the bound (q2 1.5e-6 to 7.7e-6, rel 1.9e-7 to 6.0e-7). - scales 1% off: 2.3 to 29 times the bound for bf16 outputs, 2.7 to 391 for fp32, 44 to 273 for simd_qmm. Not run: an M5, where affine_mm takes the tensor units; `zig build tf-affine-ref && ./zig-out/bin/tf-affine-ref`. The fp32 bound is the worst case for any summation order, so a changed order still passes; the measured errors sit two orders below it, and the bit checks are what pin the order. Gate on main 0efbd44 (M3 Ultra, Zig 0.17.0): `zig build test test-golden -Dcpu=apple_m1` has 309/319 goldens equal and 0 unexpected; every step passes but one that fails on main too (core.checkpoint_metal's split-checkpoint test: `get` logs an error on the DeferredTensor path the test expects). lean_check: 536 problems (main 539), no new findings.
…nst the kernels this chip runs Only the modules the engine loads are resolved, so draft-head and vision overrides never refuse a checkpoint. A width or group the projection kernels cannot run yet refuses naming the module, layer and width; fused members of different widths are admitted as runs of one width.
… width, mixed fused groups run per width Uniform 4-bit g64 keeps its bytes, kernels and launches. Other widths take the row kernel before tensor units and the non-reg lane kernel on them; fused members of different widths become runs, each its own product into the fused rows, one row a launch until the core output stride lands. The embedding takes the checkpoint's width. The Neural Engine share declines unless every MLP matrix is 4-bit g64, and the DFlash2 draft head refuses a non-4-bit embedding or head for now. Synthetic graphs load at uniform 4-bit g64.
…eckpoint widths The draft vocabulary copies whole rows (or 32-row tiles) of k * bits / 8 bytes and k / group metadata, and the borrowed embedding is validated at its resolved width. The draft-head check covers 2-, 4- and 8-bit heads.
…nce does The reference stacks fused members only when all share one width and group; otherwise each member is its own call at its own K split. Members of one width beside a member of another no longer share a product.
…before tensor units Each member is still its own product, now one row-kernel launch over all rows writing its columns at the stack width, in place of one launch a row. The row kernel gives a row the same bits in any window, so every row count still matches.
…ainst float64 without tensor units tf-qwen27-widths runs the family's own products (quant_gpu.Kernels.quant on projection.prepare's matrices: the row kernels, or simd_qmm at 4/64, as a chip without tensor units picks them) on synthetic MLX affine weights at the shape of every projection class (GDN qkv, z, b, a and out; attention q, k, v and o; MLP gate_up and down; lm_head) at 2, 3, 4, 5, 6 and 8 bits, group 64, which covers every width oQ4 and oQ2 use: - a prompt call (128 rows) and decoded rows (one row a call) against core.quant.mlx.reference in float64, the worst column within the fp32-sum bound; scales 1% off exceed the bound in every class and width; - every row's bits equal in calls of 1, 2, 3, 8, 16, 17 and 128 rows; - the mixed stacks [z4 | b5 a5], [z4 b4 | a5], [z2 b2 | a3] and [k5 | v6], built as the loader builds them: every row's bits equal in every call, each member's columns within the float64 bound and equal bit for bit to its own product (the row kernel at its width, unstrided), and to the member loaded as a matrix of its own, except a 4-bit z, which alone takes simd_qmm (no output stride). The engine module exports core.quant for the reference. The program refuses a chip with tensor units, whose lane kernels it does not check. M3 Ultra (no tensor units): PASS 10,575,872 values within 0.482 of the float64 bound (bf16 output rounding; worst-column |err| / max|y| 2.5e-3 to 3.9e-3), scales 1% off at least 2.1 times the bound in every class; 1,515,200,224 bit-equal pairs; 31 s. A 4-bit z inside a mixed stack differs from the same z as a uniform layer's simd_qmm product in 51 and 72 of 128 rows, by up to 3.1e-2; both are within the bound. zig build test test-golden -Dcpu=apple_m1: 158/160 steps, 1566/1574 tests (8 skipped), goldens 309/319 with 0 unexpected differences; the one failure is main's core.checkpoint_metal split-checkpoint test. tools/zig/lean_check.py: 536 findings, 0 new against main.
…oat64 tf-qwen27-widths now also runs the family's embedding gather (glue_gpu.Kernels.embedding, q27_embed) on a synthetic MLX affine table at the 27B's shape [248320, 5120], group 64, at 8 bits (oQ2's embedding) and 3 bits, for 64 tokens including both ends of the vocabulary and a repeat: - every gathered value equals core.quant.mlx.reference in float64 rounded once to bf16, bit for bit; - one-row calls give the same bits as one 64-row call; - the reference with scales 1% off differs in a large share of values, so the comparison sees a wrong decode. M3 Ultra (no tensor units): 8-bit 0 of 327,680 values differ, 3-bit 0 of 327,680; one-row calls equal; scales 1% off differ in 160,115 and 142,528 values. The projection checks are unchanged (PASS 10,575,872 values within 0.482 of the bound). zig build test test-golden -Dcpu=apple_m1: 158/160 steps, goldens 309/319 with 0 unexpected differences; the one failure is main's core.checkpoint_metal split-checkpoint test. tools/zig/lean_check.py: 536 findings, 0 new.
…pt chunk seams and after decode --compare-chunk compared Session.fingerprint() (every GDN layer's recurrent and conv state, the visible KV rows of every attention layer, the last row's bf16 logits) at the prompt end only, and decode by tokens. With --compare-chunk C it now also compares the fingerprint after the decoded reply in both arms (the prefill to decode seam and every decode step's state), failing with DecodeStateOrLogitsDiffer. --seams N adds, for every N-th prompt row, a prefill of the prompt up to that row at --chunk and at C, fingerprints compared byte for byte; the receipt lists each seam and seams_byte_equal, and any difference fails with SeamStateOrLogitsDiffer. Test program only; nothing in the engine. M3 Ultra (no tensor units), the exact 2048-token public prefill array, chunk 128 against 16 and against 37, seams every 256 rows, 64 decoded tokens: every seam, the prompt end and the decoded state byte-equal on the uniform 4-bit checkpoint and on oQ4 and oQ2.
…kpoint never drafts on another's costs Window costs are kept per build, chip and model shape, and every 27B checkpoint has one shape: whichever checkpoint a build served first wrote the table every later one read. An oQ2 server then chose its draft depth from the uniform 4-bit costs (wide windows nearly free) and drafted deeper than its row kernels pay for: same tokens, 28% more drafted rows, 15% slower code decode. A mixed-width checkpoint now adds a tag of its widths to the shape; a uniform 4-bit g64 checkpoint keeps its shape string, so its key and costs are unchanged.
…he row kernels, on every chip With tensor units, projection.prepare refused any run with n % tile != 0 (BadProjectionShape) and widths.check gave every 4/64 run the reg lane layout, which needs n % 32 == 0. A lone 48-row GDN in_proj_a or in_proj_b is such a run: TensorFold/Qwen3.8-27B-oQ4's layer 37 (in_proj_a at 5/64, in_proj_b a 48-row 4/64 run) was refused at admission and -oQ2 failed at load on M5. Those runs now take the row kernels, as they do before tensor units; mixed stacks already send a row run to rowStrided. A 4/64 run off the 32-row lanes stays out of regOrder. Before tensor units nothing changes (those runs already took prepareRows), and a uniform 4/64 checkpoint's matrices are multiples of 32 rows. Test: widths.zig: a 48-row 4/64 run admits on tensor units through the row kernels, a 64-row one through the lanes (fails before: the lane layout refused n % 32 != 0). Reported with the fix by Seth For Privacy (@sethforprivacy) on an M5 Ultra (ashhart#756), with oQ4 and oQ2 served exactly there.
…, and profile shapes by code width tf-qwen27-classes times every layer's matrices of one projection class in one command buffer at 1, 4, 16 and 128 rows on real weights and prints the GPU ms a step, the bytes it reads (codes, scales, biases), the code widths by share and the ratio to the bytes at a given bandwidth. Profiled dense shapes now carry the code width, so a mixed checkpoint's widths of one shape stay apart. No kernel or launch changes.
…ry mixed-stack member run on simd_qmm tiles The row kernel reads a matrix once for every two rows, so a 16-row verify window read each oQ2 matrix eight times and a 128-row prompt chunk sixty-four. simd_qmm_bits runs simd_qmm's 8x8 MMA tiles, K chunks, reduction tree and 1-2 row scalar twin at any code width in groups of 64, with exact codes as floats (4-bit keeps simd_qmm's masked nibbles), and writes rows at a stride. A row's bits depend on n and k alone, never on the window; the twin is checked against the tiles once a width and shape, as simd_qmm's is. Mixed-stack members take it at their width, 4-bit included, so a 4-bit member of a mixed stack now equals the same matrix in a uniform layer bit for bit (it took the strided row kernel before). Uniform 4-bit g64 matrices keep simd_qmm, its kernels and launches. Groups other than 64 keep the row kernel. tf-qwen27-widths (12 classes x 2/3/4/5/6/8 bits, 4 mixed stacks, embedding): PASS 10,575,872 values, worst 0.4815 of the float64 bound (0.4820 before), every class's worst column the same as before, control >= 2.1x; every row's bits equal in calls of 1, 2, 3, 8, 16, 17 and 128 rows; each stack member equals its own product and the member as its own matrix. tf-qwen27-classes, projections of a step on M3 Ultra (ms; before -> after; uniform 4-bit for reference): oQ2 1 row 18.8 -> 20.8, 4 rows 40.5 -> 28.6, 16 rows 153.4 -> 47.0, 128 rows 1223.7 -> 337.0 (uniform 23.2/33.1/58.5/419) oQ4 1 row 23.3 -> 24.1, 4 rows 36.4 -> 32.7, 16 rows 84.2 -> 55.7, 128 rows 639.1 -> 398.4
…d_qmm masks 4-bit nibbles Each 2-bit code is read as code * 2^(2s + 16h) from its word with one AND, and its input is pre-scaled by 2^-(2s + 16h), so each product is the same float as before with no shift a code, in the 8x8 tiles and the 1-2 row twin. Other widths are unchanged. tf-qwen27-widths: PASS, every class's worst column and every bit-equal pair as before (10,575,872 values, worst 0.4815 of the float64 bound; 1,516,773,088 bit-equal pairs). tf-qwen27-classes, projections of a step on M3 Ultra (ms, before -> after): oQ2 1 row 20.24 -> 16.53, 2 rows 24.23 -> 21.84, 4 rows 28.50 -> 27.70; 16 and 128 rows and oQ4 unchanged.
grantoverton
force-pushed
the
oqxe/speed-27b
branch
from
October 13, 2026 20:21
f738b48 to
badfc9d
Compare
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
ThinkOffApp
pushed a commit
to ThinkOffApp/TensorFold
that referenced
this pull request
Oct 14, 2026
…sor units only The contributor's three commits landed above with the same code. Our commit after them gives the tiles the 27B's row-major matrices before tensor units only, where they were the only ones until the 48-row fix. This keeps the PR's head in the history with no change to the tree.
Owner
|
Thank you, @grantoverton. Before tensor units, Qwen3.8-27B's projections at 2, 3, 5, 6 and 8 bits, and every member of a mixed stack, run on Your commits are in 1.0.10 under your name ( On your M3 Ultra, at |
Open
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…sor units only The contributor's three commits landed above with the same code. Our commit after them gives the tiles the 27B's row-major matrices before tensor units only, where they were the only ones until the 48-row fix. This keeps the PR's head in the history with no change to the tree.
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
The contributor's commits landed on this line earlier with the same code, rebased here onto 1.0.9; this keeps the current head in the history with no change to the tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto main c36b536 (1.0.9) and #756's rebased tip on 2026-10-13: no conflicts in this change's code; the tip passes the gate again. 27B oQ2 on the rebased tip serves exactly (concurrent equals solo, cancellation passes) at 127.2 / 70.1 tok/s (code / chat) with 1.0.9's own 27B speedups underneath; the tables below are the pre-rebase runs.
Part of #716; stacks on #756 (which stacks on #755). This branch carries their sixteen commits until they merge, so please review the last three commits here (from "tests: qwen27-classes" on). In #756 every 27B projection at a width other than 4/64, and every member of a mixed stack, runs on the row kernel, which reads a matrix once for every two rows: a 16-row verify window reads each oQ2 matrix eight times, a 128-row prompt chunk sixty-four. This runs them on
simd_qmm's tiles at their own width instead. On an M3 Ultra, oQ2 decodes 2.0× (code) and 1.6× (chat) faster and prefills 2.4-3.3× faster; oQ4 1.3× and 1.3-1.4×. The uniform 4/64 path keepssimd_qmm, its kernels and launches, and its tokens.What this changes
tf-qwen27-classes(a test program): GPU time per 27B projection class at 1, 2, 4, 8, 16 and 128 rows against the class's weight bytes at 800 GB/s, every layer's matrices of a class in one command buffer; profile shapes carry their code width.simd_qmmtiles.core/simd_qmm_bitsrunssimd_qmm's 8×8 MMA tiles, K chunks, reduction tree and 1-2 row scalar twin at any code width in groups of 64, with exact codes as floats (4-bit keepssimd_qmm's masked nibbles), and writes rows at a stride. A row's bits depend on n and k alone, never on the window; the twin is checked against the tiles once a width and shape, assimd_qmm's is. Mixed-stack members take it at their width, 4-bit included, so a 4-bit member of a mixed stack now equals the same matrix in a uniform layer bit for bit (it took the strided row kernel before). Uniform 4-bit g64 matrices keepsimd_qmm. Groups other than 64 keep the row kernel.simd_qmm_bitsmasks 2-bit codes in place, assimd_qmmmasks 4-bit nibbles. Each 2-bit code is read as code × 2^(2s + 16h) from its word with one AND, and its input is pre-scaled by 2^-(2s + 16h), so each product is the same float with no shift a code, in the tiles and the twin.M5 (tensor units): the 27B runs its lane kernels on tensor units and the row kernels before them (
widths.Chip); this replaces only the row-kernel path, so the tensor-unit path is unchanged at every width. M5 speed at oQ widths is unmeasured (no M5).Exactness
TensorFold/Qwen3.8-27B-oQ4 @fb60862, TensorFold/Qwen3.8-27B-oQ2 @086dcb0 and the uniform Vontra/Qwen3.8-27B-MLX-4bit @70ae7fa, each with z-lab/Qwen3.8-27B-DFlash2, greedy, drafts on unless
"draft": false;serve <checkpoint> --drafter <DFlash2> --parallel 4 --context 65536 --prompt-cache-gib 0. Before is #756's tip 8c05ac9, after this branch's tip f738b48.bench_concurrent's alone and serial checks at 1 and 4 requests, both prompts: 34/34 on each); cancellation, reset and partial acceptance, 25 steps, pass on each (a fresh"draft": falseprobe gives the reference hash and the drafted probe equals it, with partial acceptance seen on both prompts; then after a stream dropped mid-decode, a long prompt dropped mid-prompt, a stream dropped beside a running probe and a request past the window, the drafted and plain probes equal the fresh hashes; hashes oQ4c201c615aba6/f2e2d479cadc, oQ22ec414650f51/67cf87c9db04).70871d78a690→94b612b4e273, oQ2af21ef189db8→47a2565c099c). Fidelity againstmlx_lmimproves on both (below).tf-qwen27-widths(Qwen3.8-27B support for oQ in Zig engine #756's program; 12 classes × 2/3/4/5/6/8 bits, 4 mixed stacks, the embedding): PASS, 10,575,872 values, worst 0.4815 of the float64 bound (0.4820 on Qwen3.8-27B support for oQ in Zig engine #756), every class's worst column the same, scales 1% off at least 2.1× the bound; every row's bits equal in calls of 1, 2, 3, 8, 16, 17 and 128 rows; each stack member equals its own product and the member as its own matrix. Commit 3 leaves every value and bit-equal pair as commit 2 has them.Performance
Mac Studio, M3 Ultra (80-core GPU), 256 GB, macOS 26.4.1 (25E253), Zig 0.17.0. Fresh builds,
zig build native tf-qwen27-run -Dcpu=apple_m1. Every run starts from a fresh server, with other checkpoints' file cache evicted first; before and after alternate (before, after, after, before).Decode and concurrency.
bench_openai.py … --tokens 256 --reps 3 --temperatures 0(medians;fibonacci-raw, code, andgpu-chat-no-think, chat) andbench_concurrent.py --levels 1,4 --alone --serial(aggregate tok/s):bench_openaicodeoQ2 now decodes code 1.32× and chat 0.96× of the uniform checkpoint on the same build, and past
mlx_lm0.32.0 on the same oQ2 checkpoint (71.52 / 47.80 in #756) by 1.74× and 1.38×; oQ4 decodes code 1.20× and chat 0.95× of uniform. The uniform runs are not this PR's: their kernels are unchanged and their tokens identical, and after draws three fewer rounds on chat (72-73 against 75, deeper drafts) because each build times its own window costs once and caches them. The code prompt's rounds are the same (53-54).Cold prefill (oQ2 and oQ4; before and after). Exact token arrays: upstream's public prefill paragraph ("A lighthouse keeper writes in the log as the boats cross the bay.",
tools/zig/qwen27_gate.py) repeated and tokenized once without special tokens;/v1/completionswith the ids,max_tokens 1,--parallel 1 --context 34816 --prompt-cache-gib 0, every request cold (cached_tokens0), a 2k warmup, one rep a size:The uniform 4/64 prefills at 286.4 / 258.8 / 183.0 tok/s at these sizes in #756 (main's rate), and its prompt kernels are unchanged here. oQ2 now prefills faster than the uniform checkpoint; oQ4 at 0.96-1.01× of it. 64k and the native window not run.
Per projection class (
tf-qwen27-classes, the projections of one step, ms; the before column is #756's row kernel, profiled on the branch before its rebase onto #756's tip):Weight bytes a step: uniform 14.4 GB, oQ4 15.0 GB, oQ2 9.0 GB (18.0 / 18.8 / 11.3 ms at 800 GB/s). Commit 3 takes oQ2's 1-row step from 20.24 to 16.53 ms, under the row kernel's 18.8. Three more exact steps were tried and dropped because they were slower where they aimed: four outputs a lane at 2-3 bits (oQ2 1 row 19.87 ms, 2 rows 33.12), 32-row tiles past 16 rows (oQ2 128 rows 663.9 ms) and a 3-4 row twin (oQ2 4 rows 57.2 ms).
Memory
Physical footprint (
darwin_rusage_info_v4, the server process,--parallel 4 --context 65536, DFlash2's 3.6 GB included), the same before and after: after load oQ2 19.9 GB, oQ4 25.4 GB, uniform 24.8 GB; the four streams take 12.3 GiB more on first use, so the runs that used them (decode + concurrent) peak at oQ2 33.22 GB and oQ4 38.70 GB, as the uniform checkpoint's 38.16 GB in #756. The uniform runs here were decode only (one stream) and peak at 24.93 GB.Fidelity
Teacher-forced against
mlx_lm0.32.0 on the same checkpoints and the same fixed histories as #756 (mlx_lm's reference reused): upstream's four public teacher prompts, 1,024 positions, top two at every position fromtf-qwen27-run --run --teacher --force-length --chunk 128 --compare-chunk 16.oQ2 still fails upstream's judge, with one fewer position past the tie policy than #756; copy position 14 is the widest (this branch's pick is
mlx_lm's sixth, margin 1.1875 there).Every difference (`mlx_lm`'s pick and margin; this branch's pick and margin; its rank in `mlx_lm`'s top 20)
mlx_lmpickHost suite and lean check
zig build test test-golden -Dcpu=apple_m1at each of the three commits, from a clean detached worktree: 158 of 160 steps; goldens 309/319 equal, the 10 declared differences, 0 unexpected. The one failing step is main's owncore.checkpoint_metalsplit-checkpoint test (it logs an error on the path it expects), as in Core support for oQ in Zig engine #755 and Qwen3.8-27B support for oQ in Zig engine #756.tf-qwen27-widthspasses at the tip (above);simd_qmm_bits's twin is checked against its tiles at every width and shape it runs.python3 tools/zig/lean_check.py: 536 problems against main's 539 at every commit, no new findings. Every touched file is at or under 600 lines;zig fmtclean.Platforms
then, greedy: a request and the same with
"draft": false(same tokens), and two requests at once against each alone.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (oQ4 and oQ2 served on the tip; uniform tokens equal Qwen3.8-27B support for oQ in Zig engine #756's).Receipts: https://gist.github.com/grantoverton/75ffd02f8d28ab21bbde8022dfff4671