Repository navigation
DeepSeek V4.1: sparse attention on eight SIMD groups, the same bytes - #4
Open
cwschroeder wants to merge 2 commits into
Open
cwschroeder wants to merge 2 commits into
cwschroeder wants to merge 2 commits into
Conversation
…D groups, the same bytes dsv41_sparse_attention runs one SIMD group per head and row: it scores the 64 keys of each tile one after another, then accumulates all sixteen 32-wide output columns. dsv41_sparse_attention_parallel launches eight SIMD groups (256 threads). They score different keys of a tile into threadgroup memory; every lane then reads the tile's scores in key order, and each group accumulates two of the sixteen output columns over the same keys. Each dot product, running maximum, tile sum and value sum keeps the predecessor's order, so the outputs are the same bytes. TF_DSV41_PARALLEL_ATTENTION=1 selects it; the default stays the predecessor. Encoding refuses a pipeline that cannot take the launch. tf-dsv41-run records the choice as attention_parallel. csa2_parallel_test compares both kernels byte for byte over 51 shapes, including tile edges, nonfinite inputs and the guards around each output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h bit-equal to the one it replaces metal_dsv41_attention.zig runs dsv41_sparse_attention_parallel through gates 1 to 4: nonfinite classes against float64, five byte-identical repeats with untouched guards, 14 shapes bit-equal to dsv41_sparse_attention, and exact sums at k 640. Its launch declares [[max_total_threads_per_threadgroup(256)]] and is registered with that reservation; the predecessor is registered with its 32 threads and the M1 and M2 limits of Xcode's offline compiler. The records live in dsv41_metal_records.zig and join all_metal, as the DeltaNet slots and Flash Next vision records do, so registry.zig stays at its line count. The family compiles these kernels with precise math functions; the limits check and the offline sweep now build them the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dsv41_sparse_attentionruns one SIMD group (32 threads) per head and row and scores a tile's 64 keys one after another.dsv41_sparse_attention_parallellaunches eight SIMD groups per head and row. They score different keys of the tile into threadgroup memory; every lane then reads the scores in key order, and each group accumulates two of the sixteen 32-wide output columns over the same keys. Each dot product, running maximum, tile sum and value sum keeps the predecessor's order, so the outputs are the same bytes.TF_DSV41_PARALLEL_ATTENTION=1. The default stays the predecessor, because a 268-token teacher prefill measured 2% slower (below). Whether it should become the default is your call.[[max_total_threads_per_threadgroup(256)]]). The records live indsv41_metal_records.zigand joinall_metal, as the DeltaNet slot and Flash Next vision records do, soregistry.zigkeeps 598 lines (600 with DeepSeek V4.1: the indexer's merge and finish on a threadgroup a row, the same picks; a decode row's indexer 13-34x faster #2).pr-dsv41-bob-indexer. I leftdocs/kernel-contract.mdalone because DeepSeek V4.1: the indexer's merge and finish on a threadgroup a row, the same picks; a decode row's indexer 13-34x faster #2 rewrites the sentence that lists the DeepSeek kernels; the sparse attention can join it once DeepSeek V4.1: the indexer's merge and finish on a threadgroup a row, the same picks; a decode row's indexer 13-34x faster #2 lands.On this head (
ec1f47f2; Apple M5 Max, macOS 26.6.2, Xcode 26.6, Zig 0.17.0):zig build test-deepseek-v41 test-kernel-contract test-deepseek-v41-weights --summary all, fresh cache: 55/55 steps, 158/158 tests passed.zig build test --summary line: 356/360 steps, 4,903/4,934 tests (21 skipped, 10 failed). All 10 are server tests in the 506-test binary thattestandtest-cuda-hostboth run: Cortheon's linear-time readers, the health manifest and the benchmark CLI, run and stream tests. None is a DeepSeek or contract test.test-cuda-hoston the base fails in the same binary too (a benchmark stream timeout), and the failing set changes from run to run on both, so I read them as timing on a busy laptop rather than this change.csa2_parallel_test: 51 shapes, predecessor bytes and guards equal. Warm attention call, median of 32 samples: position 128, 2.393 to 0.685 ms; position 8191, 5.327 to 1.326 ms. These are laptop timings and vary between runs.tf-kernel-contract --only deepseek_v41:SUMMARY 1 kernels: 1 pass gates 1-4, 0 fail, 0 known findings. Gate 5 here: 256 threads and 8,192 threadgroup bytes for the parallel kernel, 32 threads and 256 bytes for the predecessor (this GPU's most 32,768). The 12 limit problems are the base's own Flash Next "no compiled thread limit recorded" findings.--offline): 161 launches, 159 on the base. The predecessor passes at its recorded 768/768. The parallel kernel reads N/A (no M1/M2 build (ToolFailed)on this Xcode, as for the tile_qmm and Qwen launches) and rests on its 256-thread reservation. The 18 problems are the base's 12 declaration findings plus the same six Flash Next pipelines measured offline (recorded 0, 0); none is a DeepSeek launch.zig fmt --checkandgit diff --checkclean.lean_check.py: 444 problems, 444 on the base, 0 new.test-deepseek-v41andtest-deepseek-v41-weights.Whole model, on our 256 GB line. This family keeps about 414 GB resident, more than our Macs hold, so the full model ran on our integration branch, which streams the Engram tables from SSD. There
csa2.metal,csa2.zig,csa2_parallel_test.zigand the contract test are byte-identical to this PR (61ab890f,cdfa5203); its other decode work differs from this line. One M3 Ultra (80-core GPU, 256 GB),LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpwatafc7aa0a, the original 1017 Engram tables, 32 prompt tokens and 127 decode forwards, three runs per mode:Token IDs, the 268-token teacher logits and one-token teacher chunks are byte-identical between the two modes, and two interleaved lanes match solo execution bit for bit. Receipt: ds1017-attention-2026-10-19.
Not run on this head: the full model. If the 512 GB M3 Ultra has a slot, a decode run with and without the switch would put the speed on your line.
🤖 Generated with Claude Code