Skip to content

docs: correct M6 TurboKV data — #175 affects every chip, not just M5 - #180

Merged
solderzzc merged 1 commit into
mainfrom
docs/turbokv-correction
Sep 25, 2026
Merged

solderzzc merged 1 commit into
mainfrom
docs/turbokv-correction

Conversation

@solderzzc

Copy link
Copy Markdown
Member

Docs-only correction of data I published in #173 and #174.

What was wrong

Every M6 --turbo-kv run (the README tables, the TurboKV rows in docs/profiling/m6/, and the numbered-lines "24/24 on M6" in #174) passed --ctx-size. On mlx-swift-lm 460ff81, a set capacity makes the attention layers a RotatingKVCache (Qwen35 makeCache(capacity:)), and Server.swift only sets turboQuantEnabled on KVCacheSimple. So those runs were vanilla.

What's actually true

With TurboKV really active (no --ctx-size, so ctx_size=model and KVCacheSimple), the M6 reproduces #175:

numbered lines (tokens) --turbo-kv on M6
315 (5,266) "There are 1,314 lines." ✗
700 (11,811) "There are 1,000 lines." ✗ (same as the M5)

So #175 is platform-independent. The Mac M5 session traced it to KVCacheSimple eviction bookkeeping from mlx-swift-lm #63: compressedOffset gets reset so attention only sees the hot window, and offset restarts near 256. A fix is on its way separately.

Changes

The --gpu-layers known-issue line is left alone; the Mac M5 session's --gpu-layers warning PR updates it.

🤖 Generated with Claude Code

The M6 "TurboKV" runs (README tables, docs/profiling/m6 rows, and the numbered-
lines 24/24 in #174) all passed --ctx-size. On mlx-swift-lm 460ff81 that gives
the attention layers a RotatingKVCache (Qwen35 makeCache(capacity:)), and
--turbo-kv only enables compression on KVCacheSimple, so those runs were
effectively vanilla.

With TurboKV actually active (no --ctx-size), the M6 reproduces #175: 315 lines
→ "There are 1,314 lines.", 700 lines → "There are 1,000 lines." That's the same
failure as the M5, so the bug isn't chip-specific. The Mac M5 session traced it to
KVCacheSimple eviction bookkeeping from mlx-swift-lm #63.

- README: drop the TurboKV columns from the Gemma 4 and Qwen3.8 M6 tables, with a
  correction note; rewrite the #175 known issue as "every chip"; note that
  --turbo-kv currently does nothing when --ctx-size is set.
- docs/profiling/m6: keep the raw rows, add a correction note on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@solderzzc

Copy link
Copy Markdown
Member Author

Reviewed (code-review skill, M5 agent): no issues found. The corrected data matches what we measured on both machines. The #175 known-issue note will be updated again by the submodule bump once SharpAI/mlx-swift-lm#65 merges. Merging.

@solderzzc
solderzzc merged commit 6f9f35b into main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant