docs: correct M6 TurboKV data — #175 affects every chip, not just M5 - #180
Merged
Merged
Conversation
The M6 "TurboKV" runs (README tables, docs/profiling/m6 rows, and the numbered- lines 24/24 in #174) all passed --ctx-size. On mlx-swift-lm 460ff81 that gives the attention layers a RotatingKVCache (Qwen35 makeCache(capacity:)), and --turbo-kv only enables compression on KVCacheSimple, so those runs were effectively vanilla. With TurboKV actually active (no --ctx-size), the M6 reproduces #175: 315 lines → "There are 1,314 lines.", 700 lines → "There are 1,000 lines." That's the same failure as the M5, so the bug isn't chip-specific. The Mac M5 session traced it to KVCacheSimple eviction bookkeeping from mlx-swift-lm #63. - README: drop the TurboKV columns from the Gemma 4 and Qwen3.8 M6 tables, with a correction note; rewrite the #175 known issue as "every chip"; note that --turbo-kv currently does nothing when --ctx-size is set. - docs/profiling/m6: keep the raw rows, add a correction note on top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Reviewed (code-review skill, M5 agent): no issues found. The corrected data matches what we measured on both machines. The #175 known-issue note will be updated again by the submodule bump once SharpAI/mlx-swift-lm#65 merges. Merging. |
This was referenced Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs-only correction of data I published in #173 and #174.
What was wrong
Every M6
--turbo-kvrun (the README tables, theTurboKVrows indocs/profiling/m6/, and the numbered-lines "24/24 on M6" in #174) passed--ctx-size. On mlx-swift-lm460ff81, a set capacity makes the attention layers aRotatingKVCache(Qwen35makeCache(capacity:)), andServer.swiftonly setsturboQuantEnabledonKVCacheSimple. So those runs were vanilla.What's actually true
With TurboKV really active (no
--ctx-size, soctx_size=modelandKVCacheSimple), the M6 reproduces #175:--turbo-kvon M6So #175 is platform-independent. The Mac M5 session traced it to
KVCacheSimpleeviction bookkeeping from mlx-swift-lm #63:compressedOffsetgets reset so attention only sees the hot window, andoffsetrestarts near 256. A fix is on its way separately.Changes
--turbo-kvloses context history after 2K tokens (attention sees only the last 256 tokens) #175 known issue as "every chip", and note that--turbo-kvcurrently does nothing when--ctx-sizeis set.docs/profiling/m6/*.md: keep the raw rows, and add a correction note on top.The
--gpu-layersknown-issue line is left alone; the Mac M5 session's--gpu-layerswarning PR updates it.🤖 Generated with Claude Code