From a709b777bb2af54e67008038cf26794a620b5ac4 Mon Sep 17 00:00:00 2001 From: simba Date: Thu, 24 Sep 2026 09:59:40 -0700 Subject: [PATCH 1/2] docs(readme): TurboKV precision known issue on M5 (not reproduced on M6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reproduced by the M5 session on main 318f712 with Qwen3.8-27B-4bit --vision at temperature 0. Prompt: n numbered lines ("Line {i}: the quick brown fox jumps over the lazy dog."), then "How many lines are there? Answer briefly." With --turbo-kv on M5: 125 lines → "125", 315 → "1,000", 500 → "14", 600 → "1,000", 700 → "1,000" (all repeated 2–3×). Without --turbo-kv it answers 315 / 500 / 700 correctly. On the M6 the same prompts are 24/24 correct in both modes, 2K–11.8K tokens. Points to a GPU-family-dependent TurboKV kernel path. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index cfd3a67..32a4f01 100644 --- a/README.md +++ b/README.md @@ -121,6 +121,8 @@ TurboKV barely changes speed on this model. Only 16 of its 64 layers use full at 3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. 4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump. +> ⚠️ **`--turbo-kv` precision on M5 (not reproduced on M6):** on an Apple M5, Qwen3.8-27B-4bit with `--turbo-kv` gets exact long-range lookups wrong from somewhere between 2K and 5K prompt tokens. Asked how many numbered lines a prompt has, it answers "1,000" or "14" instead of 315 / 500 / 700. Without `--turbo-kv` it answers correctly, and on the M6 both modes are 24/24 correct from 2K to 11.8K tokens with the same prompt. The likely cause is a GPU-family-dependent path in TurboKV's dequant or attention kernels. Until it's fixed, avoid `--turbo-kv` on M5 when exact recall matters. +> > ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. Reproduce: From a5b8f36277b02081c0cf30aed6ac0f3566e6b7a0 Mon Sep 17 00:00:00 2001 From: simba Date: Thu, 24 Sep 2026 11:05:22 -0700 Subject: [PATCH 2/2] docs(readme): link #175 / #176 from the known issues; recommend --stream-experts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - TurboKV-on-M5 note → tracking issue #175. - --gpu-layers GPU timeout: reproduced on M5 as well as M6; tracked in #176 with the fix in SharpAI/mlx-swift#17. CPU-resident MoE layers stay very slow after the fix (~0.4 tok/s prefill), so the note points 32 GB users at --stream-experts first. Co-Authored-By: Claude Opus 5.5 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 32a4f01..494a659 100644 --- a/README.md +++ b/README.md @@ -121,9 +121,9 @@ TurboKV barely changes speed on this model. Only 16 of its 64 layers use full at 3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. 4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump. -> ⚠️ **`--turbo-kv` precision on M5 (not reproduced on M6):** on an Apple M5, Qwen3.8-27B-4bit with `--turbo-kv` gets exact long-range lookups wrong from somewhere between 2K and 5K prompt tokens. Asked how many numbered lines a prompt has, it answers "1,000" or "14" instead of 315 / 500 / 700. Without `--turbo-kv` it answers correctly, and on the M6 both modes are 24/24 correct from 2K to 11.8K tokens with the same prompt. The likely cause is a GPU-family-dependent path in TurboKV's dequant or attention kernels. Until it's fixed, avoid `--turbo-kv` on M5 when exact recall matters. +> ⚠️ **`--turbo-kv` precision on M5 (not reproduced on M6):** on an Apple M5, Qwen3.8-27B-4bit with `--turbo-kv` gets exact long-range lookups wrong from somewhere between 2K and 5K prompt tokens. Asked how many numbered lines a prompt has, it answers "1,000" or "14" instead of 315 / 500 / 700. Without `--turbo-kv` it answers correctly, and on the M6 both modes are 24/24 correct from 2K to 11.8K tokens with the same prompt. The likely cause is a GPU-family-dependent path in TurboKV's dequant or attention kernels. Until it's fixed, avoid `--turbo-kv` on M5 when exact recall matters. Tracked in [#175](https://github.com/SharpAI/SwiftLM/issues/175). > -> ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. +> ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request, on both M5 and M6 (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). Tracked in [#176](https://github.com/SharpAI/SwiftLM/issues/176); the fix is [SharpAI/mlx-swift#17](https://github.com/SharpAI/mlx-swift/pull/17). Even once it's fixed, CPU-resident MoE layers are very slow (~0.4 tok/s prefill), so on a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 7.7 GB GPU). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. Reproduce: