diff --git a/README.md b/README.md index cfd3a67..494a659 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,9 @@ TurboKV barely changes speed on this model. Only 16 of its 64 layers use full at 3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. 4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump. -> ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. +> ⚠️ **`--turbo-kv` precision on M5 (not reproduced on M6):** on an Apple M5, Qwen3.8-27B-4bit with `--turbo-kv` gets exact long-range lookups wrong from somewhere between 2K and 5K prompt tokens. Asked how many numbered lines a prompt has, it answers "1,000" or "14" instead of 315 / 500 / 700. Without `--turbo-kv` it answers correctly, and on the M6 both modes are 24/24 correct from 2K to 11.8K tokens with the same prompt. The likely cause is a GPU-family-dependent path in TurboKV's dequant or attention kernels. Until it's fixed, avoid `--turbo-kv` on M5 when exact recall matters. Tracked in [#175](https://github.com/SharpAI/SwiftLM/issues/175). +> +> ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request, on both M5 and M6 (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). Tracked in [#176](https://github.com/SharpAI/SwiftLM/issues/176); the fix is [SharpAI/mlx-swift#17](https://github.com/SharpAI/mlx-swift/pull/17). Even once it's fixed, CPU-resident MoE layers are very slow (~0.4 tok/s prefill), so on a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 7.7 GB GPU). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. Reproduce: