fix(ssd): Qwen3.6 --stream-experts works again (b769/b773 crash), first-run streaming, README re-measure - #192
Merged
Merged
Conversation
- Bump mlx-swift-lm to include SharpAI/mlx-swift-lm#69 (SSD path crashes and eager weight load after the upstream sync). #71 follows before merge. - --stream-experts on a model that isn't downloaded yet now fetches it before planning, so streaming is activated for that load instead of the loader loading every expert. - README: re-measure the Qwen3.6-35B-A3B table on the fixed build and flag that b769 --stream-experts crashes on Qwen3.5/3.6. The earlier SSD numbers were from before the sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
solderzzc
changed the title
solderzzc
marked this pull request as ready for review
September 26, 2026 09:09
Review fixes for the prefetch added in this PR: - Skip it for --info, which would otherwise download the whole model. - Reuse an existing copy under the loader's hub root (localRepoLocation), so later starts don't re-list and re-HEAD every file (or re-hash them offline). - Show download progress with ProgressTracker. - Set the load phase first, so a bad or gated id reports model_load_failed instead of binary_error. - README: the crash warning covers b773 too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Release b769 crashes on the first request with
--stream-expertson Qwen3.5/3.6 MoE:The mlx-swift-lm upstream sync that came in with #167 broke the SSD path in three ways: a double unsort, evals inside compiled decode traces, and a concurrent loader that loaded every expert into memory. The README's Qwen3.6 SSD numbers came from before that sync and were never re-measured afterwards.
Changes
--mtp/draft verify, streaming scoped to the main model, end-to-end SSD tests that CI runs since chore: update mlx-swift-lm to fix/gemma4-pad-eos-token #70).--stream-experts(Server.swift): if the model isn't on disk yet,modelDirectorywas nil, soExpertStreamingConfigwas never activated and the load went eager. The model is now fetched first, with the sameHubApiroot and patterns the loader uses, so streaming activates on the first run too.docs/profiling/m6: re-measured the Qwen3.6-35B-A3B table (GPU and SSD, 548 to 40.8K tokens) on the fixed build, and added a warning that b769--stream-expertscrashes on Qwen3.5/3.6.Verification (Mac mini M6, 32 GB, median of 3, 1 run at 40.8K, needle checked in every run)
Before the sync (old README): SSD 321 / 402 / 403 / 340 prefill, 13.2 / 13.1 / 12.9 / 11.9 decode, 6.0–7.7 GB. The only remaining gap is prefill at ~550 tokens. On b769 and b773: crash. Swap growth was 0 in every run.
Final check of this exact tree (merged with main at 537a1cc, mlx-swift-lm 5596071), run back to back with the earlier fix build: SSD 548 / 2,346 tok is 252.9 / 409.7 prefill and 13.97 / 13.88 decode tok/s at 5.7 GB. The earlier build measured 251.2 / 411.2 and 14.01 / 13.93. GPU: 47.85 / 46.8 decode. Needle ok in every run.
The first-run path wasn't exercised end to end (it needs a model that isn't cached yet). It builds, and the cached path is unchanged:
modelDirectoryresolves and the new block is skipped.AI usage: written by Claude Code (Claude Opus 5.5) in the M6 benchmarking session, with the repo owner's approval to open this PR. The #71 bugs and the first-run gap were found by the SwiftLM review session.
🤖 Generated with Claude Code