diff --git a/README.md b/README.md index a3c385d..bf31a2c 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README | `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 13.2 tok/s | 40.8K tokens | 5.8 GB | | `Qwen3.8-27B-4bit` (dense) | 11.3 GB | GPU | 9.3 tok/s | 40.8K tokens | 18.4 GB | | `gemma-4-26b-a4b-it-8bit` | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) | — | — | -| `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 8.8 tok/s | 9.5K tokens (32K swapped) | 7.6 GB | +| `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 9.1 tok/s | 9.5K tokens (32K swapped) | 7.3 GB | - **MoE models are the sweet spot at 32 GB.** Only the active experts are read for each token, so they decode 5–6× faster than a dense 27B. A 4-bit MoE with up to about 22 GB of weights runs entirely on the GPU. - **Qwen3.6-35B-A3B on a base M6 reaches 76%** of the M1 Ultra 64 GB decode speed below (47.0 vs 61.7 tok/s). @@ -90,6 +90,8 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- ### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming +![Qwen3.6-35B-A3B with --stream-experts on release b782: 13.6 tok/s decode, 5.1 GB peak, no swap, on a base Mac mini M6 32 GB](docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.gif) + | Prompt tokens | GPU prefill / decode (tok/s) | GPU peak | `--stream-experts` prefill / decode (tok/s) | SSD peak | |---|---|---|---|---| | ~550 | 714 / 47.0 | 19.8 GB | 256 / 13.2 | 5.6 GB | @@ -97,7 +99,7 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- | ~9.8K | 858 / 43.4 | 20.4 GB | 401 / 12.7 | 5.6 GB | | 40.8K | 615 / 36.1 | 21.5 GB | 336 / 12.0 | 5.8 GB | -> ⚠️ **`--stream-experts` crashes on Qwen3.5/3.6 in releases b769 and b773** (`broadcast_shapes … (N,8,8,2048)` on the first request). The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. +> ⚠️ **`--stream-experts` crashes on quantized MoE models in releases b769 and b773** (`broadcast_shapes … (N,8,8,D)` on the first request). Reproduced on M6 with Qwen3.6-35B-A3B and Gemma 4 26B-A4B. The mlx-swift-lm upstream sync in #167 broke the SSD path. Earlier versions of this table were measured before that sync and were never re-checked afterwards. Fixed in SharpAI/mlx-swift-lm#69 and #71; the table above was re-measured with those fixes. ### Qwen3.8-27B-4bit (dense) @@ -121,7 +123,7 @@ Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it- 1. **The MLX buffer cache was unbounded on full-GPU loads.** It could grow to the whole 26.8 GB working set. It is now sized from the RAM left after weights and KV. 2. **The KV-cache estimate counted every layer as full attention.** Gemma 4 (25 of 30 layers use a 1,024-token sliding window) was overestimated 10×, and Qwen3.5/3.8 (48 of 64 layers are linear attention) 4×. On 32 GB that pushed Gemma into CPU/GPU layer partitioning, which crashed with a Metal GPU timeout. 3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. -4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` after that bump; the Qwen3.6 table was re-measured with SharpAI/mlx-swift-lm#69 and #71. +4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` after that bump. The `--stream-experts` rows (Qwen3.6 and Gemma 4 8-bit) were re-measured with SharpAI/mlx-swift-lm#69 and #71, because b769/b773 crash in that mode. > ℹ️ **`--turbo-kv` long-range recall is fixed** ([#175](https://github.com/SharpAI/SwiftLM/issues/175), SharpAI/mlx-swift-lm#65). Before the fix, once a prompt passed the 2,048-token compression threshold, attention only saw the recent hot window and positions restarted, so Qwen3.8-27B-4bit got exact lookups wrong. Attention now covers the compressed history too, which makes `--turbo-kv` slower than before (97 s vs 72 s on an 11.8K-token prompt on the M6). `--turbo-kv` still has no effect when `--ctx-size` is set (the attention layers use `RotatingKVCache`). > diff --git a/Sources/SwiftLM/Server.swift b/Sources/SwiftLM/Server.swift index a117434..2e606f9 100644 --- a/Sources/SwiftLM/Server.swift +++ b/Sources/SwiftLM/Server.swift @@ -281,6 +281,21 @@ final class ProgressTracker { init(modelId: String) { self.modelId = modelId } + + /// True while a `\r` progress bar is on screen without its closing newline. + nonisolated(unsafe) static var barOpen = false + + /// Stops redrawing and ends the bar's line, so later output (and the `exiting` + /// event, which the daemon parses per line) starts on a fresh line. + func finish() { + isDone = true + trackingTask?.cancel() + if Self.barOpen { + print("") + fflush(stdout) + Self.barOpen = false + } + } func getDownloadedBytes() -> Int64 { let home = FileManager.default.homeDirectoryForCurrentUser @@ -358,11 +373,14 @@ final class ProgressTracker { let msg = String(format: "\r[SwiftLM] Download: [%@] %@ %@ (%@ MB / %@ MB) %@", bars, pctStr, spinner, completedMB, totalMB, speedText) + if self.isDone { break } print(msg.padding(toLength: 100, withPad: " ", startingAt: 0), terminator: "") fflush(stdout) - + Self.barOpen = true + if fraction >= 1.0 { print("") + Self.barOpen = false self.isDone = true break } @@ -398,6 +416,10 @@ func emitEvent(_ payload: [String: Any]) { Data("[SwiftLM] failed to encode event for stdout: \(payload)\n".utf8)) return } + if ProgressTracker.barOpen { + print("") + ProgressTracker.barOpen = false + } print(json) fflush(stdout) } @@ -644,6 +666,13 @@ struct MLXServer: AsyncParsableCommand { var modelDirectory = ModelStorage.validatedContentDirectory(for: modelId) ?? resolveModelDirectory(modelId: modelId) + // resolveModelDirectory doesn't check the weights are there; streaming must not be + // activated for an empty or partial snapshot the loader won't read. + if self.streamExperts, let dir = modelDirectory, + !ModelStorage.validateLocalModelDirectory(dir) + { + modelDirectory = nil + } if self.streamExperts, !self.info, modelDirectory == nil, !FileManager.default.fileExists(atPath: modelId) { @@ -654,8 +683,10 @@ struct MLXServer: AsyncParsableCommand { .appendingPathComponent("MLX", isDirectory: true) .appendingPathComponent("HuggingFace", isDirectory: true)) let localRepo = hub.localRepoLocation(Hub.Repo(id: modelId)) - if FileManager.default.fileExists( - atPath: localRepo.appendingPathComponent("config.json").path) + // Every shard must be present: an interrupted download (or the config.json + // the architecture probe fetches) would otherwise plan with a partial size. + if FileManager.default.fileExists(atPath: localRepo.path), + ModelStorage.validateLocalModelDirectory(localRepo) { modelDirectory = localRepo } else { @@ -663,6 +694,7 @@ struct MLXServer: AsyncParsableCommand { phase = .architectureProbe print("[SwiftLM] --stream-experts: downloading \(modelId) before loading...") let prefetchTracker = ProgressTracker(modelId: modelId) + defer { prefetchTracker.finish() } modelDirectory = try await hub.snapshot( from: modelId, matching: ["*.safetensors", "*.json", "*.jinja"] ) { progress in @@ -670,6 +702,11 @@ struct MLXServer: AsyncParsableCommand { } } } + // Streaming is activated for `modelDirectory`, and only a load of that exact + // directory streams. Load from it, or a different lookup could pick another copy. + if self.streamExperts, let dir = modelDirectory { + modelConfig = ModelConfiguration(directory: dir) + } var mainModelProfile: ModelProfile? = nil if self.streamExperts, let dir = modelDirectory { mainModelProfile = ModelProfiler.profile(modelDirectory: dir, modelId: modelId) @@ -937,6 +974,7 @@ struct MLXServer: AsyncParsableCommand { return self.model }() let tracker = ProgressTracker(modelId: resolvedModelId) + defer { tracker.finish() } let isAudio = self.audio phase = .mainModelLoad @@ -1003,6 +1041,7 @@ struct MLXServer: AsyncParsableCommand { print("[SwiftLM] Note: the prompt cache is not used for VLM/Omni loads; each text request re-prefills its full prompt.") } + tracker.finish() print("[SwiftLM] Loaded model configuration. Inferred tool call format: \(String(describing: await container.configuration.toolCallFormat))") // ── Check if target model supports DFlash ── diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl index 13f4c52..08f826e 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl @@ -8,3 +8,10 @@ {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 7.58, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9543, "prefill_tps": 147.6, "ttft_s": 64.86, "decode_tps": 7.08, "gen_tokens": 128, "needle_ok": true, "degenerate": false} {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 7.19, "swap_delta_gb": 0.0, "min_free_pct": 60, "status": "OK", "prompt_tokens": 9518, "prefill_tps": 138.5, "ttft_s": 68.94, "decode_tps": 6.92, "gen_tokens": 128, "needle_ok": true, "degenerate": false} {"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 7.26, "swap_delta_gb": 0.0, "min_free_pct": 53, "status": "OK", "prompt_tokens": 9577, "prefill_tps": 136.9, "ttft_s": 70.22, "decode_tps": 6.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 5.87, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 520, "prefill_tps": 70.6, "ttft_s": 7.39, "decode_tps": 9.19, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a duration long enough for the rhythmic pulse of the Fresnel lens to become his own heartb"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 6.5, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 533, "prefill_tps": 90.6, "ttft_s": 5.93, "decode_tps": 9.13, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is MARLIN-23.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a span of time measured not by calendars, but by the rhythmic pulse of the great Fresnel l"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 6.31, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 542, "prefill_tps": 91.8, "ttft_s": 5.95, "decode_tps": 9.09, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is PELICAN-81.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a span of time that had turned his skin into something resembling cured leather and his e"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 5.6, "swap_delta_gb": 0.0, "min_free_pct": 71, "status": "OK", "prompt_tokens": 527, "prefill_tps": 88.7, "ttft_s": 5.99, "decode_tps": 9.04, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is HERON-20.\n\n***\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a span of time that had turned his skin to leather and his eyes to the color of a storm-tos"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 6.46, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9546, "prefill_tps": 148.4, "ttft_s": 64.52, "decode_tps": 8.22, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is PELICAN-68.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His wor"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 5.92, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9511, "prefill_tps": 145.4, "ttft_s": 65.61, "decode_tps": 8.18, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-85", "answer_head": "The secret code word is HERON-85.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Reach lighthouse for twenty-two years, a duration that felt less like a career and more like a slow merging of man and stone. His world"} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 7.3, "swap_delta_gb": 0.0, "min_free_pct": 59, "status": "OK", "prompt_tokens": 9549, "prefill_tps": 145.7, "ttft_s": 65.73, "decode_tps": 8.16, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-30", "answer_head": "The secret code word is MARLIN-30.\n\nElias Thorne had lived within the salt-crusted walls of the Blackwood Rock Lighthouse for twenty-two years, a duration long enough for the rhythmic pulse of the Fresnel lens to become his own heartbeat. T"} diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.md b/docs/profiling/m6/gemma4_26b_a4b_8bit.md index 74ef970..d997446 100644 --- a/docs/profiling/m6/gemma4_26b_a4b_8bit.md +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.md @@ -12,3 +12,13 @@ Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · | SSD | 2048 (2291) | 193.7 | 11.93 | 7.7 | 7.32 | 0.0 | 69 | ok | | SSD | 8192 (9543) | 138.5 | 68.94 | 6.92 | 7.58 | 0.0 | 53 | ok | | SSD | 32768 | — | — | — | 5.8 | 2.09 | 32 | **MEM_ABORT** swap grew 2.1 GB | + +#### `--stream-experts` re-measured with SharpAI/mlx-swift-lm#69 and #71 (b769/b773 crash in this mode) + +Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Δ GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| SSD | 512 (533) | 90.6 | 5.95 | 9.09 | 6.5 | 0.0 | 71 | ok | +| SSD | 8192 (9546) | 145.7 | 65.61 | 8.18 | 7.3 | 0.0 | 59 | ok | +| SSD | 32768 | — | — | — | 5.79 | 2.04 | 31 | **MEM_ABORT** swap grew 2.0 GB | diff --git a/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.cast b/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.cast new file mode 100644 index 0000000..69ebe1d --- /dev/null +++ b/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.cast @@ -0,0 +1,145 @@ +{"version":3,"term":{"cols":112,"rows":40},"timestamp":1790449820,"idle_time_limit":2.0,"command":"/Users/simba/.cache/swiftlm-bench/demo/record_ssd.sh","title":"SwiftLM · Qwen3.6-35B-A3B --stream-experts · Mac mini M6 32GB","env":{"SHELL":"/bin/zsh"}} +[0.329, "o", "\u001b[36mSwiftLM — github.com/SharpAI/SwiftLM\u001b[39m\r\n\u001b[3244mOpenAI-compatible LLM server for Apple Silicon, in Swift on MLX · release b782\u001b[39m\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m sysctl -n machdep.cpu.brand_string hw.memsize | paste -sd' ' -\r\n"] +[0.641, "o", "Apple M6 · 32 GB · macOS 27.0\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m ./SwiftLM --model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit --stream-experts --port 5431 &\r\n"] +[2.203, "o", "💾 SSD Expert Streaming enabled (lazy load + layer-sync)\r\n✅ Ready. Listening on http://127.0.0.1:5431\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py warmup # first request after load (warm-up)\r\n"] +[1.545, "o", "\r\u001b[K\u001b[32m1"] +[0.070, "o", ","] +[0.069, "o", " "] +[0.073, "o", "2"] +[0.075, "o", ","] +[0.073, "o", " "] +[0.072, "o", "3"] +[0.070, "o", ","] +[0.071, "o", " "] +[0.072, "o", "4"] +[0.071, "o", ","] +[0.072, "o", " "] +[0.072, "o", "5"] +[0.071, "o", ","] +[0.072, "o", " "] +[0.072, "o", "6"] +[0.069, "o", ","] +[0.072, "o", " "] +[0.073, "o", "7"] +[0.075, "o", ","] +[0.078, "o", " "] +[0.071, "o", "8"] +[0.072, "o", "\u001b[0m\r\n\r\n\u001b[36m\u001b[1m 22 tokens · decode 13.3 tok/s · TTFT 0.9s ·\u001b[0m\r\n\u001b[36m\u001b[1m peak SwiftLM 5.1 GB · swap +0.0 GB\u001b[0m\r\n"] +[0.004, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py short # measured request\r\n"] +[1.448, "o", "\r\u001b[K\u001b[32m//"] +[0.068, "o", " Returns"] +[0.070, "o", " the"] +[0.070, "o", " n"] +[0.070, "o", "-th"] +[0.074, "o", " Fibonacci"] +[0.079, "o", " number"] +[0.074, "o", " using"] +[0.068, "o", " an"] +[0.069, "o", " iterative"] +[0.071, "o", " approach"] +[0.072, "o", "."] +[0.068, "o", "\r\n"] +[0.071, "o", "func"] +[0.070, "o", " fib"] +[0.072, "o", "(_"] +[0.070, "o", " n"] +[0.072, "o", ":"] +[0.072, "o", " Int"] +[0.070, "o", ")"] +[0.076, "o", " ->"] +[0.081, "o", " Int"] +[0.073, "o", " {"] +[0.072, "o", "\r\n"] +[0.070, "o", " "] +[0.070, "o", " guard"] +[0.071, "o", " n"] +[0.070, "o", " >"] +[0.071, "o", " "] +[0.071, "o", "0"] +[0.069, "o", " else"] +[0.071, "o", " {"] +[0.072, "o", " return"] +[0.072, "o", " "] +[0.072, "o", "0"] +[0.080, "o", " }"] +[0.083, "o", "\r\n"] +[0.071, "o", " "] +[0.071, "o", " guard"] +[0.071, "o", " n"] +[0.069, "o", " >"] +[0.073, "o", " "] +[0.072, "o", "1"] +[0.071, "o", " else"] +[0.070, "o", " {"] +[0.072, "o", " return"] +[0.072, "o", " "] +[0.073, "o", "1"] +[0.070, "o", " }"] +[0.075, "o", "\r\n"] +[0.089, "o", " \r\n"] +[0.082, "o", " "] +[0.070, "o", " var"] +[0.070, "o", " prev"] +[0.068, "o", "2"] +[0.070, "o", " ="] +[0.070, "o", " "] +[0.071, "o", "0"] +[0.070, "o", "\r\n"] +[0.070, "o", " "] +[0.071, "o", " var"] +[0.070, "o", " prev"] +[0.070, "o", "1"] +[0.073, "o", " ="] +[0.086, "o", " "] +[0.093, "o", "1"] +[0.071, "o", "\r\n \r\n"] +[0.070, "o", " "] +[0.070, "o", " for"] +[0.069, "o", " _"] +[0.069, "o", " in"] +[0.072, "o", " "] +[0.071, "o", "2"] +[0.070, "o", "..."] +[0.070, "o", "n"] +[0.071, "o", " {"] +[0.071, "o", "\r\n"] +[0.069, "o", " "] +[0.074, "o", " let"] +[0.091, "o", " current"] +[0.076, "o", " ="] +[0.076, "o", " prev"] +[0.074, "o", "1"] +[0.072, "o", " +"] +[0.072, "o", " prev"] +[0.072, "o", "2"] +[0.070, "o", "\r\n"] +[0.072, "o", " "] +[0.069, "o", " prev"] +[0.070, "o", "2"] +[0.071, "o", " ="] +[0.072, "o", " prev"] +[0.074, "o", "1"] +[0.087, "o", "\r\n"] +[0.096, "o", " "] +[0.072, "o", " prev"] +[0.071, "o", "1"] +[0.071, "o", " ="] +[0.072, "o", " current"] +[0.071, "o", "\r\n"] +[0.073, "o", " "] +[0.071, "o", " }"] +[0.072, "o", "\r\n"] +[0.072, "o", " \r\n"] +[0.070, "o", " "] +[0.071, "o", " return"] +[0.072, "o", " prev"] +[0.087, "o", "1"] +[0.098, "o", "\r\n"] +[0.070, "o", "}"] +[0.072, "o", "\u001b[0m\r\n\r\n\u001b[36m\u001b[1m 110 tokens · decode 13.6 tok/s · TTFT 0.8s ·\u001b[0m\r\n\u001b[36m\u001b[1m peak SwiftLM 5.1 GB · swap +0.0 GB\u001b[0m\r\n"] +[0.004, "o", "\r\n"] +[0.032, "x", "0"] diff --git a/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.gif b/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.gif new file mode 100644 index 0000000..5282b33 Binary files /dev/null and b/docs/profiling/m6/media/m6_qwen36_35b_a3b_ssd_stream.gif differ