diff --git a/README.md b/README.md index 2338352..cfd3a67 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,90 @@ Then start the server (models download automatically if not cached): *(Add `--stream-experts` when running oversized MoE models to bypass macOS virtual memory swapping and stream expert layers directly from NVMe SSD.)* +## πŸ“Š Performance: Mac mini M6 (base, 32 GB) + +The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README comes from 64 GB hardware. Running the same context lengths on this M6 exposed memory bugs that 64 GB machines had been hiding, and they are fixed in this release (see [What 32 GB exposed](#what-32-gb-exposed)). + +![Gemma-4-26B-A4B streaming at 55 tok/s on a base Mac mini M6 32 GB](docs/profiling/m6/media/m6_gemma4_26b_a4b_stream.gif) + +> *Hardware:* Mac mini (Mac18,5), Apple M6, 12-core GPU, 32 GB unified memory (170 GB/s), macOS 27.0. Metal working set 26.8 GB. +> *Method:* [`scripts/profiling/m6_bench.py`](scripts/profiling/m6_bench.py). One warm-up, then the median of 3 runs (1 run at 32K and above), temperature 0. Every prompt starts with a unique nonce, so the prompt cache can't hit, and hides a code word that the answer must return. A memory guard aborts any case whose swap grows by more than 2 GB. Raw results: [`docs/profiling/m6/`](docs/profiling/m6/). + +### What runs well on a 32 GB M6 + +| Model (4-bit unless noted) | Weights | Mode | Decode, short prompt | Longest prompt that passed | Peak GPU | +|---|---|---|---|---|---| +| **`gemma-4-26b-a4b-it-4bit`** (MoE, ~4B active) | 15.3 GB | GPU | **52.2 tok/s** | 80.7K tokens | 19.5 GB | +| **`Qwen3.6-35B-A3B-UD-MLX-4bit`** (MoE, ~3B active) | 21.6 GB | GPU | **46.7 tok/s** | 40.8K tokens | 22.4 GB | +| `Qwen3.6-35B-A3B-UD-MLX-4bit` | 21.6 GB | `--stream-experts` | 13.2 tok/s | 40.8K tokens | 7.7 GB | +| `Qwen3.8-27B-4bit` (dense) | 11.3 GB | GPU | 9.3 tok/s | 40.8K tokens | 18.4 GB | +| `gemma-4-26b-a4b-it-8bit` | ~26 GB | GPU | swaps (+3.1 GB on the first prompt) | β€” | β€” | +| `gemma-4-26b-a4b-it-8bit` | ~26 GB | `--stream-experts` | 8.8 tok/s | 9.5K tokens (32K swapped) | 7.6 GB | + +- **MoE models are the sweet spot at 32 GB.** Only the active experts are read for each token, so they decode 5–6Γ— faster than a dense 27B. A 4-bit MoE with up to about 22 GB of weights runs entirely on the GPU. +- **Qwen3.6-35B-A3B on a base M6 reaches 76%** of the M1 Ultra 64 GB decode speed below (46.7 vs 61.7 tok/s). +- **Dense 27B decode is bandwidth-bound.** 9.3 tok/s Γ— 11.3 GB is about 105 GB/s, roughly 60% of the M6's rated 170 GB/s. +- **An 8-bit 26 GB model needs SSD streaming** and tops out at about 10K tokens of context. + +### Gemma-4-26B-A4B 4-bit β€” by prompt length + +| Prompt tokens | Vanilla prefill / decode (tok/s) | TTFT | `--turbo-kv` prefill / decode (tok/s) | Peak GPU Β· swap growth | +|---|---|---|---|---| +| ~530 | 733 / **52.2** | 0.8 s | 785 / 53.2 | 14.5 GB Β· 0 | +| ~2.3K | **963** / 50.2 | 2.5 s | 969 / 50.5 | 15.0 GB Β· 0 | +| ~9.5K | 959 / 45.1 | 10.1 s | 971 / 45.5 | 15.8 GB Β· 0 | +| ~39.7K | 757 / 31.0 | 53.3 s | 787 / 31.5 | 17.9 GB Β· 0 | +| ~80.7K | 622 / 24.3 | 131.3 s | 630 / 24.5 | 19.5 GB Β· 0 | + +Every needle check passed in both modes. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). + +### Qwen3.6-35B-A3B 4-bit β€” GPU vs SSD streaming + +| Prompt tokens | GPU prefill / decode (tok/s) | GPU peak | `--stream-experts` prefill / decode (tok/s) | SSD peak | +|---|---|---|---|---| +| ~550 | 808 / 46.7 | 20.4 GB | 321 / 13.2 | 6.0 GB | +| ~2.3K | 969 / 45.5 | 21.0 GB | 402 / 13.1 | 6.2 GB | +| ~9.8K | 849 / 43.4 | 21.1 GB | 403 / 12.9 | 6.6 GB | +| 40.8K | 635 / 35.1 | 22.4 GB | 340 / 11.9 | 7.7 GB | + +### Qwen3.8-27B-4bit (dense) β€” Vanilla vs TurboKV + +| Prompt tokens | Vanilla prefill / decode (tok/s) | TurboKV prefill / decode (tok/s) | Peak GPU Β· swap growth | +|---|---|---|---| +| ~550 | 233 / 9.3 | 231 / 9.3 | 15.2 GB Β· 0 | +| ~2.3K | 242 / 9.1 | 261 / 9.2 | 16.1 GB Β· 0 | +| ~9.8K | 238 / 8.9 | 250 / 8.9 | 16.9 GB Β· 0 | +| ~40.8K | 200 / 7.9 | 202 / 7.9 | 18.4 GB Β· 0 (TurboKV 16.9 GB) | + +TurboKV barely changes speed on this model. Only 16 of its 64 layers use full attention (the other 48 are GatedDeltaNet), so the KV cache is already small. It saves about 1.5 GB at 40K tokens. + +### What 32 GB exposed + +| 8.5K-token prompt, Qwen3.8-27B-4bit | Before (old pin) | After (this release) | +|---|---|---| +| Prefill | 33.4 tok/s | **~240 tok/s** (β‰ˆ7Γ—; 238 tok/s measured at 9.8K) | +| Peak memory | 38 GB process footprint | **≀19 GB** process (16.9 GB GPU peak at 9.8K) | +| Swap growth | +15 GB | **0** | + +1. **The MLX buffer cache was unbounded on full-GPU loads.** It could grow to the whole 26.8 GB working set. It is now sized from the RAM left after weights and KV. +2. **The KV-cache estimate counted every layer as full attention.** Gemma 4 (25 of 30 layers use a 1,024-token sliding window) was overestimated 10Γ—, and Qwen3.5/3.8 (48 of 64 layers are linear attention) 4Γ—. On 32 GB that pushed Gemma into CPU/GPU layer partitioning, which crashed with a Metal GPU timeout. +3. **An auto-detected VLM that failed to load exited the server.** `Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`. +4. **Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump. + +> ⚠️ **Known issues:** `--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`. + +Reproduce: + +```bash +./build.sh +.build/release/SwiftLM --model mlx-community/gemma-4-26b-a4b-it-4bit --port 5431 --ctx-size 48000 & +python3 scripts/demo/stream_client.py short +python3 scripts/profiling/m6_bench.py --model mlx-community/gemma-4-26b-a4b-it-4bit \ + --config "Vanilla=" --contexts 512,2048,8192,32768,65536 --out docs/profiling/m6/gemma4_26b_a4b_4bit +``` + +More recordings: [Gemma, 41K-token prompt (4Γ—)](docs/profiling/m6/media/m6_gemma4_26b_a4b_41k_prompt_4x.gif) Β· [Qwen3.8-27B streaming](docs/profiling/m6/media/m6_qwen38_27b_stream.gif) Β· [Qwen3.8-27B, 8.6K-token prompt (4Γ—)](docs/profiling/m6/media/m6_qwen38_27b_8k_prompt_4x.gif) + ## πŸ“Š Performance: MTP Speculative Decoding β€” Gemma 4-26B (MacBook Pro M5 Pro 64 GB) Benchmarked with `gemma-4-26b-a4b-it-4bit` running three configurations across 512 / 40K / 100K token contexts. diff --git a/Sources/SwiftLM/ModelProfiler.swift b/Sources/SwiftLM/ModelProfiler.swift index d7e8b4d..8b96858 100644 --- a/Sources/SwiftLM/ModelProfiler.swift +++ b/Sources/SwiftLM/ModelProfiler.swift @@ -31,6 +31,14 @@ struct ModelProfile: Sendable { let numActiveExperts: Int? let weightFileSizeBytes: Int let modelId: String + /// Attention layout from `layer_types`. When a config has no `layer_types`, + /// every layer counts as full attention, which is the old estimate. + var fullAttentionLayers: Int? = nil + var slidingAttentionLayers: Int = 0 + var slidingWindow: Int? = nil + /// Gemma 4 gives its full-attention layers their own KV head count and head dim. + var globalKVHeads: Int? = nil + var globalHeadDim: Int? = nil /// Estimated total parameters in billions (rough) var estimatedParamsB: Double { @@ -55,10 +63,19 @@ struct ModelProfile: Sendable { /// KV cache memory in GB for a given context length func kvCacheMemoryGB(contextLength: Int) -> Double { - // KV cache = 2 (K + V) Γ— layers Γ— kv_heads Γ— head_dim Γ— context Γ— 2 bytes (FP16) + // KV cache = 2 (K + V) Γ— layers Γ— kv_heads Γ— head_dim Γ— tokens Γ— 2 bytes (FP16). + // Only attention layers hold a KV cache. Linear-attention layers (GatedDeltaNet) + // keep a fixed-size state instead, and sliding-window layers stop growing at the + // window size. Counting every layer as full attention overestimated Gemma 4 by + // ~10Γ— and Qwen3.5/3.8 by ~4Γ—, which pushed 32 GB machines into CPU + // partitioning they didn't need. let bytesPerElement = 2 // FP16 - let kvBytes = 2 * numLayers * numKVHeads * headDim * contextLength * bytesPerElement - return Double(kvBytes) / 1e9 + let fullLayers = fullAttentionLayers ?? numLayers + let fullBytes = 2 * fullLayers * (globalKVHeads ?? numKVHeads) * (globalHeadDim ?? headDim) + * contextLength * bytesPerElement + let slidingTokens = min(contextLength, slidingWindow ?? contextLength) + let slidingBytes = 2 * slidingAttentionLayers * numKVHeads * headDim * slidingTokens * bytesPerElement + return Double(fullBytes + slidingBytes) / 1e9 } /// Total memory required in GB (weights + KV cache + overhead) @@ -177,6 +194,10 @@ enum ModelProfiler { let vocabSize: Int? let quantizationConfig: QuantConfig? let textConfig: TextConfig? + let layerTypes: [String]? + let slidingWindow: Int? + let numGlobalKeyValueHeads: Int? + let globalHeadDim: Int? enum CodingKeys: String, CodingKey { case modelType = "model_type" @@ -189,6 +210,10 @@ enum ModelProfiler { case vocabSize = "vocab_size" case quantizationConfig = "quantization_config" case textConfig = "text_config" + case layerTypes = "layer_types" + case slidingWindow = "sliding_window" + case numGlobalKeyValueHeads = "num_global_key_value_heads" + case globalHeadDim = "global_head_dim" } } @@ -200,6 +225,10 @@ enum ModelProfiler { let headDim: Int? let intermediateSize: Int? let vocabSize: Int? + let layerTypes: [String]? + let slidingWindow: Int? + let numGlobalKeyValueHeads: Int? + let globalHeadDim: Int? enum CodingKeys: String, CodingKey { case numHiddenLayers = "num_hidden_layers" @@ -209,6 +238,10 @@ enum ModelProfiler { case headDim = "head_dim" case intermediateSize = "intermediate_size" case vocabSize = "vocab_size" + case layerTypes = "layer_types" + case slidingWindow = "sliding_window" + case numGlobalKeyValueHeads = "num_global_key_value_heads" + case globalHeadDim = "global_head_dim" } } @@ -275,7 +308,8 @@ enum ModelProfiler { // Measure weight file sizes on disk (only for MoE to avoid slow walks on dense models) let weightSize = isMoE ? measureWeightFiles(directory: modelDirectory) : 0 - return ModelProfile( + let layerTypes = config.layerTypes ?? config.textConfig?.layerTypes + var profile = ModelProfile( modelType: modelType, numLayers: numLayers, hiddenSize: hiddenSize, @@ -291,6 +325,22 @@ enum ModelProfiler { weightFileSizeBytes: weightSize, modelId: modelId ) + if let layerTypes, layerTypes.count == numLayers { + // Anything that isn't sliding or linear attention is treated as full attention. + let sliding = layerTypes.filter { $0 == "sliding_attention" }.count + let linear = layerTypes.filter { $0 == "linear_attention" }.count + profile.slidingAttentionLayers = sliding + profile.fullAttentionLayers = layerTypes.count - sliding - linear + profile.slidingWindow = config.slidingWindow ?? config.textConfig?.slidingWindow + if sliding > 0 && profile.slidingWindow == nil { + // Can't cap a window we don't know, so count those layers as full. + profile.fullAttentionLayers = layerTypes.count - linear + profile.slidingAttentionLayers = 0 + } + } + profile.globalKVHeads = config.numGlobalKeyValueHeads ?? config.textConfig?.numGlobalKeyValueHeads + profile.globalHeadDim = config.globalHeadDim ?? config.textConfig?.globalHeadDim + return profile } /// Routed-expert count keys, in precedence order within a container. @@ -505,7 +555,15 @@ enum ModelProfiler { switch strategy { case .fullGPU: memoryLimit = Int(Double(system.recommendedWorkingSetBytes) * 1.5) - cacheLimit = system.recommendedWorkingSetBytes // default + // MLX's default cache limit is the whole working set. Prefill chunk + // buffers change shape as the KV offset grows, so they are rarely + // reused and the cache just keeps growing. On a 32 GB machine that + // pushed macOS into swap by ~8K tokens. Give the cache half of what + // is left after weights, KV and an 8 GB OS reserve, and never more + // than the old default. + let headroomGB = system.totalRAMGB - (weightGB + draftGB) - kvGB - 8.0 + let budget = Int(max(1.0, headroomGB / 2) * 1e9) + cacheLimit = min(system.recommendedWorkingSetBytes, budget) case .swapAssisted: memoryLimit = 200 * 1024 * 1024 * 1024 // 200 GB sentinel to bypass MLX eval_impl spin loop (let macOS swap handle it) cacheLimit = 2 * 1024 * 1024 // 2MB β€” let OS manage caching diff --git a/Sources/SwiftLM/Server.swift b/Sources/SwiftLM/Server.swift index da3e190..4e0e185 100644 --- a/Sources/SwiftLM/Server.swift +++ b/Sources/SwiftLM/Server.swift @@ -205,6 +205,28 @@ private struct TransformersTokenizerBridge: MLXLMCommon.Tokenizer, Sendable { /// Returns `nil` when the value must be dropped (JSON `null` / NSNull), otherwise a /// structure with every nested null removed. See `TransformersTokenizerBridge.applyChatTemplate`. +/// True when a VLM load failed because the checkpoint doesn't match the VLM code: +/// its config doesn't decode, its weights don't line up with the module tree, or the +/// factory doesn't know the model/processor type. Only these justify retrying an +/// auto-detected VLM as a text-only LLM. Anything else (cancellation, download, I/O) +/// would fail the same way again and just hide the real error. +func isVLMCheckpointMismatch(_ error: any Error) -> Bool { + switch error { + case is DecodingError, is UpdateError: + return true + case let factoryError as ModelFactoryError: + switch factoryError { + case .unsupportedModelType, .unsupportedProcessorType, .configurationDecodingError, + .invalidConfiguration: + return true + default: + return false + } + default: + return false + } +} + func sanitizeForJinja(_ value: any Sendable) -> (any Sendable)? { if value is NSNull { return nil } let mirror = Mirror(reflecting: value) @@ -771,7 +793,8 @@ struct MLXServer: AsyncParsableCommand { // Apply memory strategy switch plan.strategy { case .fullGPU: - print("[SwiftLM] \(plan.strategy.emoji) Memory strategy: FULL GPU (\(String(format: "%.1f", plan.weightMemoryGB))GB model, \(String(format: "%.1f", system.availableRAMGB))GB available)") + Memory.cacheLimit = plan.recommendedCacheLimit + print("[SwiftLM] \(plan.strategy.emoji) Memory strategy: FULL GPU (\(String(format: "%.1f", plan.weightMemoryGB))GB model, \(String(format: "%.1f", system.availableRAMGB))GB available, cache limited to \(plan.recommendedCacheLimit / (1024*1024))MB)") case .swapAssisted: if self.streamExperts { // SSD Streaming: expert weights are mmap'd from SSD via the OS page cache. @@ -863,7 +886,7 @@ struct MLXServer: AsyncParsableCommand { let speculativeDecodingRequested = self.draftModel != nil || self.dflash || self.mtp let autoDetectedVision = !self.audio && architecture.supportsVision && !speculativeDecodingRequested - let isVision = self.vision || autoDetectedVision + var isVision = self.vision || autoDetectedVision if architecture.supportsVision, !self.vision, !self.audio, speculativeDecodingRequested { print( "[SwiftLM] Note: \(architecture.modelType ?? "unknown") reports vision support, but speculative/MTP decoding was requested; loading as a text-only LLM." @@ -895,12 +918,29 @@ struct MLXServer: AsyncParsableCommand { } } else if isVision { print("[SwiftLM] Loading VLM (vision-language model)...") - container = try await VLMModelFactory.shared.loadContainer( - from: downloader, - using: TransformersTokenizerLoader(modelId: resolvedModelId), - configuration: modelConfig - ) { progress in - tracker.printProgress(progress) + do { + container = try await VLMModelFactory.shared.loadContainer( + from: downloader, + using: TransformersTokenizerLoader(modelId: resolvedModelId), + configuration: modelConfig + ) { progress in + tracker.printProgress(progress) + } + } catch where !self.vision && isVLMCheckpointMismatch(error) { + // Vision was only auto-detected, and the vision side of the checkpoint + // doesn't match the VLM code (e.g. a preprocessor_config.json without + // image_mean). The text model can still serve, so fall back rather than + // exit. Cancellation, network and I/O errors still propagate, as does + // any error under an explicit --vision. + print("[SwiftLM] ⚠️ Auto-detected VLM failed to load (\(error)); loading as a text-only LLM. Pass --vision to make this fatal.") + isVision = false + container = try await LLMModelFactory.shared.loadContainer( + from: downloader, + using: TransformersTokenizerLoader(modelId: resolvedModelId), + configuration: modelConfig + ) { progress in + tracker.printProgress(progress) + } } } else if isAudio { print("[SwiftLM] Loading ALM (audio-language model)...") diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl b/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl new file mode 100644 index 0000000..c6c2b24 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit.jsonl @@ -0,0 +1,24 @@ +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 14.0, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 520, "prefill_tps": 439.0, "ttft_s": 1.21, "decode_tps": 52.63, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne lived in the rhythm of the tides and the mechanical pulse of the Great North Light. For thirty years, the lighthouse had been his only constant companion, a towering sentinel of granite and g"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 14.32, "swap_delta_gb": 0.0, "min_free_pct": 44, "status": "OK", "prompt_tokens": 533, "prefill_tps": 733.3, "ttft_s": 0.77, "decode_tps": 52.48, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is: **MARLIN-23**.\n\n***\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that the sea was not a thing of water, but a th"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 14.48, "swap_delta_gb": 0.0, "min_free_pct": 44, "status": "OK", "prompt_tokens": 542, "prefill_tps": 738.4, "ttft_s": 0.77, "decode_tps": 52.24, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is **PELICAN-81**.\n\n***\n\nElias was a man of rhythm, a man of stone, and a man of salt. For thirty years, he had lived within the circular confines of the Blackwood Rock Lighthouse, a towering sentinel that stood defiant"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 14.08, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 527, "prefill_tps": 728.6, "ttft_s": 0.76, "decode_tps": 51.88, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is: **HERON-20**\n\n***\n\nThe salt spray was a constant companion to Elias, a fine mist that clung to his beard and turned the heavy wool of his coat into a weighted-down second skin. He sat in the lantern room of the Blac"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 14.86, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2273, "prefill_tps": 950.7, "ttft_s": 2.46, "decode_tps": 50.07, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is: **PELICAN-68**.\n\n***\n\nThe salt spray was a constant companion to Elias, a fine mist that clung to his beard and turned the heavy wool of his coat into a sodden weight. He sat in the lantern room of the Blackwood Roc"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 14.89, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2288, "prefill_tps": 963.0, "ttft_s": 2.45, "decode_tps": 50.24, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-77", "answer_head": "The secret code word is MARLIN-77.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 14.95, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2284, "prefill_tps": 962.9, "ttft_s": 2.45, "decode_tps": 50.31, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-75", "answer_head": "The secret code word is MARLIN-75.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 15.78, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9528, "prefill_tps": 958.8, "ttft_s": 10.14, "decode_tps": 45.0, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-70", "answer_head": "The secret code word is PELICAN-70.\n\nElias Thorne was a man of salt, iron, and silence. For thirty-two years, he had inhabited the Blackwood Rock Lighthouse, a tapering finger of granite that rose defiantly from the churning Atlantic. His w"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 15.73, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 9543, "prefill_tps": 961.0, "ttft_s": 10.14, "decode_tps": 45.26, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-30", "answer_head": "The secret code word is OSPREY-30.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow, rhythmic merging with the se"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.5, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 9557, "prefill_tps": 947.5, "ttft_s": 10.3, "decode_tps": 45.09, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-57", "answer_head": "The secret code word is PELICAN-57.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 17.91, "swap_delta_gb": 0.0, "min_free_pct": 32, "status": "OK", "prompt_tokens": 39772, "prefill_tps": 756.8, "ttft_s": 53.33, "decode_tps": 31.03, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-28", "answer_head": "The secret code word is: OSPREY-28.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for twenty-two years, a duration that felt less like a measurement of time and more like a gradual merging of man an"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 65536, "run": 0, "warmup": false, "peak_gpu_gb": 19.48, "swap_delta_gb": 0.0, "min_free_pct": 25, "status": "OK", "prompt_tokens": 80710, "prefill_tps": 621.7, "ttft_s": 131.31, "decode_tps": 24.34, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-54", "answer_head": "The secret code word is **OSPREY-54**.\n\n***\n\nThe brass lantern of the Great North Beacon was a temperamental beast, much like the sea it sought to tame. Elias, a man whose face was etched with as many lines as a nautical chart, polished the"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 13.97, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 526, "prefill_tps": 582.5, "ttft_s": 0.93, "decode_tps": 52.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-41", "answer_head": "The secret code word is MARLIN-41.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a span of time and more like a rhythmic breathing of the ti"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 14.32, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 535, "prefill_tps": 742.1, "ttft_s": 0.76, "decode_tps": 53.17, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-63", "answer_head": "The secret code word is: **HERON-63**.\n\n***\n\nElias was a man of rhythm, a man who found comfort in the predictable pulse of the world. As the keeper of the Blackwood Reach Lighthouse, his life was measured not by hours, but by the steady ro"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 14.36, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 531, "prefill_tps": 784.5, "ttft_s": 0.72, "decode_tps": 53.28, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-89", "answer_head": "The secret code word is: **HERON-89**.\n\n***\n\nElias Thorne had lived in the Blackwood Lighthouse for forty years, and in that time, he had learned that the sea was not a living thing, but a vast, breathing clock. It did not care for the ambi"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 14.37, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 531, "prefill_tps": 787.0, "ttft_s": 0.71, "decode_tps": 52.88, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-98", "answer_head": "The secret code word is: **OSPREY-98**.\n\n***\n\nThe salt spray was a constant companion to Elias, a fine, abrasive mist that coated the glass of the lantern room and the weathered creases of his face. For thirty years, the Great North Point L"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 14.75, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2279, "prefill_tps": 971.9, "ttft_s": 2.41, "decode_tps": 50.59, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-86", "answer_head": "The secret code word is: OSPREY-86.\n\n***\n\nElias lived in the rhythm of the tides and the mechanical pulse of the Great North Beacon. For forty years, the lighthouse had been his only constant, a towering sentinel of granite and glass perche"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 14.88, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2310, "prefill_tps": 965.8, "ttft_s": 2.46, "decode_tps": 50.46, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-51", "answer_head": "The secret code word is PELICAN-51.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that the sea did not speak in words, but in rhythms. The"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 14.93, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 2298, "prefill_tps": 968.5, "ttft_s": 2.44, "decode_tps": 49.02, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-43", "answer_head": "The secret code word is HERON-43.\n\nElias Thorne lived in the rhythm of the tides and the mechanical pulse of the Great North Beacon. For thirty years, the lighthouse had been his only constant, a towering sentinel of granite and glass perch"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 15.58, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9571, "prefill_tps": 970.6, "ttft_s": 10.05, "decode_tps": 45.52, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-34", "answer_head": "The secret code word is OSPREY-34.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow, rhythmic merging with the se"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 15.88, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9565, "prefill_tps": 973.2, "ttft_s": 10.03, "decode_tps": 42.01, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-43", "answer_head": "The secret code word is OSPREY-43.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a career and more like a slow, rhythmic merging with the se"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.74, "swap_delta_gb": 0.0, "min_free_pct": 39, "status": "OK", "prompt_tokens": 9602, "prefill_tps": 970.0, "ttft_s": 10.1, "decode_tps": 45.62, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-76", "answer_head": "The secret code word is HERON-76.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every m"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 32768, "run": 0, "warmup": false, "peak_gpu_gb": 17.55, "swap_delta_gb": 0.0, "min_free_pct": 32, "status": "OK", "prompt_tokens": 39672, "prefill_tps": 786.6, "ttft_s": 51.13, "decode_tps": 31.47, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-59", "answer_head": "The secret code word is MARLIN-59.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a measurement of time and more like a slow, rhythmic breath"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "TurboKV", "context": 65536, "run": 0, "warmup": false, "peak_gpu_gb": 19.28, "swap_delta_gb": 0.0, "min_free_pct": 26, "status": "OK", "prompt_tokens": 80922, "prefill_tps": 630.3, "ttft_s": 129.75, "decode_tps": 24.51, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-75", "answer_head": "The secret code word is **HERON-75**.\n\n***\n\nThe brass lantern of the Great North Beacon did not merely shine; it breathed. For Elias, a man whose skin had become as weathered and salt-crusted as the driftwood on the shores of Oakhaven, the "} diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit.md b/docs/profiling/m6/gemma4_26b_a4b_4bit.md new file mode 100644 index 0000000..bdfd4e2 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit.md @@ -0,0 +1,17 @@ +### `mlx-community/gemma-4-26b-a4b-it-4bit` + +Apple M6 Β· 32 GB Β· runs=3 (long=1) Β· warmup=1 Β· gen=128 Β· temperature 0 Β· medians + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Ξ” GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 512 (533) | 733.3 | 0.77 | 52.24 | 14.48 | 0.0 | 43 | ok | +| Vanilla | 2048 (2284) | 962.9 | 2.45 | 50.24 | 14.95 | 0.0 | 40 | ok | +| Vanilla | 8192 (9543) | 958.8 | 10.14 | 45.09 | 15.78 | 0.0 | 39 | ok | +| Vanilla | 32768 (39772) | 756.8 | 53.33 | 31.03 | 17.91 | 0.0 | 32 | ok | +| Vanilla | 65536 (80710) | 621.7 | 131.31 | 24.34 | 19.48 | 0.0 | 25 | ok | +| TurboKV | 512 (531) | 784.5 | 0.72 | 53.17 | 14.37 | 0.0 | 43 | ok | +| TurboKV | 2048 (2298) | 968.5 | 2.44 | 50.46 | 14.93 | 0.0 | 40 | ok | +| TurboKV | 8192 (9571) | 970.6 | 10.05 | 45.52 | 15.88 | 0.0 | 39 | ok | +| TurboKV | 32768 (39672) | 786.6 | 51.13 | 31.47 | 17.55 | 0.0 | 32 | ok | +| TurboKV | 65536 (80922) | 630.3 | 129.75 | 24.51 | 19.28 | 0.0 | 26 | ok | +| MTP | None | β€” | β€” | β€” | β€” | β€” | β€” | **START_FAIL** | diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.jsonl b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.jsonl new file mode 100644 index 0000000..696be97 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.jsonl @@ -0,0 +1,20 @@ +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 13.96, "swap_delta_gb": 0.0, "min_free_pct": 45, "status": "OK", "prompt_tokens": 520, "prefill_tps": 636.3, "ttft_s": 0.85, "decode_tps": 53.2, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-60", "answer_head": "The secret code word is MARLIN-60.\n\nElias Thorne lived in the rhythm of the tides and the mechanical pulse of the Great North Light. For thirty years, the lighthouse had been his only constant companion, a towering sentinel of granite and g"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 14.27, "swap_delta_gb": 0.0, "min_free_pct": 45, "status": "OK", "prompt_tokens": 533, "prefill_tps": 789.3, "ttft_s": 0.72, "decode_tps": 52.95, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-23", "answer_head": "The secret code word is: **MARLIN-23**.\n\n***\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that the sea did not speak in words, but in rhy"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 14.5, "swap_delta_gb": 0.0, "min_free_pct": 45, "status": "OK", "prompt_tokens": 542, "prefill_tps": 788.5, "ttft_s": 0.73, "decode_tps": 53.09, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-81", "answer_head": "The secret code word is **PELICAN-81**.\n\n***\n\nThe salt spray was a constant companion to Elias, a fine mist that coated the brass fittings of the lantern room and turned his beard into a coarse, white thicket. He had lived in the Blackwood "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 14.1, "swap_delta_gb": 0.0, "min_free_pct": 45, "status": "OK", "prompt_tokens": 527, "prefill_tps": 774.6, "ttft_s": 0.72, "decode_tps": 52.53, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-20", "answer_head": "The secret code word is **HERON-20**.\n\n***\n\nThe salt spray was a constant companion to Elias, a fine, microscopic mist that coated the brass fittings of the lantern room and settled into the deep creases of his weathered face. He had lived "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 15.14, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 2273, "prefill_tps": 971.3, "ttft_s": 2.41, "decode_tps": 50.72, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-68", "answer_head": "The secret code word is: **PELICAN-68**.\n\n***\n\nThe salt spray was a constant companion to Elias, a man whose skin had become as weathered and creased as the driftwood that washed up on the jagged shores of Blackwood Cove. For forty years, h"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 14.89, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 2288, "prefill_tps": 974.8, "ttft_s": 2.42, "decode_tps": 50.78, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-77", "answer_head": "The secret code word is MARLIN-77.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 14.91, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 2284, "prefill_tps": 975.0, "ttft_s": 2.41, "decode_tps": 50.76, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-75", "answer_head": "The secret code word is MARLIN-75.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 16.6, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9528, "prefill_tps": 972.9, "ttft_s": 9.99, "decode_tps": 45.96, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-70", "answer_head": "The secret code word is PELICAN-70.\n\n***\n\nElias Thorne lived in the rhythm of the tides and the heavy, rhythmic pulse of the Great North Beacon. For forty years, the lighthouse had been his only constant, a towering sentinel of granite and "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 16.71, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9543, "prefill_tps": 972.3, "ttft_s": 10.01, "decode_tps": 45.93, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-30", "answer_head": "The secret code word is OSPREY-30.\n\nElias Thorne had lived in the Blackwood Rock Lighthouse for twenty-two years, a duration that felt less like a span of time and more like a slow, rhythmic breathing of the sea. The tower was a salt-cruste"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "Vanilla", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 15.43, "swap_delta_gb": 0.0, "min_free_pct": 41, "status": "OK", "prompt_tokens": 9557, "prefill_tps": 966.6, "ttft_s": 10.09, "decode_tps": 46.06, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-57", "answer_head": "The secret code word is PELICAN-57.\n\nElias Thorne had lived within the salt-sprayed stone walls of the Blackwood Lighthouse for thirty-two years, a duration that felt less like a career and more like a slow merging of man and masonry. Every"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 16.26, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 542, "prefill_tps": 608.5, "ttft_s": 0.92, "decode_tps": 44.83, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-28", "answer_head": "The secret code word is OSPREY-28.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that silence was not the absence of sound, but the presen"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 16.28, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 536, "prefill_tps": 791.9, "ttft_s": 0.71, "decode_tps": 46.67, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-93", "answer_head": "The secret code word is HERON-93.\n\nElias Thorne lived in the rhythm of the tides and the mechanical pulse of the Great North Light. For thirty years, the lighthouse had been his only constant, a towering sentinel of white stone perched prec"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 16.33, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 519, "prefill_tps": 892.2, "ttft_s": 0.63, "decode_tps": 43.59, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-98", "answer_head": "The secret code word is PELICAN-98.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that silence was not the absence of sound, but the prese"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 16.26, "swap_delta_gb": 0.0, "min_free_pct": 40, "status": "OK", "prompt_tokens": 522, "prefill_tps": 750.0, "ttft_s": 0.73, "decode_tps": 45.18, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-97", "answer_head": "The secret code word is PELICAN-97.\n\nElias Thorne had lived within the salt-crusted stone walls of the Blackwood Rock Lighthouse for twenty-two years, and in that time, he had learned that silence was not the absence of sound, but the prese"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 15.74, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 2286, "prefill_tps": 983.6, "ttft_s": 2.4, "decode_tps": 35.63, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-48", "answer_head": "The secret code word is HERON-48.\n\nElias Thorne lived in the rhythm of the tides and the Great Crag Lighthouse were separated entities, linked to the other only by the salt spray and the heavy, rotating beam that cut through the Atlantic wi"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 16.3, "swap_delta_gb": 0.0, "min_free_pct": 42, "status": "OK", "prompt_tokens": 2293, "prefill_tps": 972.0, "ttft_s": 2.43, "decode_tps": 35.34, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "MARLIN-82", "answer_head": "The secret code word is MARLIN-82.\n\nElias Thorne had lived within the salt spray and the rhythmic pulse of the Great Lens for forty-year stretch. To were calloused, mapped of deep in his once, but now they were etched history he sea. Every "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 16.06, "swap_delta_gb": 0.0, "min_free_pct": 43, "status": "OK", "prompt_tokens": 2263, "prefill_tps": 973.5, "ttft_s": 2.39, "decode_tps": 36.23, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-42", "answer_head": "OSPREY-42\n\nElias Thorne\u2019s lighthouse for forty years, and in many ways, the stone and salt was the only heartbeat he truly understand. Every morning, before the sun had even begun to crest of the Atlantic, Elias the spiral staircase, his bo"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 16.66, "swap_delta_gb": 0.0, "min_free_pct": 38, "status": "OK", "prompt_tokens": 9582, "prefill_tps": 966.5, "ttft_s": 10.11, "decode_tps": 29.9, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "OSPREY-72", "answer_head": "The secret code word is OSPREY-72.\n\nElias had been his entire life, a weathered by salt and the relentless, rhythmic pulse of the Atlantic. Forged in the spray, and his only constant, a low growl that spoke of the deep. Every evening, and n"} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 16.65, "swap_delta_gb": 0.0, "min_free_pct": 38, "status": "OK", "prompt_tokens": 9552, "prefill_tps": 955.0, "ttft_s": 10.2, "decode_tps": 30.35, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "PELICAN-50", "answer_head": "The secret code word is PELICAN-50.\n\nElias was a man of salt, stone, and silence that follows of a great distance. For thirty, he had been the sole of the Blackwood Rock-bound to a tooth of the coast. He lived in a world of rhythmic pulses "} +{"model": "mlx-community/gemma-4-26b-a4b-it-4bit", "config": "MTP-bf16", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 16.61, "swap_delta_gb": 0.0, "min_free_pct": 38, "status": "OK", "prompt_tokens": 9487, "prefill_tps": 951.2, "ttft_s": 10.17, "decode_tps": 30.82, "gen_tokens": 128, "needle_ok": true, "degenerate": false, "code": "HERON-38", "answer_head": "The secret code word is HERON-38.\n\nElias Thorne had lived within the salt spray and the rhythmic pulse of the Great Lens for forty-year stretch. To the villagers in the valley, he was a ghost story and shadow and stone and sea. He was a man"} diff --git a/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md new file mode 100644 index 0000000..a3d73f0 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_4bit_mtp_bf16.md @@ -0,0 +1,12 @@ +### `mlx-community/gemma-4-26b-a4b-it-4bit` + +Apple M6 Β· 32 GB Β· runs=3 (long=1) Β· warmup=1 Β· gen=128 Β· temperature 0 Β· medians + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Ξ” GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 512 (533) | 788.5 | 0.72 | 52.95 | 14.5 | 0.0 | 45 | ok | +| Vanilla | 2048 (2284) | 974.8 | 2.41 | 50.76 | 15.14 | 0.0 | 41 | ok | +| Vanilla | 8192 (9543) | 972.3 | 10.01 | 45.96 | 16.71 | 0.0 | 41 | ok | +| MTP-bf16 | 512 (522) | 791.9 | 0.71 | 45.18 | 16.33 | 0.0 | 40 | ok | +| MTP-bf16 | 2048 (2286) | 973.5 | 2.4 | 35.63 | 16.3 | 0.0 | 42 | ok | +| MTP-bf16 | 8192 (9552) | 955.0 | 10.17 | 30.35 | 16.66 | 0.0 | 38 | ok | diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl new file mode 100644 index 0000000..13f4c52 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.jsonl @@ -0,0 +1,10 @@ +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": -1, "warmup": true, "peak_gpu_gb": 6.21, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 534, "prefill_tps": 105.0, "ttft_s": 5.12, "decode_tps": 9.24, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 0, "warmup": false, "peak_gpu_gb": 6.94, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 543, "prefill_tps": 219.0, "ttft_s": 2.52, "decode_tps": 9.18, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 1, "warmup": false, "peak_gpu_gb": 6.89, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 528, "prefill_tps": 232.6, "ttft_s": 2.32, "decode_tps": 8.78, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 512, "run": 2, "warmup": false, "peak_gpu_gb": 6.95, "swap_delta_gb": 0.0, "min_free_pct": 74, "status": "OK", "prompt_tokens": 526, "prefill_tps": 219.6, "ttft_s": 2.44, "decode_tps": 8.49, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 2048, "run": 0, "warmup": false, "peak_gpu_gb": 7.26, "swap_delta_gb": 0.0, "min_free_pct": 73, "status": "OK", "prompt_tokens": 2269, "prefill_tps": 321.3, "ttft_s": 7.14, "decode_tps": 7.83, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 2048, "run": 1, "warmup": false, "peak_gpu_gb": 7.32, "swap_delta_gb": 0.0, "min_free_pct": 70, "status": "OK", "prompt_tokens": 2293, "prefill_tps": 193.7, "ttft_s": 11.93, "decode_tps": 7.7, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 2048, "run": 2, "warmup": false, "peak_gpu_gb": 7.06, "swap_delta_gb": 0.0, "min_free_pct": 69, "status": "OK", "prompt_tokens": 2291, "prefill_tps": 119.6, "ttft_s": 19.24, "decode_tps": 7.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 0, "warmup": false, "peak_gpu_gb": 7.58, "swap_delta_gb": 0.0, "min_free_pct": 66, "status": "OK", "prompt_tokens": 9543, "prefill_tps": 147.6, "ttft_s": 64.86, "decode_tps": 7.08, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 1, "warmup": false, "peak_gpu_gb": 7.19, "swap_delta_gb": 0.0, "min_free_pct": 60, "status": "OK", "prompt_tokens": 9518, "prefill_tps": 138.5, "ttft_s": 68.94, "decode_tps": 6.92, "gen_tokens": 128, "needle_ok": true, "degenerate": false} +{"model": "mlx-community/gemma-4-26b-a4b-it-8bit", "config": "SSD", "context": 8192, "run": 2, "warmup": false, "peak_gpu_gb": 7.26, "swap_delta_gb": 0.0, "min_free_pct": 53, "status": "OK", "prompt_tokens": 9577, "prefill_tps": 136.9, "ttft_s": 70.22, "decode_tps": 6.64, "gen_tokens": 128, "needle_ok": true, "degenerate": false} diff --git a/docs/profiling/m6/gemma4_26b_a4b_8bit.md b/docs/profiling/m6/gemma4_26b_a4b_8bit.md new file mode 100644 index 0000000..74ef970 --- /dev/null +++ b/docs/profiling/m6/gemma4_26b_a4b_8bit.md @@ -0,0 +1,14 @@ +### `mlx-community/gemma-4-26b-a4b-it-8bit` + +Apple M6 Β· 32 GB Β· runs=3 (long=1) Β· warmup=1 Β· gen=128 Β· temperature 0 Β· medians + +| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Ξ” GB | Min free % | Checks | +|---|---|---|---|---|---|---|---|---| +| Vanilla | 512 | β€” | β€” | β€” | 3.33 | 3.06 | 28 | **MEM_ABORT** swap grew 3.1 GB | +| Vanilla | 2048 | β€” | β€” | β€” | β€” | β€” | β€” | **SKIPPED_AFTER_ABORT** | +| Vanilla | 8192 | β€” | β€” | β€” | β€” | β€” | β€” | **SKIPPED_AFTER_ABORT** | +| Vanilla | 32768 | β€” | β€” | β€” | β€” | β€” | β€” | **SKIPPED_AFTER_ABORT** | +| SSD | 512 (528) | 219.6 | 2.44 | 8.78 | 6.95 | 0.0 | 73 | ok | +| SSD | 2048 (2291) | 193.7 | 11.93 | 7.7 | 7.32 | 0.0 | 69 | ok | +| SSD | 8192 (9543) | 138.5 | 68.94 | 6.92 | 7.58 | 0.0 | 53 | ok | +| SSD | 32768 | β€” | β€” | β€” | 5.8 | 2.09 | 32 | **MEM_ABORT** swap grew 2.1 GB | diff --git a/docs/profiling/m6/media/gemma_long.cast b/docs/profiling/m6/media/gemma_long.cast new file mode 100644 index 0000000..c8c94fa --- /dev/null +++ b/docs/profiling/m6/media/gemma_long.cast @@ -0,0 +1,77 @@ +{"version":3,"term":{"cols":112,"rows":22},"timestamp":1790229271,"idle_time_limit":3.0,"command":"env CTX=48000 /Users/simba/.cache/swiftlm-bench/demo/record_demo.sh mlx-community/gemma-4-26b-a4b-it-4bit long 2500","title":"SwiftLM Β· Gemma-4-26B-A4B Β· 40K-token prompt Β· M6 32GB","env":{"SHELL":"/bin/zsh"}} +[0.007, "o", "\u001b[36mSwiftLM β€” github.com/SharpAI/SwiftLM\u001b[39m\r\n\u001b[3244mOpenAI-compatible LLM server for Apple Silicon, in Swift on MLX\u001b[39m\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m sysctl -n machdep.cpu.brand_string hw.memsize | paste -sd' ' -\r\n"] +[0.630, "o", "Apple M6 Β· 32 GB Β· macOS 27.0\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m .build/release/SwiftLM --model mlx-community/gemma-4-26b-a4b-it-4bit --port 5431 &\r\n"] +[3.254, "o", "βœ… Memory strategy: FULL GPU (15.3GB model, 30.4GB available, cache limited to 4685MB)\r\nβœ… Ready. Listening on http://127.0.0.1:5431\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[33m(this GIF plays at 4Γ— speed β€” every timing printed on screen is real)\u001b[39m\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py long 2500 # ~40K-token prompt, code word hidden inside\r\n"] +[1.796, "o", "\r\u001b[2m prefilling … 1s | SwiftLM 15.0 GB swap +0.0 GB\u001b[0m "] +[1.078, "o", "\r\u001b[2m prefilling … 2s | SwiftLM 16.0 GB swap +0.0 GB\u001b[0m "] +[1.067, "o", "\r\u001b[2m prefilling … 3s | SwiftLM 16.0 GB swap +0.0 GB\u001b[0m "] +[1.079, "o", "\r\u001b[2m prefilling … 4s | SwiftLM 16.0 GB swap +0.0 GB\u001b[0m "] +[1.080, "o", "\r\u001b[2m prefilling … 5s | SwiftLM 16.0 GB swap +0.0 GB\u001b[0m "] +[1.084, "o", "\r\u001b[2m prefilling … 6s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.081, "o", "\r\u001b[2m prefilling … 7s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.081, "o", "\r\u001b[2m prefilling … 9s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.079, "o", "\r\u001b[2m prefilling … 10s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.082, "o", "\r\u001b[2m prefilling … 11s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.086, "o", "\r\u001b[2m prefilling … 12s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.087, "o", "\r\u001b[2m prefilling … 13s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.084, "o", "\r\u001b[2m prefilling … 14s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.084, "o", "\r\u001b[2m prefilling … 15s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.086, "o", "\r\u001b[2m prefilling … 16s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.074, "o", "\r\u001b[2m prefilling … 17s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.085, "o", "\r\u001b[2m prefilling … 18s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 19s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.084, "o", "\r\u001b[2m prefilling … 20s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 22s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.093, "o", "\r\u001b[2m prefilling … 23s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.078, "o", "\r\u001b[2m prefilling … 24s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.104, "o", "\r\u001b[2m prefilling … 25s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 26s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 27s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.084, "o", "\r\u001b[2m prefilling … 28s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 29s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.102, "o", "\r\u001b[2m prefilling … 30s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.098, "o", "\r\u001b[2m prefilling … 31s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.104, "o", "\r\u001b[2m prefilling … 32s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.112, "o", "\r\u001b[2m prefilling … 34s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.093, "o", "\r\u001b[2m prefilling … 35s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 36s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.098, "o", "\r\u001b[2m prefilling … 37s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 38s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.100, "o", "\r\u001b[2m prefilling … 39s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.099, "o", "\r\u001b[2m prefilling … 40s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.093, "o", "\r\u001b[2m prefilling … 41s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.098, "o", "\r\u001b[2m prefilling … 42s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.085, "o", "\r\u001b[2m prefilling … 43s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 45s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 46s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 47s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.091, "o", "\r\u001b[2m prefilling … 48s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 49s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.108, "o", "\r\u001b[2m prefilling … 50s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 51s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.092, "o", "\r\u001b[2m prefilling … 52s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.103, "o", "\r\u001b[2m prefilling … 53s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 54s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.087, "o", "\r\u001b[2m prefilling … 56s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.089, "o", "\r\u001b[2m prefilling … 57s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[0.248, "o", "\r\u001b[K\u001b[32mThe"] +[0.043, "o", " secret"] +[0.039, "o", " code"] +[0.034, "o", " word"] +[0.035, "o", " is"] +[0.035, "o", " O"] +[0.034, "o", "SP"] +[0.034, "o", "REY"] +[0.033, "o", "-"] +[0.033, "o", "1"] +[0.034, "o", "7"] +[0.033, "o", "."] +[0.033, "o", "\u001b[0m\r\n\r\n\u001b[36m\u001b[1m 12 tokens Β· decode 26.2 tok/s Β· TTFT 56.9s Β· prompt 41,447 tok Β· prefill 728 tok/s\u001b[0m\r\n\u001b[36m\u001b[1m peak SwiftLM 21.0 GB Β· swap +0.0 GB\u001b[0m\r\n"] +[0.010, "o", "\r\n"] +[0.162, "x", "0"] diff --git a/docs/profiling/m6/media/gemma_short.cast b/docs/profiling/m6/media/gemma_short.cast new file mode 100644 index 0000000..8b3d405 --- /dev/null +++ b/docs/profiling/m6/media/gemma_short.cast @@ -0,0 +1,104 @@ +{"version":3,"term":{"cols":112,"rows":30},"timestamp":1790229095,"idle_time_limit":2.0,"command":"/Users/simba/.cache/swiftlm-bench/demo/record_demo.sh mlx-community/gemma-4-26b-a4b-it-4bit short","title":"SwiftLM Β· Gemma-4-26B-A4B Β· Mac mini M6 32GB","env":{"SHELL":"/bin/zsh"}} +[0.223, "o", "\u001b[36mSwiftLM β€” github.com/SharpAI/SwiftLM\u001b[39m\r\n"] +[0.000, "o", "\u001b[3244mOpenAI-compatible LLM server for Apple Silicon, in Swift on MLX\u001b[39m\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m sysctl -n machdep.cpu.brand_string hw.memsize | paste -sd' ' -\r\n"] +[0.630, "o", "Apple M6 Β· 32 GB Β· macOS 27.0\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m .build/release/SwiftLM --model mlx-community/gemma-4-26b-a4b-it-4bit --port 5431 &\r\n"] +[3.229, "o", "βœ… Memory strategy: FULL GPU (15.3GB model, 30.4GB available, cache limited to 4994MB)\r\nβœ… Ready. Listening on http://127.0.0.1:5431\r\n"] +[0.001, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py short # streams POST /v1/chat/completions\r\n"] +[1.338, "o", "\r\u001b[K\u001b[32m```"] +[0.016, "o", "swift"] +[0.018, "o", "\r\n"] +[0.018, "o", "///"] +[0.018, "o", " Returns"] +[0.018, "o", " the"] +[0.019, "o", " n"] +[0.017, "o", "-"] +[0.019, "o", "th"] +[0.017, "o", " Fibonacci"] +[0.017, "o", " number"] +[0.018, "o", " using"] +[0.017, "o", " an"] +[0.018, "o", " iterative"] +[0.018, "o", " approach"] +[0.019, "o", "."] +[0.016, "o", "\r\n"] +[0.018, "o", "func"] +[0.017, "o", " fib"] +[0.018, "o", "(_"] +[0.018, "o", " n"] +[0.017, "o", ":"] +[0.018, "o", " Int"] +[0.020, "o", ")"] +[0.016, "o", " ->"] +[0.018, "o", " Int"] +[0.018, "o", " {"] +[0.018, "o", "\r\n"] +[0.019, "o", " "] +[0.019, "o", "if"] +[0.018, "o", " n"] +[0.019, "o", " <="] +[0.017, "o", " "] +[0.017, "o", "1"] +[0.018, "o", " {"] +[0.017, "o", " return"] +[0.018, "o", " n"] +[0.018, "o", " }"] +[0.017, "o", "\r\n"] +[0.020, "o", " "] +[0.015, "o", "var"] +[0.019, "o", " a"] +[0.018, "o", " ="] +[0.017, "o", " "] +[0.018, "o", "0"] +[0.018, "o", "\r\n"] +[0.018, "o", " "] +[0.019, "o", "var"] +[0.017, "o", " b"] +[0.017, "o", " ="] +[0.018, "o", " "] +[0.019, "o", "1"] +[0.017, "o", "\r\n"] +[0.019, "o", " "] +[0.017, "o", "for"] +[0.019, "o", " _"] +[0.017, "o", " in"] +[0.017, "o", " "] +[0.018, "o", "2"] +[0.018, "o", "..."] +[0.017, "o", "n"] +[0.018, "o", " {"] +[0.018, "o", "\r\n"] +[0.018, "o", " "] +[0.017, "o", "let"] +[0.018, "o", " temp"] +[0.017, "o", " ="] +[0.018, "o", " a"] +[0.019, "o", " +"] +[0.018, "o", " b"] +[0.018, "o", "\r\n"] +[0.019, "o", " "] +[0.016, "o", "a"] +[0.018, "o", " ="] +[0.019, "o", " b"] +[0.018, "o", "\r\n"] +[0.018, "o", " "] +[0.017, "o", "b"] +[0.018, "o", " ="] +[0.019, "o", " temp"] +[0.016, "o", "\r\n"] +[0.018, "o", " "] +[0.018, "o", "}"] +[0.018, "o", "\r\n"] +[0.018, "o", " "] +[0.017, "o", "return"] +[0.019, "o", " b"] +[0.019, "o", "\r\n"] +[0.018, "o", "}"] +[0.018, "o", "\r\n"] +[0.019, "o", "```"] +[0.018, "o", "\u001b[0m\r\n\r\n\u001b[36m\u001b[1m 91 tokens Β· decode 55.4 tok/s Β· TTFT 0.6s Β· peak SwiftLM 15.0 GB Β· swap +0.0 GB\u001b[0m\r\n"] +[0.006, "o", "\r\n"] +[0.106, "x", "0"] diff --git a/docs/profiling/m6/media/m6_gemma4_26b_a4b_41k_prompt_4x.gif b/docs/profiling/m6/media/m6_gemma4_26b_a4b_41k_prompt_4x.gif new file mode 100644 index 0000000..bd822d4 Binary files /dev/null and b/docs/profiling/m6/media/m6_gemma4_26b_a4b_41k_prompt_4x.gif differ diff --git a/docs/profiling/m6/media/m6_gemma4_26b_a4b_stream.gif b/docs/profiling/m6/media/m6_gemma4_26b_a4b_stream.gif new file mode 100644 index 0000000..12330cb Binary files /dev/null and b/docs/profiling/m6/media/m6_gemma4_26b_a4b_stream.gif differ diff --git a/docs/profiling/m6/media/m6_qwen38_27b_8k_prompt_4x.gif b/docs/profiling/m6/media/m6_qwen38_27b_8k_prompt_4x.gif new file mode 100644 index 0000000..557fb43 Binary files /dev/null and b/docs/profiling/m6/media/m6_qwen38_27b_8k_prompt_4x.gif differ diff --git a/docs/profiling/m6/media/m6_qwen38_27b_stream.gif b/docs/profiling/m6/media/m6_qwen38_27b_stream.gif new file mode 100644 index 0000000..81de9ae Binary files /dev/null and b/docs/profiling/m6/media/m6_qwen38_27b_stream.gif differ diff --git a/docs/profiling/m6/media/qwen_long.cast b/docs/profiling/m6/media/qwen_long.cast new file mode 100644 index 0000000..5008f50 --- /dev/null +++ b/docs/profiling/m6/media/qwen_long.cast @@ -0,0 +1,106 @@ +{"version":3,"term":{"cols":112,"rows":22},"timestamp":1790227258,"idle_time_limit":3.0,"command":"/Users/simba/.cache/swiftlm-bench/demo/record_demo.sh mlx-community/Qwen3.8-27B-4bit long 540","title":"SwiftLM Β· 8.6K-token prompt Β· Mac mini M6 32GB","env":{"SHELL":"/bin/zsh"}} +[0.008, "o", "\u001b[36mSwiftLM β€” github.com/SharpAI/SwiftLM\u001b[39m\r\n\u001b[3244mOpenAI-compatible LLM server for Apple Silicon, in Swift on MLX\u001b[39m\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m sysctl -n machdep.cpu.brand_string hw.memsize | paste -sd' ' -\r\n"] +[0.631, "o", "Apple M6 Β· 32 GB Β· macOS 27.0\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m .build/release/SwiftLM --model mlx-community/Qwen3.8-27B-4bit --port 5431 &\r\n"] +[3.221, "o", "βœ… Memory strategy: FULL GPU (11.3GB model, 30.4GB available, cache limited to 5115MB)\r\nβœ… Ready. Listening on http://127.0.0.1:5431\r\n"] +[0.001, "o", "\r\n"] +[0.000, "o", "\u001b[33m(this GIF plays at 4Γ— speed β€” every timing printed on screen is real)\u001b[39m\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py long 540 # ~9K-token prompt, code word hidden inside\r\n"] +[1.803, "o", "\r\u001b[2m prefilling … 1s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.081, "o", "\r\u001b[2m prefilling … 2s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.076, "o", "\r\u001b[2m prefilling … 3s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.076, "o", "\r\u001b[2m prefilling … 4s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.073, "o", "\r\u001b[2m prefilling … 5s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.087, "o", "\r\u001b[2m prefilling … 6s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.083, "o", "\r\u001b[2m prefilling … 7s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.108, "o", "\r\u001b[2m prefilling … 9s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.087, "o", "\r\u001b[2m prefilling … 10s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.077, "o", "\r\u001b[2m prefilling … 11s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 12s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 13s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.091, "o", "\r\u001b[2m prefilling … 14s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.098, "o", "\r\u001b[2m prefilling … 15s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.085, "o", "\r\u001b[2m prefilling … 16s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.093, "o", "\r\u001b[2m prefilling … 17s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.080, "o", "\r\u001b[2m prefilling … 18s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.087, "o", "\r\u001b[2m prefilling … 19s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 21s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.099, "o", "\r\u001b[2m prefilling … 22s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 23s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.100, "o", "\r\u001b[2m prefilling … 24s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.099, "o", "\r\u001b[2m prefilling … 25s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.090, "o", "\r\u001b[2m prefilling … 26s | SwiftLM 17.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 27s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 28s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.107, "o", "\r\u001b[2m prefilling … 29s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.088, "o", "\r\u001b[2m prefilling … 30s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.100, "o", "\r\u001b[2m prefilling … 32s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 33s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.097, "o", "\r\u001b[2m prefilling … 34s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.099, "o", "\r\u001b[2m prefilling … 35s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 36s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.121, "o", "\r\u001b[2m prefilling … 37s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.097, "o", "\r\u001b[2m prefilling … 38s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.089, "o", "\r\u001b[2m prefilling … 39s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.115, "o", "\r\u001b[2m prefilling … 40s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 41s | SwiftLM 18.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 43s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 44s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.109, "o", "\r\u001b[2m prefilling … 45s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 46s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.096, "o", "\r\u001b[2m prefilling … 47s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 48s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.100, "o", "\r\u001b[2m prefilling … 49s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 50s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 51s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.109, "o", "\r\u001b[2m prefilling … 52s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 54s | SwiftLM 19.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 55s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 56s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.106, "o", "\r\u001b[2m prefilling … 57s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.102, "o", "\r\u001b[2m prefilling … 58s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 59s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 60s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 61s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.102, "o", "\r\u001b[2m prefilling … 62s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.103, "o", "\r\u001b[2m prefilling … 63s | SwiftLM 20.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 65s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.090, "o", "\r\u001b[2m prefilling … 66s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 67s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.091, "o", "\r\u001b[2m prefilling … 68s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.094, "o", "\r\u001b[2m prefilling … 69s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.098, "o", "\r\u001b[2m prefilling … 70s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.099, "o", "\r\u001b[2m prefilling … 71s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 72s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.109, "o", "\r\u001b[2m prefilling … 73s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.107, "o", "\r\u001b[2m prefilling … 74s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.104, "o", "\r\u001b[2m prefilling … 76s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 77s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.095, "o", "\r\u001b[2m prefilling … 78s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.090, "o", "\r\u001b[2m prefilling … 79s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.111, "o", "\r\u001b[2m prefilling … 80s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.117, "o", "\r\u001b[2m prefilling … 81s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 82s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.121, "o", "\r\u001b[2m prefilling … 83s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.101, "o", "\r\u001b[2m prefilling … 84s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.108, "o", "\r\u001b[2m prefilling … 85s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.105, "o", "\r\u001b[2m prefilling … 87s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[1.104, "o", "\r\u001b[2m prefilling … 88s | SwiftLM 21.0 GB swap +0.0 GB\u001b[0m "] +[0.904, "o", "\r\u001b[K\u001b[32m"] +[0.000, "o", "The"] +[0.119, "o", " secret"] +[0.122, "o", " code"] +[0.120, "o", " word"] +[0.120, "o", " is"] +[0.120, "o", " OSP"] +[0.125, "o", "RE"] +[0.126, "o", "Y"] +[0.119, "o", "-"] +[0.119, "o", "1"] +[0.123, "o", "7"] +[0.123, "o", "."] +[0.118, "o", "\u001b[0m\r\n\r\n\u001b[36m\u001b[1m 12 tokens Β· decode 7.6 tok/s Β· TTFT 88.7s Β· prefill 96.8 tok/s Β· peak SwiftLM 21.0 GB Β· swap +0.0 GB\u001b[0m\r\n"] +[0.011, "o", "\r\n"] +[0.194, "x", "0"] diff --git a/docs/profiling/m6/media/qwen_short.cast b/docs/profiling/m6/media/qwen_short.cast new file mode 100644 index 0000000..0803e30 --- /dev/null +++ b/docs/profiling/m6/media/qwen_short.cast @@ -0,0 +1,97 @@ +{"version":3,"term":{"cols":112,"rows":26},"timestamp":1790227384,"idle_time_limit":2.0,"command":"/Users/simba/.cache/swiftlm-bench/demo/record_demo.sh mlx-community/Qwen3.8-27B-4bit short","title":"SwiftLM Β· Qwen3.8-27B-4bit Β· Mac mini M6 32GB","env":{"SHELL":"/bin/zsh"}} +[0.006, "o", "\u001b[36mSwiftLM β€” github.com/SharpAI/SwiftLM\u001b[39m\r\n\u001b[3244mOpenAI-compatible LLM server for Apple Silicon, in Swift on MLX\u001b[39m\r\n\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m sysctl -n machdep.cpu.brand_string hw.memsize | paste -sd' ' -\r\n"] +[0.637, "o", "Apple M6 Β· 32 GB Β· macOS 27.0\r\n"] +[0.000, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m .build/release/SwiftLM --model mlx-community/Qwen3.8-27B-4bit --port 5431 &\r\n"] +[3.262, "o", "βœ… Memory strategy: FULL GPU (11.3GB model, 30.4GB available, cache limited to 5115MB)\r\nβœ… Ready. Listening on http://127.0.0.1:5431\r\n"] +[0.001, "o", "\r\n"] +[0.000, "o", "\u001b[3244m$\u001b[39m python3 scripts/demo/stream_client.py short # streams POST /v1/chat/completions\r\n"] +[1.859, "o", "\r\u001b[K\u001b[32m```"] +[0.115, "o", "swift"] +[0.117, "o", "\r\n"] +[0.116, "o", "///"] +[0.114, "o", " Returns"] +[0.115, "o", " the"] +[0.117, "o", " n"] +[0.114, "o", "-th"] +[0.114, "o", " Fibonacci"] +[0.116, "o", " number"] +[0.117, "o", " computed"] +[0.116, "o", " iter"] +[0.115, "o", "atively"] +[0.119, "o", "."] +[0.115, "o", "\r\n"] +[0.123, "o", "func"] +[0.119, "o", " fib"] +[0.123, "o", "(_"] +[0.117, "o", " n"] +[0.121, "o", ":"] +[0.116, "o", " Int"] +[0.113, "o", ")"] +[0.115, "o", " ->"] +[0.118, "o", " Int"] +[0.113, "o", " {"] +[0.113, "o", "\r\n"] +[0.115, "o", " "] +[0.116, "o", " guard"] +[0.118, "o", " n"] +[0.114, "o", " >="] +[0.116, "o", " "] +[0.116, "o", "0"] +[0.114, "o", " else"] +[0.116, "o", " {"] +[0.114, "o", " return"] +[0.114, "o", " "] +[0.116, "o", "0"] +[0.117, "o", " }"] +[0.115, "o", "\r\n"] +[0.116, "o", " "] +[0.116, "o", " var"] +[0.118, "o", " a"] +[0.116, "o", " ="] +[0.118, "o", " "] +[0.118, "o", "0"] +[0.119, "o", ","] +[0.119, "o", " b"] +[0.121, "o", " ="] +[0.113, "o", " "] +[0.116, "o", "1"] +[0.115, "o", "\r\n"] +[0.115, "o", " "] +[0.113, "o", " for"] +[0.115, "o", " _"] +[0.117, "o", " in"] +[0.115, "o", " "] +[0.115, "o", "0"] +[0.233, "o", " +""" +import json, re, subprocess, sys, threading, time, urllib.request + +PORT = 5431 +DIM, BOLD, CYAN, GREEN, RESET = "\033[2m", "\033[1m", "\033[36m", "\033[32m", "\033[0m" + + +def server_pid(): + out = subprocess.run(["pgrep", "-n", "-f", "SwiftLM --model"], capture_output=True, text=True).stdout.split() + return out[0] if out else None + + +def footprint_gb(pid): + out = subprocess.run(["footprint", "-p", pid], capture_output=True, text=True).stdout + m = re.search(r"phys_footprint:\s+([0-9.]+)\s+(GB|MB)", out) + if not m: + return 0.0 + v = float(m.group(1)) + return v if m.group(2) == "GB" else v / 1024 + + +def swap_gb(): + out = subprocess.run(["sysctl", "-n", "vm.swapusage"], capture_output=True, text=True).stdout + m = re.search(r"used = ([0-9.]+)M", out) + return float(m.group(1)) / 1024 if m else 0.0 + + +def main(): + kind = sys.argv[1] + if kind == "short": + prompt = "Write a Swift function `fib(_ n: Int) -> Int` that returns the n-th Fibonacci number iteratively. Code only, with a one-line doc comment." + max_tokens = 160 + else: + n = int(sys.argv[2]) + lines = [f"Sensor {i} in the greenhouse reported stable humidity and mild warmth." for i in range(n)] + lines.insert(n * 2 // 3, "Note: the secret code word is OSPREY-17.") + prompt = f"[demo-{int(time.time())}]\n" + "\n".join(lines) + "\n\nWhat is the secret code word? Answer in one short sentence." + max_tokens = 24 + + pid = server_pid() + swap0, peak_fp, peak_swap = swap_gb(), [0.0], [0.0] + stop = threading.Event() + + def sample(): + while not stop.is_set(): + peak_fp[0] = max(peak_fp[0], footprint_gb(pid)) + peak_swap[0] = max(peak_swap[0], swap_gb() - swap0) + stop.wait(1) + + threading.Thread(target=sample, daemon=True).start() + + # Client-side prefill ticker. The server's opt-in prefill_progress heartbeat + # doesn't fire on the VLM path yet, so we show elapsed time and live memory + # here until the first token arrives. + first_token = threading.Event() + out_lock = threading.Lock() # keeps the ticker from writing over the first token + t0 = time.time() + + def ticker(): + while not first_token.wait(1): + with out_lock: + if first_token.is_set(): + break + sys.stdout.write(f"\r{DIM} prefilling … {time.time() - t0:5.0f}s | SwiftLM {footprint_gb(pid):4.1f} GB " + f"swap +{max(0, swap_gb() - swap0):.1f} GB{RESET} ") + sys.stdout.flush() + + if kind != "short": + threading.Thread(target=ticker, daemon=True).start() + + body = json.dumps({"messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens, + "temperature": 0, "stream": True, + "stream_options": {"include_usage": True}}).encode() + req = urllib.request.Request(f"http://127.0.0.1:{PORT}/v1/chat/completions", body, + {"Content-Type": "application/json", "X-SwiftLM-Prefill-Progress": "true"}) + t_first = None; n_tok = 0; event = None; n_prompt = None + with urllib.request.urlopen(req, timeout=3600) as r: + for raw in r: + line = raw.decode().strip() + if line.startswith("event:"): + event = line[6:].strip(); continue + if not line.startswith("data:") or line.endswith("[DONE]"): + continue + d = json.loads(line[5:]) + if event == "prefill_progress": + n_prompt = d.get("n_prompt_tokens") or n_prompt + sys.stdout.write(f"\r{DIM} prefilling {n_prompt or '?':,} tokens … {d.get('elapsed_seconds', time.time() - t0):5.0f}s " + f"| SwiftLM {footprint_gb(pid):4.1f} GB swap +{max(0, swap_gb() - swap0):.1f} GB{RESET} ") + sys.stdout.flush(); event = None; continue + event = None + if d.get("usage"): + n_prompt = d["usage"].get("prompt_tokens", n_prompt) + delta = (d.get("choices") or [{}])[0].get("delta", {}).get("content") + if delta: + if t_first is None: + with out_lock: + t_first = time.time(); first_token.set() + sys.stdout.write("\r\033[K" + GREEN) + n_tok += 1 + sys.stdout.write(delta); sys.stdout.flush() + t_end = time.time(); stop.set(); first_token.set() + sys.stdout.write(RESET + "\n\n") + decode = (n_tok - 1) / (t_end - t_first) if n_tok > 1 else 0 + ttft = t_first - t0 + pf = f" prompt {n_prompt:,} tok Β· prefill {n_prompt / ttft:.0f} tok/s Β·" if n_prompt and kind != "short" else "" + print(f"{CYAN}{BOLD} {n_tok} tokens Β· decode {decode:.1f} tok/s Β· TTFT {ttft:.1f}s Β·{pf.rstrip(' Β·')}{RESET}") + print(f"{CYAN}{BOLD} peak SwiftLM {peak_fp[0]:.1f} GB Β· swap +{max(0, peak_swap[0]):.1f} GB{RESET}") + + +if __name__ == "__main__": + main() diff --git a/scripts/profiling/m6_bench.py b/scripts/profiling/m6_bench.py new file mode 100644 index 0000000..d5f975a --- /dev/null +++ b/scripts/profiling/m6_bench.py @@ -0,0 +1,330 @@ +#!/usr/bin/env python3 +"""Memory-guarded SwiftLM benchmark harness (built for the 32 GB M6 Mac mini). + +For each config it starts one SwiftLM server, then for each context length it +runs a warm-up plus N measured requests and reports medians. It guards memory +on a small-RAM machine: + + * Before each request it records swap usage and free memory as a baseline. + * While a request runs it polls GPU in-use memory (ioreg), swap and free memory + every 0.5 s. + * If swap grows by more than --swap-abort-gb, or free memory drops below + --min-free-pct, it kills the server, records a MEM_ABORT row, and skips the + larger contexts for that config. + +Every prompt starts with a unique nonce, so SwiftLM's prompt cache can't turn a +repeated run into a cache hit. Each prompt also contains a planted code word, +and the answer must reproduce it. + +Example: + python3 scripts/profiling/m6_bench.py --model mlx-community/Qwen3.8-27B-4bit \ + --config "Vanilla=" --config "TurboKV=--turbo-kv" \ + --contexts 512,2048,8192,32768 --out docs/profiling/m6/qwen38_27b +""" +import argparse +import json +import os +import random +import re +import signal +import statistics +import subprocess +import threading +import time +import urllib.request + +SWIFTLM_PATH = ".build/release/SwiftLM" +FILLER = [ + "The harbor master logged {n} vessels before noon and noted calm water.", + "Sensor {n} in the greenhouse reported stable humidity and mild warmth.", + "Ledger entry {n}: shipment of copper wire received, inspected, and shelved.", + "Trail marker {n} points north along the ridge past the old pine grove.", +] + + +# ── system memory probes ───────────────────────────────────────────────────── + +def swap_used_gb(): + out = subprocess.run(["sysctl", "-n", "vm.swapusage"], capture_output=True, text=True).stdout + m = re.search(r"used = ([0-9.]+)M", out) + return float(m.group(1)) / 1024 if m else 0.0 + + +def free_pct(): + out = subprocess.run(["memory_pressure"], capture_output=True, text=True).stdout + m = re.search(r"free percentage: (\d+)%", out) + return int(m.group(1)) if m else -1 + + +def gpu_in_use_gb(): + out = subprocess.run(["ioreg", "-r", "-d", "1", "-w", "0", "-c", "AGXAccelerator"], + capture_output=True, text=True, timeout=5).stdout + m = re.search(r'"In use system memory"=(\d+)', out) + return int(m.group(1)) / 1024**3 if m else 0.0 + + +class MemWatch: + """Polls memory while a request runs. Sets .tripped when a guard fires.""" + + def __init__(self, swap_abort_gb, min_free_pct, on_trip): + self.swap0 = swap_used_gb() + self.swap_abort_gb, self.min_free_pct, self.on_trip = swap_abort_gb, min_free_pct, on_trip + self.peak_gpu = self.peak_swap_delta = 0.0 + self.min_free = 100 + self.tripped = None + self._stop = threading.Event() + self._t = threading.Thread(target=self._run, daemon=True) + + def _run(self): + tick = 0 + while not self._stop.is_set(): + self.peak_gpu = max(self.peak_gpu, gpu_in_use_gb()) + self.peak_swap_delta = max(self.peak_swap_delta, swap_used_gb() - self.swap0) + if tick % 4 == 0: # memory_pressure is slower; sample every 2 s + fp = free_pct() + if fp >= 0: + self.min_free = min(self.min_free, fp) + tick += 1 + if self.tripped is None: + if self.peak_swap_delta > self.swap_abort_gb: + self.tripped = f"swap grew {self.peak_swap_delta:.1f} GB" + elif 0 <= self.min_free < self.min_free_pct: + self.tripped = f"free memory {self.min_free}%" + if self.tripped: + self.on_trip() + self._stop.wait(0.5) + + def __enter__(self): + self._t.start() + return self + + def __exit__(self, *_): + self._stop.set() + self._t.join(timeout=5) + + +# ── server lifecycle ───────────────────────────────────────────────────────── + +def start_server(model, flags, port, ctx_size, log_path): + cmd = [SWIFTLM_PATH, "--model", model, "--port", str(port), "--ctx-size", str(ctx_size)] + flags + log = open(log_path, "w") + proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT) + deadline = time.time() + 900 + while time.time() < deadline: + if proc.poll() is not None: + return None + try: + urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=2) + return proc + except Exception: + time.sleep(1) + proc.kill() + return None + + +def stop_server(proc): + if proc and proc.poll() is None: + proc.send_signal(signal.SIGTERM) + try: + proc.wait(timeout=20) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() + # Let the Metal heap drain before the next config loads. + deadline = time.time() + 60 + while time.time() < deadline and gpu_in_use_gb() > 2.0: + time.sleep(1) + + +def log_lines_since(log_path, offset): + with open(log_path) as f: + f.seek(offset) + return f.read() + + +# ── one request ────────────────────────────────────────────────────────────── + +def build_prompt(target_tokens, rng): + nonce = f"run-{rng.getrandbits(48):012x}" + code = f"{rng.choice(['PELICAN', 'MARLIN', 'OSPREY', 'HERON'])}-{rng.randint(10, 99)}" + # ~15 tokens per filler line; leave room for the instructions. + n_lines = max(1, (target_tokens - 80) // 15) + lines = [rng.choice(FILLER).format(n=i) for i in range(n_lines)] + lines.insert(rng.randint(0, len(lines)), f"Note: the secret code word is {code}.") + prompt = (f"[{nonce}]\n" + "\n".join(lines) + + "\n\nFirst, state the secret code word from the notes above. " + "Then write a detailed story of at least 300 words about a lighthouse keeper.") + return prompt, code + + +def run_request(port, prompt, max_tokens): + body = json.dumps({"messages": [{"role": "user", "content": prompt}], + "max_tokens": max_tokens, "temperature": 0, "stream": True}).encode() + req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/chat/completions", body, + {"Content-Type": "application/json"}) + t0 = time.time() + t_first, text = None, [] + with urllib.request.urlopen(req, timeout=3600) as r: + for raw in r: + line = raw.decode().strip() + if not line.startswith("data:") or line.endswith("[DONE]"): + continue + d = json.loads(line[5:]) + delta = (d.get("choices") or [{}])[0].get("delta", {}).get("content") + if delta: + t_first = t_first or time.time() + text.append(delta) + return t0, t_first, time.time(), "".join(text) + + +def parse_server_stats(log_text): + pre = re.findall(r"prefill done \| n_tokens=(\d+), t=([0-9.]+)s, ([0-9.]+)t/s", log_text) + done = re.findall(r"slot done: id 0 \| gen_tokens=(\d+)", log_text) + return (pre[-1] if pre else None), (int(done[-1]) if done else None) + + +def is_degenerate(text): + words = text.split() + if len(words) < 20: + return False + grams = [" ".join(words[i:i + 4]) for i in range(len(words) - 3)] + return len(set(grams)) / len(grams) < 0.5 + + +# ── main loop ──────────────────────────────────────────────────────────────── + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True) + ap.add_argument("--config", action="append", required=True, + help='NAME=FLAGS, e.g. "TurboKV=--turbo-kv" (repeatable)') + ap.add_argument("--contexts", default="512,2048,8192") + ap.add_argument("--runs", type=int, default=3) + ap.add_argument("--long-runs", type=int, default=1, help="runs for contexts >= --long-threshold") + ap.add_argument("--long-threshold", type=int, default=16384) + ap.add_argument("--warmup", type=int, default=1) + ap.add_argument("--gen", type=int, default=128) + ap.add_argument("--port", type=int, default=5431) + ap.add_argument("--swap-abort-gb", type=float, default=2.0) + ap.add_argument("--min-free-pct", type=int, default=10) + ap.add_argument("--seed", type=int, default=7) + ap.add_argument("--out", required=True, help="output path prefix (writes .jsonl and .md)") + args = ap.parse_args() + + contexts = [int(x) for x in args.contexts.split(",")] + ctx_size = max(contexts) + args.gen + 1024 + rng = random.Random(args.seed) + os.makedirs(os.path.dirname(args.out) or ".", exist_ok=True) + raw_path, md_path = args.out + ".jsonl", args.out + ".md" + rows = [] + + for spec in args.config: + name, _, flag_str = spec.partition("=") + flags = flag_str.split() + log_path = f"{args.out}.{re.sub(r'[^A-Za-z0-9]+', '_', name)}.server.log" + print(f"\n=== {name} flags={flags or '(none)'} swap={swap_used_gb():.1f}GB free={free_pct()}%") + proc = start_server(args.model, flags, args.port, ctx_size, log_path) + if not proc: + print(" server failed to start; see", log_path) + rows.append({"config": name, "context": None, "status": "START_FAIL"}) + continue + with open(log_path) as f: + load = re.search(r"\(([0-9.]+)GB model", f.read()) + print(f" loaded ({load.group(1) if load else '?'} GB weights), GPU in-use {gpu_in_use_gb():.1f} GB") + + aborted = False + for ctx in contexts: + if aborted: + rows.append({"config": name, "context": ctx, "status": "SKIPPED_AFTER_ABORT"}) + continue + n_runs = args.long_runs if ctx >= args.long_threshold else args.runs + n_warm = args.warmup if ctx == contexts[0] else 0 + for i in range(n_warm + n_runs): + warm = i < n_warm + prompt, code = build_prompt(ctx, rng) + offset = os.path.getsize(log_path) + watch = MemWatch(args.swap_abort_gb, args.min_free_pct, on_trip=proc.kill) + row = {"config": name, "context": ctx, "run": i - n_warm, "warmup": warm} + with watch: + try: + t0, t_first, t_end, text = run_request(args.port, prompt, args.gen) + ok = True + except Exception as e: + ok, err = False, str(e) + row.update(peak_gpu_gb=round(watch.peak_gpu, 2), swap_delta_gb=round(watch.peak_swap_delta, 2), + min_free_pct=watch.min_free) + if watch.tripped or not ok: + row["status"] = "MEM_ABORT" if watch.tripped else "REQUEST_FAIL" + row["reason"] = watch.tripped or err + print(f" ctx={ctx} {row['status']}: {row['reason']}") + rows.append(row) + aborted = True + break + time.sleep(0.5) # let the server flush its "slot done" line + pre, gen_tokens = parse_server_stats(log_lines_since(log_path, offset)) + gen_tokens = gen_tokens or max(1, len(text.split())) + row.update( + status="OK", + prompt_tokens=int(pre[0]) if pre else None, + prefill_tps=float(pre[2]) if pre else None, + ttft_s=round(t_first - t0, 2) if t_first else None, + decode_tps=round((gen_tokens - 1) / (t_end - t_first), 2) if t_first and gen_tokens > 1 else None, + gen_tokens=gen_tokens, + needle_ok=code in text, + degenerate=is_degenerate(text), + code=code, + answer_head=text[:240], # enough to see what a needle miss actually said + ) + rows.append(row) + tag = "warm" if warm else f"run{row['run']}" + print(f" ctx={ctx:>6} {tag:<5} prompt={row['prompt_tokens']} prefill={row['prefill_tps']} t/s " + f"ttft={row['ttft_s']}s decode={row['decode_tps']} t/s gen={gen_tokens} " + f"needle={'ok' if row['needle_ok'] else 'MISS'}{' DEGEN' if row['degenerate'] else ''} " + f"| gpu={row['peak_gpu_gb']}GB swapΞ”={row['swap_delta_gb']}GB freeβ‰₯{row['min_free_pct']}%") + with open(raw_path, "a") as f: + f.write(json.dumps({"model": args.model, **row}) + "\n") + stop_server(proc) + + write_markdown(md_path, args, rows) + print("\nwrote", md_path, "and", raw_path) + + +def med(vals): + vals = [v for v in vals if v is not None] + return statistics.median(vals) if vals else None + + +def write_markdown(path, args, rows): + hw = subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string", "hw.memsize"], + capture_output=True, text=True).stdout.split("\n") + with open(path, "w") as f: + f.write(f"### `{args.model}`\n\n") + f.write(f"{hw[0]} Β· {int(hw[1]) / 1024**3:.0f} GB Β· runs={args.runs} (long={args.long_runs}) Β· " + f"warmup={args.warmup} Β· gen={args.gen} Β· temperature 0 Β· medians\n\n") + f.write("| Config | Context (prompt tok) | Prefill tok/s | TTFT s | Decode tok/s | Peak GPU GB | Swap Ξ” GB | Min free % | Checks |\n") + f.write("|---|---|---|---|---|---|---|---|---|\n") + keys = [] + for r in rows: + k = (r["config"], r["context"]) + if k not in keys: + keys.append(k) + for cfg, ctx in keys: + every = [r for r in rows if (r["config"], r["context"]) == (cfg, ctx)] + # A case that aborted during its warm-up has no measured rows; report the warm-up. + group = [r for r in every if not r.get("warmup")] or every + bad = [r for r in group if r.get("status") != "OK"] + if bad: + f.write(f"| {cfg} | {ctx} | β€” | β€” | β€” | {bad[0].get('peak_gpu_gb', 'β€”')} | " + f"{bad[0].get('swap_delta_gb', 'β€”')} | {bad[0].get('min_free_pct', 'β€”')} | " + f"**{bad[0]['status']}** {bad[0].get('reason', '')} |\n") + continue + checks = "ok" if all(r["needle_ok"] and not r["degenerate"] for r in group) else \ + f"needle {sum(r['needle_ok'] for r in group)}/{len(group)}, degen {sum(r['degenerate'] for r in group)}" + f.write(f"| {cfg} | {ctx} ({med([r['prompt_tokens'] for r in group])}) | " + f"{med([r['prefill_tps'] for r in group])} | {med([r['ttft_s'] for r in group])} | " + f"{med([r['decode_tps'] for r in group])} | {max(r['peak_gpu_gb'] for r in group)} | " + f"{max(r['swap_delta_gb'] for r in group)} | {min(r['min_free_pct'] for r in group)} | {checks} |\n") + + +if __name__ == "__main__": + main() diff --git a/tests/SwiftLMTests/VLMFallbackTests.swift b/tests/SwiftLMTests/VLMFallbackTests.swift new file mode 100644 index 0000000..63f072e --- /dev/null +++ b/tests/SwiftLMTests/VLMFallbackTests.swift @@ -0,0 +1,44 @@ +import XCTest +import Foundation +import MLXLMCommon +import MLXNN +@testable import SwiftLM + +/// An auto-detected VLM that fails to load falls back to text-only only for +/// checkpoint mismatches. Other failures must surface as themselves. +final class VLMFallbackTests: XCTestCase { + + private struct Probe: Decodable { let image_mean: [Double] } + + func testConfigDecodingErrorFallsBack() { + // The real case: unsloth/Qwen3.6-35B-A3B's preprocessor_config.json has no image_mean. + do { + _ = try JSONDecoder().decode(Probe.self, from: Data("{}".utf8)) + XCTFail("expected a DecodingError") + } catch { + XCTAssertTrue(isVLMCheckpointMismatch(error)) + } + } + + func testWeightMismatchFallsBack() { + let error = UpdateError.unhandledKeys(path: [], modules: ["Vision"], keys: ["pre_projection"]) + XCTAssertTrue(isVLMCheckpointMismatch(error)) + } + + func testUnsupportedModelTypeFallsBack() { + XCTAssertTrue(isVLMCheckpointMismatch(ModelFactoryError.unsupportedModelType("qwen4_exp"))) + } + + func testCancellationDoesNotFallBack() { + XCTAssertFalse(isVLMCheckpointMismatch(CancellationError())) + } + + func testNetworkErrorDoesNotFallBack() { + XCTAssertFalse(isVLMCheckpointMismatch(URLError(.notConnectedToInternet))) + } + + func testMissingConfigFileDoesNotFallBack() { + let io = CocoaError(.fileReadNoSuchFile) + XCTAssertFalse(isVLMCheckpointMismatch(ModelFactoryError.configurationFileError("config.json", "m", io))) + } +}