⚡ Zero-Stall MoE Inference via Lookahead Prediction & Async DMA Prefetching. Optimized for SSD I/O with Hybrid MLA+Sliding Window Attention.
-
Updated
Nov 13, 2026 - Python
⚡ Zero-Stall MoE Inference via Lookahead Prediction & Async DMA Prefetching. Optimized for SSD I/O with Hybrid MLA+Sliding Window Attention.
Run DeepSeek-V4.1-Flash (551B MoE, IQ2_XXS) locally on Apple Silicon with 128K context, even though it's bigger than RAM. Patched llama.cpp Metal runtime streams experts from SSD with a GPU expert cache. Tested on M5 Pro 64 GB: 252 tok/s prefill, 16.3 tok/s decode.
125B MoE (Qwen3.8-Flash-Next) on 16GB RAM + 8GB GPU: expert cache, next-layer read-ahead and GPU keep-alive for llama.cpp on Windows
From-scratch Metal inference engine that runs DeepSeek-V4.1-Flash (233 GB GGUF) with 200K context on a 64 GB Mac — 7x faster prefill, 12x faster decode than llama.cpp. OpenAI-compatible server + web UI.
Run the full 2.8T-parameter Kimi-K3 checkpoint from SSD on a 64 GB M4 Pro Mac mini: all 93 layers, per-expert streaming, native MLX/Metal. Slow (~32 s/token) but complete.
A llama.cpp fork that runs Qwen3.8-Flash-Next (180B MoE, 97.5 GB at true Q4) on a 64 GB M1 Max at ~30 tok/s: experts streamed from the SSD, Metal 4, sparse attention, MTP drafts. 1.9-6x faster than other engines on the same file.
senior thesis: trillion-param moe inference on a few ordinary pcs, with experts streamed nvme → dram → gpu. fork of llama.cpp.
Purpose-built Rust inference engines for Snapdragon X CPUs: a static-plan MoE engine with MTP verify lanes, SSD expert streaming for a 125B MoE in 32 GB, and an int8 image engine.
Kimi K3 (2.8T MoE) on a 16GB-RAM Windows PC: SSD streaming for llama.cpp, 45 -> 4.5 s/token
To associate your repository with the ssd-offloading topic, visit your repo's landing page and select "manage topics."