Restoring Speculative Speedups for Offloaded Mixture-of-Experts Inference
BriskMoE is a lightweight vLLM plugin that makes speculative decoding (SD) an effective latency accelerator for offloaded MoE serving on a single GPU. Without BriskMoE, enabling SD on offloaded MoE models yields negligible speedup (1.05×) due to cache-path overhead and SD-induced transient overflow. BriskMoE eliminates these bottlenecks, achieving up to 4.80× speedup on Qwen3-30B-A3B and 5.00× on GPT-OSS-120B.
| Model | AR Baseline | SD + Cache (naive) | SD + BriskMoE | Speedup |
|---|---|---|---|---|
| Qwen3-30B-A3B (BF16, top-8) | 2.01 tok/s | 7.67 tok/s | 9.98 tok/s | 4.80× |
| GPT-OSS-120B (MXFP4, top-4) | 2.40 tok/s | 11.52 tok/s | 12.01 tok/s | 5.00× |
Hardware: NVIDIA RTX A6000 (48 GB), PCIe Gen4, 503 GB host RAM.
BriskMoE addresses two bottlenecks unique to SD + expert caching:
- Unblock — eliminates cache-hit path overhead via Pool-Direct execution, synchronization-free cache routing (TASER), cross-layer prefetch, and fused-MoE kernel tuning.
- Protect — prevents SD-induced cache overflow via DIPP (Draft-Informed Prioritized Preloading), which exploits the draft-generation interval to pre-stage experts.
BriskMoE is implemented as a non-intrusive vLLM general plugin (adapters/vllm_elmm_plugin.py). It monkey-patches the model worker after loading and requires zero modification to vLLM internals.
adapters/ # Core BriskMoE plugin
elmm_plugin.py # ELMM: Expert-Level Memory Management (main module)
vllm_elmm_plugin.py # vLLM plugin entry point (registered via pyproject.toml)
draft_prefetch_hook.py # Draft-guided expert prefetch hook for EAGLE-3
accept_reject_tracker.py # Per-expert accept/reject attribution tracker
sacr.py # SACR: Speculation-Aware Cache Replacement
elp.py # ELP: Expert Lifecycle Partitioning
dipp.py # DIPP: Draft-Informed Prioritized Preloading
pred_cache.py # PredCache: Predictive expert cache manager
briskmoe_cache.py # Unified cache facade (SACR + ELP + DIPP)
scripts/ # Benchmark and experiment scripts
tests/ # Unit tests for each module
data/ # Benchmark datasets (HumanEval, GSM8K)
docs/paper/ # ATC / SC paper sources
- Python ≥ 3.10
- NVIDIA GPU with ≥ 48 GB VRAM (tested on RTX A6000)
- CUDA 12.x
- Model weights downloaded to a local path
# Create conda environment
conda create -n briskmoe python=3.10 -y
conda activate briskmoe
# Install dependencies
pip install -r requirements.txt
# Install BriskMoE as editable package (registers the vLLM plugin)
pip install -e .python -c "from adapters.vllm_elmm_plugin import register; print('BriskMoE plugin ready')"All benchmarks use real HumanEval prompts (not random tokens) to produce realistic expert routing patterns. Results are written to results/<experiment>/ as JSON files.
BriskMoE is controlled entirely via environment variables:
| Variable | Default | Description |
|---|---|---|
VLLM_PLUGINS |
"" |
Set to elmm to activate BriskMoE |
ELMM_CACHE_GB |
8 |
GPU expert cache size in GB |
ELMM_POOL_DIRECT |
1 |
Enable Pool-Direct execution (skip scratchpad copy) |
ELMM_STALE_REMAP |
4 |
TASER stale-remap threshold |
ELMM_DIRECT_DISPATCH |
0 |
Enable direct dispatch optimization |
BRISKMOE_DIPP |
0 |
Enable DIPP (Draft-Informed Prioritized Preloading) |
BRISKMOE_SACR |
0 |
Enable SACR (Speculation-Aware Cache Replacement) |
BRISKMOE_ELP |
0 |
Enable ELP (Expert Lifecycle Partitioning) |
The core benchmark runner feeds HumanEval prompts through vllm.LLM.generate() and measures per-prompt latency.
# AR baseline (no SD, no BriskMoE)
VLLM_PLUGINS="" \
python scripts/bench_humaneval_runner.py \
--model /path/to/Qwen3-30B-A3B-Instruct-2507 \
--dataset data/humaneval_bench.jsonl \
--output-len 128 \
--num-prompts 50 \
--warmup-prompts 5 \
--gpu-memory-utilization 0.90 \
--cpu-offload-gb 30 \
--max-model-len 4096 \
--enforce-eager \
--trust-remote-code \
--dtype bfloat16 \
--output-json results/ar_vanilla/result.jsonParameters explained:
--model: Path to the HuggingFace model directory.--dataset: JSONL file where each line contains{"prompt": "..."}.--output-len 128: Generate 128 tokens per prompt.--num-prompts 50: Number of timed prompts (after warmup).--warmup-prompts 5: Prompts used for JIT warmup (not timed).--gpu-memory-utilization 0.90: Fraction of GPU memory vLLM may use.--cpu-offload-gb 30: Amount of model weights offloaded to CPU (GB).--max-model-len 4096: Maximum sequence length (KV cache budget).--enforce-eager: Disable CUDA graphs for deterministic offloading.--trust-remote-code: Required for Qwen3 model code.--dtype bfloat16: Model precision.--output-json: Path to save result JSON (TPS mean/median, latency stats).
To enable speculative decoding, add:
--speculative-config '{"method":"eagle3","model":"/path/to/speculator.eagle3","num_speculative_tokens":3}'This tells vLLM to use EAGLE-3 with K=3 draft tokens per step.
Runs 7 configurations sequentially to produce the ablation table:
| # | Config | ELMM | SACR | ELP | DIPP | Description |
|---|---|---|---|---|---|---|
| 1 | ar_vanilla |
— | — | — | — | AR baseline (no SD) |
| 2 | sd_vanilla |
✓ | — | — | — | SD + LRU cache only |
| 3 | sd_sacr |
✓ | ✓ | — | — | + Speculation-Aware Replacement |
| 4 | sd_elp |
✓ | — | ✓ | — | + Expert Lifecycle Partitioning |
| 5 | sd_dipp |
✓ | — | — | ✓ | + Draft-Informed Preloading |
| 6 | sd_sacr_elp |
✓ | ✓ | ✓ | — | SACR + ELP combined |
| 7 | sd_briskmoe |
✓ | ✓ | ✓ | ✓ | Full BriskMoE |
# Edit paths in the script header, then:
bash scripts/bench_briskmoe_humaneval.shResults are saved to results/briskmoe_humaneval/<config>/result.json.
Validates BriskMoE on GPT-OSS-120B (MXFP4 quantization, 128 experts, top-4 routing):
bash scripts/bench_gptoss_cross_arch.shRuns 3 configurations: ar_elmm, sd_elmm, sd_briskmoe. Results in results/gptoss_cross_arch/.
Offline trace-based simulation of cache replacement policies (LRU, SACR, ELP, DIPP) across multiple cache sizes. Does not require GPU — operates on pre-recorded routing traces.
python scripts/bench_elp_dipp_ablation.pyGenerates the paper's motivation figure data (working set expansion, throughput comparison):
python scripts/collect_and_plot_real_motivation.py# Run all BriskMoE unit tests
pytest tests/ -v
# Run a specific test module
pytest tests/test_dipp.py -vPaper under submission to USENIX ATC 2026. git add -A git commit -m "refactor: clean up repository for BriskMoE final framework
- Remove 189 legacy files (collectors, controllers, configs, tools, etc.)
- Update core adapters (elmm_plugin, draft_prefetch_hook, vllm_elmm_plugin)
- Add new BriskMoE modules (dipp, elp, sacr, pred_cache, briskmoe_cache)
- Add benchmark scripts and test suite
- Rewrite README with comprehensive documentation
- Update requirements.txt and pyproject.toml"