exllamav3

Serve EXL3 (ExLlamaV3 trellis) quantized models on vLLM fork runtimes — any architecture, mixed per-layer bitrates, composable with source-format non-routed weights

Run a 177B model (Qwen3.8-Flash-Next) on an 8 GB laptop GPU. Measured: 6.6 GiB VRAM, 47.8 GiB RAM, 34-35 tok/s. The n-gram table stays on disk; the experts run on CPU.

Qwen3.8-27B EXL3 (4.00 bpw) + DFlash2 speculative decoding for ExLlamaV3, validated at 262k context on a 24 GB RTX 3090

LLM inference server for ExLlamaV3 / EXL3, with OpenAI- and Anthropic-compatible APIs optimized for Agent workloads.

LLM / AI inference server for Windows + NVIDIA (EXL3/ExLlamaV3). OpenAI-compatible API, multi-GPU, Blazor admin. Ollama-like, no Docker.

Windows dual-16GB GPU experiments for Qwen3.8-Flash-Next EXL3: workload-aware MoE placement, 64K context, MTP, benchmarks and agent integrations.

Run BLACKFROST 3.8 Flash DERISKED EXL3 3.05 BPW on one 96GB RTX PRO 6000 with vLLM, ExLlamaV3 and MTP.

A llama-server-compatible HTTP front for ExLlamaV3: /props, /health, /slots and /v1/chat/completions with llama-server's timings object, so llama.cpp tooling drives an EXL3 model unchanged.

Serve Qwen3.8-Flash-Next (uncensored EXL3) on one NVIDIA DGX Spark: one Docker image, OpenAI-compatible endpoint, MTP speculative decoding, 262k context. ~80 tok/s.

Blackwell (sm120) build of Mia'a AI Lab's Qwen3.8-27B EXL3 deployment kit. Measured on one RTX 5090: 193 tok/s decode, full native 262k context in 25.4 GiB.

Run Qwen3.8-Flash-Next EXL3 (turboderp pack) on one AMD Strix Halo / Framework Desktop (gfx1151) via the exllamav3-amd runtime fork. 41 tok/s mean, 47.5 peak, MTP self-speculation. Setup encodes the five ROCm/gfx1151 traps.

FastAPI LLM pool for local and OpenAI-compatible remote inference, with multimodal input, scheduling, replicas, metrics, and admin APIs.