Skip to content

About

Lightweight, single-binary, production-grade Rust LLM routing gateway & dashboard with health-aware failover, Claude Code compatibility, and MCP server.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Repository files navigation

Vibe Router

A single-binary, lightweight, production-grade Rust LLM routing gateway and dashboard with health-aware failover, rate-limit management, multi-key rotation, Claude Code compatibility, and portable zero-wear execution.

Vibe Router provides unified OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) compatibility and routes requests across any combination of local and cloud providers with data-driven fallback chains (combos), automatic retries, exponential backoff, micro-dollar cost tracking, and an embedded Web UI.


Key Features

  • Single Lightweight Binary: Compiled to a ~8.5–9.5 MB standalone executable (≈8.5 MB on x86_64 Linux, ≈9.5 MB on Windows MSVC). No Node.js runtime, no Python, no external database dependencies.
  • Tiny Memory Footprint: Reported ~9.2 MB RSS under live traffic plus the dashboard, versus ~265 MB for Node/Next.js gateways. On 1–2 GiB VPS instances, tune cache.max_memory_mb down (e.g. 16 or 32) to bound the LRU cache; the vps runtime profile already caps it at 4 MB.
  • Hardware-Aware Adaptive Runtime:
    • Disk-First VPS Mode: Minimizes RSS footprint (< 30 MB) and restricts cache to 4 MB for small $5/mo VPS instances.
    • RAM-First Portable Mode: Hydrates SQLite into host memory (:memory:) with periodic crash-safe atomic flush back to disk, providing zero flash wear on USB drives.
    • Drive-Letter Independence: Normalizes paths with {{PORTABLE_ROOT}} so changing USB drive letters on Windows never breaks the configuration.
  • Universal Client Compatibility:
    • OpenAI Compatible: Direct drop-in for OpenAI SDK, Cursor, Cline, OpenCode, and local agents at http://localhost:20128/v1.
    • Claude Code (/v1/messages): Native Anthropic protocol adapter supporting both JSON completions and real-time SSE streaming. Run vibe-router env claude-code to connect Claude CLI in seconds.
  • Native Vibe-Link Synergy Protocol (with Vibe Assembly):
    • Telemetry response headers: X-Vibe-Router-Cost-Micros, X-Vibe-Router-Latency-Ms, X-Vibe-Router-Cache, and X-Vibe-Router-Provider.
    • Inbound role tracking (X-Vibe-Role: boss | code | debug | judge | chat | hermes) with per-role token & spend accounting.
    • Judge Retry Bypass (X-Vibe-Retry: 1): Bypasses the cache and rotates to the next healthy candidate in the chain so code rejected by evaluation judges is retried on an alternative model.
    • Zero-401 Local Handshake: Synchronizes .vibe_router_token for seamless local agent authentication.
  • Multi-Key Pool Rotation, 429 Backoff & Auth-Fault Retirement:
    • Supports RoundRobin, LeastLoaded, and Weighted selection strategies across multiple API keys per provider.
    • Upstream 429 rate-limit responses trigger automatic exponential backoff cooldowns and instantly retry the next available healthy key transparently.
    • A 401/403 on a pooled key is treated as a key fault, not a request fault: that key is retired for a long auth cooldown (auth.auth_cooldown_ms, default 30 minutes, cleared on successful use or config reload) and the request rotates to the next healthy key — on both non-streaming and streaming paths. Only when every key is exhausted does the last 401/403 surface, unchanged and without a candidate-level cooldown wait.
    • Request-scoped errors (400/404/422 and validation failures) fail fast: they neither rotate keys nor retry.
  • High-Capacity In-Memory Prompt & Stream Cache:
    • Bounded LRU cache (256 MB default; profile-tuned) hashing the full response-shaping surface — model, messages, tools, tool_choice, temperature, top_p, max_tokens, stop, seed, n, penalties, response_format, logit_bias — via SHA-256 with sub-millisecond cache hits.
    • Opt-in stream caching cleanly buffers and replays completed SSE streams; cache hits are recorded as zero-cost usage rows.
  • Virtual Model Aliasing:
    • Map arbitrary model names (e.g., gpt-4o, claude-3-7-sonnet) to underlying combos or provider targets via REST, CLI, or Web UI.
  • Automatic Vibe Local Node Discovery:
    • Discovers local worker nodes on this PC and across the LAN via a UDP probe (VIBE_DISCOVER_V1 on port 20131, loopback + broadcast) with an HTTP fallback (GET http://127.0.0.1:20129/v1/models) for hosts that block UDP.
    • Responding nodes are registered as passthrough OpenAI providers (vibe-local for the canonical loopback node, vibe-local-<host> for LAN workers) and removed automatically when they stop responding.
    • Controlled by server.auto_discover_local (default true). Set VIBE_LOCAL_TOKEN in the environment to authenticate against secured nodes; the token is referenced as an env: secret and resolved at request time, never stored literally.
  • Remote Catalog Sync & Auto-Combo (daily-free):
    • Point server.remote_catalog_url at a Vibe_Radar JSON feed; the router refreshes it every server.remote_catalog_interval_secs (default 900s, 30s floor) on a bounded 15s / 2 MB budget.
    • Advertised models are merged onto existing providers without touching their configured endpoints or credentials; new catalog-only providers are SSRF-guarded (public http(s) hosts only) and registered anonymously.
    • The feed's daily_free_chain is filtered to resolvable provider/model entries and upserted as a runtime combo named daily-free, with a free alias registered when unset. Stale catalog providers and an empty chain are removed on the next pass.
  • Circuit Breakers & Health Probes:
    • Per-provider circuit breaker automatically trips to OPEN after repeated 5xx errors or timeouts, transitions to HALF_OPEN for probing, and resets to CLOSED on success.
  • Tagged Proxy Pools & Cloudflare Worker Relay:
    • Support proxy tier tags (cf-worker, datacenter, residential, direct).
    • Deterministic 64-bit FNV-1a sticky IP affinity ensures an API key consistently uses the same proxy to avoid anti-fraud bans.
    • Built-in ready-to-deploy Cloudflare Worker reverse-proxy script (vibe-router relay worker-script).
  • Interactive Terminal UI & MCP Server:
    • vibe-router tui (or top): Live terminal dashboard displaying RAM profile, spend, latencies, and circuit states.
    • vibe-router mcp: Built-in stdio Model Context Protocol (MCP) server exposing tools to external AI assistants (vibe_router_status, vibe_router_costs, vibe_router_health, vibe_router_set_alias, vibe_router_add_proxy).
  • Embedded Web Dashboard:
    • Vanilla HTML/CSS/JavaScript interface embedded via rust-embed with industrial dark/yellow branding (#F5B800), live token counters (distinguishing reported vs. estimated), USD cost & p50/p95/p99 latency metrics, combo editor, and API playground.
    • Session-key authenticated: the dashboard stores the admin key in sessionStorage and sends it as Authorization: Bearer on every /api/* call. All dynamic values are rendered with safe DOM APIs (no innerHTML interpolation) under a strict Content-Security-Policy.
  • Hardened Admin Boundary:
    • Every /api/* route requires the configured API key; state-mutating requests must carry Content-Type: application/json or X-Vibe-Admin: 1 (CSRF guard).
    • Runtime-created providers cannot use env:/file: secret references or shadow config-file provider IDs, and their base_url/proxy URLs are SSRF-guarded (loopback, link-local, and RFC1918 targets blocked unless server.allow_local_upstreams = true).
  • Unified Storage:
    • A single SQLite owner serves both the sync request path and the async admin API under one schema (providers, combos, aliases, proxies, rates, usage), so every request is recorded exactly once and priced from one persisted rate table.

Workspace Architecture

Vibe Router is structured as a modular 11-crate Rust workspace with strictly downward, acyclic dependency flow:

vibe-router/
├── Cargo.toml                      # Workspace manifest
├── ARCHITECTURE.md                 # Architecture reference
├── vibe-router.toml.example        # Documented sample configuration
├── src/main.rs                     # Minimal CLI driver entrypoint
└── crates/
    ├── vibe-router-core/           # Core traits, Model, Combo, Candidate, WireFormat, HardwareProfile
    ├── vibe-router-config/         # TOML parsing, validation, and error classification rule engine
    ├── vibe-router-health/         # Availability tracking: cooldowns, backoff levels, RPM limits, circuit breakers
    ├── vibe-router-routing/        # Fallback engine: candidate resolution, retry loop, failover driver
    ├── vibe-router-translate/      # Wire format translation (OpenAI, Anthropic /v1/messages, SSE decoders)
    ├── vibe-router-providers/      # HTTP transport, multi-key rotation pools, sticky proxies, CF relay
    ├── vibe-router-storage/        # Single SQLite owner (sync Store + async StorageBackend), WAL, portable hydration
    ├── vibe-router-cache/          # In-memory bounded LRU prompt & SSE stream cache with SHA-256 keys
    ├── vibe-router-ui/             # Embedded static Web UI assets (HTML/CSS/JS) via rust-embed
    ├── vibe-router-server/         # Axum server: /v1/*, /api/*, embedded dashboard, and Vibe-Link handlers
    └── vibe-router-cli/            # Command-line interface, TUI, and stdio MCP server

Quick Start

1. Build from Source

Ensure Rust 1.85+ is installed:

cargo build --release

The resulting binary will be at target/release/vibe-router (or target/release/vibe-router.exe on Windows).

2. Configure Providers

Copy the example configuration:

cp vibe-router.toml.example vibe-router.toml

Edit vibe-router.toml with your providers and keys:

[server]
host = "127.0.0.1"
port = 20128
api_keys = [] # Optional local auth keys

[[providers]]
id = "groq"
display_name = "Groq Cloud"
base_url = "https://api.groq.com/openai/v1/chat/completions"
auth = { scheme = "Bearer", secret = "env:GROQ_API_KEY" }
models = [
    { id = "llama-3.3-70b-versatile" }
]

[[providers]]
id = "ollama"
display_name = "Local Ollama"
base_url = "http://localhost:11434/v1/chat/completions"
auth = { secret = "" }
passthrough = true

[[combos]]
name = "smart-fallback"
description = "Cloud Groq first, local Ollama backup"
models = [
    "groq/llama-3.3-70b-versatile",
    "ollama/llama3.2"
]

Validate your configuration:

./target/release/vibe-router config validate

3. Run the Server

./target/release/vibe-router serve
┌──────────────────────────────────────────────────────────┐
│                    VIBE ROUTER v0.1.0                    │
├──────────────────────────────────────────────────────────┤
│ Endpoint:  http://127.0.0.1:20128/v1                     │
│ Claude:    http://127.0.0.1:20128/v1/messages            │
│ Dashboard: http://127.0.0.1:20128/                       │
│ Auth:      Open (no API key required)                    │
│ Profile:   workstation (RAM: 8192MB, Cores: 4)           │
│ Providers: 2                                             │
│ Storage:   vibe-router.db                                │
└──────────────────────────────────────────────────────────┘

Open http://localhost:20128 in your browser to access the dashboard.


Integration Guides

Claude Code

Run Vibe Router and print the shell environment:

./target/release/vibe-router env claude-code

Then export the variables in your shell before launching claude:

export ANTHROPIC_BASE_URL="http://127.0.0.1:20128"
export ANTHROPIC_API_KEY="vibe-router"
claude

OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:20128/v1",
    api_key="none"
)

response = client.chat.completions.create(
    model="smart-fallback",  # Target a combo, alias, or provider/model
    messages=[{"role": "user", "content": "Explain Rust borrowing in one sentence."}],
    stream=True
)

for chunk in response:
    content = chunk.choices[0].delta.content or ""
    print(content, end="", flush=True)

Model Context Protocol (MCP)

To connect Vibe Router as an MCP tool server for Claude Desktop, Cursor, or Hermes:

{
  "mcpServers": {
    "vibe-router": {
      "command": "/path/to/vibe-router",
      "args": ["mcp", "--config", "/path/to/vibe-router.toml"]
    }
  }
}

Local Discovery & Remote Catalog

Both features are configured under [server] and are off-the-shelf optional:

[server]
# Discover Vibe Local worker nodes on this PC and the LAN (UDP 20131 + HTTP 20129).
auto_discover_local = true

# Sync a Vibe_Radar model catalog and auto-build the "daily-free" combo (alias "free").
remote_catalog_url = "https://example.com/vibe-radar/catalog.json"
remote_catalog_interval_secs = 900   # 15 minutes; clamped to a 30s floor
  • VIBE_LOCAL_TOKEN (environment variable): when set, discovery attaches it as Authorization: Bearer and X-Vibe-Local-Token on the model probe, and registered nodes reference it as an env: secret resolved at request time. It is never written to disk or stored literally in the registry.

  • Catalog feed shape (Vibe_Radar):

    {
      "updated_at": "2026-11-18T00:00:00Z",
      "providers": [
        { "id": "catpub", "base_url": "https://api.catpub.example/v1/chat/completions",
          "models": [{ "id": "m1", "context": 8000 }] }
      ],
      "daily_free_chain": ["catpub/m1"]
    }

    Existing providers keep their configured endpoint and credentials — the catalog only merges new model metadata. New providers must use a public http(s) host (loopback, link-local, and RFC1918 targets are rejected). Target the auto-built chain with the model name daily-free or its alias free.


CLI Commands

Commands:
  serve      Start the OpenAI/Anthropic proxy server and dashboard (default)
  config     Inspect and validate configuration (e.g., config validate)
  providers  Inspect upstream providers and manage key pools
  models     List all addressable models, combos, and aliases
  combos     List configured combos and fallback chains
  status     Display usage and storage status
  costs      Display USD cost breakdown and latency metrics (p50/p95/p99)
  alias      Manage virtual model aliases (set, list)
  proxy      Manage tagged proxies (add, list)
  relay      Cloudflare Worker reverse-proxy relay utilities (worker-script)
  env        Print client environment variables (claude-code)
  tui        Live terminal status dashboard (alias: top)
  mcp        Model Context Protocol (MCP) stdio JSON-RPC 2.0 server
  cache      Inspect and purge in-memory prompt cache (stats, purge)

Testing & Quality

Vibe Router maintains 100% passing tests across all crates with zero warnings:

cargo test --workspace --all-features
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
cargo build --release --workspace

All integration tests are hermetic and deterministic using an in-process TCP mock upstream that tests real HTTP sockets, chunked SSE streaming, timeouts, disconnects, failover chains, rate limits, and error status classification without relying on external networks or live accounts.


License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

Lightweight, single-binary, production-grade Rust LLM routing gateway & dashboard with health-aware failover, Claude Code compatibility, and MCP server.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages