CachyLlama

A self-hosted, cache-aware LLM inference stack:

• CachyLLama — Vulkan-accelerated OpenAI-compatible server (fork of llama.cpp with CachyOS patches)
• Harrier — separate embedding microservice (GGUF embedding model)
• Redis Stack — vector store with RediSearch / RedisJSON, used by LiteLLM for the semantic cache
• LiteLLM — unified gateway exposing both models behind an OpenAI-compatible API, with Redis-backed semantic caching

Bring your own GGUF files; the stack does not download models.

Quick start

# 1. Place GGUF files in ./data/models/
mkdir -p data/models
# (copy or symlink your LLM and embedder GGUF files here)

# 2. Configure
cp .env.example .env
# edit .env so LLM_MODEL_FILE and EMBED_MODEL_FILE match the GGUF filenames
# you placed in step 1. Replace REDIS_PASSWORD and LITELLM_MASTER_KEY.

# 3. Build and run
docker compose up --build

The LiteLLM gateway listens on http://localhost:4000.

Models

Two models are required:

Role	Default file	Size	Notes
Text generation	Ornith-1.5-35B-A3B-CRACK-Q4_K_M.gguf	21.7 GB	Qwen-3.5 hybrid GatedDeltaNet + attention MoE (35 B total / 3 B active, 256 experts / 8 routed). CRACK-abliterated (uncensored). Reasoning ON by default — disable via LLM_CHAT_TEMPLATE_KWARGS={"enable_thinking": false}. Publisher sampling: temp=1.0, top_p=0.95, top_k=20.
Embedding	harrier-oss-v1-0.6B-Q5_K_M.gguf	424 MiB	Microsoft Harrier-OSS-v1-0.6B. 1024-dim, 32K context, last-token pooling. Q5_K_M is the model card's recommended sweet spot for retrieval workloads.

After changing embedders, delete the existing RediSearch index if the new model has a different embedding dimension (otherwise vector dimension mismatches silently break the cache):

docker exec -it redis-vector-store redis-cli -a "$REDIS_PASSWORD" \
  FT.DROPINDEX litellm_semantic_cache_index

The current defaults (Harrier, Q5_K_M) preserve the 1024-dim embedding shape, so no migration is required for the recommended setup.

VRAM budget for the default LLM

On a 24 GB GPU:

• Q4_K_M (21.7 GB) — recommended. Leaves ~2-3 GB for KV cache at 8K context.
• Q3_K_M (17.2 GB) — drop to this if you want more KV headroom for longer context or partial offload tolerance.
• Q5_K_M and up — exceeds the 24 GB VRAM budget before KV cache.

Sampling and chat template

Ornith uses llama.cpp's Jinja renderer (--jinja) for its chat template. The default flags set in docker-compose.yml are:

--temp 1.0 --top-p 0.95 --top-k 20

These map to LLM_TEMP, LLM_TOP_P, LLM_TOP_K in .env.

Reasoning mode is gated by LLM_CHAT_TEMPLATE_KWARGS. The default {"enable_thinking": false} gives direct answers; flip to {"enable_thinking": true} to enable chain-of-thought reasoning (slower, longer responses).

API

The LiteLLM gateway exposes an OpenAI-compatible API on port 4000. Authenticate with LITELLM_MASTER_KEY.

Chat completion (text generation)

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "custom-llm",
    "messages": [{"role":"user","content":"hello"}]
  }'

Embeddings

curl -s http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"custom-embed","input":"hello"}'

The embedder used for generation caching and the one you query through /v1/embeddings are the same EMBED_MODEL_NAME model.

Caching

Two-tier caching:

1. Semantic cache (LiteLLM → Redis Stack). Prompts are embedded by the embedder microservice; if the cosine similarity to any cached prompt exceeds SEMANTIC_SIMILARITY_THRESHOLD (default 0.85), the cached response is returned without hitting CachyLLama. Entries expire after SEMANTIC_CACHE_TTL seconds (default 86400 = 24h).
2. SSD KV cache (CachyLLama). Long-context KV state is spilled to ./data/ssd_cache/ so context beyond VRAM stays warm. Tunable via LLM_CACHE_SSD_* and LLM_CACHE_RAM env vars.

Inspect cached prompts:

docker exec -it redis-vector-store redis-cli -a "$REDIS_PASSWORD" \
  FT.SEARCH litellm_semantic_cache_index "*"

Verification

After docker compose ps shows all services healthy:

# 1. Health endpoints
curl -fsS http://localhost:8080/health        # CachyLLama
curl -fsS http://localhost:8081/health        # Embedder
curl -fsS http://localhost:4000/health/liveliness  # LiteLLM

# 2. Authenticated Redis access
docker exec -it redis-vector-store redis-cli -a "$REDIS_PASSWORD" ping

Configuration reference

See .env.example for the full list of environment variables.

Upgrading

• LiteLLM: bump the FROM tag in Dockerfile.litellm and re-run the verification step in the file's header comment to align the redisvl pin.
• CachyLLama: bump CACHYLLAMA_COMMIT in .env (or override at build time with --build-arg CACHYLLAMA_COMMIT=<sha>). The Dockerfile ARG default can lag behind.
• Embedder: only change EMBED_MODEL_FILE. If the new model has a different embedding dim, drop the RediSearch index (see above).
• LLM: change LLM_MODEL_FILE, LLM_CONTEXT_SIZE, and the GPU-layer flag. On smaller GPUs, drop to a lower quant (Q3_K_M or below) and reduce LLM_CONTEXT_SIZE.

CI / images

The GitHub Actions workflow at .github/workflows/docker-build.yml builds and publishes two images to GHCR on every push/tag and weekly:

• ghcr.io/<repo>/cachyllama-vulkan — the Vulkan inference engine.
• ghcr.io/<repo>/cachyllama-litellm — the LiteLLM gateway with redisvl pre-installed.

Both images are linted (hadolint), smoke-tested (llama-server --help), vulnerability-scanned (Trivy, fail on CRITICAL), and pushed with SLSA-style provenance attestation.

Known limitations

• Cosine similarity threshold tuning: 0.85 is a reasonable default for general chat; bump to 0.92+ for technical/code-completion queries where false positives are costly.
• Multi-turn cache boundary: LiteLLM caches at the prompt level; long multi-turn conversations with shifting context can produce low-similarity lookups and cache misses.
• SSD endurance: The SSD KV cache writes on every checkpoint (LLM_CACHE_SSD_CHECKPOINTS). Use a consumer-grade SSD only for dev; for production, prefer an enterprise or Optane-class drive, or shrink LLM_CACHE_SSD_WARM_WINDOW to reduce write amplification.
• Embedder swap: Changing EMBED_MODEL_FILE requires dropping the existing RediSearch index (vector dimensions must match).