Skip to content

api-reference.md

Latest commit

 

History

History
178 lines (132 loc) · 7.25 KB

api-reference.md

File metadata and controls

178 lines (132 loc) · 7.25 KB
Raw

ExLlamaSharp API Reference

Overview of the HTTP surface. Interactive OpenAPI/Swagger is available in Development (/swagger). The Blazor UI also includes an API guide page.

Base URL default: http://localhost:14563

Authentication for protected routes: Authorization: Bearer <api_key>.

JSON for OpenAI routes uses snake_case field names where applicable.

OpenAI-compatible (/v1)

Primary contract is OpenAI Chat Completions / Completions. Clients that speak OpenAI (SDKs, Continue, Open WebUI in OpenAI mode, etc.) work against /v1/* with Authorization: Bearer <key>.

Ollama-style extras are also accepted on the same routes via an options object (and a few aliases). There is no separate Ollama HTTP API (/api/chat, /api/generate).

Method Path Description
POST /v1/chat/completions Chat completions (streaming SSE when stream: true)
POST /v1/completions Text completions
GET /v1/models List models known to the server
GET /v1/models/{id} Model detail
POST /v1/embeddings Embeddings (ONNX required; 503 if missing unless CI fallback env)
POST /v1/tokenize Encode text → token ids
POST /v1/detokenize Decode token ids → text
GET /v1/metrics Engine metrics (JSON)

Unimplemented /v1/** paths (including images/audio generation) return 501 with an OpenAI-shaped error object (not_implemented_error).

Chat supports OpenAI tools / tool_choice and response_format:

Header Values
X-ExLlamaSharp-Tools-Mode prompt_parse (system hint + post-parse) or constrained (LLGuidance JSON schema on the worker when llguidance is installed)
X-ExLlamaSharp-Structured-Output prompt_only or constrained when response_format / JSON schema is used

When LLGuidance is unavailable, tools stay in prompt-parse mode. tool_choice: required with a mock engine returns 400 tools_constrained_unavailable. Otherwise tool_choice: required returns 400 if the model does not emit valid tool_calls. Streaming does not emit incremental tool_calls deltas; tool calls are validated on the accumulated text.

Multimodal image_url requires a loaded EXL3 VLM (max 8 images per request, else 413). Text-only models return 400 vision_not_supported. Unsupported fields: logit_bias, logprobs, n > 1 → 400. A/B routing returns 409 if the chosen variant is not already loaded. Unimplemented /v1/* (e.g. /v1/images/*, /v1/audio/*) → 501 not_implemented_error. Responses include X-ExLlamaSharp-Engine: worker|mock.

Generation length & context

Field Role
max_tokens / max_completion_tokens OpenAI — max reply tokens
options.num_predict Ollama alias for reply length
(omitted) Uses Settings → Default max tokens
options.num_ctx Ollama context / KV size — must match Settings → Max batched tokens; changing it requires model reload (per-request mismatch is logged and ignored)

Priority for reply length: max_tokens → max_completion_tokens → options.num_predict → settings default.

OpenAI top-level sampling fields (temperature, top_p, top_k, min_p, penalties, seed, stop) override the same keys inside options when both are present.

Chat completions (sketch)

POST /v1/chat/completions
Authorization: Bearer sk-...
Content-Type: application/json

{
  "model": "office-assistant",
  "messages": [
    { "role": "system", "content": "You are helpful." },
    { "role": "user", "content": "Hello" }
  ],
  "temperature": 0.7,
  "max_tokens": 512,
  "stream": false,
  "options": {
    "num_predict": 512,
    "num_ctx": 8192,
    "top_k": 40,
    "repeat_penalty": 1.1
  }
}

options may include: num_predict, num_ctx, temperature, top_p, top_k, min_p, seed, stop, presence_penalty, frequency_penalty, repeat_penalty (alias for frequency_penalty).

Streaming responses use text/event-stream chunks compatible with OpenAI clients.

Ops (no API key)

Method Path Description
GET /health Component health report
GET /ready Readiness (ready, model_loaded, engine_running)
GET /metrics Prometheus exposition format

Admin (/api/v1)

Requires API key (except GET /api/v1/about, which is public for version/GPU discovery).

Settings

Method Path Description
GET /api/v1/settings Current settings
POST /api/v1/settings Replace settings
PATCH /api/v1/settings Partial update

Models & jobs

Method Path Description
GET /api/v1/models/library Curated / cached library entries
POST /api/v1/models/library Add library entry
POST /api/v1/models/load Load model into engine
POST /api/v1/models/unload Unload
POST /api/v1/models/pull Start pull job
POST /api/v1/models/quantize Start quantize job
POST /api/v1/models/import Import from path
POST /api/v1/models/alias Set alias
GET/PUT /api/v1/models/{id}/modelfile Modelfile metadata
GET /api/v1/models/jobs/{job_id} Job status
GET /api/v1/jobs List jobs
POST /api/v1/jobs/{id}/cancel Cancel job

Keys & users

Method Path Description
GET/POST /api/v1/keys List / create API keys
DELETE /api/v1/keys/{id} Revoke key
GET/POST /api/v1/users List / create users
PATCH/DELETE /api/v1/users/{id} Update / delete user

Moderation, backup, ops

Method Path Description
GET/POST /api/v1/moderation/rules List / create rules
DELETE /api/v1/moderation/rules/{id} Delete rule
GET /api/v1/about Version, runtime, engine, GPU
GET /api/v1/logs/stream Log stream (SSE-style)
POST /api/v1/backup Export backup ZIP
POST /api/v1/backup/restore Restore from ZIP
POST /api/v1/restart Request process restart

| GET/POST | /api/v1/ab | List / create A/B tests | | GET | /api/v1/ab/{id} | Get A/B test | | POST | /api/v1/ab/vote | Preview route assignment (request_id + optional preferred) | | GET/POST | /api/v1/tenants | List / create tenants | | GET | /api/v1/tenants/{id} | Get tenant | | GET/POST | /api/v1/adapters | List / register LoRA adapter metadata | | GET/DELETE | /api/v1/adapters/{id} | Get / delete adapter |

OpenAI chat/completions accept X-Ab-Test-Id or model: "ab:<guid>" to assign a variant (headers X-Ab-Variant / X-Ab-Test-Id on the response; audit stores AbTestId). GPU weights are not swapped automatically.

Errors

OpenAI routes return:

{
  "error": {
    "message": "...",
    "type": "...",
    "code": "..."
  }
}

Common HTTP codes: 400 validation, 401 auth, 404 missing, 429 rate limit, 501 not implemented, 503 unhealthy/not ready.

Related

  • admin-guide.md — keys, webhooks, multi-GPU
  • UI: /diagnostics, in-app API guide