CNV_BitNet2B_Model

The one place Microsoft's BitNet b1.58 2B4T is served. Projects send it chat turns over HTTP instead of loading the model themselves. It has the same admin page, tokens, rate limits, metrics, settings and update checks as CNV_MiniCPM52B_Model and DSC_Laya_Model.

BitNet runs on the CPU with one of two engines (setting Engine, SERVICE_ENGINE, restart):

• transformers (default): in this process. It has no BitNet kernels, so expect the small memory footprint, not the speed the model card shows. Works everywhere, supports project models.
• bitnet.cpp: a llama-server built from Microsoft's bitnet.cpp into the Docker image only (/usr/local/bin/llama-server, SERVICE_CPP_SERVER overrides the name). It uses the i2_s CPU kernels and is much faster. It reads the GGUF weights (microsoft/BitNet-b1.58-2B-4T-gguf, about 1.2 GB): after switching, press Download on the Updates card. Project models are not supported with it (a call with project falls back to the base model). dtype has no effect. CPU threads apply by restarting the server (the next call pays the load). Roughly 5 times the tokens per second of transformers on one test machine (about 17 against 3, a few short prompts; not a benchmark).

Ollama and stock llama.cpp cannot load Microsoft's GGUF (unsupported tensor "blk.0.ffn_down.weight" size overflows); only the bitnet.cpp fork can.

API

Call	Purpose
POST /v1/chat	{"messages", "project", "temperature", "max_tokens", "stream"} → {"content", "finish_reason", "model", "fallback", "usage", "duration_ms"}.
GET /health	{"status": "ok", "model", "ready", "num_thread", "project_models"}; ready is true once the base model is loaded. No token needed.

Streaming. With "stream": true the reply is application/x-ndjson: one JSON object per line while the answer is made, so the first words arrive at once instead of after the whole answer. Lines are {"type": "content", "text"} and finally {"type": "done", ...} holding the same body a call without stream returns. A refusal before the first piece (a conversation that is too long, a full queue) is still a normal HTTP error; a failure after it is sent as {"type": "error", "status", ...} and ends the stream. A kept answer (see below) is streamed back in one piece.

Cache. The Answers kept per replica setting (SERVICE_CACHE_SIZE, default 256, 0 = off) keeps the last answers for identical requests and sends them again without generating (X-Cache: hit). Only requests with temperature 0 (the default) are kept, because a higher temperature asks for a different answer each time. Project model files are not watched (see the note on project models), so a replaced checkpoint is only noticed after a restart; new base weights empty the cache. The cache is per replica, so with N replicas the hit rate is about 1/N. stream does not change the key: a streamed answer fills the cache for plain calls and the other way round.

messages use the roles system, user and assistant. temperature 0 (the default) is greedy. max_tokens is at most 4096 and the whole conversation must fit the model's 4096-token window (422 otherwise).

Not supported. tools, schema and thinking are refused with 422, not ignored: the model has no tool-calling or thinking mode and the service has no constrained decoding. Use CNV_MiniCPM52B_Model for those.

Project models. Optional "project": "my-project" (^[a-z0-9][a-z0-9_-]{0,47}$) uses the checkpoint folder <project models folder>/my-project when it holds a complete Hugging Face checkpoint of the same architecture (config.json, weights, tokenizer). Up to Project models loaded per replica stay in memory, least recently used out first. A missing or unloadable project uses the base model and returns "fallback": true. model names the model that answered. Replacing the files of an already loaded project needs a replica restart.

Admin page

/admin/ (password SERVICE_ADMIN_PASSWORD): per-project tokens with project scope, rate limit and expiry, usage numbers, CPU threads, the Settings card (update check hours, in-flight cap, answers kept, project models), and the Updates card (Check now and Download new weights for the Hugging Face repository). New weights are downloaded once into the shared models volume; the other replicas notice within a few seconds and load them on their next request. /metrics (admin password) is Prometheus text per token. bitnet_request_seconds is a histogram of how long each /v1 call took on the replica that answers (to the first byte for a streamed reply); it starts from 0 when the replica restarts.

Run

On first run, open /admin/ → Updates → Download. Until weights are downloaded, /health reports ready: false and inference returns 503. Loading never downloads. See download steps and curl alternative.

docker compose --env-file .env.compose up -d --build        # http://<host>:29201

Without Docker: run.bat (uv, listens on 127.0.0.1:29201). The admin Download action downloads about 1.2 GB. The admin Model variant card chooses the number format this machine can hold. Each replica holds the unpacked model in RAM (about 5 GB), so SERVICE_REPLICAS multiplies memory use. See deployment.md.

Tests

uv sync
uv run pytest

Limits worth knowing

• One generation at a time per replica; extra calls wait, and more than the in-flight cap get 429.
• The rate limit is per replica.
• A generation stops after 120 s and returns what it has with finish_reason: "length".
• TORCHDYNAMO_DISABLE=1 is set by the service; unpacking the weights otherwise tries to compile C++ and fails on machines without a compiler.

Try it from the admin page

Open /admin/ and use Try it to call the public API with a service token held only in memory. Runs use normal token scopes, limits and caches; the panel shows HTTP status, latency, responses, cancellation and a curl equivalent with a token placeholder. See the web UI guide.

Chat sends messages to POST /v1/chat, with optional project, temperature, max_tokens and streaming. BitNet rejects tools, schema and thinking with 422; its form does not send those MiniCPM-only fields.

Documentation

• Purpose: input, output and what it is for
• Deployment
• Changelog
• Docker on another machine
• Docker on this PC
• Run without Docker
• Test it from the command line

Admin pages and navigation

Open /admin/ and sign in. Overview opens first; each sidebar destination has its own URL fragment and supports browser Back/Forward and direct links:

• #overview: health, model readiness, replicas, active tokens and request counts.
• #playground: service-specific request forms and streamed or media responses.
• #tokens: create, edit, expire and revoke scoped access tokens.
• #runtime: CPU threads where supported, replica information and live/restart settings.
• #updates: check versions and explicitly download weights.
• #connections: caller addresses and public endpoints.
• #models and #voices: project models and cloned voices, shown only when supported.

Collapse/Expand switches between labeled sidebar and icon rail on desktop and mobile. The non-secret sidebar preference is saved per service in this browser. Navigation preserves form drafts and active Playground requests; sign-out still clears credentials and cancels work. Page names appear once in the main header. No preview data or cross-service selector ships in the real admin UI. Existing authentication, quotas, settings precedence and explicit-download policy remain in effect.