A stateful inference engine for large language models, built for long-running, multi-user conversations on a single server. A conversation keeps its KV on the server, so a client sends only the new message of a turn.
The source code and documentation will be published in this repository. Until then, this page reports the engine's measurements against vLLM and SGLang.
- Engine: C++17 and CUDA (own kernels), OpenAI-compatible HTTP API
- Targets: NVIDIA data-center GPUs, on-premises deployment
- License: to be announced with the source release
| Item | What it shows | Section |
|---|---|---|
| Token throughput at a fixed load | generated tokens per second at 8, 32 and 64 concurrent requests | 2 |
| Multi-user workloads | requests per second and time to the first token while the number of users grows: text, image, video, tool calls, slide generation | 3 |
| Tokens sent per request | what a client sends with a server-side session and without one | 4 |
| Cost and energy | $ per 1,000 requests, $ per 1M tokens, GPU Wh per 1,000 requests, output tokens per joule | 5 |
| Billing by tokens sent | the price of the same work when input tokens are billed | 6 |
Every number is one run on one NVIDIA H200, one engine at a time, measured from 2026-11-13 to 2026-11-17. The numbers of vLLM and SGLang were recorded once and not measured again when qml-engine changed; where a qml-engine row comes from a later run, the text says so.
| GPU | one NVIDIA H200 (141 GB) |
| Engines | qml-engine, vLLM 0.32.0 (with LMCache), SGLang 0.5.22 (with HiCache and Mooncake) |
| Models | Qwen3-30B-A3B, BF16 (section 2); Qwen3-VL-30B-A3B-Instruct, BF16, the same folder for the three engines (sections 3 to 6) |
| Requests | the OpenAI chat API, streamed, the same requests to every engine |
| Context | 32,768 tokens |
Differences between the engines (read these before the results):
| qml-engine | vLLM | SGLang | |
|---|---|---|---|
| What the client sends | a session id and the new message | the whole conversation | the whole conversation |
| Reuse of earlier work | the session's KV stays on the server (and in an int8 store on disk) | prefix KV cache, LMCache (60 GB memory, 100 GB disk) | prefix KV cache, HiCache and Mooncake (100 GB) |
| Vision encoder weights | F16 | BF16 | BF16 |
| Limit of requests at once | 256 (64 where noted) | 256 | 256 |
| Other | conversation memory on (retrieval of archived turns, bge-m3) in sections 3 to 6 | --gpu-memory-utilization 0.85 |
--mem-fraction-static 0.85 |
qml-engine ran in the mode that does not keep a result independent of the batch a request runs in (its default keeps it); vLLM and SGLang have no such guarantee either.
Qwen3-30B-A3B, BF16. Every request is a new conversation: a prompt of about 785 tokens, an answer of 256 tokens, temperature 0. Eight requests warm up for 20 s, then 60 s are measured. Generated tokens per second:
| Concurrent requests | qml-engine | SGLang | qml-engine / SGLang |
|---|---|---|---|
| 8 | 1,045 | 997 | 1.05 |
| 32 | 2,612 | 2,484 | 1.05 |
| 64 | 3,945 | about 3,678 | 1.07 |
The SGLang numbers are from 2026-11-13 with its default settings. The difference between two runs of qml-engine at 8 requests is about 1.7%. Conversation memory was off in this test.
A closed loop: every user sends a request, reads the answer, waits 3 s and sends the next one. The number of users grows in stages of 100 s; the last 80 s of a stage are counted. A stage passes with no error and a time to the first token of at most 2.0 s at the 95th percentile.
| Users | qml-engine req/s | first token p50 / p95 (s) | vLLM req/s | p50 / p95 | SGLang req/s | p50 / p95 |
|---|---|---|---|---|---|---|
| 16 | 3.10 | 0.03 / 0.06 | 2.69 | 0.07 / 0.21 | 2.64 | 0.10 / 0.32 |
| 32 | 5.12 | 0.05 / 0.07 | 4.65 | 0.10 / 0.33 | 4.32 | 0.21 / 0.40 |
| 64 | 8.27 | 0.06 / 1.71 | 7.43 | 0.15 / 0.54 | 7.75 | 0.28 / 0.47 |
| 96 | 8.30 | 0.70 / 1.40 | 9.70 | 0.24 / 1.15 | 10.31 | 0.34 / 0.60 |
| 128 | 13.80 | 0.08 / 0.16 | 10.91 | 0.82 / 2.42 | 12.93 | 0.33 / 0.68 |
| 192 | 16.90 | 0.10 / 0.33 | 17.54 | 0.50 / 1.09 | ||
| 256 | 17.41 | 0.14 / 1.88 | 20.72 | 0.49 / 2.86 |
The qml-engine rows for 16 to 96 users were recorded with a limit of 64 requests at once, which holds them back from 64 users on; they were not measured again with the limit of 256. After a later change, the stages of 192 and 256 users run alone gave 18.78 and 21.04 req/s (1.07 and 1.02 times SGLang; first token p95 0.44 s at 256 users); a stage run alone and a stage of a growing run are not the same condition. Largest number of users that passes: qml-engine 256, SGLang 192, vLLM 96.
| Users | qml-engine req/s | first token p50 / p95 (s) | vLLM req/s | p50 / p95 | SGLang req/s | p50 / p95 |
|---|---|---|---|---|---|---|
| 16 | 4.85 | 0.03 / 0.11 | 4.67 | 0.10 / 0.29 | 4.50 | 0.15 / 0.30 |
| 32 | 9.46 | 0.04 / 0.12 | 8.63 | 0.20 / 0.63 | 8.60 | 0.23 / 0.43 |
| 64 | 17.99 | 0.06 / 0.19 | 14.35 | 0.50 / 1.28 | 16.43 | 0.33 / 0.43 |
| 96 | 25.56 | 0.12 / 0.35 | 17.38 | 0.96 / 1.61 | 23.58 | 0.34 / 0.55 |
| 128 | 30.94 | 0.27 / 0.98 | 19.40 | 1.52 / 2.24 | 30.11 | 0.40 / 0.68 |
qml-engine has the highest throughput at every stage (1.03 to 1.10 times SGLang). Its first token is the shortest at the median; at 128 users SGLang is shorter at the 95th percentile (0.68 against 0.98 s). Stages above 128 users were not measured.
| Users | qml-engine req/s | first token p50 / p95 (s) | vLLM req/s | p50 / p95 | SGLang req/s | p50 / p95 |
|---|---|---|---|---|---|---|
| 16 | 4.81 | 0.03 / 0.32 | 4.75 | 0.08 / 0.36 | 3.05 | 2.80 / 3.04 |
| 32 | 9.49 | 0.05 / 0.29 | 9.22 | 0.08 / 0.45 | ||
| 64 | 15.10 | 0.31 / 0.85 | 14.87 | 0.29 / 2.01 | ||
| 96 | 21.60 | 0.65 / 1.25 |
The qml-engine rows are with the prefix of a conversation shared between the users of a tenant (the test repeats 512 videos among its users, and vLLM's prefix cache reuses them in the same way). Without that option qml-engine gave 4.81, 8.52 and 12.80 req/s (0.86 to 1.01 times vLLM). SGLang did not pass the first stage: its time to the first token was 2.80 s at 16 users (0.35 s with one user), with no error.
| Workload | 16 | 32 | 64 | 96 | 128 | 192 | 256 | |
|---|---|---|---|---|---|---|---|---|
| Tool call | qml-engine req/s | 9.76 | 16.59 | 28.20 | 40.34 | 50.98 | 64.47 | 72.32 |
| vLLM req/s | 8.90 | 15.65 | 26.67 | 35.02 | 41.04 | 51.59 | 59.17 | |
| first content p95, qml-engine / vLLM (s) | 0.03 / 0.11 | 0.04 / 0.15 | 0.05 / 0.18 | 0.06 / 0.30 | 0.07 / 0.44 | 0.09 / 0.63 | 0.12 / 0.96 | |
| Slide | qml-engine req/s | 3.02 | 4.97 | 7.87 | 10.52 | 12.24 | 15.31 | 14.36 |
| vLLM req/s | 2.60 | 4.09 | 6.08 | 8.05 | 9.34 | 10.84 | 12.10 | |
| first content p95, qml-engine / vLLM (s) | 0.05 / 0.20 | 0.05 / 0.22 | 0.06 / 0.23 | 0.07 / 0.28 | 0.09 / 0.39 | 0.09 / 0.74 | 0.19 / 1.26 |
No request failed. In the slide workload the model's tool arguments were wrong or the tool was not called in 563 requests on qml-engine and 566 on vLLM over the whole run. SGLang has no load record for these two workloads.
Averages at 128 users (text, image) and 64 users (video):
| Workload | qml-engine | vLLM (of which cache hits) | SGLang (of which cache hits) |
|---|---|---|---|
| Text | 259 | 1,461 (1,226) | 1,449 (1,104) |
| Image | 252 | 757 (537) | 756 (528) |
| Video | 988 | 1,990 (about 1,840) | 1,988 (1,617) |
With a session on the server the client does not send the conversation again, so nothing has to be found in a cache.
Cost per hour = $6.06 (the GPU) + $0.12 per kWh x the GPU's power. Cost of 1,000 requests = cost per hour / 3600 /
(requests per second) x 1000. Cost of 1M tokens = the same cost over the tokens of the whole conversation (input) and
the output. Power is the GPU board's draw read every 2 s (nvidia-smi); the host's power is not included.
| Workload, users | Engine | req/s | GPU W | $ / 1,000 requests | $ / 1M tokens | GPU Wh / 1,000 requests | Output tokens / J |
|---|---|---|---|---|---|---|---|
| Text, 128 | qml-engine | 13.80 | 666 | 0.124 | 0.072 | 13.4 | 5.40 |
| SGLang | 12.93 | 602 | 0.132 | 0.077 | 12.9 | 5.60 | |
| vLLM | 10.91 | 488 | 0.156 | 0.090 | 12.4 | 5.82 | |
| Text, 192 | qml-engine | 18.79 | 681 | 0.091 | 0.053 | 10.1 | 7.17 |
| SGLang | 17.54 | 648 | 0.097 | 0.057 | 10.3 | 7.08 | |
| Text, 256 | qml-engine | 21.04 | 688 | 0.081 | 0.047 | 9.1 | 8.04 |
| SGLang | 20.72 | 666 | 0.082 | 0.048 | 8.9 | 8.10 | |
| Image, 128 | qml-engine | 32.56 | 641 | 0.052 | 0.065 | 5.5 | 2.22 |
| SGLang | 30.11 | 520 | 0.057 | 0.071 | 4.8 | 2.48 | |
| vLLM | 19.40 | 299 | 0.087 | 0.109 | 4.3 | 2.81 | |
| Video, 64 | qml-engine | 15.09 | 305 | 0.112 | 0.055 | 5.6 | 1.86 |
| vLLM | 14.87 | 301 | 0.114 | 0.056 | 5.6 | 1.83 | |
| Tool call, 64 | qml-engine | 27.99 | 594 | 0.061 | 0.062 | 5.9 | 2.72 |
| vLLM | 26.67 | 523 | 0.064 | 0.065 | 5.5 | 3.05 | |
| Slide, 64 | qml-engine | 7.72 | 677 | 0.221 | 0.132 | 24.4 | 3.80 |
| vLLM | 6.08 | 545 | 0.280 | 0.166 | 24.9 | 3.78 |
- The cost of a request follows the throughput, because the GPU's price is 99% of the cost per hour: qml-engine is the lowest in every row (1 to 7% below SGLang, 1 to 40% below vLLM).
- The GPU's energy per request does not follow it. qml-engine uses as much or less in the rows text 192, video 64 and slide 64, and 2 to 28% more in text 128 and 256, image 128 and tool call 64. The difference is the cost of the engine's conversation memory, which ran in every qml-engine row and which the other engines do not have: TAWT (the front of a long conversation is archived instead of dropped) and RIG (every turn is indexed, and the archived turns a question needs are found by BM25 and by bge-m3 semantic search on the GPU and put back). In a separate run of 32 users on the same GPU, qml-engine drew 583 W with this memory on and 526 W with it off; SGLang drew 504 W.
- A low power reading is not always an efficient engine: where a GPU waits for work on the host (vLLM with images: 46% GPU utilization), the reading is low and so is the throughput.
- The qml-engine rows text 192 and 256, image 128 and video 64 are stages run alone after later changes; the other rows are stages of the growing runs of section 3.
The same work priced by tokens, at $0.12 per 1M input tokens and $0.50 per 1M output tokens, in $ per 1,000 requests. The discounted columns apply a cached-input price of 0.25 or 0.1 times the input price to the other engines' cache hits.
| Workload, users | qml-engine | Whole conversation, list price | Cached input at 0.25 | Cached input at 0.1 |
|---|---|---|---|---|
| Text, 192 | 0.164 | 0.306 | 0.201 | 0.181 |
| Image, 128 | 0.053 | 0.112 | 0.065 | 0.055 |
| Tool call, 32 | 0.050 | 0.139 | 0.067 | 0.053 |
| Slide, 32 | 0.201 | 0.328 | 0.245 | 0.228 |
Video is left out of this table: its test repeats videos between users, and how much of that reuse a price list discounts differs by provider.
- Stages above 128 users with images; text at 16 to 96 users with qml-engine's limit of 256 requests at once.
- Tool calls and slide generation on SGLang.
- The host's power. The energy of the conversation memory by part (archive, index, search) and per workload; the decision model that picks among the retrieved turns is off by default and did not run in these tests.
- More than one run per number.
The measurements above use Qwen3-30B-A3B and Qwen3-VL-30B-A3B-Instruct (mixture of experts, BF16). The engine also serves Qwen3-32B, Qwen3-Coder-30B-A3B, Llama-3.1-8B, gpt-oss-20b and Gemma 4 31B; a table of the supported models and weight formats comes with the source release.
KPST (Korea Platform Service Technology Co., Inc.)