Skip to content

About

Stateful LLM inference engine for long-running, multi-user conversations on one server: session KV restore, turn archive and recall, own CUDA kernels for Hopper and Blackwell, OpenAI-compatible API.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

QML Engine

A stateful inference engine for large language models, built for long-running, multi-user conversations on a single server. A conversation keeps its KV on the server, so a client sends only the new message of a turn.

Status

The source code and documentation will be published in this repository. Until then, this page reports the engine's measurements against vLLM and SGLang.

  • Engine: C++17 and CUDA (own kernels), OpenAI-compatible HTTP API
  • Targets: NVIDIA data-center GPUs, on-premises deployment
  • License: to be announced with the source release

What is measured

Item What it shows Section
Token throughput at a fixed load generated tokens per second at 8, 32 and 64 concurrent requests 2
Multi-user workloads requests per second and time to the first token while the number of users grows: text, image, video, tool calls, slide generation 3
Tokens sent per request what a client sends with a server-side session and without one 4
Cost and energy $ per 1,000 requests, $ per 1M tokens, GPU Wh per 1,000 requests, output tokens per joule 5
Billing by tokens sent the price of the same work when input tokens are billed 6

Every number is one run on one NVIDIA H200, one engine at a time, measured from 2026-11-13 to 2026-11-17. The numbers of vLLM and SGLang were recorded once and not measured again when qml-engine changed; where a qml-engine row comes from a later run, the text says so.

1. Setup

GPU one NVIDIA H200 (141 GB)
Engines qml-engine, vLLM 0.32.0 (with LMCache), SGLang 0.5.22 (with HiCache and Mooncake)
Models Qwen3-30B-A3B, BF16 (section 2); Qwen3-VL-30B-A3B-Instruct, BF16, the same folder for the three engines (sections 3 to 6)
Requests the OpenAI chat API, streamed, the same requests to every engine
Context 32,768 tokens

Differences between the engines (read these before the results):

qml-engine vLLM SGLang
What the client sends a session id and the new message the whole conversation the whole conversation
Reuse of earlier work the session's KV stays on the server (and in an int8 store on disk) prefix KV cache, LMCache (60 GB memory, 100 GB disk) prefix KV cache, HiCache and Mooncake (100 GB)
Vision encoder weights F16 BF16 BF16
Limit of requests at once 256 (64 where noted) 256 256
Other conversation memory on (retrieval of archived turns, bge-m3) in sections 3 to 6 --gpu-memory-utilization 0.85 --mem-fraction-static 0.85

qml-engine ran in the mode that does not keep a result independent of the batch a request runs in (its default keeps it); vLLM and SGLang have no such guarantee either.

2. Token throughput at a fixed load

Qwen3-30B-A3B, BF16. Every request is a new conversation: a prompt of about 785 tokens, an answer of 256 tokens, temperature 0. Eight requests warm up for 20 s, then 60 s are measured. Generated tokens per second:

Concurrent requests qml-engine SGLang qml-engine / SGLang
8 1,045 997 1.05
32 2,612 2,484 1.05
64 3,945 about 3,678 1.07

The SGLang numbers are from 2026-11-13 with its default settings. The difference between two runs of qml-engine at 8 requests is about 1.7%. Conversation memory was off in this test.

3. Multi-user workloads

A closed loop: every user sends a request, reads the answer, waits 3 s and sends the next one. The number of users grows in stages of 100 s; the last 80 s of a stage are counted. A stage passes with no error and a time to the first token of at most 2.0 s at the 95th percentile.

Text (conversations of several turns)

Users qml-engine req/s first token p50 / p95 (s) vLLM req/s p50 / p95 SGLang req/s p50 / p95
16 3.10 0.03 / 0.06 2.69 0.07 / 0.21 2.64 0.10 / 0.32
32 5.12 0.05 / 0.07 4.65 0.10 / 0.33 4.32 0.21 / 0.40
64 8.27 0.06 / 1.71 7.43 0.15 / 0.54 7.75 0.28 / 0.47
96 8.30 0.70 / 1.40 9.70 0.24 / 1.15 10.31 0.34 / 0.60
128 13.80 0.08 / 0.16 10.91 0.82 / 2.42 12.93 0.33 / 0.68
192 16.90 0.10 / 0.33 17.54 0.50 / 1.09
256 17.41 0.14 / 1.88 20.72 0.49 / 2.86

The qml-engine rows for 16 to 96 users were recorded with a limit of 64 requests at once, which holds them back from 64 users on; they were not measured again with the limit of 256. After a later change, the stages of 192 and 256 users run alone gave 18.78 and 21.04 req/s (1.07 and 1.02 times SGLang; first token p95 0.44 s at 256 users); a stage run alone and a stage of a growing run are not the same condition. Largest number of users that passes: qml-engine 256, SGLang 192, vLLM 96.

Image (one image and a question)

Users qml-engine req/s first token p50 / p95 (s) vLLM req/s p50 / p95 SGLang req/s p50 / p95
16 4.85 0.03 / 0.11 4.67 0.10 / 0.29 4.50 0.15 / 0.30
32 9.46 0.04 / 0.12 8.63 0.20 / 0.63 8.60 0.23 / 0.43
64 17.99 0.06 / 0.19 14.35 0.50 / 1.28 16.43 0.33 / 0.43
96 25.56 0.12 / 0.35 17.38 0.96 / 1.61 23.58 0.34 / 0.55
128 30.94 0.27 / 0.98 19.40 1.52 / 2.24 30.11 0.40 / 0.68

qml-engine has the highest throughput at every stage (1.03 to 1.10 times SGLang). Its first token is the shortest at the median; at 128 users SGLang is shorter at the 95th percentile (0.68 against 0.98 s). Stages above 128 users were not measured.

Video (16 frames, 1,951 tokens, and a question)

Users qml-engine req/s first token p50 / p95 (s) vLLM req/s p50 / p95 SGLang req/s p50 / p95
16 4.81 0.03 / 0.32 4.75 0.08 / 0.36 3.05 2.80 / 3.04
32 9.49 0.05 / 0.29 9.22 0.08 / 0.45
64 15.10 0.31 / 0.85 14.87 0.29 / 2.01
96 21.60 0.65 / 1.25

The qml-engine rows are with the prefix of a conversation shared between the users of a tenant (the test repeats 512 videos among its users, and vLLM's prefix cache reuses them in the same way). Without that option qml-engine gave 4.81, 8.52 and 12.80 req/s (0.86 to 1.01 times vLLM). SGLang did not pass the first stage: its time to the first token was 2.80 s at 16 users (0.35 s with one user), with no error.

Tool calls and slide generation (qml-engine and vLLM)

Workload 16 32 64 96 128 192 256
Tool call qml-engine req/s 9.76 16.59 28.20 40.34 50.98 64.47 72.32
vLLM req/s 8.90 15.65 26.67 35.02 41.04 51.59 59.17
first content p95, qml-engine / vLLM (s) 0.03 / 0.11 0.04 / 0.15 0.05 / 0.18 0.06 / 0.30 0.07 / 0.44 0.09 / 0.63 0.12 / 0.96
Slide qml-engine req/s 3.02 4.97 7.87 10.52 12.24 15.31 14.36
vLLM req/s 2.60 4.09 6.08 8.05 9.34 10.84 12.10
first content p95, qml-engine / vLLM (s) 0.05 / 0.20 0.05 / 0.22 0.06 / 0.23 0.07 / 0.28 0.09 / 0.39 0.09 / 0.74 0.19 / 1.26

No request failed. In the slide workload the model's tool arguments were wrong or the tool was not called in 563 requests on qml-engine and 566 on vLLM over the whole run. SGLang has no load record for these two workloads.

4. Tokens sent per request

Averages at 128 users (text, image) and 64 users (video):

Workload qml-engine vLLM (of which cache hits) SGLang (of which cache hits)
Text 259 1,461 (1,226) 1,449 (1,104)
Image 252 757 (537) 756 (528)
Video 988 1,990 (about 1,840) 1,988 (1,617)

With a session on the server the client does not send the conversation again, so nothing has to be found in a cache.

5. Cost and energy

Cost per hour = $6.06 (the GPU) + $0.12 per kWh x the GPU's power. Cost of 1,000 requests = cost per hour / 3600 / (requests per second) x 1000. Cost of 1M tokens = the same cost over the tokens of the whole conversation (input) and the output. Power is the GPU board's draw read every 2 s (nvidia-smi); the host's power is not included.

Workload, users Engine req/s GPU W $ / 1,000 requests $ / 1M tokens GPU Wh / 1,000 requests Output tokens / J
Text, 128 qml-engine 13.80 666 0.124 0.072 13.4 5.40
SGLang 12.93 602 0.132 0.077 12.9 5.60
vLLM 10.91 488 0.156 0.090 12.4 5.82
Text, 192 qml-engine 18.79 681 0.091 0.053 10.1 7.17
SGLang 17.54 648 0.097 0.057 10.3 7.08
Text, 256 qml-engine 21.04 688 0.081 0.047 9.1 8.04
SGLang 20.72 666 0.082 0.048 8.9 8.10
Image, 128 qml-engine 32.56 641 0.052 0.065 5.5 2.22
SGLang 30.11 520 0.057 0.071 4.8 2.48
vLLM 19.40 299 0.087 0.109 4.3 2.81
Video, 64 qml-engine 15.09 305 0.112 0.055 5.6 1.86
vLLM 14.87 301 0.114 0.056 5.6 1.83
Tool call, 64 qml-engine 27.99 594 0.061 0.062 5.9 2.72
vLLM 26.67 523 0.064 0.065 5.5 3.05
Slide, 64 qml-engine 7.72 677 0.221 0.132 24.4 3.80
vLLM 6.08 545 0.280 0.166 24.9 3.78
  • The cost of a request follows the throughput, because the GPU's price is 99% of the cost per hour: qml-engine is the lowest in every row (1 to 7% below SGLang, 1 to 40% below vLLM).
  • The GPU's energy per request does not follow it. qml-engine uses as much or less in the rows text 192, video 64 and slide 64, and 2 to 28% more in text 128 and 256, image 128 and tool call 64. The difference is the cost of the engine's conversation memory, which ran in every qml-engine row and which the other engines do not have: TAWT (the front of a long conversation is archived instead of dropped) and RIG (every turn is indexed, and the archived turns a question needs are found by BM25 and by bge-m3 semantic search on the GPU and put back). In a separate run of 32 users on the same GPU, qml-engine drew 583 W with this memory on and 526 W with it off; SGLang drew 504 W.
  • A low power reading is not always an efficient engine: where a GPU waits for work on the host (vLLM with images: 46% GPU utilization), the reading is low and so is the throughput.
  • The qml-engine rows text 192 and 256, image 128 and video 64 are stages run alone after later changes; the other rows are stages of the growing runs of section 3.

6. Billing by tokens sent

The same work priced by tokens, at $0.12 per 1M input tokens and $0.50 per 1M output tokens, in $ per 1,000 requests. The discounted columns apply a cached-input price of 0.25 or 0.1 times the input price to the other engines' cache hits.

Workload, users qml-engine Whole conversation, list price Cached input at 0.25 Cached input at 0.1
Text, 192 0.164 0.306 0.201 0.181
Image, 128 0.053 0.112 0.065 0.055
Tool call, 32 0.050 0.139 0.067 0.053
Slide, 32 0.201 0.328 0.245 0.228

Video is left out of this table: its test repeats videos between users, and how much of that reuse a price list discounts differs by provider.

Not measured

  • Stages above 128 users with images; text at 16 to 96 users with qml-engine's limit of 256 requests at once.
  • Tool calls and slide generation on SGLang.
  • The host's power. The energy of the conversation memory by part (archive, index, search) and per workload; the decision model that picks among the retrieved turns is off by default and did not run in these tests.
  • More than one run per number.

Models

The measurements above use Qwen3-30B-A3B and Qwen3-VL-30B-A3B-Instruct (mixture of experts, BF16). The engine also serves Qwen3-32B, Qwen3-Coder-30B-A3B, Llama-3.1-8B, gpt-oss-20b and Gemma 4 31B; a table of the supported models and weight formats comes with the source release.

Contact

KPST (Korea Platform Service Technology Co., Inc.)

About

Stateful LLM inference engine for long-running, multi-user conversations on one server: session KV restore, turn archive and recall, own CUDA kernels for Hopper and Blackwell, OpenAI-compatible API.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors