fix(engine): reserve the pool's dummy page in the --moe-cache-auto KV floor

[dejay2]

What

One-line fix to the boot-time --moe-cache-auto budget split, plus the
--kv-reserve-tokens help text that documents it:

kv_reserve_pages = div_ceil(kv_reserve_tokens, page_size) + 1

Why

plan_cache_budget reasons in usable pages, but create_kv_pool(config, num_pages, ...) allocates num_pages + 1: every pool family keeps page 0 as an
unreachable dummy/sentinel.

The plan prices the KV half as num_pages * cache_per_page and then lets experts
greedily take everything the reserve does not protect (the fill is deliberately
headroom-free; see the total <= budget_bytes assert immediately below it). Because
the priced count is the usable one, the plan under-counted the KV pool by exactly one
page's bytes, and that page came out of memory already promised to expert slots. On a
model with a large cache_per_page that is not a rounding error, and the failure mode
is precisely the one --moe-cache-auto exists to prevent: an arithmetic plan that
claims to fit, followed by a CUDA OOM in the KV allocation at boot.

The same off-by-one made --kv-reserve-tokens N under-deliver. It is documented as a
KV-cache token floor, but ceil(N / page_size) usable pages minus the dummy is less
than N tokens of reachable capacity. Hence also the help-text change.

Provenance

This hunk was extracted from a larger fork commit whose message carries no recorded
symptom. The reasoning above is reconstructed from the code paths named, not from a
logged OOM; the new test shows the arithmetic changes exactly as described.

How it was tested

Windows 11, RTX 5090.

python -m pytest -q tests/engine tests/server

• before this commit: 643 passed, 2 failed
• after this commit: 644 passed, 2 failed (1 new test)

The 2 failures are pre-existing on this box and unrelated to this change:
tests/engine/test_cache_budget.py::test_adjust_config_resolves_num_tokens_generic and
::test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend both
require flashinfer, which is not installed here.

New test, tests/engine/test_cache_budget.py::test_resolve_auto_reserves_usable_tokens_beyond_the_dummy_page:
pins that --kv-reserve-tokens 256 at page_size=64 yields pages - 1 usable pages
covering at least 256 tokens. On origin/main this returns (size=9, pages=4); with the
fix it returns (size=8, pages=14), so the test genuinely discriminates.

What is NOT included

The originating fork commit also clamps the auto-sized KV pool to one full-length
context. That bounds aggregate KV under --max-running-requests > 1 (the default is 4)
and trades cross-request cache residency for VRAM, which is a policy choice rather than
a bug fix, so it is deliberately left out of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RG8BXfsSZi1nh4wMZnhJQK

[gdevenyi]

Tested on a Qwen3.8-Flash-Next deployment (--moe-cache-auto default, --num-tokens 262144): boots, and the plan moves exactly as described.

	main 86214a9	+ this PR
resolved plan	moe_cache_size=8984, num_pages=4096	moe_cache_size=8983, num_pages=4097
free after init	2.59 GiB	2.59 GiB
single-stream / 8-conc decode	58.6 / 132.8 tok/s	57.8 / 130.6 tok/s (noise)

One page here is 64 tokens x 12 QSA layers (~1.5 MB), so the one-slot trade is invisible; the fix is right regardless.

Related bug in the same resolver, which you may want to fold in since it is the same "plan vs. what the pool allocates" class: with --num-tokens N (num_page_override), Engine._resolve_auto_moe_cache_size still plans the KV half against kv_reserve_tokens only, so the expert fill takes the budget and the pool then allocates N pages on top of it. At N=262144 on this model that is a guaranteed OOM in LinearStatePool/KV allocation after the experts are resident. Fix I am running with: clamp the reserve to the override before planning,

if config.num_page_override is not None:
    kv_reserve_tokens = max(kv_reserve_tokens, config.num_page_override * page_tokens)

plus a regression test in tests/engine/test_cache_budget.py (fails on main, passes with the clamp). Both of the runs above carry that fix; happy to open it as a separate PR or hand it to you.

Measured on a 2x RTX 6000 Ada (48 GiB, sm_89, PCIe Gen4, no NVLink) / 2x Xeon Gold 6526Y / 503 GiB box, CUDA 13.3, torch 2.11+cu130, sgl_kernel 0.4.5, model RadixArk/Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, NVFP4 experts + bf16 dense). The PR cherry-picks cleanly onto main 86214a9; both runs also carry a local fix so --moe-cache-auto honours --num-tokens (it OOMs otherwise at this KV size). One GPU, one run each, same flags:

ft serve --model <RadixArk snapshot> --moe-backend offload --ple-backend pinned --num-tokens 262144 \
  --memory-ratio 0.94 --moe-prefill-hit-d2d --max-running-requests 8 --cuda-graph-max-bs 8

Single-stream = median of three 64-vs-256-token completion pairs, aggregate = 8 concurrent 256-token completions, TTFT on a ~1k-token prompt. Run-to-run spread of the baseline on this box is about +-4% single-stream.