Repository navigation
[Engine] 0.2.7 PLE residency policy turns the automatic conversation cache off on a 128 GB two-GPU PC #4518
Description
Area
Engine
Desktop app, Terminal or source?
Source: stock setup.py at the v0.2.7 tag with the 0.1.52 Windows CUDA 13 engine package (0.2.7.1 keeps the Windows engine at 0.1.52).
What happened
At start the new PLE residency policy locks the 26.8 GiB PLE table in RAM, and the automatic conversation cache then turns itself off:
strata generate: --ple-io ram chosen by the residency policy (the table is 26.8 GiB, the expert arena still to fill 46.8 GiB and 117.8 GiB of RAM is available; STRATA_PLE_AUTO_RAM=0 keeps the direct reads, --ple-io direct forces them)
strata generate: PLE table locked in RAM (--ple-io ram chosen by the residency policy) in 31.4 s
...
strata serve: conversation cache off: not enough free RAM to park conversations (--conversation-cache-mib N asks for one anyway)
On 0.2.3 (engine 0.1.48) the same PC, model and flags sized the automatic conversation cache to 8,192 MiB (4 parked conversations). With STRATA_PLE_AUTO_RAM=0 on 0.2.7 the cache is back at 8,192 MiB and parks again.
The locked table also comes out of Windows commit: with the engine serving, commit headroom was 1.03 GiB of a 135.7 GiB limit on 0.2.7 (127.7 GiB RAM + 8 GiB page file), against 27.3 GiB on 0.2.3 with the same model and flags; a later bench sampled 0.63-1.33 GiB. Anything else that allocates on that PC (a browser, a game) then risks running out of commit.
Smallest steps or command that reproduce it
- Stock setup for two cards with
--gpu-roles 0=main,1=helper(engine flag--pipeline-windows 2, no batch slots), Qwen3.8-Flash-Next IQ3_S at 262,144 tokens, text only. - Start the server and read the start log, or
automatic_conversation_cache.budget_mibinGET /metrics(0).
Reproduction status
Every default 0.2.7 start we made on this PC: a full requalification run (budget_mib 0) and two back-to-back bench starts. The one start with STRATA_PLE_AUTO_RAM=0 had the 8,192 MiB cache.
Strata version
0.2.7 (tag commit 5e8a4cca), engine 0.1.52.
Graphics card, memory and system
RTX 4090 24 GB + RTX 3090 24 GB layer split, 128 GB DDR4, Windows 11, NVIDIA driver 621.01.
Measured
One request per depth, 512 output tokens, back to back on the same PC:
| Prompt | 0.2.7 default (PLE in RAM, no cache) TTFT | 0.2.7 STRATA_PLE_AUTO_RAM=0 (cache 8 GiB) TTFT |
0.2.3 TTFT |
|---|---|---|---|
| 8,192 | 5.60 s | 5.63 s | 5.69 s |
| 32,768 | 11.36 s | 11.53 s | 11.54 s |
| 65,536 | 19.92 s | 20.21 s | 20.24 s |
| 102,400 | 30.40 s | 31.15 s | 31.23 s |
Decode is the same in all three (161-172 tokens/s). So on this PC the policy trades parking (parking a 102K-token conversation took 1.1 s; returning to a parked conversation took 1.5-3.4 s in our earlier 0.1.43.1 measurements on this PC, against ~30 s to read it again) for ~2-3% faster prompt reading and a 31-37 s longer start. On a 192 GB RTX PRO 6000 both fit (PLE locked and the 8 GiB cache) and prompt reading did not get faster either (single samples, within noise).
Ask
Could the residency admission also count the automatic conversation cache's budget and Windows commit headroom (or size the cache first), so a PC that parks conversations today keeps doing so, with commit to spare, after 0.2.7?
Related: #4421 (the same policy's admission on the Mac).
Activity
udqe commentedon Oct 28, 2026
Same machine shape (Windows, source install, 2x RTX 4090, 127.9 GiB RAM, stock engine 0.1.52, Qwen3.8-Flash-Next UD-Q4_K_XL, layer split) - so here is our side of the same wall, plus the one number that seems to decide it.
1) The residency policy made the same choice here. Our 22:29 start log:
We never hit your "conversation cache off" line because we ask
--conversation-cache-mib 12288explicitly - the cache is visibly alive (parked 124563 tokens in 1222.4 ms; parked=2 bytes=9692123400) across 42 engine starts on Oct 28 including a silent restart at 22:29, prompt reuse 99.6-99.9%. Your point stands: machines on the automatic budget lose the cache silently.2) The decisive number looks like the commit limit, and pagefile sizing is the lever. Measured while serving:
The engine alone wants a lot of commit: process private (committed) 134.2 GiB, working set 96.7 GiB, plus
expert arena: locked 50253 MiB via working-set minimum + VirtualLock. So on a 128 GiB Windows box the working rule seems to be commit limit > engine private (~134 here) + everything else - and the residency policy's 26.8 GiB lands at engine start, before the model can serve anything. A policy that prices its added lock against the commit limit (or a docs line "on 128 GiB Windows, keep a pagefile that covers arena + PLE") would have prevented both of us finding this.3) The same amplifier likely bites the updater path too (filed today as #4463). With the serving engine at ~93 GiB,
strata update's test start loads a second copy and the new engine's memory guard refuses it -limit 127.9 GiB, current 93.0 GiB, available 34.9 GiB ... error 1450- and the advised retry fails identically while the first engine holds. A second engine that first commits 26.8 GiB to the PLE table has even less room to fail into; stopping the serving model and re-running the identical command exits 0 (control).(AI-assisted on our side too - static analysis and measurements cross-checked; the answering model is Qwen3.8-Flash-Next UD-Q4_K_XL served by this repo's engine 0.1.52; all numbers measured locally.)
Niko1221 commentedon Oct 29, 2026
Confirmed on Linux with the same numbers. The policy locks the 26.8 GiB table at start; what the rest of the load takes afterwards (KV pools, pinned staging, the host buffers of two GPUs) is not known then, so the conversation cache gets what is left, and on your box that is nothing. Fixed on our side; the fix ships in the next update (0.2.8). If giving the lock back lets the automatic cache grow, the table is unlocked, the log says so, and the cache is sized again (1536 MiB became 8192 MiB in our run).
--conversation-cache-miband an explicit--ple-io ramare never touched.Not done: the Windows commit-limit side, which we could not measure on Linux. Until then
STRATA_PLE_AUTO_RAM=0gives the 0.2.6 behaviour.Qwen: the PLE residency policy gives its lock back when the lock is w…
Niko1221 commentedon Oct 29, 2026
Confirmed and fixed in 0.2.8 on Linux: when the locked PLE table leaves the automatic conversation cache too little, the lock is given back and the cache is sized again (1536 MiB became 8192 MiB in our run). An explicit
--conversation-cache-mibor--ple-io ramis never touched. We could not measure the Windows commit-limit side, so please say so here if you see it on Windows. Runstrata update. Closing the Linux case.Merge upstream v0.2.8 into memory-core: the Arc-pair and four-card ro…