[rochaudhari]

Overview

A new session has following limitations which reusing prefix that another slot holds:

1. It cannot use cells from a slot that is busy, which is the concurrent-session case,
2. Two sessions cannot share one copy; each slot that has the prefix keeps its own copy in the KV cache.

With --slot-attach and unified cache, a new session takes the shared prefix from another slot with seq_cp, which shares the KV cells instead of copying them. Hybrid models attach at one of the other slot's checkpoints.

For hybrid models, because the recurrent state cannot be cut back to an earlier token, when a prompt matches the cached state only up to a point before the first user message, the server saves a checkpoint there. Later sessions can then resume exactly where prompts differ.

Additional information

Original discussion: #30664

Design

• Attach: with a unified KV cache, seq_cp within one stream adds the new slot's tag to the donor's cells without copying data. The whole sequence is copied (llama_memory_hybrid_idx only supports that), then seq_rm(slot, n, -1) drops the tags after the attach point.
• Recurrent and hybrid memory: between those two steps, the donor's checkpoint at n is loaded. This first removes the new slot from the recurrent cell it shares with the donor, so the donor's state is untouched. The slot's own state then ends at n-1, so the final seq_rm leaves it alone.
• Divergence checkpoints (recurrent and hybrid, chat requests only): a checkpoint is saved where a prompt splits from a prompt held by any slot or by the RAM cache, before the first user message, at least 1,024 tokens in, and more than 512 tokens past where the request resumes. Later sessions attach up to it.

Numbers

Setup: NVIDIA A40 (48 GB), CUDA 12.8, q8_0 KV, 4 slots with unified KV, 3 concurrent sessions, MTP off. Prompts use Hermes's real tools and system prompt; the project context, skills and memory parts are synthetic. The shared prefix is 14.1K tokens. Times are the mean of 3 runs. The KV cell counts come from one separate run.

Values are "off -> attach":

Model	TTFT, session 3	Decode, 3 sessions	KV cells used	KV saved
Qwen3.8-27B (dense hybrid)	12.82 -> 0.53 s	60 -> 72 tok/s (+20%)	42,925 -> 14,682	66%, 0.92 GiB
Qwen3.6-35B-A3B (hybrid MoE)	3.80 -> 0.27 s	169 -> 200 tok/s (+18%)	43,480 -> 15,332	65%, 0.29 GiB
Qwen3-30B-A3B (attention only)	6.50 -> 0.13 s	95 -> 159 tok/s (+67%)	42,545 -> 14,914	65%, 1.34 GiB

A checkpoint copy on the 35B is 62.8 MiB and takes about 29 ms per attach.

Tests

Three new test functions, five cases, in tools/server/tests/unit/test_completion.py:

• test_slot_attach, with and without idle-slot clearing:
  • attach happens;
  • the result matches same-slot reuse within 1e-5;
  • without idle clearing, the donor's logprobs for its own prompt are unchanged after another slot attached and wrote cells.
• test_slot_attach_off, for non-unified KV and for speculative decoding: nothing attaches.
• test_slot_attach_hybrid:
  • attach at a checkpoint;
  • a divergence checkpoint;
  • the same result as that slot restoring the checkpoint itself.

The server suite tools/server/tests without slow tests: 438 passed, 4 skipped.

Related and follow-ups

• RAM prefix entries: for sessions on later days, after the slots have been reused, restore a copy of the shared prefix from the RAM cache. That is a copy, not an attach, since sharing cells only works between slots (Sharing prompt prefixes across sessions in llama-server #30664)
• Paged KV cache and scheduler: Phase 1 (design feedback requested) #21961: paged KV cache and scheduler discussion. A comment there tested the same seq_cp sharing idea on attention-only models.
• server : add --cache-disk for a persistent prompt cache #28092: disk prompt cache (--cache-dir). ngxson mentions a block-hash prompt cache there that would replace it.
• llama-server: Use reference-counted checkpoints instead of deep-copies #30137: reference-counted checkpoints. It would remove the checkpoint copy per attach.
• abort_all_slots releases slots without clearing their memory, so a slot's token list can be ahead of its KV cache. The attach code guards against this by capping the attach at seq_pos_max + 1.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Claude wrote the change and made the tests. I reviewed the change and ran the tests. I also wrote the text for the PR description.

[joelagnel]

Why should this be an option? It should be the default IMO, is there a reason we do not want to do this optimization?

[rochaudhari]

some of the features, such as --cache-reuse and --context-shift cannot go along-with --slot-attach since these features require sliding and shifting of cells. To enable all of these together will require copy-on-write (making private copies) which would be follow-up for current MR (avoided that purposely in current MR to keep scope minimal).
Additionally, IMO providing an option also helps in narrowing down any perf gaps/regressions.

[joelagnel]

But we can disable the feature automatically if something incompatible is turned on right? I don't see why it needs to be yet another option. It should be "let's do this if we can" automatically. Did I miss something?

[rochaudhari]

yes, we can keep it on by default. If we have to follow convention of other features, we can keep --no-slot-attach as kill switch. WDYT?

[joelagnel]

Yes makes sense to me. I like this approach better because it makes it easier for user to not have to know about the option.