[Engine] Opt-in: keep the conversation already read when a request switches the control vector on or off #4585

Area

[Engine]

What do you want to do?

Switch a loaded control vector on and off between turns of one conversation without the engine reading the whole conversation again.

Today a request whose experimental_speed_projection differs from the previous one drops the live session and its checkpoints (src/program/generate.cpp:14427-14430 at 46c152a1, live_ok = false; checks.clear();), and the conversation cache only offers entries saved in the same state (ConversationCache::best(..., cvec), include/strata/core/conversation_cache.hpp:312). That is the right default: switched off, the output is token for token the stock model's (docs/DETAILS.md, "Experimental speed projection").

Some users want the other behavior, the one antirez's ds4 has with /steer: the scale changes between turns and the session is kept. The cells already read keep the state they were read with, only the new tokens get the new state. A typical use is steering the start of a conversation and then switching the vector off for the rest, or the other way round, on a long agent context.

Proposed behavior, opt-in, default unchanged (name up to you; locally I called it --dir-steering-keep-cache):

• a request of the other state that continues the live conversation (its prompt starts with all of it) resumes from the live session instead of token 0;
• a conversation parked in the other state is continued the same way, but only whole (its live end), never through one of its checkpoints;
• every checkpoint is still dropped at the switch, so a new conversation that only shares a prefix (the same system prompt, say) never reads cells computed in the other state;
• batch slots keep matching their own state only;
• the switch is logged with the number of tokens kept.

Without the option nothing changes: same cache keys, same drops, same output.

I have this working as a local patch on engine 0.1.45 (two files, generate.cpp only for this part, about 30 lines). On main the same idea also has to cover the paths added since then: the slot checkpoint move (STRATA_SLOT_CKPT_MOVE), borrowed checkpoints and the async park drops (drops_all includes want_cvec != cvec_cached). I would keep all of them on the current rule (own state only) and change just the live session and whole parked conversations.

Tradeoffs

• With the option on, a cached conversation can hold cells read in both states. Its tag is the state of its last request. Only a continuation of that same conversation can reuse those cells; that is the point of the option, and why it must stay opt-in.
• With the option on, "switched off = stock output" holds for a fresh conversation only, not for a conversation that was steered earlier. The log line makes that visible.
• No change to the kernels, to the cvec flag in VRAM, or to cvec_set_enabled: the switch still happens between requests.

Related: #4465 / #4466 (cvec auto) want the same thing for the automatic gate ("No re-prefill when the gate flips"), and list as an open point that steered decode rows end up in a stock-tagged cache. This proposal is the explicit, user-chosen version for the plain true/false switch, and does not depend on auto. If you prefer one mechanism for both, I am happy to align with whatever stamp you choose there.

Measured (local patch on engine 0.1.45, 2× Pascal: GTX 1080 Ti 11 GB + GTX 1070 8 GB, Swift 1.5 IQ2_XS pack, a project-mode vector on layers 4-44, 22/10/2026):

• with the option: the turn that switches the vector off reused 78 of the 99 prompt tokens of the steered conversation; a new conversation started afterwards reused 0;
• without the option (same engine, flag absent): the same switch reused 0 of 99, as on main.

Not measured yet: prefill time saved per switch. For scale, on this machine a 15K-token prompt takes about 95-98 s to read (v0.2.7.1, median of 3, temperature 0), which is what every switch costs today on a long conversation.

If the scope is fine, I would open one PR against main with the option, a server-level test (three turns on/off/off: resume reported in the log, prefill limited to the new message; same conversation without the option = today's behavior) and a line in DETAILS.md.

What do you do today instead?

A locally patched engine on 0.1.45. On the released engine the only choices are paying a full re-read at every switch, or running two engines (one with the vector, one without), which doubles memory and does not share the conversation.