Skip to content

Team state injected via experimental.chat.system.transform invalidates the provider prefix cache on every state change #53

Open

Description

@pkuqiuning

Team state is pushed through experimental.chat.system.transform on every LLM call (src/index.ts:382, output.system.push(prompt) at src/index.ts:391). The pushed block carries member statuses, task counts, in-progress/recent tasks and undelivered messages (src/system-prompt.ts:24 buildLeadSystemPrompt, src/system-prompt.ts:169 buildTeammateSystemPrompt), and it mutates whenever a teammate transitions busy↔ready, a task is claimed/completed, or a message is delivered (markDelivered at src/system-prompt.ts:97 and :194 flips the pending list between turns).

Providers cache by prefix, so a 1-character change anywhere in the system prompt invalidates the entire conversation for that request: everything after the first changed byte is re-billed at full input price. The lead is only woken when state changed, so nearly every lead turn pays this; teammates pay it too via buildTeammateSystemPrompt.

Measured on a local proxy that logs provider usage (same mechanism, different plugin mutating the system prompt — the invalidation is a property of prefix caching, not of the proxy):

request usage cache hit
steady state input=94427 cached=88832 94–100%
first request after the system prompt changed input=91313 cached=24576 27%
worst observed input=96833 cached=0 0%

At typical cache-read pricing (~0.1× input), that is ~7× the input cost of a hot turn, on every churned turn. It recovers on the next request only if the block stops changing.

Related upstream work in the same class: anomalyco/opencode #58485 ("Moving plan/build reminders invalidate the prompt cache and increase API costs"), #54935 ("Parallel sub-agents start a fresh session with their own cache key and no concurrency cap, so they cost more than one agent"), #50777, #56422.

Suggestion

The wake path is already cache-safe: src/index.ts:288-291 sends the tickle as a trailing promptAsync text part ([System: N new team message(s) available]), with wakeLeadTimestamps / WAKE_LEAD_COOLDOWN_MS dedupe. The message payload and the state summary could ride the same way — appending at the tail only costs the new bytes, everything before stays cached.

The reason the payload sits in the system prompt looks like compaction survival (README:303 "the lead's system prompt carries member statuses and task counts, so it stays aware across turns including after compaction"), and that constraint is already covered by the hook you implement at src/index.ts:395 (experimental.session.compacting → buildTeamCompactionContext): re-inject the state there after compaction. The Plan→Build agent flip in #37 is orthogonal (fixed by 42a29ec / 5919ee0 via explicit agent+model on promptAsync).

Minimal alternative if the current placement is intentional: a config knob (systemPromptTeamState: full | state-only | off, or a switch that moves message payloads to the trailing wake message) so cache-sensitive setups can opt out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions