What happens
LLMDescriptionScorer._create_prompt (llm_encoder.py:155-200) lays the single user message out as
- two-line header,
<text_sample>{utterance}</text_sample>,<possible_intent_descriptions> — all N descriptions, identical for every utterance of a run and the bulk of the tokens (77–150 descriptions on banking77/clinc150),<instructions>.
Provider prompt/prefix caching (OpenAI automatic prompt caching, Anthropic cache, OpenRouter pass-through) only matches the longest identical prefix, and here the prefix diverges at the utterance, so the descriptions block is re-billed at full input price on every call. Input tokens are essentially the whole cost of each call (the output is a short JSON of indices).
Proposed
Order the message header → descriptions → instructions → utterance (utterance last). The shared prefix then covers everything but the utterance, and cached input tokens are billed at the provider's discounted rate across different utterances, not just exact repeats. Darinochka/AutoIntent-experiments#43 saw the exact-repeat effect already: a byte-identical dry-run repeat cost $0.0054/call vs $0.0308 cold on OpenRouter.
Caveats: the prompt text is part of the StructuredOutputCache key, so existing caches miss once; and the change moves the question relative to the context, so quality should be re-checked on one dataset before switching the default.
Follow-up from #350.
What happens
LLMDescriptionScorer._create_prompt(llm_encoder.py:155-200) lays the single user message out as<text_sample>{utterance}</text_sample>,<possible_intent_descriptions>— all N descriptions, identical for every utterance of a run and the bulk of the tokens (77–150 descriptions on banking77/clinc150),<instructions>.Provider prompt/prefix caching (OpenAI automatic prompt caching, Anthropic cache, OpenRouter pass-through) only matches the longest identical prefix, and here the prefix diverges at the utterance, so the descriptions block is re-billed at full input price on every call. Input tokens are essentially the whole cost of each call (the output is a short JSON of indices).
Proposed
Order the message header → descriptions → instructions → utterance (utterance last). The shared prefix then covers everything but the utterance, and cached input tokens are billed at the provider's discounted rate across different utterances, not just exact repeats. Darinochka/AutoIntent-experiments#43 saw the exact-repeat effect already: a byte-identical dry-run repeat cost $0.0054/call vs $0.0308 cold on OpenRouter.
Caveats: the prompt text is part of the
StructuredOutputCachekey, so existing caches miss once; and the change moves the question relative to the context, so quality should be re-checked on one dataset before switching the default.Follow-up from #350.