Repository navigation
🔖 fix: Add a Second Bedrock Message Cache Point So Rebuilt Turns Reach the Previous Turn's Cache - #631
Open
Haarris wants to merge 1 commit into
Open
🔖 fix: Add a Second Bedrock Message Cache Point So Rebuilt Turns Reach the Previous Turn's Cache#631Haarris wants to merge 1 commit into
Haarris wants to merge 1 commit into
Conversation
…h the Previous Turn's Cache Bedrock only looks back about 20 content blocks from a cachePoint. With a single tail checkpoint, a turn whose history is rebuilt differently from what was sent live (e.g. reasoning that is not replayed) can only match the prefix ending at the previous turn's opening human message, which is out of reach after a tool-heavy turn, so the whole conversation is written to the cache again. addBedrockTailCacheControl takes an optional maxCachePoints (default 1, unchanged for existing callers). With 2, the same backward pass also anchors the latest real human message before the tail. getBedrockMessageCachePointBudget returns 2 only when that fits under Bedrock's limit of 4 alongside the system checkpoints in messages[0] and one reserved for the tool list; the graph's serving path passes it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the lookback part of #617.
Bedrock only looks back about 20 content blocks from a
cachePoint(21 added blocks still hit, 22 miss). With the single tail checkpoint fromaddBedrockTailCacheControl, a turn whose history is rebuilt differently from what was sent live (today: reasoning that is not replayed) can only match the prefix that ends at the previous turn's opening human message. After a tool-heavy turn that prefix is out of reach, so the first call of the next turn writes the whole conversation to the cache again.addBedrockTailCacheControltakes an optionalmaxCachePoints(default 1, so existing callers such as summarization are unchanged). With 2, the same backward pass also anchors the latest real human message before the tail (never a tool result, synthetic message or the tail itself).getBedrockMessageCachePointBudgetreturns 2 only when that fits under Bedrock's limit of 4 next to the system checkpoints inmessages[0]and one reserved for the tool list, so it stays valid if #613 adds a second system checkpoint. The graph's serving path passes it.Live on Bedrock (
us.anthropic.claude-sonnet-4-6, us-west-2), two runs each, cost in full-price input tokens per prompt token (reads 0.1, writes 1.25):With an unchanged history the tail alone already reaches the previous turn (each tool round leaves an entry close to the next turn), so this only changes cost when the rebuilt history differs. Once signed reasoning is replayed (#619) that case gets rarer, but the second checkpoint still covers hosts that don't persist signatures, and it never costs more in the runs above.
Validation: 14 new tests in
cache.tail.test.ts, including a Converse-payload lookback check for rebuilt and unchanged histories (8 fail on main). 37/37 tail tests, 1,241 message/graph tests, and 205 tracing tests pass;tsc --noEmit, ESLint (no new warnings) and the build pass.