Skip to content

fix(bedrock): fold cache tokens into non-streaming converse usage - #13

Open
teriyakichild wants to merge 1 commit into
mezmofrom
fix/bedrock-converse-cache-usage
Open

teriyakichild wants to merge 1 commit into
mezmofrom
fix/bedrock-converse-cache-usage

Conversation

@teriyakichild

Copy link
Copy Markdown

Follow-up to #12, surfaced in downstream adversarial review of mezmo/aura#631 (finding 3): the non-streaming Converse path copied inputTokens raw and dropped cacheReadInputTokens/cacheWriteInputTokens, even though the internal TokenUsage parses them. AWS defines inputTokens as exclusive of cached tokens, so non-streaming prompt usage was understated whenever caching was active. The streaming path already folds correctly — this aligns the non-streaming conversion, with a regression test.

aura is streaming-only, so no aura behavior changes; the pin can move on its next routine bump.

The non-streaming Converse conversion copied inputTokens raw, dropping
cacheReadInputTokens / cacheWriteInputTokens even though the internal
TokenUsage parses them — AWS defines inputTokens as exclusive of cached
tokens, so prompt usage was understated whenever caching was active.
Fold them in, matching the streaming path.
@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown

Greptile Summary

Aligns non-streaming Amazon Bedrock Converse token accounting with the streaming path.

  • Includes cache-read and cache-write tokens in normalized prompt usage.
  • Adds a regression test covering a cache-enabled Converse response.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security issues identified.

The new non-streaming conversion matches the existing streaming arithmetic, handles absent cache counters as zero, preserves provider-reported output and total usage, and has regression coverage for cache-active responses.

Important Files Changed

Filename Overview
rig-integrations/rig-bedrock/src/types/assistant_content.rs Correctly folds optional cache token counts into non-streaming input usage and verifies the normalized totals with a focused regression test.

Reviews (1): Last reviewed commit: "fix(bedrock): fold cache tokens into non..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant