Skip to content

Add prompt caching, default to Sonnet 5.5, and offer Haiku 5.5 - #49

Open
aikhanjum wants to merge 2 commits into
feat/weekly-usage-limitfrom
feat/prompt-caching
Open

aikhanjum wants to merge 2 commits into
feat/weekly-usage-limitfrom
feat/prompt-caching

Conversation

@aikhanjum

Copy link
Copy Markdown
Collaborator

PI currently defaults to Opus and sends reusable prompt prefixes without cache markers. This makes Sonnet 5.5 the default and adds Haiku 5.5, including the request in TIG-311.

  • Apply five-minute cache breakpoints to the system prefix and final conversation block for every Claude call, including tool-result follow-ups and budget downgrades.
  • Share model selection between browser preferences and the server. Preserve existing Opus choices and migrate old Sonnet 5 or retired choices to Sonnet 5.5.
  • Keep Haiku selected when the weekly budget restricts Opus to Sonnet. Leave the existing dollar allowances unchanged.
  • Add Haiku pricing, applying its higher rates only to individual requests above 100,000 prompt tokens. Correct Sonnet 5.5 cache-read pricing using Anthropic's price list.

Stacked on #34, which depends on #33. Merge order is #33, #34, then this PR.

Validation passed with the repository's example development variables.

  • npx wrangler types
  • npm run typecheck
  • npm test, 78 passed and 0 failed
  • npm run build
  • git diff --check

Tests capture real SDK request bodies for all three models, including streaming and tool results, and cover budget selection and Haiku pricing thresholds. Provider responses are mocked. Live inference, signed-in UI behavior, and actual cache hits have not been verified. The installed Anthropic SDK accepts Haiku 5.5 through its compatibility path and emits an informational 128,000-token output-limit warning.

Implemented with Codex assistance.

aikhanjum and others added 2 commits October 29, 2026 19:53
Every Claude call now goes through an AI SDK middleware that sets two
Anthropic cache breakpoints: the last system message (covers tool
definitions and the system prompt) and the last message (rolling, so each
step and each turn reads the previous prefix from cache). The production
key showed zero cache reads over two months, so every step paid full input
price. Wire tests capture the request body and check the breakpoints land.

Sonnet 5 is the default model in the server, shared settings and the
client store, per Ibraheem; Opus stays a per-chat choice. The budget note
no longer calls tier 1 "the Opus budget", since a Sonnet-only student can
use it up too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbJwLPGAUb8rEEDz2ZR2iq
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant