diff --git a/README.md b/README.md index fd2da52..4c86517 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,22 @@ It works with Claude through the Anthropic API, through proxies that speak the A The Cache section of the OpenCode sidebar: status warm, cached 92% · 141k / 153k, expires 14:52 (52m), ttl 1h (detected), next ping 14:42 (42m), pings 2/5 ✓ hit ($0.05), saved $0.42 · 2 rewrites, return 9× ping cost

+## Contents + +- [Why](#why) +- [The three modes](#the-three-modes) + - [Modes per provider](#modes-per-provider) +- [Supported providers and endpoints](#supported-providers-and-endpoints) +- [Install](#install) +- [Usage](#usage) + - [Per-model override](#per-model-override) + - [Proxies and other provider ids](#proxies-and-other-provider-ids) +- [Keepalive](#keepalive) +- [Settings](#settings) +- [Sidebar](#sidebar) + - [Savings](#savings) +- [Contributing](#contributing) + ## Why OpenCode marks Claude's cache breakpoints on its own (tools, system prompt, the last message) but always sends `ttl: "5m"`, and there is no config option for it. Step away for ten minutes and your next prompt writes the whole context into the cache again at 1.25x the base input price (2x for the 1h cache) instead of reading it at about 0.1x. GPT-5.6+ has the same problem: OpenAI caches for 30 minutes on its own, and the 12-hour cache has to be requested per breakpoint, which OpenCode never does. @@ -203,7 +219,7 @@ The effective value of each setting is the first one set, in this order: 1. the session's override (`/cache-manager` typed argument, or the picker's Session section) 2. the global value stored from the picker's Global section -3. the plugin option (`defaultMode` or `keepalive.*`) +3. the plugin option (`defaultMode`, `keepalive.*` or `savings`) 4. the built-in default A global value stored from the picker overrides the plugin option for the same setting, so changing `defaultMode` in `opencode.jsonc` has no effect once the picker has stored a global value for that setting. @@ -234,7 +250,7 @@ Cache - `cached` describes the latest primary turn: the share of its prompt that was read from cache, then the tokens read / the prompt total, such as `92% · 14.8k / 16.1k`. Pings, titles and other one-off requests don't change it. It shows for every session with cache usage, Claude or not, once a response has arrived. - `next ping` is when the next ping goes out. It reads `none` once the limit is reached. - `pings 2/5` counts pings used in this idle stretch, followed by the last result: `✓ hit`, `✗ miss` or `✗ failed`. No result is shown before the first ping. -- `pings` also shows what the pings of the current idle stretch cost, such as `($0.05)`, when the model has a price. +- `pings` also shows what the pings of the current idle stretch cost, such as `($0.05)`, when every ping in the session was priced. - `saved` and `return` are the session's savings totals; see [Savings](#savings). - When keepalive isn't running, a `keepalive` row says why: `off · 31k < 50k` (context below the minimum), `off · disabled`, `waiting · session busy` (waiting on a tool call), or `stopped · last ping missed`. @@ -269,7 +285,7 @@ Anything else is ping cost only: a session you never came back to, a return befo - `saved` is the session total. In dollars it is net: avoided cost minus the cost of every ping, wasted ones included. Without a price it is `2 rewrites · 282k`, the rewrites and the tokens they avoided. `saved nothing yet` shows once you've paid for a ping and nothing has been saved. - `return` is avoided cost divided by ping cost, such as `9× ping cost`. It appears after the first save. The label may change. -Dollars need a price for every model the session used. OpenCode's catalog prices the models of known providers. For a model under a custom provider, add `cost` to the model in `opencode.jsonc` (`input`, `output`, `cache.read`, `cache.write`, in dollars per million tokens); [the OpenAI-compatible example](docs/examples/opencode.openai-compatible.jsonc) shows where. Or set `modelsDevPrices` to `true` to look up models the catalog has no price for in models.dev. That is the only network call the plugin makes, once a day at most (after a failed fetch, the next try waits at least 15 minutes), and a price from there is the list price, which may differ from what a subscription-backed route charges you. A failed lookup, or a model with no clear match, leaves the model unpriced. If OpenCode's model catalog can't be loaded, events are unpriced and models.dev is not consulted. +Dollars need a price for every model the session used. OpenCode's catalog prices the models of known providers. For a model under a custom provider, add `cost` to the model in `opencode.jsonc` (`input`, `output`, `cache.read`, `cache.write`, in dollars per million tokens; `cache.write` prices 5m and 30m writes, while 1h and 12h writes are priced from `input` at 2x and 2.5x); [the OpenAI-compatible example](docs/examples/opencode.openai-compatible.jsonc) shows where. Or set `modelsDevPrices` to `true` to look up models the catalog has no price for in models.dev. That is the only network call the plugin makes, once a day at most (within one run of OpenCode, a failed fetch is retried after 15 minutes at the earliest; that wait isn't kept across restarts), and a price from there is the list price at the model's base tier, with no long-context tier, which may differ from what a subscription-backed route charges you. A failed lookup, or a model with no clear match, leaves the model unpriced. If OpenCode's model catalog can't be loaded, events are unpriced and models.dev is not consulted. If any event in a session has no price, the whole session shows the unit-free form, since a partial dollar figure would mislead. Each event is priced with the model that served it, so a session that switches models still adds up. Dollar amounts round to the cent, with `<$0.01` for a non-zero amount below a cent and a minus sign (`-$0.02`) for a negative net. diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index eb16fc1..a8943e3 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -82,7 +82,7 @@ Where things go: brew install charmbracelet/tap/freeze # other package managers: see the freeze README ``` -After changing the sidebar component or a screen's samples, regenerate both images and commit them: +After changing the sidebar component or a screen's samples, regenerate all three images (`sidebar.svg`, `sidebar-states.svg`, `picker.svg`) and commit them. A change to `src/ui/picker.ts` or `scripts/capture/picker-dialog.tsx` needs it too: ```sh bun scripts/capture-sidebar.ts --all diff --git a/docs/UPGRADE.md b/docs/UPGRADE.md index 386890e..c692dfb 100644 --- a/docs/UPGRADE.md +++ b/docs/UPGRADE.md @@ -4,7 +4,7 @@ Update the version range in your `plugins` entry to `^0.7.0`: on 0.x, `^0.6.0` does not pick up 0.7. -0.7.0 adds a savings ledger. The sidebar gains `saved` and `return` rows by default, and `pings` shows the cost of the current stretch when the model has a price. Turn them off with the `savings` option, the picker's Savings row, or `/cache-manager savings off`; the ledger keeps counting while hidden. See [Savings](../README.md#savings). +0.7.0 adds a savings ledger. The sidebar gains `saved` and `return` rows by default, and `pings` shows the cost of the current stretch when every ping in the session was priced. Turn them off with the `savings` option, the picker's Savings row, or `/cache-manager savings off`; the ledger keeps counting while hidden. See [Savings](../README.md#savings). `modelsDevPrices` is opt-in (default `false`). Leave it off and the plugin makes no network calls of its own; sessions on models the OpenCode catalog doesn't price show the unit-free form. diff --git a/docs/examples/opencode.openai-compatible.jsonc b/docs/examples/opencode.openai-compatible.jsonc index 2425ae2..ea4b967 100644 --- a/docs/examples/opencode.openai-compatible.jsonc +++ b/docs/examples/opencode.openai-compatible.jsonc @@ -58,6 +58,8 @@ }, // Optional: a price lets the sidebar show savings in dollars. Per million // tokens; a custom provider has no price unless you add one here. + // cache.write prices 5m and 30m writes; 1h and 12h writes are priced from + // input (2x and 2.5x). // "cost": { "input": 3, "output": 15, "cache": { "read": 0.3, "write": 3.75 } }, "variants": [ { diff --git a/package.json b/package.json index a73f327..6c313c0 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@noppu-labs/opencode-cache-manager", - "version": "0.7.0", + "version": "0.7.1", "description": "Prompt-cache manager for OpenCode V2: per-session 5m / 1h / hybrid TTL and keepalive pings for Claude and OpenAI GPT-5.6+ on the Anthropic Messages, Chat Completions and Responses APIs (OpenRouter, proxies), plus a sidebar cache status", "keywords": [ "anthropic", diff --git a/scripts/capture/ansi.ts b/scripts/capture/ansi.ts index 61918bd..7b7a02e 100644 --- a/scripts/capture/ansi.ts +++ b/scripts/capture/ansi.ts @@ -97,7 +97,7 @@ function boldRuns(rows: readonly string[]): string[] { function restoreBold(svg: string, rows: readonly string[]): string { let out = svg; for (const text of new Set(boldRuns(rows))) { - out = out.replaceAll(`>${text}`, ` font-weight="bold">${text}`); + out = out.replaceAll(`>${text}`, () => ` font-weight="bold">${text}`); } return out; } diff --git a/scripts/capture/screens.tsx b/scripts/capture/screens.tsx index 6b5ada1..d39dc0e 100644 --- a/scripts/capture/screens.tsx +++ b/scripts/capture/screens.tsx @@ -216,7 +216,18 @@ const STATES: readonly Block[] = [ }, { caption: 'Savings turned off', - snapshot: snapshot({ showSavings: false }), + snapshot: snapshot({ + savings: { + avoidedTokens: 120_000, + netUsd: 0.27, + pings: 2, + priced: true, + rescues: 1, + returnRatio: 4.6, + stretchUsd: 0.05, + }, + showSavings: false, + }), }, ]; diff --git a/src/constants/savings.ts b/src/constants/savings.ts index 5345176..42da91d 100644 --- a/src/constants/savings.ts +++ b/src/constants/savings.ts @@ -40,6 +40,12 @@ const MODELS_DEV_STORAGE_KEY = 'models-dev'; const MODELS_DEV_FETCH_TIMEOUT_MS = 10_000; /** A `ctx.model.list()` call that has not settled by then counts as a failed catalog load. */ const CATALOG_LOAD_TIMEOUT_MS = 10_000; +/** After a failed catalog load, wait this long before another reload for a missing model. */ +const CATALOG_RETRY_MS = 60_000; +/** A trailing `-YYYYMMDD` release date on a model name. */ +const DATE_SUFFIX = /-\d{8}$/u; +/** A trailing `-latest` alias on a model name. */ +const LATEST_SUFFIX = /-latest$/u; /** Vendors whose own listing wins over resellers. */ const MODELS_DEV_VENDORS: readonly string[] = ['anthropic', 'openai']; @@ -57,4 +63,7 @@ export { MODELS_DEV_FETCH_TIMEOUT_MS, MODELS_DEV_VENDORS, CATALOG_LOAD_TIMEOUT_MS, + CATALOG_RETRY_MS, + DATE_SUFFIX, + LATEST_SUFFIX, }; diff --git a/src/models/keepalive.ts b/src/models/keepalive.ts index 94fbff1..a03e7bd 100644 --- a/src/models/keepalive.ts +++ b/src/models/keepalive.ts @@ -36,7 +36,6 @@ type KeepaliveSnapshot = { contextTokens: number | null; /** Cache reads and total prompt tokens of the latest non-ping response; null before one arrives. */ lastCache: CacheReadout | null; - /** The `savings` setting. */ showSavings: boolean; /** The session's ledger readout; null with savings off, for unmanaged or child sessions and before the ledger loads. */ savings: SavingsReadout | null; diff --git a/src/models/savings.ts b/src/models/savings.ts index 9645c0f..fadb74a 100644 --- a/src/models/savings.ts +++ b/src/models/savings.ts @@ -28,7 +28,7 @@ interface SavingsReadout { avoidedTokens: number; /** `avoidedUsd - pingUsd` when priced. */ netUsd: number | null; - /** Null until rescues > 0; dollars when priced, units otherwise. */ + /** Avoided / spent, in dollars when priced and units otherwise; null while rescues is 0 or the spend is 0. */ returnRatio: number | null; /** Ping cost in the current stretch; null when unpriced or none. */ stretchUsd: number | null; diff --git a/src/services/keepalive-engine.ts b/src/services/keepalive-engine.ts index 548b337..f983341 100644 --- a/src/services/keepalive-engine.ts +++ b/src/services/keepalive-engine.ts @@ -120,7 +120,6 @@ interface SessionState { pingTtl: Ttl | null; /** Bumped by every real request; a ping charged under another value was answered after its stretch ended. */ stretch: number; - /** The `stretch` the latest ping was sent in. */ pingStretch: number; /** The latest primary or compaction request ran in `hybrid` mode. */ split: boolean; @@ -155,12 +154,10 @@ class KeepaliveEngine { state.requestSentAt = state.lastRequestAt; const kind = info.kind ?? 'primary'; const sessionTurn = kind === 'primary' || kind === 'compaction'; - // Any earlier pending return is overtaken by this request. - state.pendingReturn = sessionTurn - ? this.returnCheck(state, state.lastRequestAt, info.split === true) - : null; - this.startStretch(sessionID, state); if (sessionTurn) { + // Only a session turn overtakes an earlier pending return and ends the stretch; title and generate calls do neither. + state.pendingReturn = this.returnCheck(state, state.lastRequestAt, info.split === true); + this.startStretch(sessionID, state); state.split = info.split === true; if (info.model) state.model = info.model; state.requestedTtl = info.requestedTtl; @@ -309,19 +306,26 @@ class KeepaliveEngine { } private learnPrefix(state: SessionState, usage: CacheUsage, kind: RequestKind): void { - if (state.requestedTtl !== null && OPENAI_TTLS.has(state.requestedTtl)) { + if (this.estimatesPrefix(state, usage)) { const context = contextTokens(usage); // A zero context says nothing about the prefix; subtracting it would count the whole read. if (kind === 'primary' && state.firstPrimaryContext === null && context > 0) { state.firstPrimaryContext = context; } - } else { + } else if (state.split) { // A later turn may rewrite only the tail of the prefix; the largest write errs toward fewer avoided tokens. + // A 1h-mode full write is no prefix, so only hybrid turns count. const write1h = usage.write1hTokens ?? 0; if (write1h > 0) state.prefix1hTokens = Math.max(state.prefix1hTokens ?? 0, write1h); } } + /** OpenAI TTLs and wires that report no 1h/5m split give no measured prefix, so the first primary context stands in. */ + private estimatesPrefix(state: SessionState, usage: CacheUsage): boolean { + const openai = state.requestedTtl !== null && OPENAI_TTLS.has(state.requestedTtl); + return openai || usage.write1hTokens === undefined; + } + /** * A real primary or compaction request may rescue its stretch when the stretch had a hit * ping and the request goes out past the margin after the cache would have expired without pings. @@ -338,7 +342,10 @@ class KeepaliveEngine { const openai = state.requestedTtl !== null && OPENAI_TTLS.has(state.requestedTtl); return { contextBefore: state.contextTokens, - prefix: openai ? state.firstPrimaryContext : state.prefix1hTokens, + // Claude prefers the measured 1h write and falls back to the estimate when its wire reports no split. + prefix: openai + ? state.firstPrimaryContext + : (state.prefix1hTokens ?? state.firstPrimaryContext), // Either side being split means the long-lived prefix may have stayed warm on its own. split: state.split || split, ttl: state.ttl, @@ -399,7 +406,7 @@ class KeepaliveEngine { try { void Promise.resolve(call(ledger)).catch(() => {}); } catch { - // A failing ledger must not break scheduling. + // Ledger failures never reach scheduling. } } diff --git a/src/services/ledger-store.ts b/src/services/ledger-store.ts index 4cad2d7..6e564d4 100644 --- a/src/services/ledger-store.ts +++ b/src/services/ledger-store.ts @@ -6,16 +6,39 @@ import { isRecord } from '../utils/is-record.ts'; const LEDGER_KEY_PREFIX = 'ledger/'; const ledgerKey = (sessionID: string): string => `${LEDGER_KEY_PREFIX}${sessionID}`; -const COUNTER_FIELDS = [ - 'pings', - 'rescues', - 'avoidedTokens', - 'pingUnits', - 'avoidedUnits', - 'pingUsd', - 'avoidedUsd', - 'unpricedEvents', -] as const; +type CounterField = Exclude; + +// A Record, so a counter added to LedgerTotals fails to compile until it is listed here. +const COUNTERS: Record = { + avoidedTokens: true, + avoidedUnits: true, + avoidedUsd: true, + pingUnits: true, + pingUsd: true, + pings: true, + rescues: true, + unpricedEvents: true, +}; +const isCounterField = (key: string): key is CounterField => key in COUNTERS; +const COUNTER_FIELDS = Object.keys(COUNTERS).filter((key) => isCounterField(key)); + +type Change = (totals: LedgerTotals) => LedgerTotals; + +/** `change` applied to `totals`; a throwing change leaves them as they were. */ +function applyChange(totals: LedgerTotals, change: Change): LedgerTotals { + try { + return change(totals); + } catch { + return totals; + } +} + +/** `changes` applied in order; a throwing change is skipped. */ +function applyAll(totals: LedgerTotals, changes: readonly Change[]): LedgerTotals { + let result = totals; + for (const change of changes) result = applyChange(result, change); + return result; +} /** Stored totals when well-formed; anything else is `EMPTY_TOTALS`. */ function parseTotals(stored: unknown): LedgerTotals { @@ -33,8 +56,10 @@ function parseTotals(stored: unknown): LedgerTotals { /** Per-session savings totals, kept in memory and mirrored to plugin storage. */ class LedgerStore { private readonly totals = new Map(); - private readonly loading = new Map>(); + private readonly loading = new Map>(); private readonly queues = new Map>(); + /** Changes made while the stored record was unreadable, applied once a read succeeds. */ + private readonly deferred = new Map(); public constructor(private readonly kv: KV) {} @@ -43,35 +68,31 @@ class LedgerStore { return this.totals.get(sessionID); } - /** Loads once per session; corrupt or missing data loads as `EMPTY_TOTALS`. Never rejects. */ + /** + * Loads once per session; corrupt or missing data loads as `EMPTY_TOTALS`. Never rejects. + * A storage read error resolves as `EMPTY_TOTALS` without caching it, so the next call retries. + */ public async load(sessionID: string): Promise { - const loaded = this.totals.get(sessionID); - if (loaded !== undefined) return loaded; - let pending = this.loading.get(sessionID); - if (pending === undefined) { - pending = this.read(sessionID); - this.loading.set(sessionID, pending); - } - return pending; + return (await this.fetch(sessionID)) ?? EMPTY_TOTALS; } /** * Applies `change` to the loaded totals and persists. Never rejects: a throwing `change` - * leaves the totals as they were, and a persistence failure keeps memory. + * leaves the totals as they were, and a persistence failure keeps memory. While the stored + * record is unreadable the change is held back and nothing is written, so a stored record is + * never overwritten blind; held changes apply on the first update after a successful read. */ - public async update( - sessionID: string, - change: (totals: LedgerTotals) => LedgerTotals - ): Promise { + public async update(sessionID: string, change: Change): Promise { const previous = this.queues.get(sessionID) ?? Promise.resolve(); const run = previous.then(async () => { - const current = await this.load(sessionID); - let next: LedgerTotals; - try { - next = change(current); - } catch { - return current; + const current = await this.fetch(sessionID); + const held = this.deferred.get(sessionID) ?? []; + if (current === undefined) { + this.deferred.set(sessionID, [...held, change]); + return applyAll(EMPTY_TOTALS, [...held, change]); } + this.deferred.delete(sessionID); + const next = applyAll(current, [...held, change]); this.totals.set(sessionID, next); try { await this.kv.set(ledgerKey(sessionID), next); @@ -87,16 +108,30 @@ class LedgerStore { return run; } - private async read(sessionID: string): Promise { - let totals = EMPTY_TOTALS; + private async fetch(sessionID: string): Promise { + const loaded = this.totals.get(sessionID); + if (loaded !== undefined) return loaded; + let pending = this.loading.get(sessionID); + if (pending === undefined) { + pending = this.read(sessionID); + this.loading.set(sessionID, pending); + } + return pending; + } + + /** The stored totals, or undefined when storage could not be read (nothing is cached then). */ + private async read(sessionID: string): Promise { try { - totals = parseTotals(await this.kv.get(ledgerKey(sessionID))); + // Yield first so `fetch` registers this promise in `loading` before the `finally` clears it. Without it a synchronous `kv.get` throw would clear first and leave a stale entry. + await Promise.resolve(); + const totals = parseTotals(await this.kv.get(ledgerKey(sessionID))); + this.totals.set(sessionID, totals); + return totals; } catch { - // Unreadable storage loads as empty. + return undefined; + } finally { + this.loading.delete(sessionID); } - this.totals.set(sessionID, totals); - this.loading.delete(sessionID); - return totals; } } diff --git a/src/services/model-prices.ts b/src/services/model-prices.ts index cbe2289..0638ab5 100644 --- a/src/services/model-prices.ts +++ b/src/services/model-prices.ts @@ -1,6 +1,6 @@ -import { CATALOG_LOAD_TIMEOUT_MS } from '../constants/savings.ts'; -import type { ModelRef } from '../models/cache'; -import type { ModelPrice } from '../models/savings'; +import { CATALOG_LOAD_TIMEOUT_MS, CATALOG_RETRY_MS } from '../constants/savings.ts'; +import type { ModelRef } from '../models/cache.ts'; +import type { ModelPrice } from '../models/savings.ts'; import { isRecord } from '../utils/is-record.ts'; interface CatalogCost { @@ -22,6 +22,8 @@ interface PriceBookDeps { fallback?: (names: readonly string[]) => Promise; /** Defaults to `CATALOG_LOAD_TIMEOUT_MS`. */ loadTimeoutMs?: number; + /** Clock for the reload backoff; defaults to `Date.now`. */ + now?: () => number; } /** The tier whose size is the largest at or below `contextTokens`, else the untiered entry, else the smallest tier. */ @@ -125,10 +127,12 @@ const refKey = (ref: ModelRef): string => `${ref.providerID}\u0000${ref.id}`; /** * Per-million-token prices from OpenCode's model catalog, with an optional injected fallback. * The catalog is cached; a model missing from it triggers at most one reload per distinct model. + * A failed reload is not remembered, but no new one starts within `CATALOG_RETRY_MS` of the failure. */ class PriceBook { private catalog: readonly CatalogModel[] | undefined; private inflight: Promise | undefined; + private failedAt: number | undefined; private readonly reloads = new Map>(); public constructor(private readonly deps: PriceBookDeps) {} @@ -173,7 +177,13 @@ class PriceBook { const key = refKey(ref); let reload = this.reloads.get(key); if (reload === undefined) { - reload = this.load(true).then((list) => match(list)); + reload = this.load(true).then((list) => { + if (this.failedAt !== undefined) { + // A failed reload says nothing about the model; let a later lookup try again. + this.reloads.delete(key); + } + return match(list); + }); this.reloads.set(key, reload); } return { entry: await reload }; @@ -184,6 +194,9 @@ class PriceBook { if (!force && this.catalog !== undefined) { return this.catalog; } + if (force && this.failedAt !== undefined && this.now() - this.failedAt < CATALOG_RETRY_MS) { + return this.catalog; + } this.inflight ??= this.fetchCatalog().finally(() => { this.inflight = undefined; }); @@ -200,14 +213,20 @@ class PriceBook { }); try { this.catalog = await Promise.race([this.deps.listModels(), timeout]); + this.failedAt = undefined; } catch { - // Leave any previous catalog in place; the next lookup retries. + // Leave any previous catalog in place; the next lookup retries once the backoff passes. + this.failedAt = this.now(); } finally { clearTimeout(timer); } return this.catalog; } + private now(): number { + return (this.deps.now ?? Date.now)(); + } + private async askFallback(names: readonly string[]): Promise { if (this.deps.fallback === undefined) { return undefined; diff --git a/src/services/models-dev.ts b/src/services/models-dev.ts index 570213e..6dd2a65 100644 --- a/src/services/models-dev.ts +++ b/src/services/models-dev.ts @@ -1,4 +1,6 @@ import { + DATE_SUFFIX, + LATEST_SUFFIX, MODELS_DEV_MAX_AGE_MS, MODELS_DEV_RETRY_MS, MODELS_DEV_STORAGE_KEY, @@ -25,9 +27,6 @@ interface Cache { body: unknown; } -const DATE_SUFFIX = /-\d{8}$/u; -const LATEST_SUFFIX = /-latest$/u; - /** Lowercased name without a `provider/` prefix, a `-YYYYMMDD` suffix or `-latest`. */ function normaliseModelName(name: string): string { let result = name.trim().toLowerCase(); @@ -147,7 +146,9 @@ function pruneBody(body: unknown): Record { }; } } - pruned[provider] = { models }; + if (Object.keys(models).length > 0) { + pruned[provider] = { models }; + } } return pruned; } @@ -223,7 +224,11 @@ class ModelsDevPrices { if (!isRecord(body)) { throw new TypeError('models.dev body is not an object'); } - this.cache = { body: pruneBody(body), fetchedAt: this.deps.now() }; + const pruned = pruneBody(body); + if (Object.keys(pruned).length === 0) { + throw new TypeError('models.dev body has no priced models'); + } + this.cache = { body: pruned, fetchedAt: this.deps.now() }; this.index = undefined; this.failedAt = undefined; await this.deps.kv.set(MODELS_DEV_STORAGE_KEY, this.cache).catch(() => null); diff --git a/tests/scripts/capture-sidebar.test.ts b/tests/scripts/capture-sidebar.test.ts index 0aa3d97..47a50e6 100644 --- a/tests/scripts/capture-sidebar.test.ts +++ b/tests/scripts/capture-sidebar.test.ts @@ -173,6 +173,7 @@ describe('renderScreen', () => { const rows = (await screens.renderScreen('sidebar-states')).map((row) => stripAnsi(row)); const off = rows.slice(rows.indexOf('# Savings turned off')); expect(off).toContain(' pings 2/5 ✓ hit'); + expect(off.some((row) => row.includes('($'))).toBe(false); expect(off.some((row) => row.includes('saved') || row.includes('return'))).toBe(false); }); diff --git a/tests/server.test.ts b/tests/server.test.ts index 18044c0..12a6fd0 100644 --- a/tests/server.test.ts +++ b/tests/server.test.ts @@ -610,6 +610,29 @@ describe('keepalive wiring', () => { responseText: messagesSse(resolvedModel, usage), ...turn, }); + /** Routes every ping for `alias` through the hooks with a diverging first message; returns that message and the sent bodies. */ + const capturePings = (host: MockCtx, alias: string) => { + const diverging = { role: 'user', content: [{ type: 'text', text: 'rewritten 0' }] }; + const pings: Record[] = []; + host.generate.mockImplementation(async (input: any) => { + const body = { + ...sampleBody(), + model: alias, + messages: [diverging, { role: 'user', content: [{ type: 'text', text: PING_PROMPT }] }], + } as Record; + const ping = await roundTrip( + host, + proxyTurn(alias, READ_HIT, 'claude-haiku-5-5', { + sessionID: input.sessionID, + kind: 'generate', + body, + }) + ); + pings.push((await parsedBodyOf(ping.requestEvt.request)) as Record); + return {}; + }); + return { diverging, pings }; + }; test('a DeepSeek session is observed but never pings', async () => { const host = await start({ keepalive: { minContextTokens: 0 } }); @@ -634,25 +657,7 @@ describe('keepalive wiring', () => { test('an alias is untouched on turn 1, then rewritten and kept warm once its response names Claude', async () => { const host = await start({ defaultMode: '1h', keepalive: { minContextTokens: 0 } }); const alias = 'auto/fast'; - const diverging = { role: 'user', content: [{ type: 'text', text: 'rewritten 0' }] }; - const pings: Record[] = []; - host.generate.mockImplementation(async (input: any) => { - const body = { - ...sampleBody(), - model: alias, - messages: [diverging, { role: 'user', content: [{ type: 'text', text: PING_PROMPT }] }], - } as Record; - const ping = await roundTrip( - host, - proxyTurn(alias, READ_HIT, 'claude-haiku-5-5', { - sessionID: input.sessionID, - kind: 'generate', - body, - }) - ); - pings.push((await parsedBodyOf(ping.requestEvt.request)) as Record); - return {}; - }); + const { diverging, pings } = capturePings(host, alias); const turn = proxyTurn(alias, WRITE_1H, 'claude-haiku-5-5'); const raw = JSON.stringify(turn.body); @@ -686,6 +691,29 @@ describe('keepalive wiring', () => { expect(lastSnapshot(host)).toMatchObject({ lastPing: 'hit', pingsUsed: 1 }); }); + test('the first ping after a learned alias is aligned even when time passed since the implicit turn', async () => { + const host = await start({ defaultMode: '1h', keepalive: { minContextTokens: 0 } }); + const alias = 'auto/aged'; + const { diverging, pings } = capturePings(host, alias); + + await roundTrip(host, proxyTurn(alias, WRITE_1H, 'claude-haiku-5-5')); + await flush(); + expect(lastSnapshot(host)).toMatchObject({ skipReason: 'implicit' }); + + // The request below changes lastRequestAt, so the engine's synchronous commit is a real emission. + jest.advanceTimersByTime(60_000); + const second = await roundTrip(host, proxyTurn(alias, WRITE_1H, 'claude-haiku-5-5')); + await flush(); + const primary = (await parsedBodyOf(second.requestEvt.request)) as Record; + + jest.advanceTimersByTime(FIFTY_MIN); + await flush(); + + expect(host.generate).toHaveBeenCalledTimes(1); + expect(pings[0]?.messages[0]).toEqual(primary.messages[0]); + expect(pings[0]?.messages[0]).not.toEqual(diverging); + }); + test('a learned alias whose response names a non-Claude model stops pinging and is left alone next turn', async () => { const host = await start({ defaultMode: '1h', keepalive: { minContextTokens: 0 } }); const alias = 'auto/best'; diff --git a/tests/services/fetch-json.test.ts b/tests/services/fetch-json.test.ts index 32d578f..ea0216d 100644 --- a/tests/services/fetch-json.test.ts +++ b/tests/services/fetch-json.test.ts @@ -21,11 +21,11 @@ describe('fetchJson', () => { test('aborts the request once the timeout passes', async () => { fetchSpy.mockResolvedValue(Response.json({})); - await fetchJson('https://example.test/api.json', 5); + await fetchJson('https://example.test/api.json', 50); const signal = fetchSpy.mock.calls[0]?.[1]?.signal; expect(signal).toBeInstanceOf(AbortSignal); expect(signal?.aborted).toBe(false); - await Bun.sleep(20); + await Bun.sleep(100); expect(signal?.aborted).toBe(true); }); diff --git a/tests/services/keepalive-engine.test.ts b/tests/services/keepalive-engine.test.ts index 0002659..0b09203 100644 --- a/tests/services/keepalive-engine.test.ts +++ b/tests/services/keepalive-engine.test.ts @@ -1161,13 +1161,16 @@ describe('keepalive engine: implicit-cache sessions', () => { test('a request with a TTL does not emit the implicit skip reason it is about to clear', async () => { const h = harness(); - await h.request('s1', pureRead, null); + await h.request('s1', write1h, null); await h.advance(MIN); const before = h.changes.length; h.engine.requestSent('s1', { requestedTtl: '1h', isPing: false, kind: 'primary' }); const emitted = h.changes.slice(before); expect(emitted).toHaveLength(1); expect(emitted[0]?.skipReason).toBeNull(); + await h.flush(); + expect(h.engine.snapshot('s1')).toMatchObject({ skipReason: null }); + expect(h.engine.snapshot('s1').nextPingAt).not.toBeNull(); }); test('a request without a TTL keeps an implicit session implicit', async () => { @@ -1361,6 +1364,15 @@ describe('keepalive engine: savings', () => { return h; } + /** A split turn with `turn`, a hit ping 4m later, then a split return with `back` past the original expiry. */ + async function splitPingAndReturn(h: H, turn: CacheUsage, back: CacheUsage = pureRead) { + await h.request('s1', turn, '5m', { split: true }); + await h.advance(4 * MIN); + expect(h.engine.snapshot('s1').lastPing).toBe('hit'); + await h.advance(3 * MIN); + await h.request('s1', back, '5m', { split: true }); + } + describe('rescued stretches', () => { test('a warm return after the original expiry counts its cache reads at the stretch TTL', async () => { const h = await pingedStretch(); @@ -1448,7 +1460,7 @@ describe('keepalive engine: savings', () => { test('a hit ping of an earlier stretch does not carry over', async () => { const h = await pingedStretch(); await h.advance(MIN / 2); - await h.request('s1', pureRead, '5m', { kind: 'title' }); + await h.request('s1', pureRead, '5m'); // Past the original expiry and its margin, before the next ping. await h.advance(2 * MIN); expect(h.pings).toHaveLength(1); @@ -1456,6 +1468,15 @@ describe('keepalive engine: savings', () => { expect(rescues(h)).toEqual([]); }); + test('a title call inside the stretch neither ends it nor overtakes the return', async () => { + const h = await pingedStretch(); + await h.advance(MIN / 2); + await h.request('s1', pureRead, '5m', { kind: 'title' }); + await h.advance(2 * MIN); + await h.request('s1', pureRead); + expect(rescues(h)).toEqual([['stretchRescued', 's1', priced(), 80_000, '5m', 80_100]]); + }); + test('a return that fails is no rescue, and neither is the request after it', async () => { const h = await pingedStretch(); await h.advance(3 * MIN); @@ -1509,11 +1530,7 @@ describe('keepalive engine: savings', () => { await h.request('s1', hybridWrite, '5m', { split: true }); await h.advance(MIN); // No 1h write on this turn: the prefix stays at 20_000. - await h.request('s1', pureRead, '5m', { split: true }); - await h.advance(4 * MIN); - expect(h.engine.snapshot('s1').lastPing).toBe('hit'); - await h.advance(3 * MIN); - await h.request('s1', pureRead, '5m', { split: true }); + await splitPingAndReturn(h, pureRead); expect(rescues(h)).toEqual([['stretchRescued', 's1', priced(), 60_000, '5m', 80_100]]); }); @@ -1529,11 +1546,7 @@ describe('keepalive engine: savings', () => { write1hTokens: 2000, write5mTokens: 60_000, }; - await h.request('s1', partial, '5m', { split: true }); - await h.advance(4 * MIN); - expect(h.engine.snapshot('s1').lastPing).toBe('hit'); - await h.advance(3 * MIN); - await h.request('s1', pureRead, '5m', { split: true }); + await splitPingAndReturn(h, partial); expect(rescues(h)).toEqual([['stretchRescued', 's1', priced(), 60_000, '5m', 80_100]]); }); @@ -1544,6 +1557,37 @@ describe('keepalive engine: savings', () => { expect(rescues(h)).toEqual([]); }); + test('Claude hybrid on a wire without the 1h split estimates the prefix from the first primary context', async () => { + const h = harness({ requestedTtl: '5m' }); + const first: CacheUsage = { + ...writeNoBreakdown, + inputTokens: 100, + cacheWriteTokens: 19_900, + }; + await h.request('s1', first, '5m', { split: true }); + await h.advance(MIN); + await splitPingAndReturn(h, writeNoBreakdown, hit); + expect(rescues(h)).toEqual([ + ['stretchRescued', 's1', priced(), 80_100 - 20_000, '5m', 80_200], + ]); + }); + + test('a 1h-mode full write is not a prefix for a later hybrid turn', async () => { + const h = harness({ requestedTtl: '1h' }); + const full: CacheUsage = { + ...hybridWrite, + cacheWriteTokens: 80_000, + write5mTokens: 0, + write1hTokens: 80_000, + }; + await h.request('s1', full, '1h'); + await h.advance(MIN); + await h.request('s1', hybridWrite, '5m', { split: true }); + await h.advance(MIN); + await splitPingAndReturn(h, pureRead); + expect(rescues(h)).toEqual([['stretchRescued', 's1', priced(), 60_000, '5m', 80_100]]); + }); + test('a split stretch stays split when the return is not', async () => { const h = harness({ requestedTtl: '5m' }); await h.request('s1', write5m, '5m', { split: true }); @@ -1657,17 +1701,26 @@ describe('keepalive engine: savings', () => { expect(h.ledger.calls.map(([call]) => call)).toEqual(['pingCharged', 'stretchStarted']); }); - test('child and implicit sessions make no ledger calls and load no ledger', async () => { + test('child sessions make no ledger calls and load no ledger', async () => { const h = harness({ child: true }); await h.request('s1', write1h); await h.advance(60 * MIN); await h.request('s1', pureRead); + expect(h.pings).toHaveLength(0); + expect(h.ledger.calls).toEqual([]); + expect(h.ledger.readoutCalls).toEqual([]); + expect(h.engine.snapshot('s1').savings).toBeNull(); + }); + + test('implicit sessions make no ledger calls and load no ledger', async () => { + const h = harness(); await h.request('s2', write1h, null); + await h.advance(60 * MIN); await h.request('s2', pureRead, null); expect(h.pings).toHaveLength(0); expect(h.ledger.calls).toEqual([]); expect(h.ledger.readoutCalls).toEqual([]); - expect(h.engine.snapshot('s1').savings).toBeNull(); + expect(h.engine.snapshot('s2').savings).toBeNull(); }); }); diff --git a/tests/services/ledger-store.test.ts b/tests/services/ledger-store.test.ts index 496b8e9..b8b997e 100644 --- a/tests/services/ledger-store.test.ts +++ b/tests/services/ledger-store.test.ts @@ -34,11 +34,19 @@ describe('LedgerStore', () => { expect(totals.pings).toBe(5); }); - test('update never mutates the previous totals', async () => { - const { kv } = memoryKv(); + test('change receives the stored totals unmodified and not aliased', async () => { + const { kv, data } = memoryKv(); + const stored = { ...EMPTY_TOTALS, pings: 4 }; + data.set('ledger/ses_a', stored); const store = new LedgerStore(kv); - await store.update('ses_a', addPing); - expect(EMPTY_TOTALS.pings).toBe(0); + let received: LedgerTotals | undefined; + await store.update('ses_a', (totals) => { + received = totals; + return { ...totals, pings: 9 }; + }); + expect(received).toEqual({ ...EMPTY_TOTALS, pings: 4 }); + expect(received).not.toBe(stored); + expect(stored.pings).toBe(4); }); test.each([ @@ -69,6 +77,44 @@ describe('LedgerStore', () => { expect(await new LedgerStore(kv).load('ses_a')).toEqual(EMPTY_TOTALS); }); + test('a failed read is not cached and does not clobber the stored record', async () => { + const { kv, data } = memoryKv(); + data.set('ledger/ses_a', { ...EMPTY_TOTALS, pings: 40 }); + const original = kv.get; + let failNext = true; + kv.get = async (key) => { + if (failNext) { + failNext = false; + throw new Error('transient'); + } + return original(key); + }; + const store = new LedgerStore(kv); + await store.update('ses_a', addPing); + expect((data.get('ledger/ses_a') as LedgerTotals).pings).toBe(40); + expect(store.peek('ses_a')).toBeUndefined(); + const after = await store.update('ses_a', addPing); + expect(after.pings).toBe(42); + expect((data.get('ledger/ses_a') as LedgerTotals).pings).toBe(42); + }); + + test('a synchronously throwing get leaves no stale load, so a later load retries', async () => { + const { kv, data } = memoryKv(); + data.set('ledger/ses_a', { ...EMPTY_TOTALS, pings: 7 }); + const original = kv.get; + let throwNext = true; + kv.get = (key) => { + if (throwNext) { + throwNext = false; + throw new Error('sync'); + } + return original(key); + }; + const store = new LedgerStore(kv); + expect(await store.load('ses_a')).toEqual(EMPTY_TOTALS); + expect((await store.load('ses_a')).pings).toBe(7); + }); + test('concurrent loads share one read', async () => { const { kv } = memoryKv(); let reads = 0; diff --git a/tests/services/model-prices.property.test.ts b/tests/services/model-prices.property.test.ts new file mode 100644 index 0000000..47a12d4 --- /dev/null +++ b/tests/services/model-prices.property.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, test } from 'bun:test'; + +import fc from 'fast-check'; + +import { selectTier, toCatalogModels, type CatalogCost } from '../../src/services/model-prices.ts'; + +const tieredCost: fc.Arbitrary = fc.record({ + cache: fc.record({ read: fc.nat(), write: fc.nat() }), + input: fc.nat(), + output: fc.nat(), + tier: fc.record({ size: fc.nat({ max: 1_000_000 }), type: fc.constant('context' as const) }), +}); + +const anyCost: fc.Arbitrary = fc.oneof( + tieredCost, + fc.record({ + cache: fc.record({ read: fc.nat(), write: fc.nat() }), + input: fc.nat(), + output: fc.nat(), + }) +); + +describe('toCatalogModels properties', () => { + test('never throws and only returns entries with string ids', () => { + fc.assert( + fc.property(fc.jsonValue(), (data) => { + for (const entry of toCatalogModels(data)) { + expect(typeof entry.id).toBe('string'); + expect(typeof entry.providerID).toBe('string'); + } + }) + ); + }); +}); + +describe('selectTier properties', () => { + test('returns a member of its input, or undefined for no input', () => { + fc.assert( + fc.property(fc.array(anyCost), fc.nat({ max: 2_000_000 }), (costs, tokens) => { + const picked = selectTier(costs, tokens); + if (picked === undefined) { + expect(costs).toHaveLength(0); + } else { + expect(costs).toContain(picked); + } + }) + ); + }); + + test('the selected tier size never shrinks as the context grows', () => { + fc.assert( + fc.property( + fc.array(tieredCost, { minLength: 1 }), + fc.nat({ max: 2_000_000 }), + fc.nat({ max: 2_000_000 }), + (costs, first, second) => { + const [low, high] = first <= second ? [first, second] : [second, first]; + const sizeAt = (tokens: number): number => selectTier(costs, tokens)?.tier?.size ?? -1; + const smallest = Math.min(...costs.map((entry) => entry.tier?.size ?? 0)); + // Below every tier the smallest tier is the fallback, so compare only where some tier fits. + fc.pre(low >= smallest); + expect(sizeAt(high)).toBeGreaterThanOrEqual(sizeAt(low)); + } + ) + ); + }); +}); diff --git a/tests/services/model-prices.test.ts b/tests/services/model-prices.test.ts index a1da1a8..1c5e9ad 100644 --- a/tests/services/model-prices.test.ts +++ b/tests/services/model-prices.test.ts @@ -1,13 +1,14 @@ import { describe, expect, spyOn, test } from 'bun:test'; -import type { ModelPrice } from '../../src/models/savings'; +import { CATALOG_RETRY_MS } from '../../src/constants/savings.ts'; +import type { ModelPrice } from '../../src/models/savings.ts'; import { PriceBook, selectTier, toCatalogModels, type CatalogCost, type CatalogModel, -} from '../../src/services/model-prices'; +} from '../../src/services/model-prices.ts'; const cost = (input: number, size?: number, output = input * 5): CatalogCost => ({ ...(size === undefined ? {} : { tier: { type: 'context' as const, size } }), @@ -317,3 +318,59 @@ describe('toCatalogModels', () => { expect(toCatalogModels({ data: [info()] })).toEqual([]); }); }); + +describe('PriceBook reload backoff', () => { + const other = { providerID: 'anthropic', id: 'other' }; + + function setup(results: (() => Promise)[]) { + const clock = { t: 1_000_000 }; + let calls = 0; + const book = new PriceBook({ + listModels: () => { + const next = results[Math.min(calls, results.length - 1)]; + calls += 1; + return next?.() ?? Promise.resolve([]); + }, + now: () => clock.t, + }); + return { book, calls: () => calls, clock }; + } + + test('a failed reload is not cached: the model is found once the backoff passes', async () => { + const { book, calls, clock } = setup([ + () => Promise.resolve([model('claude-x', [cost(3)])]), + () => Promise.reject(new Error('down')), + () => Promise.resolve([model('claude-x', [cost(3)]), model('other', [cost(4)])]), + ]); + expect((await book.lookup(ref, 1))?.input).toBe(3); + expect(await book.lookup(other, 1)).toBeUndefined(); + clock.t += CATALOG_RETRY_MS; + expect((await book.lookup(other, 1))?.input).toBe(4); + expect(calls()).toBe(3); + }); + + test('failed reloads inside the backoff do not call listModels again', async () => { + const { book, calls } = setup([ + () => Promise.resolve([model('claude-x', [cost(3)])]), + () => Promise.reject(new Error('down')), + ]); + await book.lookup(ref, 1); + await book.lookup(other, 1); + await book.lookup(other, 1); + await book.lookup({ providerID: 'anthropic', id: 'third' }, 1); + expect(calls()).toBe(2); + }); + + test('concurrent lookups of one missing model share the in-flight reload', async () => { + const { book, calls } = setup([ + () => Promise.resolve([model('claude-x', [cost(3)])]), + async () => { + await Bun.sleep(5); + throw new Error('down'); + }, + ]); + await book.lookup(ref, 1); + await Promise.all([book.lookup(other, 1), book.lookup(other, 1)]); + expect(calls()).toBe(2); + }); +}); diff --git a/tests/services/models-dev.property.test.ts b/tests/services/models-dev.property.test.ts new file mode 100644 index 0000000..11160c7 --- /dev/null +++ b/tests/services/models-dev.property.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, test } from 'bun:test'; + +import fc from 'fast-check'; + +import { normaliseModelName } from '../../src/services/models-dev.ts'; + +describe('normaliseModelName properties', () => { + test('never throws and returns lowercase', () => { + fc.assert( + fc.property(fc.string(), (name) => { + const result = normaliseModelName(name); + expect(result).toBe(result.toLowerCase()); + }) + ); + }); + + test('strips a -latest or -YYYYMMDD suffix from a plain name', () => { + const base = fc.stringMatching(/^[a-z][a-z0-9.]{0,12}$/u); + const suffix = fc.oneof( + fc.constant('-latest'), + fc.integer({ min: 10_000_000, max: 99_999_999 }).map((date) => `-${date}`) + ); + fc.assert( + fc.property(base, suffix, (name, tail) => { + expect(normaliseModelName(`${name}${tail}`)).toBe(name); + }) + ); + }); +}); diff --git a/tests/services/models-dev.test.ts b/tests/services/models-dev.test.ts index 3d5a864..d7d8e71 100644 --- a/tests/services/models-dev.test.ts +++ b/tests/services/models-dev.test.ts @@ -140,6 +140,18 @@ describe('ModelsDevPrices.match', () => { expect(calls).toHaveLength(2); }); + test('a 200 body with no priced models keeps the cache and the stored copy, and backs off', async () => { + const { calls, clock, data, prices } = setup([BODY, { error: 'x' }]); + expect(await prices.match(['claude-sonnet-4-5'])).toEqual(SONNET); + const stored = JSON.stringify(data.get('models-dev')); + clock.t += MODELS_DEV_MAX_AGE_MS; + expect(await prices.match(['claude-sonnet-4-5'])).toEqual(SONNET); + expect(calls).toHaveLength(2); + expect(JSON.stringify(data.get('models-dev'))).toBe(stored); + await prices.match(['claude-sonnet-4-5']); + expect(calls).toHaveLength(2); + }); + test('concurrent matches share one fetch', async () => { const { calls, prices } = setup(); await Promise.all([prices.match(['a']), prices.match(['b'])]); diff --git a/tests/ui/sidebar-lines.test.ts b/tests/ui/sidebar-lines.test.ts index c15bc48..c1d6187 100644 --- a/tests/ui/sidebar-lines.test.ts +++ b/tests/ui/sidebar-lines.test.ts @@ -332,6 +332,12 @@ describe('sidebarLines: savings rows', () => { ); }); + test('a negative net shows a signed dollar figure while rescues are counted', () => { + expect(rows(warm(readout({ netUsd: -0.05, returnRatio: 0.5 })))[1]).toBe( + 'saved -$0.05 · 2 rewrites' + ); + }); + test('no return row without a ratio', () => { expect(rows(warm(readout({ returnRatio: null })))).toHaveLength(2); });