Skip to content

Improve model usage accounting across providers - #1887

Merged
AbirAbbas merged 8 commits into
devfrom
fix/model-usage-accounting
Oct 21, 2026
Merged

AbirAbbas merged 8 commits into
devfrom
fix/model-usage-accounting

Conversation

@santoshkumarradha

Copy link
Copy Markdown
Member

Local and custom model endpoints could finish calls without token usage, leaving gaps in usage accounting. This improves compatible-provider handling with requested streaming receipts and clearly separated estimates when measured counts are unavailable.

  • Request streamed usage, with a narrowly matched fallback for older APIs that reject the option.
  • Preserve reported zeroes, partial counts, total-only receipts and native input/output aliases.
  • Estimate missing text components across providers, including observed tool definitions, tool calls and visible reasoning. Mark interrupted output as partial; leave media and unseen work unknown.
  • Keep estimates separate from measured counts and billing. A hashed receipt key lets later provider measurements replace estimates in analytics without adding both.
  • Cover raw streaming calls as well as normal chat/task calls, preserve Ollama connection identity, and document the contract and receipt-aware PostHog query.

Companion: https://github.com/Agent-Field/website-open-infra/pull/82. Deploy that relay change first, then release this client change. Existing measured-only charts remain valid.

Validation: focused provider/config/accounting regressions, privacy/allowlist and disclosure checks, the exact shared client-to-relay fixture, a 50-event body-limit check, production build, vet, and formatting pass. The companion relay's 35 route tests pass with PostHog mocked. No synthetic production events were sent.

The broad local test gate was stopped by automatic approval review when it attempted an OpenRouter request with an unestablished payload. Targeted mock-based checks and build/vet completed; CI remains the full changed-package check.

santoshkumarradha and others added 8 commits October 21, 2026 16:36
Streamed requests had gained stream_options on the default router road,
changing its established wire bytes. Only connected services now ask for
that option, following the actual endpoint after a billing-road change.
The router retains its existing usage.include request.

The usage option had also admitted unrelated streamed 400s to learn(),
and a connected endpoint refusing it with 422 failed a call dev answered.
Keep the old repair eligibility independent of the usage option and retry
only an explicit unsupported-option refusal on 400 or 422. Remember that
refusal for the endpoint without changing unrelated errors.

Cover router and connected wire shapes, fallback and refusal memoization,
dev error/request-count parity, raw-stream single settlement, and the
separation between token estimates and billing. Update the manual and
usage disclosures with the precise roads and refusal statuses.

Ollama 0.9.0 reuses generation ids from a small random range. The former
receipt key collapsed separate answers and reused event ids with different
counts. Include the physical attempt in that key, and carry it through late
recovery so estimates still pair with their measured receipt. Cover repeated
provider ids and raw-stream recovery without duplicate rows or bills.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously a caller's status could disagree with emitted components, unsafe
integers could reach the relay, and late receipts could turn absent counts
into measured zeroes. Derive status from safe emitted components, omit invalid
counts and unsafe sums, and require a receipt identity for estimates.

Keep late token presence through a session-bound recorder. Cost-only recovery
emits no token update; one-sided recovery stays partial. Extend the shared
fixture and test its outputs against the relay's validation rules.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously whitespace generation ids split recovery from observation, repeated
tool names inflated estimates, opaque replay counted as input text, and a
metadata-only cut invented observed zero output. Canonicalize the generation
inside the physical-attempt receipt helper and count only honest observations.

Compose independent repairs once each without widening unrelated refusal
handling. Associate usage-option refusal wording with that option. Preserve
late token presence and recovered prices. Keep unmetered direct-service
answers off receipt lookup because recovery retains the original bearer.

Prove native cache aliases match canonical billing counts and token pricing;
no estimate is banked or priced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Previously the disclosure omitted the total-only exception and overstated
measurement and recovery behavior. Describe component-preserving recovery,
canonical receipt identities, safe numeric limits, opaque input, empty output,
and bounded composed repairs.

Record the deliberate native-alias measured-count change and unchanged
no-lookup behavior for unmetered direct-service overflow. Estimates remain
separate from billing, response usage and spend.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AbirAbbas

Copy link
Copy Markdown
Collaborator

Took this over to merge-ready. Four commits on top of Santosh's, each with tests named for the behaviour they prove:

  • d506b8c — the red: stream_options went on every streamed request, including the default OpenRouter road. It now goes only to connected services, following the actual attempt's road, and the byte-for-byte wire law passes unedited. A service refusing the option with 422 failed a call dev answered; an explicit refusal on 400/422 now retries once and is remembered per endpoint, while an unrelated 400/422 keeps dev's exact error and request count. Ollama 0.9.0 picks completion ids from a range of 999, so the receipt key collapsed different answers into one event id with different counts; the key now includes the physical attempt.
  • 78448c0 — UsageReceipt derives the status from the counts it emits, never turns an unsafe or negative count into a measured zero, and emits estimates only with a receipt, so every shape it can produce passes the relay's validator. The shared fixture grew to 24 events (relay copy in website-open-infra Onboarding asks for spend limits (daily, per plan, per task, standing) with 'no limit' as a first-class option #83).
  • 0928ce1 — the generation id is canonicalised the same way for the estimate and the late receipt; repeated tool names count once; opaque reasoning sidecars no longer grow a text estimate; a role-only frame before a cut observes nothing; the stream-options repair composes with the existing repairs in either order, bounded once per kind; refusal matching must qualify the option itself; a late receipt keeps token presence (cost-only adds no token event; one count stays partial). The unmetered direct-service answer keeps dev's no-lookup behaviour, because that lookup would carry the direct vendor's key to the overflow host.
  • 9445d02 — docs, manual and change entry match the code, including the native input_tokens/output_tokens aliases now landing as measured counts (dev recorded 0/0).

Verified: CI green on this head; make pr-ready green; the real binary against a stub matrix (reported, none, zeros, total-only, one-sided, 400/422 refusals, unrelated 400, cut, reasoning, tool calls, repeated names, both repair orders, false-positive refusal, native cache aliases) and a local Ollama 0.9.0, compared against a dev build; every scenario that answered on dev still answers. 493 captured client events replayed through the relay route handler from website-open-infra #82: 0 dropped. No prompt, path, model, endpoint or key in any captured body.

Merge order stays as Santosh set it: this goes in once the relay deploy from #82 is confirmed live.

@AbirAbbas AbirAbbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified; see the comment above. Merge waits on the relay deploy confirmation.

@AbirAbbas
AbirAbbas merged commit 663dfbf into dev Oct 21, 2026
5 checks passed
@AbirAbbas
AbirAbbas deleted the fix/model-usage-accounting branch October 21, 2026 22:46
AbirAbbas added a commit that referenced this pull request Oct 21, 2026
Hotfix of #1887 onto v0.8.0, carrying only that change to main. Connected
services ask for streamed token counts with stream_options.include_usage
(the default router road keeps its existing usage request), an explicit
400/422 refusal of the option retries once and is remembered per endpoint,
and missing text counts gain separately labelled estimates that never enter
billing. Each physical answer keeps its own hashed receipt so a later
provider receipt replaces its estimate in analytics. The relay that accepts
the new usage statuses shipped as website-open-infra #82.

The settings page keeps v0.8.0's resident practice paragraph; the rest of
the change is byte-identical to #1887's head 9445d02 except where dev-only
commits had moved the surrounding lines.

Co-Authored-By: Abir Abbas <abirabbas1998@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas added a commit that referenced this pull request Oct 21, 2026
The v0.8.1 hotfix carries #1887 alone on top of v0.8.0, so its section
holds that change and the v0.8.0 reconciliation entry (#1825) that was
still loose on main. Both entries leave the unreleased folder here, and
the merge back into staging and dev removes them there too, so the next
roll-up on dev cannot list them a second time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas added a commit that referenced this pull request Oct 21, 2026
main receives #1887 alone as v0.8.1; merging that hotfix here keeps every
later staging-to-main promotion a fast-forward and lets the release guard
find the rc commit on staging. Where staging already matched dev, the
merged files are byte-identical to #1887's head.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AbirAbbas added a commit that referenced this pull request Oct 21, 2026
#1887 reached dev as 663dfbf and main as the v0.8.1 hotfix. This merge
brings that hotfix's history and changelog roll-up here, so dev keeps
containing staging and main. Code is unchanged; the v0.8.1 section lands in
CHANGELOG.md, and the #1825 and #1887 entries it rolled up leave the
unreleased folder so the next roll-up does not list them again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants