Skip to content

feat: BYOK provider routing, token metering, and the model catalog - #31

Merged
man4ish merged 1 commit into
mainfrom
feature/m16-byok-provider-routing
Oct 4, 2026
Merged

man4ish merged 1 commit into
mainfrom
feature/m16-byok-provider-routing

Conversation

@man4ish

@man4ish man4ish commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes design audit gap #4 in full, as three paired PRs across omnibioai-auth (key-reveal endpoint), omnibioai-rag (actual Claude/OpenAI client routing), and this repo (request validation, the reveal-and-forward step, token-event emission, and the model catalog).

  • build_rag_query now accepts model="claude"/"openai" -- only together with use_own_key: true, since there is no platform-wide key for either provider, only an organization's own BYOK key. Either alone is rejected with a clear field pointing at whichever one is missing.
  • _handle_billable_literature_call reveals the organization's decrypted key from omnibioai-auth's new internal endpoint immediately before forwarding a BYOK-routed call -- resolved before the concurrency slot and quota are touched, so a request that can't get a key never consumes either. The key is attached to the one RAG request body only; never stored, logged, or visible in the headers forwarded downstream.
  • A BYOK-routed call that gets real token counts back from RAG also emits llm.tokens.input/llm.tokens.output usage events (V1Store.emit_usage gained quantity/unit params for this, defaulting to the unchanged 1/"requests") -- never fabricated for the default path, where these stay absent exactly as before.
  • model_source/token counts in the public /v1/literature/answers response now relay whatever RAG actually reports, instead of the old hardcoded "omnibioai_gpu"/None/None.
  • GET /v1/models lists claude/openai alongside default, price: null for all three -- BYOK means an org pays its own provider directly; this platform still charges nothing on top (same "don't invent a dollar figure" rule as everywhere else this gap has touched pricing).

Test plan

  • pytest tests/test_literature_contract.py -- 57 passed, including the full BYOK validation matrix (model without use_own_key, use_own_key without a BYOK model, both together accepted and passed through, default path's RAG body has no model key at all)
  • pytest tests/test_v1_literature.py -- 78 passed (71 existing + 7 new: reveal call + RAG forward with the key attached, 503/400/502 error mapping for reveal failures, token-usage event emission alongside the answer event, confirming the default path never emits token events, and confirming /search still ignores model/use_own_key entirely)
  • Full repo suite -- 406 passed

🤖 Generated with Claude Code

Closes design audit gap #4 in full (paired with omnibioai-auth's
key-reveal endpoint and omnibioai-rag's Claude/OpenAI client routing):

- build_rag_query accepts model="claude"/"openai" only together with
  use_own_key: true -- there is no platform-wide key for either
  provider, only an organization's own BYOK key.
- _handle_billable_literature_call reveals the organization's
  decrypted key from omnibioai-auth immediately before forwarding a
  BYOK-routed call to RAG (never stored, never logged, attached to
  that one request body only), resolved before the concurrency slot
  and quota are touched so a request that can't get a key never
  consumes either.
- A BYOK-routed call that gets real token counts back from RAG also
  emits llm.tokens.input/llm.tokens.output usage events (V1Store.
  emit_usage gains quantity/unit params for this) -- never fabricated
  for the default path, where these stay absent exactly as before.
  model_source/token counts in the public response now relay whatever
  RAG actually reports, rather than the old hardcoded "omnibioai_gpu"/
  None/None.
- GET /v1/models lists claude/openai alongside the default, price:
  null for all three (BYOK means an org pays its own provider
  directly; this platform still charges nothing on top).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@man4ish
man4ish merged commit b32ade5 into main Oct 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant