Skip to content

docs: record Data's model and the AI Gateway provider - #20

Closed
EthanThatOneKid wants to merge 2 commits into
mainfrom
docs/zo-credit-model
Closed

EthanThatOneKid wants to merge 2 commits into
mainfrom
docs/zo-credit-model

Conversation

@EthanThatOneKid

@EthanThatOneKid EthanThatOneKid commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Data's inference was failing on provider capacity, not on a broken service: the OpenRouter free pool's daily cap and Google's per-minute free-tier limit each surfaced as a 502 from /ask, with no fallback.

The agent now runs openai-compatible/openai/gpt-5-mini through a Vercel AI Gateway provider registered at https://ai-gateway.vercel.sh/v1, so turns are metered to the account's AI credits rather than a separate provider key.

Recorded, because the obvious route does not work:

  • The local backend's own vercel-ai-gateway/* handles fail on every turn with 400 max_tokens: Invalid input: expected number, received null — that provider speaks the Anthropic Messages API, where max_tokens is required, and its catalog entries carry a context window but no max-output field. Reproduced on this host before switching.
  • The openai-compatible route sets max_tokens itself (32000 here), and the same gateway key works through it.
  • The provider credential is host-local (~/.letta/lc-local-backend/providers/), so the service needs no gateway key in its environment.

Verified live: a direct POST /v1/chat/completions to the gateway returned 200; a headless Letta turn on the agent answered; and POST /ask on the restarted data-http returned 200 with an answer. No credential value is recorded in either file.

Data's inference now runs through an openai-compatible provider pointed at the
Vercel AI Gateway, so turns are metered to the account's credits instead of a
separate provider key. Record why the local backend's own vercel-ai-gateway/*
handles cannot be used (Anthropic Messages API requires max_tokens, and the
catalog carries no max-output field, so every turn fails with
'max_tokens: Invalid input: expected number, received null').
@wazoocomputer

wazoocomputer Bot commented Sep 26, 2026

Copy link
Copy Markdown

This pull request updates the documentation to record Data's current model and AI Gateway provider. The change addresses past issues with provider capacity by switching to a Vercel AI Gateway model, openai-compatible/google/gemini-2.5-flash-lite, which meters turns against the account's AI credits instead of individual provider keys. The documentation explains the setup for this new model and clarifies why the local backend's own vercel-ai-gateway/* handles are not suitable.

File Change Summary
channels/http/README.md modified Added a new section detailing the agent's model, provider, and setup instructions.
services/http-api.md modified Updated to reference the new AI Gateway model and provide a historical record of the change.

Review the added documentation for accuracy and clarity regarding the model and provider configuration.

@wazoocomputer

wazoocomputer Bot commented Sep 26, 2026

Copy link
Copy Markdown

This pull request updates the documentation to record Data's current model and AI Gateway provider. The change addresses past issues with provider capacity by switching to a Vercel AI Gateway model, openai-compatible/google/gemini-2.5-flash-lite, which meters turns against the account's existing AI credits instead of individual provider keys. The documentation explains the setup for this new model and clarifies why the local backend's own vercel-ai-gateway/* handles are not suitable.

File Change Summary
channels/http/README.md modified Added a new section detailing the agent's model, provider, and setup instructions.
services/http-api.md modified Updated to reference the new AI Gateway model and provide a historical record of the change.

Review the added documentation for accuracy and clarity regarding the model and provider configuration.

@EthanThatOneKid

Copy link
Copy Markdown
Contributor Author

Superseded by #22. This documents the Letta model and provider setup; that whole configuration moved to archives/letta-brain/services/http-api.md with the service it described. Closing.

@EthanThatOneKid
EthanThatOneKid deleted the docs/zo-credit-model branch September 26, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant