feat(ai): a second Groq link, on its own meter - #1012
Merged
Conversation
The standing rule in this codebase is that a same-vendor fallback is not one — a
second model at the same vendor draws on the same daily budget, so it is already
dead when that budget is spent. Groq measurably does NOT work that way. Read
from response headers on one key, within the same minute, 2026-09-13:
openai/gpt-oss-20b 591 / 1000 requests remaining (what production runs)
openai/gpt-oss-120b 999 / 1000
qwen/qwen3.8-27b 999 / 1000
Separate counters, and separate 8000-token minutes. So a second Groq model draws
on a budget the first one cannot spend. That is real headroom, not a longer
queue for the same allowance.
It matters most exactly when it is reached. The vendor after Groq is
OpenRouter's free tier, and that tier was answering every model with "Rate limit
exceeded: free-models-per-day" the same afternoon — the allowance is 50 requests
a day without credits. Leaving Groq early is expensive right now.
CHOSEN BY PROBE, NOT BY SPEC SHEET, and that changed the answer.
`groq/compound-mini` advertises 70000 TPM — nearly nine times the others, and
the obvious pick for Cat's prompt, which the 8000 TPM window squeezes. It
answers:
`tool calling` is not supported with this model
Cat drives a tool loop, so all of that headroom is unreachable.
`qwen/qwen3.8-27b` returned a native tool_call for a real function definition,
which is the bar. It was already in GROQ_TPM_LIMIT_BY_MODEL at 8000, so the
pre-flight sizes it correctly with no guess.
The pair is de-duplicated, so setting PLATFORM_GROQ_FALLBACK_MODEL to the
primary's id disables the second link with an env var and no deploy.
Verified by mutation, each reverted separately with the control green:
only the primary pushed (feature removed) -> 1 test fails
fallback points at the same model -> 2 tests fail
de-duplication dropped -> 1 test fails
One of the seven tests was decoration on the first pass — it asserted an inline
copy of the de-duplication rule and would have passed with the feature deleted.
It now re-imports the module under a changed env and checks the chain it
actually builds.
2962 unit tests pass (7 new), type-check, lint and file sizes clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJRuvHJEBd8t7iRA9Sb1iw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The standing rule in this codebase is that a same-vendor fallback is not one — a second model at the same vendor draws on the same daily budget, so it's already dead when that budget is spent.
Groq measurably does not work that way. Read from response headers on one key, within the same minute:
openai/gpt-oss-20bopenai/gpt-oss-120bqwen/qwen3.8-27bSeparate counters, and separate 8000-token minutes. So a second Groq model draws on a budget the first one cannot spend. That's real headroom, not a longer queue for the same allowance.
It matters most exactly when it's reached: the vendor after Groq is OpenRouter's free tier, and that tier was answering every model with
Rate limit exceeded: free-models-per-daythe same afternoon — 50 requests/day without credits. Leaving Groq early is expensive right now.Chosen by probe, not by spec sheet — and that changed the answer
groq/compound-miniadvertises 70,000 TPM, nearly nine times the others, and was the obvious pick for Cat's prompt, which the 8000 TPM window squeezes. It answers:Cat drives a tool loop, so all of that headroom is unreachable.
qwen/qwen3.8-27breturned a nativetool_callfor a real function definition, which is the bar. It was already inGROQ_TPM_LIMIT_BY_MODELat 8000, so the pre-flight sizes it correctly with no guess.The pair is de-duplicated, so setting
PLATFORM_GROQ_FALLBACK_MODELto the primary's id disables the second link with an env var and no deploy.Verified by mutation
Each reverted separately, control green:
One of the seven tests was decoration on the first pass — it asserted an inline copy of the de-duplication rule and would have passed with the feature deleted. It now re-imports the module under a changed env and checks the chain it actually builds.
2962 unit tests pass (7 new), type-check, lint and file-size checks clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HJRuvHJEBd8t7iRA9Sb1iw