Repository navigation
feat(llm): add OpenAI Responses and Anthropic Messages API support - #88
Open
garfield-oo7 wants to merge 5 commits into
Open
garfield-oo7 wants to merge 5 commits into
garfield-oo7 wants to merge 5 commits into
Conversation
Needed for the Anthropic Messages API support in LLMClient.
body.get("usage", {}) returns None when the key is present with a null
value, which crashed token/cost aggregation. Fall back to {} instead.
Newer OpenAI models (e.g. gpt-6.1-sol, gpt-6-astra) reject function tools
with reasoning on /v1/chat/completions; they only support them on
/v1/responses. Add an `api` option (chat | responses | anthropic; default
chat, so OpenRouter usage is unchanged), settable via YAML, BALATROLLM_API
or --api.
LLMClient translates the existing Chat Completions request (messages,
TOOLS.json tools, model_config) for the chosen API and converts the reply
back into a ChatCompletion, so the bot, strategies and run output format
are unchanged and BalatroBench keeps reading runs as before:
- responses: input_text parts, flat function tools, reasoning.effort,
store=false; truncated/incomplete tool calls are dropped, failed
responses are retried, refusals are kept, chat-only params (e.g. seed)
are rejected with a clear error.
- anthropic: official anthropic SDK (reads ANTHROPIC_API_KEY when no key
is configured); cache_control is kept, adaptive thinking summaries are
stored as message.reasoning, parallel_tool_calls maps to
disable_parallel_tool_use.
- "openai/" (on api.openai.com) and "anthropic/" model prefixes are
stripped before the call, so runs are stored under the same
runs/.../{vendor}/{model}/ layout BalatroBench uses, and body.provider
is set (OpenAI / Anthropic).
- Neither API reports cost, so it is estimated from per-model prices
(pricing.py, sourced from the providers' pricing pages) into the
usage.cost / cost_details fields BalatroBench reads, marked estimated.
base_url now defaults to None: OpenRouter for chat/responses, the
Anthropic API for anthropic.
Author
|
Open items for this PR:
|
This was referenced Oct 20, 2026
Merged
OpenAI's implicit caching on GPT-5.6+ only places a breakpoint at the end of the prompt, so the static strategy prefix was never reused when the game state changed (0 cached tokens, full cache-write cost on every call). Marking the strategy part as an explicit breakpoint makes later calls read it from cache (~10.6k of ~10.6k prompt tokens cached in testing).
This was referenced Oct 21, 2026
Merged
garfield-oo7
added a commit
to garfield-oo7/balatrobench
that referenced
this pull request
Oct 21, 2026
5 real Balatro games (default strategy, RED/WHITE, seeds AAAAAAA-EEEEEEE x1, medium reasoning) via the Anthropic Messages API (coder/balatrollm#88 at deb2d1a; the AAAAAAA run also had the transport-error loop fix). Average final round 21.0, 3 wins (AAAAAAA, BBBBBBB, CCCCCCC), est. cost $23.86 for the 5 counted runs. AAAAAAA froze the game twice at ante 7 round 20 when a Celestial pack opened with Black Hole first (coder/balatrobot#199); the counted run used BalatroBot v1.5.2 with that fix applied to buy.lua and pack.lua. The About page notes this and the smaller run count. Leaderboard regenerated from all v1.1.1 runs (other models unchanged). Data compacted with compact_benchmarks.py (116 MB -> 17 MB).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Newer OpenAI models (
gpt-6.1-sol,gpt-6-astra) can't be benchmarked with BalatroLLM today. On/v1/chat/completionsthey reject function tools:and
reasoning_effort: "none"isn't allowed for these models either. On/v1/responsesboth models call tools correctly. This PR also adds the Anthropic Messages API, so Claude models can be run directly with an Anthropic key.What
A new
apioption:chat(default, unchanged),responses, oranthropic, set with--api,BALATROLLM_API, orapi:in YAML.LLMClienttranslates the existing Chat Completions request (messages,TOOLS.jsontools,model_config) for the chosen API, then converts the reply back into aChatCompletion. Strategies,bot.pyand the run output format don't change, so BalatroBench reads these runs the same way it reads existing ones:responsesanthropicruns/.../{vendor}/{model}/openai/...(prefix stripped before callingapi.openai.com)anthropic/...(prefix stripped before calling the API)body.providerOpenAIAnthropicrequests.jsonlbody.messagestool_calls,message.reasoningusage.prompt_tokens/completion_tokensusage.cost/cost_details*Neither API reports cost. It's estimated from per-model prices in
pricing.py(sourced from the providers' pricing pages, with long-context tiers) and markedcost_details.estimated: true. Models without a listed price get no cost. A cost that's already reported (e.g. OpenRouter's) is kept.Robustness:
seed) are rejected with a clear error.store: false.Also:
base_urlnow defaults toNone: OpenRouter forchat/responses, the Anthropic API foranthropic.fix(collector): handleusage: nullin responses, which previously crashed stats aggregation.Usage
balatrollm --api responses --base-url https://api.openai.com/v1 --model openai/gpt-6.1-sol balatrollm --api anthropic --model anthropic/claude-haiku-5-5 # reads ANTHROPIC_API_KEYTesting
make qualitypasses.test_llm.py,test_config.pyandtest_pricing.py. They're built from real OpenAI/Anthropic SDK response objects (Response,Message), not mocks.defaultstrategy prompt (~12k tokens) and tools with a hand-builtSELECTING_HANDgamestate. Two decisions per model went through the realCollector, and the output files were parsed with BalatroBench'sextractor.py: prompts, tool calls, reasoning, tokens, provider, vendor/model and cost all came out correctly. Models checked:gpt-6.1-solandgpt-6-astra(Responses API)claude-haiku-5-5(Anthropic). Prompt caching works: 17.5k tokens are read from cache on the second call.main. 11 need game-generated fixtures (tests/unit/fixtures/), and 2 are staleseedassertions inTestGetModelConfigleft over from d642f7d. None are touched by this PR.