[vlad-perevezentsev]

Summary

Fixes #136. With --provider openai, a model name that litellm does not take for OpenAI's now goes out as openai/<model>, which means to OPENAI_API_BASE. A name litellm already sends to OpenAI keeps its route. On provider openai the response-cache key holds the model name as given, as on main, so every stored entry still hits.

The --judges spec is split correctly. The provider is lost in llm.resolve_model, which sent every openai model to litellm bare. litellm infers OpenAI only for names on its own lists and for ft:gpt-* fine-tunes. It refused any other name before sending a request ("LLM Provider NOT provided"): glm-5.3 behind OPENAI_API_BASE, meta-llama/x, a model newer than the installed litellm. A bare name it assigns to another cloud (claude-*, amazon.nova-pro-v1:0, gemini-*) went to that cloud with the OpenAI key. Every --provider openai / --judge-provider openai path goes through resolve_model: answers, judge, panel, novelty, question and persona generation.

 resolve_model(model, "openai")
-  "/" in model  -> model
-  else          -> model
+  "/" in model  -> model if the prefix is a litellm provider (its provider_list or JSON registry: openai/, azure/, ollama/, crusoe/ …), else openai/<model>
+  else          -> model if litellm takes it for OpenAI's (its chat or text-completion lists, ft:gpt-3.5-turbo / ft:gpt-4 fine-tunes), else openai/<model>

The openai/openai/<model> workaround keeps working.

Also:

• Response-cache key on provider openai (6296697): the model name as given, not the route. The route of a new model depends on litellm's price map. The remote map and the installed backup (LITELLM_LOCAL_MODEL_COST_MAP=True, or offline) disagree: gpt-6.1-terra is on the remote OpenAI list (206 names) and not on the backup (197), so it goes out bare under one and as openai/gpt-6.1-terra under the other. Both reach the same endpoint, and the key stays the same under either map. It is also the key main wrote, because main sent every openai name bare.
• docs/reviewing.md (the README paragraph moved there in docs: illustrated intro on docs home, short README with start-here #271): how to use an OpenAI-compatible server. It covers OPENAI_API_BASE / OPENAI_BASE_URL and which names reach the server. A name with a litellm provider prefix (deepseek/, groq/, ollama/) keeps that route, so it should be written openai/<name>. OPENAI_API_KEY must be set even if the server ignores it. Cost is unknown for such a model, so novelty --max-usd stops after one call.
• llm.py is at its growth baseline: two passages that repeat others are dropped (the docstring's closing paragraph and a comment), and the file stays at 1118 lines.

Evidence

Live eval cross-judge (3 questions, 2 arms) against a loopback stub /v1/chat/completions via OPENAI_API_BASE. On main b226dba, --judges openai/glm-5.3 gives skill: NOT MEASURED and exit 0, and litellm prints its "Provider List" banner 12 times. "After" is head 6296697.

--judges	requests before	requests after
openai/glm-5.3	0	6
openai/meta-llama/Llama-3.1-8B	0	6
openai/amazon.nova-pro-v1:0	0 (sent to Bedrock)	6
openai/openai/glm-5.3 (workaround)	6	6
openai/gpt-6.1-sol (control)	6	6
openai/ft:gpt-4o-mini:org::id (control)	6	6, model sent bare

With --provider openai on main and the transport intercepted (litellm 1.106.0), a bare name of another cloud reached it with the OpenAI key:

bare name	OPENAI_API_KEY set	unset
claude-sonnet-4-5	api.anthropic.com, x-api-key: <OpenAI key>	ANTHROPIC_API_KEY
amazon.nova-pro-v1:0	bedrock-runtime, Authorization: Bearer <OpenAI key>	AWS SigV4

Tests:

• tests/test_openai_compatible_route.py (new) runs the production adapter with litellm's real provider lookup (mock_response, nothing sent). It covers LLM(...).ask for names that get openai/ and for names kept bare, eval cross-judge --judges openai/glm-5.3 end to end, and a reply stored with the model on the price map that replays once it is off the map.
• tests/test_llm.py: route rows for the prefixed names (including claude-* and a bare Bedrock id) and for the names that must not change (gpt-6.1-sol, gpt-3.5-turbo-instruct, both ft: families, openai/glm-5.3, azure/gpt-4o, ollama/llama3, crusoe/x). test_an_entry_main_wrote_is_hit gets main's key for gpt-6.1-terra, which is routed openai/ under the installed map.
• tests/test_grader_characterization.py (refactor(eval): grade the judge and panel loops with llm.ask_parsed #251) is as on main: its pinned cache-key file names hold for gpt-x, now routed openai/gpt-x.
• On main: 10 failed, 101 passed (BadRequestError: LLM Provider NOT provided, and the routes). On this branch: 111 passed.
• Each mutation fails the tests: dropping the text-completion list, dropping the ft: test, loosening it to any ft:, always prefixing, taking any / as a route, keeping bare names off the lists, keying the cache on the route. Replacing the provider_list test with in ("openai", "azure") is caught by the ollama/ row.
• make ci-local at 6296697: 27 passed, 1 failed, 7 skipped (pytest 10146 passed). The FAIL is pip-audit, which reports filelock 3.32.7 advisories from the lock files; this PR does not touch them. The docker smoke passed in round 1 and was not rerun, since nothing docker-related changed.

Merge Danger

Door: two-way. A revert restores the bare route; no records or schemas change.

Blast Radius: names that failed before every request, plus one set that changes route. A bare name litellm assigns to another cloud now goes to OPENAI_API_BASE: about 450 in litellm 1.106 (Bedrock 351, Vertex 46, Anthropic 22, Cohere 16, AI21 12, NLP Cloud 2). On main it reached that cloud only while OPENAI_API_KEY was unset. With the key set, the OpenAI key went to that cloud and was refused. Those clouds have their own providers.

Overlap: main is merged in (f8279bf, after #271). #251's characterization test is unchanged from main. #255 edits other hunks of llm.py, and a one-line merge may be needed next to it.

Not in this PR:

• utils.get_llm's openai branch has the same bare route, but no caller in the package reaches it, and Consolidate duplicated subsystems (stats, LLM routing, judge loop, containers, digests, writers) #229 P1-2 removes it.
• Found while testing, for separate fixes:
  • eval/cross_judge.py:343 swallows judge errors with no log, so a judge failing every call gives NOT MEASURED with no reason.
  • "Missing credentials" is retried as transient (14 s per call).
  • litellm prints its banners to stdout.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.26%. Comparing base (0a63f4e) to head (6296697).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #256      +/-   ##
==========================================
- Coverage   93.26%   93.26%   -0.01%     
==========================================
  Files         272      272              
  Lines       35117    35121       +4     
  Branches     5964     5964              
==========================================
+ Hits        32752    32754       +2     
- Misses       1547     1548       +1     
- Partials      818      819       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/llm.py	95.82% <100.00%> (+0.03%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

The fix for glm-5.3 and meta-llama/… works, and the tests pass (100 in test_llm and test_openai_compatible_route; ruff, growth ratchet and import-architecture clean).

One thing to settle before approval: the openai branch in dokimi/llm.py:146-149 also reroutes bare names that litellm sends to other providers today. That is 449 catalog names: bedrock 351 (e.g. amazon.nova-pro-v1:0), vertex_ai 46, anthropic 22, cohere 16, ai21 12, nlp_cloud 2. They work on main with the cloud's credentials, and after this PR they would go to OPENAI_API_BASE. The body's "numbers do not change for a model that worked before" does not cover them. I checked this against litellm 1.106.0 from the lock; I did not run it against live Bedrock. Options: narrow the rule to names litellm cannot route, or list the affected set in the body and changelog and add a test row that pins one bare Bedrock id.

Nits:

• README.md:152-153 says any served model name works. An org prefix that matches a litellm provider (deepseek/, groq/, ollama/, together_ai/) still routes natively and ignores OPENAI_API_BASE. Say "a name without a litellm provider prefix", or tell users to write openai/<name>.
• README.md:155 describes --max-usd for every --provider openai run. Only novelty takes it (SpendCap, novelty/client.py:46-80).

[antonwolfy]

The fix does what it says for glm-5.3 and meta-llama/…, and CI is green. One point from @napetrov is still open and I confirm it, so this does not approve yet. One nit inline.

I agree with @napetrov that the openai branch also reroutes bare names that litellm sends to other providers. New evidence: at this head resolve_model returns openai/claude-sonnet-4-5, openai/amazon.nova-pro-v1:0 and openai/gemini-2.5-flash for provider openai, where litellm.get_llm_provider on the bare name gives anthropic, bedrock and vertex_ai, and ft:gpt-4o-mini:org::id becomes openai/ft:gpt-4o-mini:org::id.

I also agree with @napetrov's README points: the "any model name works" sentence is false for deepseek/, groq/ and ollama/ prefixes, and --max-usd exists only on novelty.

What I ran, on a Linux review host at 81c0d01:

• make ci-local: 28 passed, 0 failed, 7 skipped (jobs that need a PR event or tools not installed locally).
• A reproduction of the reroute above with the LLM transport mocked.
• Mutations of the new branch: dropping the text-completion list, always prefixing, and letting any / pass are all caught; replacing the provider_list check with in ("openai", "azure") passes the full test_llm.py and test_openai_compatible_route.py.

Checked, no issues

• Callers: resolve_model feeds pricing, usage and LLM._route; openrouter, openai-codex, google-vertex and amazon-bedrock return before the new branch and are unchanged.
• Kept routes: gpt-4o, o3, gpt-5, gpt-3.5-turbo-instruct, azure/gpt-4o and openai/glm-5.3 come out unchanged, so the openai/openai/<model> workaround still works.
• Repo rules: the changelog fragment passes its check, llm.py is at its 1118-line growth baseline, new files carry the SPDX header, and the new tests run offline.
• Env vars: litellm reads both OPENAI_BASE_URL and OPENAI_API_BASE, as the README says.

[vlad-perevezentsev]

@napetrov thanks. Taken as your second option: the rule stays, and the set is named and pinned. 1f13597 adds the bare Bedrock row (amazon.nova-pro-v1:0 -> openai/amazon.nova-pro-v1:0), and a6063b5 adds a changelog entry for those names.

I kept the rule rather than narrowing it because those names do not work on main once OPENAI_API_KEY is set. dokimi passes the openai provider's key to litellm as api_key, and litellm hands it to whichever cloud it picked. With the transport intercepted (litellm 1.106.0), --provider openai on main sends:

bare name	OPENAI_API_KEY set	unset
claude-sonnet-4-5	api.anthropic.com, x-api-key: <OpenAI key>	ANTHROPIC_API_KEY
amazon.nova-pro-v1:0	bedrock-runtime, Authorization: Bearer <OpenAI key>	AWS SigV4

So they reach their cloud only while OPENAI_API_KEY is unset, and otherwise the OpenAI key goes to another vendor and is refused. Narrowing the rule would keep that. Those clouds have their own providers (anthropic, amazon-bedrock, google-vertex), and the changelog points to them. I did not check Vertex: it needs the Google SDK, which the lock does not install.

README nits, both fixed in feb0d7e. The paragraph now says a name with a litellm provider prefix (deepseek/, groq/, ollama/) keeps that route and should be written openai/<name>. It also says novelty --max-usd.

[vlad-perevezentsev]

@antonwolfy thanks for the mutation run and the ollama/ row (9372736). On the reroute, see my reply to napetrov: the rule stays, with the Bedrock row in 1f13597 and the changelog entry in a6063b5.

Your ft:gpt-4o-mini:org::id example was a real regression: the name kept its endpoint but got a new response-cache key. 598fbfa keeps a bare name bare when litellm itself takes it for OpenAI's, which means its lists or its ft:gpt-3.5-turbo / ft:gpt-4 test. Two new tests guard this. The names kept bare still route through litellm's real lookup, and an ft: name litellm does not place (ft:gpt-5-mini:org::id) still gets openai/, so a looser ft: check fails them. I did not call litellm.get_llm_provider because on an unknown name, which is the #136 case, it prints its banner to stdout.

Also: 13165af merges main, and ea2acaa re-pins two cache-key file names in #251's test_grader_characterization.py. Its gpt-x at provider openai now routes as openai/gpt-x. The new names are the keys main computes for model openai/gpt-x, so only the route changed.

[xaleryb]

• Ran: full suite, ruff, mypy and bandit on the head and on the head plus the fix below, plus end-to-end routing with the HTTP transport intercepted.
• --provider openai, model crusoe/deepseek-ai/DeepSeek-R1-0528 or sarvam/sarvam-m, OPENAI_API_KEY unset -> openai/ is added and the call fails with "Missing credentials". On main it reached the provider's own endpoint. Expected: the prefix keeps its route, as the README and changelog say. litellm routes these 8 providers from its JSON registry, and they are not in provider_list.
• gpt-6.1-terra on provider openai -> bare with litellm's remote price map, openai/gpt-6.1-terra with LITELLM_LOCAL_MODEL_COST_MAP=True. So there are two response-cache keys for one model and prompt, and replay_only misses. Expected: one key, whatever the map or litellm version.
• Pushed 4cc10a6: it also checks litellm.JSONProviderRegistry.exists(prefix), adds a crusoe/x row, and adds no lines.
• For the author: the cache-key scheme. One option is to hash the caller's bare name on provider openai, which keeps main's keys. Another is to always use openai/<model>, which changes every stored key once.

[xaleryb]

Merged main after #271 (f8279bf): #271 moved the README credentials paragraph this PR extends into dokimi/docs/reviewing.md, so the README is main's and the OpenAI-compatible server text is added to that paragraph in reviewing.md, unchanged. The changelog fragment still says "README: running against an OpenAI-compatible server"; it may want to name docs/reviewing.md instead. Full suite, ruff, mypy and bandit pass on Linux.

[antonwolfy]

LGTM. The route rule holds up at this head: the ft:gpt-* regression, the ollama/ row and the Bedrock row from the earlier round are in, and the JSON-registry prefix fix (crusoe/) is right. Two nits inline.

@xaleryb's question on the response-cache key waits for the author: the route now depends on litellm's cost map, so a model that is in the remote map but not in the installed backup gets a different key depending on whether the fetch succeeds.

What I ran, on a Linux review host at f8279bf:

• make ci-local: 27 passed, 1 failed, 7 skipped; 10,196 tests; the changed-line coverage gate passes. The one FAIL is pip-audit, which lists filelock 3.32.7 advisories from the lock files; this PR does not touch them.
• resolve_model on edge names against litellm's own get_llm_provider (bare and prefixed names, JSON-registry and provider_list prefixes, ft: variants), with the LLM transport mocked.
• A mutation run over the new branch against test_llm.py, test_openai_compatible_route.py and test_grader_characterization.py.

Checked, no issues

• Callers: resolve_model feeds pricing, usage and LLM._route; openai/<x> is a valid litellm route for cost and completion, and openrouter and openai-codex still return before the new branch.
• Repo rules: all ten non-merge commits carry Signed-off-by; llm.py stays at its 1118-line baseline; the new test file has the SPDX header and the tests write nothing outside tmp_path; the changelog fragment check passes.
• Docs: --judges, novelty --max-usd (SpendCap stops after the first call when cost is unknown) and both base-URL variables match the code.

[antonwolfy]

LGTM. I pushed a commit per nit, up to 5b6b90c, which fix the two nits from my review:

• 01406a0: dokimi/tests/test_llm.py:62: added the ft:gpt-3.5-turbo:org::id row after the ft:gpt-4o-mini row.
• 5b6b90c: dokimi/changelog.d/20261027_120000_vlad-perevezentsev_openai-compatible-model-route.md:23: the entry's lead-in now names docs/reviewing.md instead of README.

make ci-local at 5b6b90c: 27 passed, 1 failed, 7 skipped.

Not merging yet: @xaleryb on the cache-key scheme: the openai route depends on litellm's cost map (remote vs installed backup), so one model and prompt can have two response-cache keys; hash the caller's bare name, or always use openai/ waits for your answer, @vlad-perevezentsev.

[vlad-perevezentsev]

@xaleryb thanks for 4cc10a6 and the merge. On the cache key I took your first option: 6296697 keys a provider-openai reply on the model name as given. That is what main keyed on, since main sent every openai name bare. So every key main wrote still hits, including those of the names this PR reroutes, and litellm's price map no longer moves the key. The route can still be bare or openai/ depending on the map, but both reach the same endpoint, and OPENAI_API_BASE was never part of the key. Other providers are unchanged.

Your second option would change every stored OpenAI key once and the route of every OpenAI model, so I left it out.

Tests:

• A reply stored with gpt-6.1-terra on the map replays once it is off the map and routed openai/.
• test_an_entry_main_wrote_is_hit gets main's key for that model, recorded by running main's code.
• refactor(eval): grade the judge and panel loops with llm.ask_parsed #251's test_grader_characterization.py is back to main's file names. My re-pin in ea2acaa is undone.

Keying on the route again fails all three. make ci-local: 1 failed (pip-audit, the filelock advisories antonwolfy saw), 27 passed. llm.py stays at 1118 lines.

[vchamarthi]

🔍 OpenCodeReview found 2 issue(s) in this PR.

• ✅ Successfully posted inline: 2 comment(s)

[vchamarthi]

Nit: this one-liner mixes a walrus, or and a trailing conditional expression, and ours is computed even when the name has a /. It parses as intended: (p in provider_list or JSONProviderRegistry.exists(p)) if "/" in model else ours. Separate branches would be easier to review. It also reads litellm.JSONProviderRegistry unguarded, and pyproject.toml only pins litellm>=1.105.0. If that attribute is missing in some supported version, every provider=openai name containing / (e.g. meta-llama/Llama-3.1-8B) raises AttributeError instead of being routed. Confirm the attribute exists at the minimum pin, or raise the pin.

Suggestion:

Suggested change

	routed = (p := model.split("/", 1)[0]) in litellm.provider_list or litellm.JSONProviderRegistry.exists(p) if "/" in model else ours
	if "/" in model:
	prefix = model.split("/", 1)[0]
	routed = prefix in litellm.provider_list or litellm.JSONProviderRegistry.exists(prefix)
	else:
	routed = model in known or "ft:gpt-3.5-turbo" in model or "ft:gpt-4" in model # as litellm

[vchamarthi]

Nit: this entry is named openai-rerouted, but the key is built from the caller's name on openai, so the hash is the same whether resolve_model returns gpt-6.1-terra or openai/gpt-6.1-terra. If resolve_model stopped rerouting, the test would still pass. It only pins key stability, which is fine. The comment is also fragile: it stops being true once litellm lists gpt-6.1-terra. Rename the entry and reword the comment, or leave routing to the route tests.