Pin the RubyLLM provider and seed the model registry from LLM::Stages - #53
Merged
Merged
Conversation
No call site passed `provider:`, so RubyLLM ranked candidates by its own PROVIDER_PREFERENCE, where perplexity outranks openrouter. Both catalogues list the dot-less 5-series ids (anthropic/claude-sonnet-5, -opus-5), so a registry carrying both rows -- rc3's bundle does -- hands the chat to Perplexity and it dies with ConfigurationError. On the current pin the dev store (13 providers) can reach the same state after a refresh. LLM::Stages::PROVIDER is the one provider hifumi talks to; the three plain-chat sites (both planners, the template picker), the agent's creation-time kwargs and ChatRespondJob's per-turn with_model all pass it. Same kwarg signature on the pin and on 2.0.0.rc3, so this lands before the gem bump and is unchanged by it. Tests: the agent's chat kwargs reader, the picker's captured kwargs, the job's with_model kwargs, and one test per planner that runs the real invoke_llm with RubyLLM.context swapped -- the first coverage that line has had; every other planner test stubs invoke_llm wholesale.
The store was populated by hand -- RubyLLM.models.refresh! per Runbook 03 -- and empty everywhere that procedure had not run: test, CI, every fresh clone. An empty store resolves against the gem's bundled models.json, whose provider set drifts between releases and which lacks anthropic/claude-opus-5 at the current pin, so a fresh environment could not resolve one of the five offered ids. On 2.0.0.rc3 the same state copies all 1 669 bundled rows into the store on the first chat of every test. The store now holds exactly what LLM::Stages::AVAILABLE_MODELS offers, under LLM::Stages::PROVIDER, in every environment: an ERB fixture derived from the hash for test, db/seeds.rb (find_or_create_by!, idempotent, offline, five upserts) everywhere else. Bare rows -- model_id, provider, name -- are all RubyLLM needs to resolve a chat; a refresh enriches them in place. Suite drops from 4.4 s to 3.0 s: find_or_create_model finds the fixture row instead of writing one per chat. Tests: the stages test resolves every offered id from the store under the pinned provider (red on main for opus-5 and for the exists? check on every id); the seeds test covers an empty store and idempotency on a populated one.
…b:seed The check moves into LLM::RegistryCheck (lib/llm/, plain Ruby, returns a report, no writes, no network); the bin is now a printer over it. Semantics change with the seeded store: an empty store is a FAIL rather than a warning, every id resolves under LLM::Stages::PROVIDER, the remediation is `bin/rails db:seed`, and the refresh! text is gone. Candidate ids still exit 1 when they do not resolve, but the hint no longer suggests db:seed for them: a non-empty store shadows the bundle, so a candidate can only resolve once it is offered and seeded -- the message says so. Tests cover each branch: ok, empty store, one offered id missing, failing candidate. Each resolves once against the intact fixture store in setup, so the destructive tests cannot memoise a poisoned registry for the rest of the worker.
Runbook 03 is rewritten around `bin/rails db:seed` as the population step: local and production procedures, the provider pin's rationale, and the "adding a model" steps in the order the code now enforces (add the id, run the suite -- the fixture derives from AVAILABLE_MODELS -- deploy, seed, restart). RubyLLM.models.refresh! moves to an optional metadata-enrichment section that keeps the SQLite write-lock warning. The "test environment resolves against the bundle" caveat is gone because the state is gone. CLAUDE.md's model-selection bullet says the same in one paragraph; the follow-ups file records seed-on-boot as the deferred next step and no longer describes the store as refresh-maintained.
…ries the fix bundler-audit's advisory db gained CVE-2026-67991 on 2026-09-16 -- a ReDoS in RubyLLM::Utils.underscore, patched in >= 2.0.0.rc1 -- and scan_ruby went red for every branch on the current pin. The pin is not vulnerable: the fix commit 9d75b033 is an ancestor of c45ebd78 and the checked-out utils.rb has the non-backtracking UNDERSCORE_BOUNDARY regex. The gemspec at that commit still reports 1.16.0, and the version string is all the range check reads. Ignored with the evidence beside it. The entry is deleted with the 2.0.0.rc3 bump, where the released version satisfies the range and the gate is live again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Registry hardening on the current
ruby_llmpin, ahead of the 2.0.0.rc3 bump (PR B). Independent of the gem version; closes a live defect: a fresh environment cannot resolveanthropic/claude-opus-5today, because the store is empty everywhere the hand-runRubyLLM.models.refresh!never happened (test, CI, every fresh clone) and the pin's bundledmodels.jsonlacks that id.Three changes, one per commit
Every RubyLLM call pins
provider: LLM::Stages::PROVIDER(:openrouter). No call site passed a provider, so RubyLLM ranked candidate rows by its ownPROVIDER_PREFERENCE, whereperplexityoutranksopenrouter. Both catalogues list the dot-less 5-series ids, so a store carrying both providers' rows — a refreshed one, or rc3's bundle — handssonnet-5/opus-5chats to Perplexity and they die withConfigurationError. Same kwarg signature on the pin and on rc3, so this lands first and the bump changes nothing here.The store is seeded from
LLM::Stages::AVAILABLE_MODELS.db/seeds.rbwrites one bare row per offered id underPROVIDER(find_or_create_by!, idempotent, offline, five upserts);test/fixtures/ruby_llm_models.ymlderives the test store from the same hash via ERB. Bare rows are all RubyLLM needs to resolve a chat;refresh!is demoted to optional metadata enrichment. Side effect: the suite drops from 4.4 s to 3.0 s becausefind_or_create_modelfinds the fixture row instead of writing one per chat.bin/verify-model-registrytells the truth. The check moves intoLLM::RegistryCheck(lib/llm/, plain Ruby, returns a report); the bin prints it. An empty store is now a FAIL rather than a warning, resolution is provider-pinned, and the remedy isbin/rails db:seed. A failing candidate id still exits 1 but gets its own message — a non-empty store shadows the bundle, so a candidate resolves only once it is offered and seeded.Docs
Runbook 03 is rewritten around seeds (local, production, adding a model, optional refresh with the SQLite write-lock warning kept).
CLAUDE.md's model-selection bullet says the same in one paragraph. Seed-on-boot is recorded as the deferred follow-up.Rollout
No migration. After
kamal deploy:kamal app exec --reuse "bin/verify-model-registry"→ fiveOK … provider=openrouter, store 410, exit 0. Seeds are not needed in production (all five ids already haveopenrouterrows) butdb:seedis safe to run as an idempotency proof. Rollback iskamal rollback <previous>.Coverage
Eleven new tests, one extended, each red before its code change:
test/agents/generator_agent_test.rb,test/lib/templates/picker_test.rb,test/jobs/chat_respond_job_test.rb, bothad_hoc_llm_test.rb— the provider kwarg at all five call sites. The planner tests are the first coverage thectx.chat(...)line has had; every other test there stubsinvoke_llmwholesale.test/lib/llm/stages_test.rb— every offered id resolves from the store under the pinned provider.test/db/seeds_test.rb— empty store, idempotency.test/lib/llm/registry_check_test.rb— ok / empty / missing row / failing candidate.bin/rails test753 runs / 3464 assertions / 0 failures ·bin/rubocopclean ·bin/brakemanno warnings.Plan:
thoughts/shared/plans/2026-09-17/ruby-llm-2-0-0-rc3-upgrade.md(PR A, phases 1–4).