[bartsdev]

Summary

The eval dataset now reaches the model through OpenRouter instead of the Google provider. The model is still google/gemini-2.5-flash, so pass rates stay comparable, and EVAL_MODEL swaps it for any OpenRouter model ID.

 data/evals/
-└── gemini-comprehensive.json   # provider: google, model: gemini-2.5-flash
+└── comprehensive.json          # provider: openrouter, model: google/gemini-2.5-flash
 tests/mcp/
-└── evals-gemini.spec.mjs       # requires GOOGLE_GENERATIVE_AI_API_KEY
+└── evals.spec.mjs              # requires OPENROUTER_API_KEY, optional EVAL_MODEL
 .github/workflows/
+└── mcp-evals.yml               # weekly (Mon 07:00 UTC) + workflow_dispatch(model)
 package.json
+  devDependencies: ai, @openrouter/ai-sdk-provider

The workflow is modeled on azure-smoke.yml. It adds what the cases need on the host:

mcp-evals
  install deps
  setup-terraform (wrapper off)       # scenario-deploy-terraform-project → tflocal
  pipx install terraform-local, snowflake-cli
  docker pull localstack-pro, snowflake   # so in-case starts don't time out
  yarn test:mcp:evals
  upload .mcp-test-results (always)   # pass rate is the signal, not just failures

Evidence

• Before: The evals needed ai and @ai-sdk/google, but neither was in package.json, so they only ran after a manual install. No workflow ran them.
  After: yarn installs the provider, and the tester resolves it:

  $ node -e "openrouter('google/gemini-2.5-flash')"
  openrouter google/gemini-2.5-flash
  $ playwright test tests/mcp/evals.spec.mjs --list
  [localstack-mcp-server] › evals.spec.mjs › OpenRouter comprehensive eval › single comprehensive eval dataset passes
  

• Not yet run: the full eval through OpenRouter, locally or in CI. That needs the OPENROUTER_API_KEY secret. Once this PR is merged, the workflow can be started by hand.

Merge Danger

Door: two-way

Only test code, test data, dev dependencies and a new workflow change. Reverting the PR restores the Gemini setup.

Blast Radius: CI-only

• Before the first scheduled run (Monday), add the OPENROUTER_API_KEY repository secret. Without it the job fails immediately with Missing required environment variable.
• Each run uses OpenRouter credits: 13 cases × 5 iterations, plus Snowflake × 3.
• The new dev dependencies don't ship in the published package.

🤖 Generated with Claude Code