Skip to content

fix(search-eval): skip llm-rubric assertions when judge API key is absent - #474

Draft
arabold with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-evaluate-search-quality-job
Draft

arabold with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-evaluate-search-quality-job

Conversation

Copilot AI commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

When OPENAI_API_KEY is unset, promptfoo throws on every llm-rubric assertion grading call. This doesn't just fail the LLM-judged metrics — it wipes out all namedScores for the affected test, including the deterministic IR metrics (MRR, Recall@k, nDCG@k). Result: 59/59 tests counted as errors, all metrics at 0.000, spurious 100% regression detected, job exits 1.

Changes

  • judges.ts — adds two helpers:

    • judgeApiKeyVar(judge) → returns the env var name (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY) for a resolved judge
    • isJudgeKeyAvailable(judge, env?) → returns true iff that key is present and non-empty
  • run.ts — instead of always passing the static promptfoo.yaml, now generates a runtime config at results/promptfoo.runtime.yaml (already gitignored) before each promptfoo invocation:

const judgeAvailable = isJudgeKeyAvailable(judge);
if (!judgeAvailable) {
  console.warn(`⚠  ${judgeApiKeyVar(judge)} is not set — LLM-judged metrics will be skipped.`);
}
writeFileSync(PATHS.runtimeConfig, buildRuntimeConfig(judgeAvailable), "utf8");

When judgeAvailable is false, buildRuntimeConfig produces a config without the three llm-rubric assertions and the defaultTest.options.provider block. The deterministic javascript assertions (ir-metrics, structural) always run regardless of key availability.

…sent

When the judge API key (e.g. OPENAI_API_KEY) is not configured, promptfoo
throws in every llm-rubric assertion grading call. This wipes out
namedScores for the whole test — including the deterministic IR metrics
(MRR, Recall@k, nDCG@k) — and marks all 59 tests as errors, which the
comparison step then reports as a 100% regression against the baseline.

Fix by generating a runtime promptfoo config at each run that conditionally
omits llm-rubric assertions when the judge's API key env var is absent.
The deterministic JavaScript assertions (ir-metrics, structural) always run.

- Add judgeApiKeyVar() and isJudgeKeyAvailable() helpers to judges.ts
- Add buildRuntimeConfig(judgeAvailable) to run.ts that produces the
  appropriate promptfoo YAML with or without LLM-judged rubrics
- Write the runtime config to results/promptfoo.runtime.yaml (already
  gitignored via results/.gitignore) before each promptfoo invocation
- Emit a clear warning when LLM grading is skipped

Co-authored-by: arabold <1548229+arabold@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix failing GitHub Actions job Evaluate search quality fix(search-eval): skip llm-rubric assertions when judge API key is absent Aug 18, 2026
Copilot AI requested a review from arabold August 18, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants