Skip to content

feat(evaluations): integrate offline judges and deterministic scorers - #44

Draft
donei003 wants to merge 6 commits into
devin/1786604824-evaluations-scaffoldfrom
feature/evaluations-judging-foundation
Draft

feat(evaluations): integrate offline judges and deterministic scorers#44
donei003 wants to merge 6 commits into
devin/1786604824-evaluations-scaffoldfrom
feature/evaluations-judging-foundation

Conversation

@donei003

@donei003 donei003 commented Aug 21, 2026

Copy link
Copy Markdown

Summary

Adds the typed offline judge and deterministic scorer foundations for evaluations from code, stacked on #39.

  • runs typed JudgeReference and Scorer values alongside each successful generation
  • passes generation output plus row index, rendered input, expected output, variables, and metadata into evaluation logic
  • resolves LaunchDarkly judges with pinned judge identity and diagnosable failure results
  • avoids online metric emission from offline evaluations
  • exposes the typed evaluations API from launchdarkly_ai_server and the convenience package
  • documents and tests the judge/scorer integration without live LaunchDarkly or provider calls

Generation-result publishing and ingestion behavior is inherited from #39 and is intentionally not implemented or modified by this PR.

Stack

Validation

  • uv run pytest packages/client/tests/test_evaluation_judges.py packages/client/tests/test_evaluation_scorers.py packages/client/tests/test_evaluations.py packages/client/tests/test_evaluations_run.py -v — 48 passed
  • make lint
  • make format-check
  • make typecheck currently reaches the repository's pre-existing duplicate conftest module error.

@donei003
donei003 changed the base branch from main to devin/1786604824-evaluations-scaffold August 21, 2026 05:19
@donei003
donei003 marked this pull request as draft August 21, 2026 05:20

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

@donei003 donei003 changed the title feat(evaluations): gate ingestion and integrate judges/scorers feat(evaluations): integrate offline judges and deterministic scorers Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant