feat: record judge scores as gen_ai.evaluation.result - #43
Draft
ccschmitz-launchdarkly wants to merge 5 commits into
Draft
feat: record judge scores as gen_ai.evaluation.result#43ccschmitz-launchdarkly wants to merge 5 commits into
ccschmitz-launchdarkly wants to merge 5 commits into
Conversation
Judge scores land as a `gen_ai.evaluation.result` span event (and mirrored attributes) on the judge `invoke_agent` span, so conversation turn badges can render. `with_judge_evaluation` holds that span open until scoring finishes — `execute_and_track` returns after the handler has already called `span.end()`, so without the delay the event would be dropped. Existing `track(evaluation_metric_key)` behavior is unchanged. TELEMETRY-CONTRACT.md section 4a documents the event contract. Stacked on the conversation-id PR: the two halves share conversation.py and nothing else, so they review apart and merge together. O11Y-1888 Co-Authored-By: Claude <noreply@anthropic.com>
11 tasks
…Y-1888-judge-evals # Conflicts: # packages/client/tests/test_conversation.py
…nd time Review follow-ups on the judge half. - Do not emit `gen_ai.evaluation.explanation`. The judge's reasoning is model-generated prose about the user's conversation — content — and AGENTS.md gates content attributes behind `capture_content`, a handler-factory option this layer never receives. It was exported unconditionally, including for callers who left capture off. The reasoning is still returned to the caller in `judge_results`; only the telemetry copy is withheld. - Replace `float(score)` with a non-raising finite-number guard. `float()` was a new raise site sitting ahead of the existing `client.track(evaluation_metric_key, …)` call, so a judge returning "0.9 (high)" silently killed the metric this PR claims is unchanged — and in `run_judge` it escaped uncaught, breaking that function's documented "returns None" contract. - Freeze the end time when the handler calls `end()`. Replaying a no-arg `end()` at release let the SDK stamp `time_ns()` then, inflating every judge span by the tracking and parsing work that runs in between. - Detach the judge capture even if the deferred end raises. O11Y-1888 Co-Authored-By: Claude <noreply@anthropic.com>
…Y-1888-judge-evals
One run that exercises everything O11Y-1888 touches: three turns bound to a single conversation id, each with inline judge evaluation. Prints the id so it can be opened directly in the Conversations view. It is also the manual check for the content decision — the judge's reasoning is printed from `judge_results` (the caller's copy) and should appear nowhere in the exported telemetry. O11Y-1888 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Judge scores land as a
gen_ai.evaluation.resultspan event (and mirrored span attributes) on the judge'sinvoke_agentspan, so conversation turn badges can render.with_judge_evaluationholds the judgeinvoke_agentspan open until scoring finishes.execute_and_trackreturns after the handler has already calledspan.end(), so without the delay the event would be dropped.gen_ai.evaluation.nameandgen_ai.evaluation.score.value, mirrored as span attributes.score.labelis not invented, and only a finite number is recorded.track(evaluation_metric_key)behavior is unchanged and still feeds AI Config Monitoring.TELEMETRY-CONTRACT.mdsection 4a documents the event contract; the evaluation keys are added to the cross-handler vocabulary test.Mirrors js-ai-sdk#27. Part of O11Y-1888.
Stacking
O11Y-1888-emit-conversation-id-and-evaluation(#42), notmain. Review apart, merge together — merge #42 first, then this.Split out per review: this half overrides
span.endon a live span, which is the riskiest surface in the original combined PR and deserves its own read. Note it shares onlyconversation.pywith #42.Reviewer notes
The
end()patch is the part worth scrutiny. Nothing breaks today — judges only take the blocking path — but it is the one place the SDK mutates a live span's lifecycle, and span lifecycle is this SDK's worst historical surface.Test plan
uv run pytest— 1073 pass, 11 skippeduv run mypy packages/*/srcclean;ruff check/ruff format --checkcleaninvoke_agentspan after the handler has already ended itscore.labelis not invented$ld:ai:metric track still firesReview follow-ups (pushed)
gen_ai.evaluation.explanation. The judge's reasoning is model prose about the user'sconversation — content — and
captureContent/capture_contentis a handler-factory optionthis layer never receives, so it was being exported for callers who had capture switched off.
The reasoning still reaches the caller in
judge_results; only the telemetry copy is withheld.Re-adding it needs its own opt-in.
try, so a judgeprovider/auth/network failure rejects as it did before instead of being swallowed and logged as
a parse failure. (Python: the
float(score)raise site ahead of the metrictrackis gone.)evaluationMetricKey/evaluation_metric_keytrack guard is untouched.end()call, so the deferred end no longer stamps releasetime and inflate the judge span by the tracking and parsing work in between. Test asserts it.
Still open, deliberately deferred — the
span.endpatch itself. Three reviewers independentlysuggested the same deeper fix: let the judge own its own span (or emit on a dedicated linked span)
instead of mutating a handler-owned live span. That would also retire the single-slot capture and
the
is_recording()divergence. Worth a decision before this lands, not a quiet rewrite.Staging verification
The
conversationexample added on this branch (three turns under one id, judge output printed fromjudge_results) was run againstdefault/staging:https://ld-stg.launchdarkly.com/projects/default/traces?selected-env=staging&tab=traces&startDate=2026-08-20T17%3A30%3A00Z&endDate=2026-08-20T19%3A00%3A00Z&query=gen_ai.conversation.id%3Dconversation-example-23bcd8fa
Six spans across three traces, grouped by
gen_ai.conversation.id, allOk.The judge path in this PR is still unverified end to end. The AI Config used has no
judge_configuration, so no judges ran — the example reported that explicitly rather than appearingto pass. Confirming the
gen_ai.evaluation.resultevent in the product needs a judge-enabled config(a second AI Config wired as the judge). Unit tests cover the event, the score guard, the withheld
explanation, and the frozen end time; the product-level check is outstanding.