Skip to content

feat: record judge scores as gen_ai.evaluation.result - #43

Draft
ccschmitz-launchdarkly wants to merge 5 commits into
O11Y-1888-emit-conversation-id-and-evaluationfrom
O11Y-1888-judge-evals
Draft

feat: record judge scores as gen_ai.evaluation.result#43
ccschmitz-launchdarkly wants to merge 5 commits into
O11Y-1888-emit-conversation-id-and-evaluationfrom
O11Y-1888-judge-evals

Conversation

@ccschmitz-launchdarkly

@ccschmitz-launchdarkly ccschmitz-launchdarkly commented Aug 19, 2026

Copy link
Copy Markdown

Summary

Judge scores land as a gen_ai.evaluation.result span event (and mirrored span attributes) on the judge's invoke_agent span, so conversation turn badges can render.

  • with_judge_evaluation holds the judge invoke_agent span open until scoring finishes. execute_and_track returns after the handler has already called span.end(), so without the delay the event would be dropped.
  • Event attributes: gen_ai.evaluation.name and gen_ai.evaluation.score.value, mirrored as span attributes. score.label is not invented, and only a finite number is recorded.
  • Existing track(evaluation_metric_key) behavior is unchanged and still feeds AI Config Monitoring.
  • TELEMETRY-CONTRACT.md section 4a documents the event contract; the evaluation keys are added to the cross-handler vocabulary test.

Mirrors js-ai-sdk#27. Part of O11Y-1888.

Stacking

⚠️ Based on O11Y-1888-emit-conversation-id-and-evaluation (#42), not main. Review apart, merge together — merge #42 first, then this.

Split out per review: this half overrides span.end on a live span, which is the riskiest surface in the original combined PR and deserves its own read. Note it shares only conversation.py with #42.

Reviewer notes

The end() patch is the part worth scrutiny. Nothing breaks today — judges only take the blocking path — but it is the one place the SDK mutates a live span's lifecycle, and span lifecycle is this SDK's worst historical surface.

Test plan

  • uv run pytest — 1073 pass, 11 skipped
  • uv run mypy packages/*/src clean; ruff check / ruff format --check clean
  • Evaluation event lands on the invoke_agent span after the handler has already ended it
  • score.label is not invented
  • Against staging OTLP: judge enabled → judge turn shows a score badge; $ld:ai: metric track still fires

Review follow-ups (pushed)

  • No gen_ai.evaluation.explanation. The judge's reasoning is model prose about the user's
    conversation — content — and captureContent / capture_content is a handler-factory option
    this layer never receives, so it was being exported for callers who had capture switched off.
    The reasoning still reaches the caller in judge_results; only the telemetry copy is withheld.
    Re-adding it needs its own opt-in.
  • Error semantics restored. The provider call is back outside the try, so a judge
    provider/auth/network failure rejects as it did before instead of being swallowed and logged as
    a parse failure. (Python: the float(score) raise site ahead of the metric track is gone.)
  • Score validated. Only a finite number reaches the span; the existing
    evaluationMetricKey / evaluation_metric_key track guard is untouched.
  • End time frozen at the handler's end() call, so the deferred end no longer stamps release
    time and inflate the judge span by the tracking and parsing work in between. Test asserts it.

Still open, deliberately deferred — the span.end patch itself. Three reviewers independently
suggested the same deeper fix: let the judge own its own span (or emit on a dedicated linked span)
instead of mutating a handler-owned live span. That would also retire the single-slot capture and
the is_recording() divergence. Worth a decision before this lands, not a quiet rewrite.


Staging verification

The conversation example added on this branch (three turns under one id, judge output printed from
judge_results) was run against default / staging:

https://ld-stg.launchdarkly.com/projects/default/traces?selected-env=staging&tab=traces&startDate=2026-08-20T17%3A30%3A00Z&endDate=2026-08-20T19%3A00%3A00Z&query=gen_ai.conversation.id%3Dconversation-example-23bcd8fa

Six spans across three traces, grouped by gen_ai.conversation.id, all Ok.

The judge path in this PR is still unverified end to end. The AI Config used has no
judge_configuration, so no judges ran — the example reported that explicitly rather than appearing
to pass. Confirming the gen_ai.evaluation.result event in the product needs a judge-enabled config
(a second AI Config wired as the judge). Unit tests cover the event, the score guard, the withheld
explanation, and the frozen end time; the product-level check is outstanding.

Judge scores land as a `gen_ai.evaluation.result` span event (and mirrored
attributes) on the judge `invoke_agent` span, so conversation turn badges can
render. `with_judge_evaluation` holds that span open until scoring finishes —
`execute_and_track` returns after the handler has already called `span.end()`,
so without the delay the event would be dropped.

Existing `track(evaluation_metric_key)` behavior is unchanged.
TELEMETRY-CONTRACT.md section 4a documents the event contract.

Stacked on the conversation-id PR: the two halves share conversation.py and
nothing else, so they review apart and merge together.

O11Y-1888

Co-Authored-By: Claude <noreply@anthropic.com>
Vega and others added 4 commits August 19, 2026 17:09
…Y-1888-judge-evals

# Conflicts:
#	packages/client/tests/test_conversation.py
…nd time

Review follow-ups on the judge half.

- Do not emit `gen_ai.evaluation.explanation`. The judge's reasoning is
  model-generated prose about the user's conversation — content — and AGENTS.md
  gates content attributes behind `capture_content`, a handler-factory option
  this layer never receives. It was exported unconditionally, including for
  callers who left capture off. The reasoning is still returned to the caller
  in `judge_results`; only the telemetry copy is withheld.
- Replace `float(score)` with a non-raising finite-number guard. `float()` was
  a new raise site sitting ahead of the existing
  `client.track(evaluation_metric_key, …)` call, so a judge returning
  "0.9 (high)" silently killed the metric this PR claims is unchanged — and in
  `run_judge` it escaped uncaught, breaking that function's documented
  "returns None" contract.
- Freeze the end time when the handler calls `end()`. Replaying a no-arg
  `end()` at release let the SDK stamp `time_ns()` then, inflating every judge
  span by the tracking and parsing work that runs in between.
- Detach the judge capture even if the deferred end raises.

O11Y-1888

Co-Authored-By: Claude <noreply@anthropic.com>
One run that exercises everything O11Y-1888 touches: three turns bound to a
single conversation id, each with inline judge evaluation. Prints the id so it
can be opened directly in the Conversations view.

It is also the manual check for the content decision — the judge's reasoning is
printed from `judge_results` (the caller's copy) and should appear nowhere in
the exported telemetry.

O11Y-1888

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant