[krisztianfekete]

A turn that pauses for an approval or a question and resumes as a new invoke_agent span, often in a new trace, is now scored as one turn. The resumed span must link to the agent span that started the turn, both must carry the same gen_ai.agent.name, and the resumed span must record no input of its own. The turn keeps the first part's user input and the last part's answer, and takes tools from both. A tool call recorded in both parts (same call id, tool and agent) counts once: the span that finished wins and keeps arguments only the paused span recorded. Linked traces are evaluated together in every group_by mode. Before, each part was its own turn and the tool was counted twice.

TurnRef gains evaluated_trace_id and evaluated_span_id: the span the turn's gen_ai.evaluation.result records attach to. They come from the same function the emitter uses, so API clients can attach results exactly where agentevals does.

EvaluateTracesRequest gains an optional run_id (a UUID), stamped as agentevals.eval.run.id on every record of the call. With run storage, the evaluation is recorded under it, and an id that already exists is rejected with ALREADY_EXISTS. Evaluations recorded in run storage now choose their run id before evaluating, so their records carry the stored run's id on every route, including /api/streaming/evaluate-sessions. Before, those records had no run id. Recording never writes into a run that another run already holds: if the id is taken by then (two calls racing with the same run_id, say), the evaluation is not recorded and the existing run is left untouched.

All four gen_ai.evaluation.score.label values are documented (pass, fail, not_evaluated, informational).

Breaking changes

• The resource attribute agentevals.eval_set_id is now agentevals.eval.set.id. Update OTEL_RESOURCE_ATTRIBUTES; the old name is no longer read. The SDK sets the new name.
• Evaluation result records carry agentevals.eval.set.id instead of agentevals.eval_set.id. Update queries and dashboards.

Not in this PR

• The persisted Result.span_id still names the turn's anchor span.
• Runtimes that resume a turn without a span link between the agent spans are not joined.
• A Weaver registry for the agentevals.* attributes.

Testing

2067 unit tests and 29 integration tests pass. Each commit passes on its own. Ruff, uv lock --check, buf lint, buf breaking, the Go build and vet, and the UI type check, lint and build all pass. New tests cover resumed turns and grouping, the evaluated span against the emitted record's parent, run ids on gRPC and REST, and run id conflicts on both the memory and Postgres repositories (the storage tests pass against Postgres 16).

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Concurrent evaluations can claim the same run ID and overwrite or mix results because the ID is not atomically reserved before records are emitted.

5 open issues

• Reserve run IDs atomically before evaluation · New
• Require a named agent to accept a resume link · New
• Preserve paused tool arguments when replacing the call · New
• Scope call deduplication to matching cross-segment calls · New
• Pass the run ID through session evaluation routes · New

2 files not reviewed

• gen/go/agentevals/api/v1alpha1/evaluation.pb.go: Generated file
• src/agentevals/api/v1alpha1/evaluation_pb2.py: Generated file

What changed in this PR

This PR improves how agentevals evaluates paused and resumed agent turns and correlates evaluation results with telemetry.

Changes:

• Joins linked turn segments across traces and deduplicates repeated tool calls.
• Exposes evaluated span IDs and propagates run IDs through evaluation and storage.
• Renames the eval-set telemetry attribute to agentevals.eval.set.id and updates documentation, examples, generated bindings, and tests.

File	Description
ui/​src/​lib/​types.ts	Adds evaluated span identifiers to turn references.
ui/​src/​generated/​agentevals/​api/​v1alpha1/​evaluation_pb.js	Regenerates descriptors for new API fields.
ui/​src/​generated/​agentevals/​api/​v1alpha1/​evaluation_pb.d.ts	Adds generated types for span and run IDs.
tests/​test_sdk.py	Updates SDK resource-attribute assertions.
tests/​test_otlp_receiver.py	Updates receiver fixtures and assertions for the renamed attribute.
tests/​otel/​test_emit.py	Tests labels, attribute names, and evaluated span references.
tests/​otel/​builders.py	Adds span-link support to test builders.
tests/​integration/​test_live_agents.py	Uses the renamed attribute in agent environments.
tests/​integration/​conftest.py	Updates integration trace resource attributes.
tests/​genai/​test_resumed_turns.py	Tests resumed-turn merging and trace grouping.
tests/​api/​test_grpc_api.py	Tests requested run IDs, validation, and emitted records.
tests/​api/​test_evaluate_persistence.py	Tests stored run IDs on emitted records.
src/​agentevals/​runner.py	Exposes evaluated span references and forwards run IDs.
src/​agentevals/​run/​service.py	Accepts a preselected ID when recording evaluations.
src/​agentevals/​otel/​sdk_export.py	Updates the attribute name in module documentation.
src/​agentevals/​otel/​identity.py	Renames the shared eval-set attribute constant.
src/​agentevals/​genai/​grouping.py	Groups traces connected by resume links.
src/​agentevals/​genai/​extract.py	Joins resumed turns and deduplicates tool calls.
src/​agentevals/​evaluation_events.py	Shares evaluated-span selection and aligns attribute names.
src/​agentevals/​api/​v1alpha1/​evaluation_pb2.pyi	Regenerates Python type stubs.
src/​agentevals/​api/​v1alpha1/​evaluation_pb2.py	Regenerates Python protobuf descriptors.
src/​agentevals/​api/​rpc/​services.py	Validates and forwards caller-supplied run IDs.
src/​agentevals/​api/​rpc/​convert.py	Serializes expanded turn references.
src/​agentevals/​api/​routes.py	Chooses storage-backed run IDs before evaluation.
src/​agentevals/​api/​evaluation.py	Carries run IDs through streaming evaluation and persistence.
README.md	Documents the renamed eval-set attribute.
proto/​agentevals/​api/​v1alpha1/​evaluation.proto	Defines evaluated span fields and optional request run ID.
gen/​go/​agentevals/​api/​v1alpha1/​evaluation.pb.go	Regenerates Go protobuf bindings.
examples/​zero-code-examples/​strands/​run.py	Updates the Strands resource attribute.
examples/​zero-code-examples/​pydantic-ai/​run.py	Updates the Pydantic AI resource attribute.
examples/​zero-code-examples/​openai-agents/​run.py	Updates the OpenAI Agents resource attribute.
examples/​zero-code-examples/​ollama/​run.py	Updates the Ollama resource attribute.
examples/​zero-code-examples/​langchain/​run.py	Updates the LangChain resource attribute.
examples/​zero-code-examples/​adk/​run.py	Updates the ADK resource attribute.
examples/​README.md	Updates telemetry setup instructions.
examples/​kubernetes/​README.md	Updates the sample evaluation event attribute.
docs/​otel-compatibility.md	Documents resumed-turn telemetry requirements.
docs/​opentelemetry-pipeline.md	Documents evaluated spans, run IDs, and all score labels.
docs/​grpc-api.md	Explains evaluated span references and request run IDs.
docs/​eval-set-format.md	Documents grouping behavior for resumed traces.

_{Review effort: Balanced}