feat: join resumed turns, expose the evaluated span and run id, rename the eval set id

README.md

@@ -118,7 +118,7 @@ python your_agent.py
 
 Sessions appear in the UI at `http://localhost:8001` as the agent runs, with tool calls, inputs and outputs, and can be evaluated there. For gRPC exporters use port 4317 with `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.
 
-Sessions are grouped by `agentevals.session_name`, then `gen_ai.conversation.id`, then `session.id`. `service.instance.id` tells a rerun from the next turn. Set `agentevals.eval_set_id` to associate a session with an eval set.
+Sessions are grouped by `agentevals.session_name`, then `gen_ai.conversation.id`, then `session.id`. `service.instance.id` tells a rerun from the next turn. Set `agentevals.eval.set.id` to associate a session with an eval set.
 
 Per producer settings, such as turning on message content capture, are in [OpenTelemetry Compatibility](docs/otel-compatibility.md). Working setups for each framework are in [examples/zero-code-examples/](examples/zero-code-examples/).
 

docs/eval-set-format.md

@@ -293,6 +293,8 @@ Turns are compared in order. Extra or missing turns are reported as `unexpected_
 
 Use `conversation` when each turn of a multi turn conversation is its own trace.
 
+In every mode, a trace that resumes a paused turn of another trace (a span link between their `invoke_agent` spans, see [Sending telemetry](otel-compatibility.md#what-agentevals-needs)) is evaluated with that trace.
+
 ## Usage
 
 ### CLI

docs/grpc-api.md

@@ -20,6 +20,10 @@ Sessions and conversations come back as views: turns with their user text, tool
 
 Traces go in as OTLP protobuf (`TracesData` and `LogsData`) or as files (OTLP/JSON, OTLP/JSONL or Jaeger JSON), the same formats the REST API takes.
 
+`EvaluateTraces` lists the judged turns of each group in `turn_refs`. `evaluated_trace_id` and `evaluated_span_id` name the span the turn's results belong to, the same span agentevals attaches its own [evaluation result records](opentelemetry-pipeline.md#send-evaluation-results-from-agentevals) to. Use it when you attach results to telemetry yourself.
+
+`EvaluateTraces` takes an optional `run_id` (a UUID). Every result record of the call carries it as `agentevals.eval.run.id`, and with run storage the evaluation is recorded under it. An id that is already recorded is rejected with `ALREADY_EXISTS`. Without `run_id`, a call recorded in run storage stamps its records with the recorded run's id.
+
 ## Calling it
 
 With reflection on, `grpcurl` can list and call everything:

docs/opentelemetry-pipeline.md

@@ -158,17 +158,29 @@ agentevals run traces.json -e eval_set.json -m tool_trajectory_avg_score --emit-
 
 `OTEL_EXPORTER_OTLP_PROTOCOL=grpc` (with port 4317) switches to gRPC. `OTEL_SDK_DISABLED=true` or `OTEL_LOGS_EXPORTER=none` turns emitting off even when it is enabled. `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` work as usual.
 
-Each result is a `gen_ai.evaluation.result` log event attached to the span it scores (the turn's agent span, or its last model call). Metrics that score each turn give one event per turn; others give one event on the last turn.
+Each result is a `gen_ai.evaluation.result` log event attached to the span it scores (the turn's agent span, or its last model call). Metrics that score each turn give one event per turn; others give one event on the last turn. The evaluation APIs name the same span in each turn reference (`evaluated_trace_id`, `evaluated_span_id`).
 
 | Attribute | Value |
 |---|---|
 | `gen_ai.evaluation.name` | Evaluator name |
 | `gen_ai.evaluation.score.value` | Score, when there is one |
-| `gen_ai.evaluation.score.label` | `pass`, `fail` or `not_evaluated` |
+| `gen_ai.evaluation.score.label` | `pass`, `fail`, `not_evaluated` or `informational`, see below |
 | `error.type` | Exception class when the evaluator failed (no score or label then) |
 | `gen_ai.evaluation.explanation` | Only with `AGENTEVALS_EVALUATION_EVENTS_EXPLANATION=true` |
 | `gen_ai.agent.name`, `gen_ai.conversation.id`, `gen_ai.response.id` | Copied from the evaluated span when present |
 | `agentevals.evaluator.type` | `llm_judge` or `deterministic` for built in metrics |
-| `agentevals.eval.run.id`, `agentevals.eval.case.id`, `agentevals.eval_set.id` | When known |
+| `agentevals.eval.run.id` | The run: a queued run, an evaluation recorded in run storage, or the `run_id` of an `EvaluateTraces` call |
+| `agentevals.eval.case.id`, `agentevals.eval.set.id` | When known |
+
+| Label | Meaning |
+|---|---|
+| `pass` | The score met the evaluator's threshold |
+| `fail` | The score missed the threshold |
+| `not_evaluated` | The evaluator gave no score, for example because the metric needs an eval set |
+| `informational` | A value to track that never passes or fails, such as `token_usage_v1` |
+
+The GenAI conventions give `pass` and `fail` as example labels and ask each implementation to document its own; `not_evaluated` and `informational` are agentevals values.
+
+The ids have the same names as the resource attributes you set: `agentevals.eval.set.id` (and `agentevals.eval.case.id`) mean the same thing on your telemetry and on these events.
 
 Events never contain prompts, responses or tool arguments. The explanation can, which is why it is off by default.

docs/otel-compatibility.md

@@ -32,6 +32,8 @@ Working setups for each are in [examples/zero-code-examples](../examples/zero-co
 
 If a framework only instruments the model client (each model call is its own trace), agentevals joins a call that answers a tool result to the turn before it. This needs message content.
 
+If a runtime pauses a turn (for an approval or a question) and runs the rest as a new `invoke_agent` span, often in a new trace, add a span link from the new span to the `invoke_agent` span that started the turn. agentevals then scores both parts as one turn: the user input of the first, the final answer of the last, and the tools of both. Both spans must carry the same `gen_ai.agent.name`, and the new span no input messages of its own. Linked traces are evaluated together whatever the grouping.
+
 ## Sessions
 
 Live traces are grouped into sessions by the first key present:

examples/README.md

@@ -10,7 +10,7 @@ The simplest way to connect any agent to agentevals. Point your standard OTel OT
 
 ```bash
 export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
-export OTEL_RESOURCE_ATTRIBUTES="agentevals.session_name=my-agent,agentevals.eval_set_id=my-eval"
+export OTEL_RESOURCE_ATTRIBUTES="agentevals.session_name=my-agent,agentevals.eval.set.id=my-eval"
 python your_agent.py
 ```
 
@@ -38,7 +38,7 @@ This approach works with any framework that has OTel instrumentation: LangChain,
 | Attribute | Required | Description |
 |-----------|----------|-------------|
 | `agentevals.session_name` | no | Groups spans into a named session. Without it, sessions are named `otlp-<traceId prefix>`. |
-| `agentevals.eval_set_id` | no | Associates the session with an eval set for scoring. |
+| `agentevals.eval.set.id` | no | Associates the session with an eval set for scoring. |
 
 Set them via `OTEL_RESOURCE_ATTRIBUTES` (env var) or `Resource.create()` in code.
 

examples/kubernetes/README.md

@@ -169,7 +169,7 @@ Body: Empty()
 Attributes:
      -> gen_ai.evaluation.name: Str(tool_trajectory_avg_score)
      -> agentevals.eval.case.id: Str(case_1)
-     -> agentevals.eval_set.id: Str(manual-1)
+     -> agentevals.eval.set.id: Str(manual-1)
      -> agentevals.evaluator.type: Str(deterministic)
      -> gen_ai.evaluation.score.value: Double(1)
      -> gen_ai.evaluation.score.label: Str(pass)

examples/zero-code-examples/adk/run.py

@@ -48,7 +48,7 @@ async def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "adk-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=dice_agent_eval,agentevals.session_name=adk-zero-code",
+        "agentevals.eval.set.id=dice_agent_eval,agentevals.session_name=adk-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

examples/zero-code-examples/langchain/run.py

@@ -52,7 +52,7 @@ def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "langchain-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=langchain_agent_eval,agentevals.session_name=langchain-zero-code",
+        "agentevals.eval.set.id=langchain_agent_eval,agentevals.session_name=langchain-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

examples/zero-code-examples/ollama/run.py

@@ -116,7 +116,7 @@ def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "ollama-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=langchain_local_ollama_openai_eval,agentevals.session_name=langchain-ollama-openai-zero-code",
+        "agentevals.eval.set.id=langchain_local_ollama_openai_eval,agentevals.session_name=langchain-ollama-openai-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

examples/zero-code-examples/openai-agents/run.py

@@ -62,7 +62,7 @@ def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "openai-agents-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=openai_agents_eval,agentevals.session_name=openai-agents-zero-code",
+        "agentevals.eval.set.id=openai_agents_eval,agentevals.session_name=openai-agents-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

examples/zero-code-examples/pydantic-ai/run.py

@@ -58,7 +58,7 @@ def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "pydantic-ai-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=pydantic_ai_eval,agentevals.session_name=pydantic-ai-zero-code",
+        "agentevals.eval.set.id=pydantic_ai_eval,agentevals.session_name=pydantic-ai-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

examples/zero-code-examples/strands/run.py

@@ -44,7 +44,7 @@ def main():
     os.environ.setdefault("OTEL_SERVICE_NAME", "strands-agent")
     os.environ.setdefault(
         "OTEL_RESOURCE_ATTRIBUTES",
-        "agentevals.eval_set_id=strands_agent_eval,agentevals.session_name=strands-zero-code",
+        "agentevals.eval.set.id=strands_agent_eval,agentevals.session_name=strands-zero-code",
     )
     # A process level identity, so agentevals can tell a rerun of this script from the next turn.
     if "service.instance.id=" not in os.environ["OTEL_RESOURCE_ATTRIBUTES"]:

proto/agentevals/api/v1alpha1/evaluation.proto

@@ -146,8 +146,13 @@ message MetricResult {
 }
 
 message TurnRef {
+  // The turn's anchor: its agent or workflow span, else the root span of its trace.
   string trace_id = 1;
   string span_id = 2;
+  // The span the turn's gen_ai.evaluation.result records attach to: the anchor, or the turn's
+  // final model call when the anchor is not a GenAI span. Attach results here.
+  string evaluated_trace_id = 3;
+  string evaluated_span_id = 4;
 }
 
 // The results of one evaluation group: a trace, or a conversation of several traces.
@@ -194,6 +199,10 @@ message EvaluateTracesRequest {
   // {"kind": "kubernetes", "namespace": "...", "name": "...", "key": "..."} or
   // {"kind": "env", "name": "..."}. Values are resolved per request and never returned.
   map<string, google.protobuf.Struct> credential_refs = 4;
+  // Optional. A UUID stamped as agentevals.eval.run.id on every gen_ai.evaluation.result record of
+  // this evaluation. With run storage the evaluation is recorded under this id, and an id that is
+  // already recorded is rejected; without it, storage picks the id and the records carry that.
+  string run_id = 5;
 }
 
 message EvaluationProgress {

src/agentevals/api/evaluation.py

@@ -11,21 +11,20 @@
 import contextlib
 import logging
 from collections.abc import AsyncIterator
-from dataclasses import dataclass, field
-from typing import TYPE_CHECKING, Any
+from dataclasses import dataclass, field, replace
+from typing import Any
+from uuid import UUID, uuid4
 
 from google.adk.evaluation.eval_set import EvalSet
 
 from ..config import EvalParams
 from ..genai.extract import extract_conversation
 from ..otel.model import Trace
 from ..resolvers import reset_resolved_credentials, resolve_credential_refs, set_resolved_credentials
+from ..run.service import RunIdTaken, RunService
 from ..runner import RunResult, TraceResult, evaluation_groups, run_evaluation_from_traces
 from ..trace_metrics import extract_performance_metrics, extract_trace_metadata
 
-if TYPE_CHECKING:
-    from ..run.service import RunService
-
 logger = logging.getLogger(__name__)
 
 
@@ -73,6 +72,8 @@ class Evaluation:
     eval_set_dict: dict[str, Any] | None = None
     trace_format: str | None = None
     upload_filenames: list[str] | None = None
+    # Stamped on the evaluation result records, and the recorded run's id.
+    run_id: str | None = None
 
 
 def performance_events(traces: list[Trace], group_by: str, eval_set: EvalSet | None) -> list[Performance]:
@@ -99,7 +100,12 @@ async def stream_evaluation(
 
     Raises :class:`CredentialResolutionError` after the performance events when a credential
     reference cannot be resolved; nothing is evaluated then.
+
+    With ``run_service``, the run id is chosen before evaluating, so the records emitted during the
+    evaluation carry the id the run is recorded under.
     """
+    if run_service is not None and evaluation.run_id is None:
+        evaluation = replace(evaluation, run_id=str(uuid4()))
     for event in performance_events(evaluation.traces, evaluation.params.group_by, evaluation.eval_set):
         yield event
     resolved = await _resolve(evaluation.credential_refs)
@@ -122,6 +128,7 @@ async def run() -> None:
                     eval_set=evaluation.eval_set,
                     progress_callback=on_progress,
                     trace_progress_callback=on_group,
+                    run_id=evaluation.run_id,
                 )
             else:
                 result = RunResult(errors=["No traces loaded."])
@@ -181,13 +188,17 @@ async def persist_evaluation(
         return None
     try:
         run = await run_service.record_eval_run(
+            run_id=UUID(evaluation.run_id) if evaluation.run_id else None,
             params=evaluation.params,
             eval_set_dict=evaluation.eval_set_dict,
             trace_format=evaluation.trace_format,
             upload_filenames=evaluation.upload_filenames,
             run_result=result,
         )
         return str(run.run_id)
+    except RunIdTaken as exc:
+        logger.warning("Not recording the evaluation: %s", exc)
+        return None
     except Exception:
         logger.exception("Failed to record the evaluation as a run; the result is still returned")
         return None