feat(evaluations): integrate offline judges and deterministic scorers

AGENTS.md

@@ -703,3 +703,10 @@ response = await graph(
     },
 ).invoke(user_input, context)
 ```
+
+## Maintaining this file
+
+Keep this file for knowledge useful to almost every future agent session in this project.
+Do not repeat what the codebase already shows; point to the authoritative file or command instead.
+Prefer rewriting or pruning existing entries over appending new ones.
+When updating this file, preserve this bar for all agents and keep entries concise.

CLAUDE.md

@@ -0,0 +1,2 @@
+<!-- Points Claude at AGENTS.md via import; edit AGENTS.md, not this file. -->
+@AGENTS.md

packages/ai/README.md

@@ -57,7 +57,7 @@ Never raises. Returns `{"enabled": bool, "config": dict | None, "meta": dict | N
 `init_evaluations` and the evaluations result types are also re-exported:
 
 ```python
-from launchdarkly_ai_python import init_evaluations
+from launchdarkly_ai_python import Accuracy, Scorer, init_evaluations
 
 evals = init_evaluations()
 result = await evals.run(
@@ -66,10 +66,15 @@ result = await evals.run(
     dataset="golden-dataset",
     handler=my_handler,
     generation={"provider": "OpenAI", "model": "gpt-4o"},
+    judges=[
+        Accuracy(),
+        Scorer(name="exact-match", fn=lambda row, output: output == row.expected_output),
+    ],
 )
+print(result.evaluation_results)
 ```
 
-`LD_API_TOKEN` is required. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
+`LD_API_TOKEN` is required, and LaunchDarkly judges also require `LD_SDK_KEY`; deterministic `Scorer` values do not. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
 
 ---
 

packages/client/README.md

@@ -44,18 +44,18 @@ No code changes are required — `init_client()` detects the packages at runtime
 
 ### Run an evaluation from code
 
-The generation-only evaluations harness reads an LD-hosted dataset, creates a new evaluation and client-source run, invokes your handler once per row, uploads the generations, and returns LaunchDarkly's stored verdict. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
+The evaluations harness reads an LD-hosted dataset, creates a new evaluation and client-source run, invokes your handler once per row, optionally runs typed LaunchDarkly judges and deterministic scorers in the same worker, uploads the generations, and returns LaunchDarkly's stored verdict. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
 
 ```python
 import asyncio
 import sys
 
 from launchdarkly_ai_openai_messages import create_openai_messages_handler
-from launchdarkly_ai_server import init_evaluations
+from launchdarkly_ai_server import Accuracy, Judge, Scorer, init_evaluations
 
 
 async def main() -> int:
-    evals = init_evaluations()  # LD_API_TOKEN required; LD_SDK_KEY optional
+    evals = init_evaluations()  # LD_API_TOKEN + LD_SDK_KEY for LD judges
     result = await evals.run(
         project_key="my-project",
         key="support-qa-2026-08-20",
@@ -66,8 +66,17 @@ async def main() -> int:
             "model": "gpt-4o",
             "instructions": "You are a support agent.",
         },
+        judges=[
+            Accuracy(),
+            Judge(key="security-judge", threshold=0.7),
+            Scorer(
+                name="exact-match",
+                fn=lambda row, output: output == row.expected_output,
+            ),
+        ],
     )
     print(result.url, result.summary)
+    print(result.evaluation_results)
     return 0 if result.passed else 1
 
 
@@ -76,6 +85,8 @@ sys.exit(asyncio.run(main()))
 
 `project_key` is supplied per run rather than during initialization. `generation.instructions` is shorthand for one system message; use `generation.messages` instead for a full message list, but do not supply both. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.
 
+`judges` accepts only typed `JudgeReference` values (`Accuracy`, `AnswerRelevancy`, `Likeness`, `Bias`, `Toxicity`, `Misinformation`, or `Judge`) and `Scorer` values. LaunchDarkly judges require `LD_SDK_KEY` and a generation handler built with `create_handler()` so the resolved judge model can be routed safely. Scorers may be synchronous or asynchronous and receive `(row, generation_output)`; `row` includes the rendered row index, input, expected output, variables, and metadata. Results are returned in `EvalRunResult.evaluation_results`. This judging foundation does not yet submit those local scores to LaunchDarkly's evaluation-results endpoint, so `result.passed` remains the stored generation-run verdict.
+
 The client uses **lazy initialization**: importing the package does not connect to LaunchDarkly. The singleton is created automatically on the first API call that needs it (`config().invoke()`, `graph().invoke()`, `resolve_graph()`, etc.), as long as `LD_SDK_KEY` is set in the environment.
 
 Call `init_client()` explicitly when you want to:

packages/client/agents.md

@@ -30,7 +30,7 @@ No other `launchdarkly-ai-*` package may define or duplicate these. They import
 | `src/launchdarkly_ai_server/utils.py` | `parse_template`, `parse_json_with_possible_fences`, `create_handler`, `parse_usage`, `make_track_data`, `to_ld_context` |
 | `src/launchdarkly_ai_server/registry.py` | `Registry`, `global_registry`, `compose`, `resolve_handlers`, `resolve_tools` |
 | `src/launchdarkly_ai_server/judges.py` | `run_judges`, `build_judge_tasks`, `run_judge` |
-| `src/launchdarkly_ai_server/evaluations/` | `init_evaluations`, the private management API operations, and generation-only `EvaluationsModule.run()` orchestration |
+| `src/launchdarkly_ai_server/evaluations/` | `init_evaluations`, the private management API operations, offline judge/scorer resolution, and `EvaluationsModule.run()` orchestration |
 | `src/launchdarkly_ai_server/__init__.py` | Public barrel — the only surface handler packages import from |
 
 ---
@@ -126,9 +126,9 @@ Handlers may return any of these — the client normalizes them before emitting
 
 ## SDK-run evaluations
 
-`init_evaluations()` creates an evaluations harness using `LD_API_TOKEN` and the management API host `LD_API_BASE_URI`. Do not reuse `LD_BASE_URI`: that variable configures SDK flag delivery and may point at a relay proxy. `LD_SDK_KEY` is optional for generation-only runs and enables the normal handler observability path.
+`init_evaluations()` creates an evaluations harness using `LD_API_TOKEN` and the management API host `LD_API_BASE_URI`. Do not reuse `LD_BASE_URI`: that variable configures SDK flag delivery and may point at a relay proxy. `LD_SDK_KEY` is optional for generation-only or deterministic-scorer runs, and required when `judges` contains a LaunchDarkly judge reference.
 
-`await EvaluationsModule.run(...)` takes `project_key` per call. Dataset lookup/row pagination, evaluation creation, and run creation are private helpers; only `run()` is public. Each call creates a new evaluation with `POST`, so its key must be unique. The harness directly invokes the supplied handler once per row, never retries it, batches generation ingest, and trusts only the server's stored verdict.
+`await EvaluationsModule.run(...)` takes `project_key` per call. Dataset lookup/row pagination, evaluation creation, and run creation are private helpers; only `run()` is public. Each call creates a new evaluation with `POST`, so its key must be unique. The harness directly invokes the supplied handler once per row, never retries it, then runs typed `JudgeReference` / `Scorer` values with the generated output and full rendered row context. Generation results are batch-ingested; local evaluation outcomes are returned in `EvalRunResult.evaluation_results`, while `passed` remains the server's stored generation-run verdict until evaluation-results ingest lands.
 
 ## OTel Setup
 

packages/client/pyproject.toml

@@ -2,7 +2,7 @@
 name = "launchdarkly-ai-server"
 version = "0.1.3"
 requires-python = ">=3.12"
-dependencies = ["opentelemetry-api>=1.25"]
+dependencies = ["opentelemetry-api>=1.25", "pydantic>=2"]
 description = "LaunchDarkly AI SDK core client for Python"
 readme = "README.md"
 license = "Apache-2.0"

packages/client/src/launchdarkly_ai_server/__init__.py

@@ -17,12 +17,32 @@
     to_semconv_finish_reason,
 )
 from .evaluations import (
+    Accuracy,
+    AnswerRelevancy,
+    Bias,
     EvalRunResult,
+    EvaluationMethod,
     EvaluationsError,
     EvaluationsModule,
     GenerationConfig,
+    Judge,
+    JudgeEvaluationError,
+    JudgeEvaluationResult,
+    JudgeIdentity,
+    JudgeReference,
+    JudgeUsage,
+    LaunchDarklyJudgeEvaluation,
+    Likeness,
+    Misinformation,
     RunSummary,
+    Scorer,
+    ScorerError,
+    ScorerResult,
+    ScorerRow,
+    ScoreValue,
+    Toxicity,
     init_evaluations,
+    resolve_launchdarkly_judges,
 )
 from .graph import GraphInstance, graph, resolve_graph
 from .judges import build_judge_tasks, run_judge, run_judges
@@ -159,12 +179,32 @@
     "to_semconv_finish_reason",
     "VariationMeta",
     # evaluations
+    "Accuracy",
+    "AnswerRelevancy",
+    "Bias",
     "EvalRunResult",
+    "EvaluationMethod",
     "EvaluationsError",
     "EvaluationsModule",
     "GenerationConfig",
+    "Judge",
+    "JudgeEvaluationError",
+    "JudgeEvaluationResult",
+    "JudgeIdentity",
+    "JudgeReference",
+    "JudgeUsage",
+    "LaunchDarklyJudgeEvaluation",
+    "Likeness",
+    "Misinformation",
     "RunSummary",
+    "ScoreValue",
+    "Scorer",
+    "ScorerError",
+    "ScorerResult",
+    "ScorerRow",
+    "Toxicity",
     "init_evaluations",
+    "resolve_launchdarkly_judges",
     # utils
     "create_handler",
     "make_track_data",

packages/client/src/launchdarkly_ai_server/evaluations/__init__.py

@@ -9,21 +9,59 @@
     Transport,
     urllib_transport,
 )
+from .judges import (
+    Accuracy,
+    AnswerRelevancy,
+    Bias,
+    EvaluationMethod,
+    Judge,
+    JudgeEvaluationError,
+    JudgeEvaluationResult,
+    JudgeIdentity,
+    JudgeReference,
+    JudgeUsage,
+    LaunchDarklyJudgeEvaluation,
+    Likeness,
+    Misinformation,
+    Toxicity,
+    resolve_launchdarkly_judges,
+)
 from .module import EvaluationsModule, init_evaluations
+from .scorers import Scorer, ScorerError, ScorerResult, ScorerRow, ScoreValue
 from .types import EvalRunResult, GenerationConfig, RunSummary, Usage
 
 __all__ = [
     "DEFAULT_BASE_URI",
+    "Accuracy",
+    "AnswerRelevancy",
+    "Bias",
     "EvalRunResult",
+    "EvaluationMethod",
     "EvaluationsError",
     "EvaluationsModule",
     "GenerationConfig",
     "HttpResponse",
+    "Judge",
+    "JudgeEvaluationError",
+    "JudgeEvaluationResult",
+    "JudgeIdentity",
+    "JudgeReference",
+    "JudgeUsage",
     "LDApiClient",
     "LDApiError",
+    "LaunchDarklyJudgeEvaluation",
+    "Likeness",
+    "Misinformation",
     "RunSummary",
+    "ScoreValue",
+    "Scorer",
+    "ScorerError",
+    "ScorerResult",
+    "ScorerRow",
+    "Toxicity",
     "Transport",
     "Usage",
     "init_evaluations",
+    "resolve_launchdarkly_judges",
     "urllib_transport",
 ]

packages/client/src/launchdarkly_ai_server/evaluations/flags.py

@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+import inspect
+import logging
+from typing import Any, Final
+
+from ..utils import to_ld_context
+
+logger = logging.getLogger(__name__)
+
+ENABLE_BATCH_INGEST_IN_EVALS_FROM_CODE_FLAG_KEY: Final[str] = (
+    "enable-batch-ingest-in-evals-from-code"
+)
+"""Canonical rollout flag for generation-result batch ingestion."""
+
+
+async def is_generation_result_batch_ingest_enabled(
+    client: Any,
+    project_key: str,
+) -> bool:
+    """Return whether the rollout flag enables generation-result batch ingest.
+
+    Flag evaluation is fail-safe: false, malformed, or failed evaluations disable
+    the gated batch-ingest path.
+    """
+    try:
+        context = to_ld_context(
+            client,
+            {"kind": "project", "key": project_key},
+        )
+        result = client.variation(
+            ENABLE_BATCH_INGEST_IN_EVALS_FROM_CODE_FLAG_KEY,
+            context,
+            False,
+        )
+        value = await result if inspect.isawaitable(result) else result
+        return value is True
+    except Exception:
+        logger.warning(
+            "Unable to evaluate %s; generation results will not be batch ingested",
+            ENABLE_BATCH_INGEST_IN_EVALS_FROM_CODE_FLAG_KEY,
+            exc_info=True,
+        )
+        return False