Skip to content

About

Scores AI agent answers against declarative rubrics. FastAPI, PostgreSQL, Docker, 34 tests in CI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

agent-grader

Русская версия

A service that grades AI agents' answers against declarative rubrics. A task is a prompt plus a rubric, which is a list of checks. The agent submits an answer, and the service returns a verdict that explains every check and keeps per-task stats.

FastAPI · PostgreSQL · SQLAlchemy 2 (async) · pytest · Docker · GitHub Actions

Why

When you run an agent over a set of tasks, it's usually the bookkeeping around the model that breaks, not the model. A retry after a timeout gets counted as a second attempt and drags the pass rate down. A broken check looks like "the agent failed". p95 gets interpolated and reports a latency nobody ever saw. This service is built around those spots, not around the happy path.

Three decisions worth noticing

Nothing submitted is executed. Checks are declarative: contains, not_contains, regex, json_equals, numeric_range, max_latency_ms. That limits what a rubric can express, but the grading harness never becomes a way to run arbitrary code on your side.

The rubric is validated when the task is created. A broken check returns 422 with the reason to the task author, instead of surfacing a week later as "the agent didn't pass".

Retries collapse on idempotency_key. The same key within a task returns the original result with 200 instead of 201, and a unique index in the database holds that even under concurrent retries. A test that fires four parallel requests with one key found a real bug here; it is explained in a comment in app/main.py.

API

Method Path What it does
GET /health liveness check
POST /tasks create a task with a rubric (validated right away)
GET /tasks list tasks
POST /tasks/{slug}/submissions grade an agent's answer
GET /tasks/{slug}/stats pass rate, mean score, p95 latency, failure breakdown

Interactive schema: /docs.

Example

curl -X POST localhost:8000/tasks -H 'content-type: application/json' -d '{
  "slug": "json-status",
  "prompt": "Return {\"status\": \"done\"}",
  "rubric": [
    {"kind": "json_equals", "path": "status", "value": "done"},
    {"kind": "not_contains", "value": "api_key", "weight": 3},
    {"kind": "max_latency_ms", "value": 500}
  ]
}'

curl -X POST localhost:8000/tasks/json-status/submissions \
  -H 'content-type: application/json' \
  -d '{"agent": "claude", "output": "{\"status\": \"done\"}", "latency_ms": 120, "idempotency_key": "run-1"}'

The response has passed (whether all checks passed), score (the share of weight that passed, so "completely off" and "almost right" look different) and checks, with an explanation for each one.

Running it

docker compose up --build      # API on :8000, Postgres on :5432

Locally without Docker:

pip install -r requirements-dev.txt
DATABASE_URL="postgresql+asyncpg://grader:grader@localhost:5432/grader" \
  uvicorn app.main:app --reload

Tests

pytest                                   # SQLite, fast
DATABASE_URL=postgresql+asyncpg://... pytest   # the way CI runs them

34 tests: the grading engine on its own, and the HTTP layer end to end through an ASGI transport. CI runs them on PostgreSQL 16 (postgres:16-alpine), the same engine as production, because a SQLite run won't catch the differences between databases. It also builds the image, brings up docker compose and checks /health with a live request.

Deliberately left out

  • No Alembic. The schema is created from the models. For a service this size that's a reasonable trade; once the schema starts changing on live data, it stops being one.
  • No authentication. The service is meant to run internally. Don't expose it without an auth layer in front.
  • No queue. Grading is synchronous and fast because it doesn't execute code. Once slow checks show up, it will need a worker, and that's a different project.

About

Scores AI agent answers against declarative rubrics. FastAPI, PostgreSQL, Docker, 34 tests in CI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages