A service that grades AI agents' answers against declarative rubrics. A task is a prompt plus a rubric, which is a list of checks. The agent submits an answer, and the service returns a verdict that explains every check and keeps per-task stats.
FastAPI · PostgreSQL · SQLAlchemy 2 (async) · pytest · Docker · GitHub Actions
When you run an agent over a set of tasks, it's usually the bookkeeping around the model that breaks, not the model. A retry after a timeout gets counted as a second attempt and drags the pass rate down. A broken check looks like "the agent failed". p95 gets interpolated and reports a latency nobody ever saw. This service is built around those spots, not around the happy path.
Nothing submitted is executed. Checks are declarative: contains,
not_contains, regex, json_equals, numeric_range, max_latency_ms. That
limits what a rubric can express, but the grading harness never becomes a way to
run arbitrary code on your side.
The rubric is validated when the task is created. A broken check returns 422
with the reason to the task author, instead of surfacing a week later as "the agent
didn't pass".
Retries collapse on idempotency_key. The same key within a task returns the
original result with 200 instead of 201, and a unique index in the database holds
that even under concurrent retries. A test that fires four parallel requests with one
key found a real bug here; it is explained in a comment in app/main.py.
| Method | Path | What it does |
|---|---|---|
GET |
/health |
liveness check |
POST |
/tasks |
create a task with a rubric (validated right away) |
GET |
/tasks |
list tasks |
POST |
/tasks/{slug}/submissions |
grade an agent's answer |
GET |
/tasks/{slug}/stats |
pass rate, mean score, p95 latency, failure breakdown |
Interactive schema: /docs.
curl -X POST localhost:8000/tasks -H 'content-type: application/json' -d '{
"slug": "json-status",
"prompt": "Return {\"status\": \"done\"}",
"rubric": [
{"kind": "json_equals", "path": "status", "value": "done"},
{"kind": "not_contains", "value": "api_key", "weight": 3},
{"kind": "max_latency_ms", "value": 500}
]
}'
curl -X POST localhost:8000/tasks/json-status/submissions \
-H 'content-type: application/json' \
-d '{"agent": "claude", "output": "{\"status\": \"done\"}", "latency_ms": 120, "idempotency_key": "run-1"}'The response has passed (whether all checks passed), score (the share of
weight that passed, so "completely off" and "almost right" look different) and
checks, with an explanation for each one.
docker compose up --build # API on :8000, Postgres on :5432Locally without Docker:
pip install -r requirements-dev.txt
DATABASE_URL="postgresql+asyncpg://grader:grader@localhost:5432/grader" \
uvicorn app.main:app --reloadpytest # SQLite, fast
DATABASE_URL=postgresql+asyncpg://... pytest # the way CI runs them34 tests: the grading engine on its own, and the HTTP layer end to end through an
ASGI transport. CI runs them on PostgreSQL 16 (postgres:16-alpine), the same engine
as production, because a SQLite run won't catch the differences between databases.
It also builds the image, brings up docker compose and checks /health with a live
request.
- No Alembic. The schema is created from the models. For a service this size that's a reasonable trade; once the schema starts changing on live data, it stops being one.
- No authentication. The service is meant to run internally. Don't expose it without an auth layer in front.
- No queue. Grading is synchronous and fast because it doesn't execute code. Once slow checks show up, it will need a worker, and that's a different project.