API.md

API reference and database schema

Checked against the route decorators in backend/src/agentprobe/infrastructure/api/routes/ and the models in backend/src/agentprobe/infrastructure/persistence/models/tables.py. The interactive OpenAPI docs are served at /docs when the backend runs.

API

Agent Execution

Method	Endpoint	Description
POST	/api/v1/run	Start agent run (SSE stream)

Run Management

Method	Endpoint	Description
GET	/api/v1/runs	List runs (paginated, filterable)
GET	/api/v1/runs/{id}	Run detail with full step trace
DELETE	/api/v1/runs/{id}	Delete a run
GET	/api/v1/runs/{id}/replay	Replay run as SSE stream

Benchmarking

Method	Endpoint	Description
GET	/api/v1/benchmarks/cases	List benchmark test cases
GET	/api/v1/benchmarks/cases/{id}	One benchmark test case
POST	/api/v1/benchmarks/cases	Create custom test case
POST	/api/v1/benchmarks/suites	Start benchmark suite (SSE progress)
GET	/api/v1/benchmarks/suites	List completed suites
GET	/api/v1/benchmarks/suites/{id}	Suite detail with per-case results

Analytics & Providers

Method	Endpoint	Description
GET	/api/v1/analytics/failures	Failure type breakdown
GET	/api/v1/analytics/models	Cross-model performance stats
GET	/api/v1/providers	Available providers + models
GET	/api/v1/tools	Built-in tool list
GET	/api/v1/health	Health check with provider status

Custom Tools & Prompts

Method	Endpoint	Description
POST	/api/v1/tools/custom	Create custom tool
GET	/api/v1/tools/custom	List custom tools
DELETE	/api/v1/tools/custom/{id}	Delete custom tool
POST	/api/v1/prompts	Create prompt template
GET	/api/v1/prompts	List prompt templates
PUT	/api/v1/prompts/{id}	Update prompt template
DELETE	/api/v1/prompts/{id}	Delete prompt template

Auth & Export

Method	Endpoint	Description
POST	/auth/register	Register new user
POST	/auth/login	Login (returns JWT)
POST	/auth/api-keys	Generate API key
GET	/auth/me	Current user info
GET	/api/v1/exports/runs/{id}/csv	Export run as CSV
GET	/api/v1/exports/benchmarks/{id}/csv	Export benchmark as CSV
GET	/api/v1/exports/benchmarks/{id}/pdf	Export benchmark as PDF

---

Database schema

erDiagram
    users ||--o{ api_keys : has
    users ||--o{ runs : owns
    users ||--o{ custom_tools : creates
    users ||--o{ prompt_templates : creates
    users ||--o{ memory_entries : stores
    runs ||--o{ steps : contains
    runs ||--o{ failures : records
    benchmark_suites ||--o{ benchmark_results : contains
    benchmark_cases ||--o{ benchmark_results : tested_by
    runs ||--o{ benchmark_results : produces

    users {
        string id PK
        string email UK
        string hashed_password
        datetime created_at
    }
    runs {
        string id PK
        string query
        string model_id
        string provider
        string status
        string final_answer
        int total_tokens
        float duration_ms
        bool succeeded
        string user_id FK
    }
    steps {
        int id PK
        string run_id FK
        int step_index
        string step_type
        string content
        string tool_name
        string failure_type
    }
    failures {
        int id PK
        string run_id FK
        int step_id FK
        string failure_type
        string context
    }
    benchmark_cases {
        string id PK
        string query
        string category
        string difficulty
    }
    benchmark_suites {
        string id PK
        string model_id
        string provider
        float success_rate
    }
    custom_tools {
        string id PK
        string user_id FK
        string name
        string tool_type
    }
    memory_entries {
        string id PK
        string user_id FK
        string key
        string value
    }

Loading

11 tables (the diagram shows the main columns of 8 of them; api_keys, prompt_templates and benchmark_results appear as relationships only). SQLite for local development, PostgreSQL for production. Alembic manages migrations.