llama-swap Docs Agent Eval

An evaluation harness for the llama-swap Docs Agent.

This project runs a set of YAML-defined questions against a Docs Agent connected through an OpenAI-compatible /v1/chat/completions API, executes the documentation tools locally, and grades the agent's responses and tool usage against explicit expectations.

The goal is to make it easy to test whether a model can reliably answer questions about llama-swap using its documentation and configuration schema.

What it tests

The evaluator checks both what the agent says and how it gets there.

Evaluation cases can assert:

• Expected text or regex matches in the final answer
• At least one of several acceptable answers
• Text that must not appear in the answer
• Required tool calls
• One-of-many acceptable tool calls
• Tools that must not be called
• Whether the agent actually used tools
• Maximum number of agent iterations
• API/tool-call errors

This makes it useful for evaluating things such as:

• Documentation retrieval
• Configuration questions
• TTL and model lifecycle behavior
• Routing behavior
• Correct use of the configuration schema
• Tool selection
• Avoiding hallucinated configuration options
• Following the Docs Agent's instructions
• Comparing different models or model configurations

How it works

The evaluator connects to an OpenAI-compatible endpoint:

/v1/chat/completions

For each evaluation case, it:

1. Sends the question to the model with the llama-swap Docs Agent system prompt.
2. Provides the Docs Agent's tools through OpenAI-compatible function calling.
3. Executes those tools locally against the llama-swap documentation and config schema.
4. Continues the agent/tool loop until the model produces an answer or reaches the iteration limit.
5. Grades the final answer and tool usage against the case's assertions.
6. Prints human-readable results to stderr.
7. Prints a JSON summary to stdout for scripting or CI.

The four documentation tools are implemented locally from the llama-swap repository, so the evaluator does not need the llama-swap Go server's documentation tools to be running separately.

Repository layout

When used from a llama-swap checkout, the expected layout is:

llama-swap/
├── config-schema.json
├── docs/
│   └── kb/
└── evals/
    └── docs-agent/
        └── cases/
            ├── ...
            └── ...

The evaluator locates the llama-swap repository automatically when run from within the checkout.

Requirements

• Python 3.9+
• A running OpenAI-compatible API endpoint
• A model available through that endpoint
• A llama-swap checkout containing:
  • config-schema.json
  • docs/kb/
  • evals/docs-agent/cases/

Python dependencies:

pip install pyyaml requests

Usage

From the llama-swap repository:

python eval_openai_compatible.py

Specify a model and API endpoint:

python eval_openai_compatible.py \
  --model your-model \
  --base-url http://localhost:8080

You can also explicitly provide the llama-swap repository root:

python eval_openai_compatible.py \
  --repo-root /path/to/llama-swap \
  --model your-model \
  --base-url http://localhost:8080

The evaluator performs a preflight check against /v1/models before running any cases and will fail early if the server cannot be reached or the requested model is unavailable.

Selecting evaluation cases

Cases are organized into groups.

Run a specific group:

python eval_openai_compatible.py --group ttl

Run multiple groups:

python eval_openai_compatible.py \
  --group routing \
  --group ttl

Run a single case or a subset of cases by ID:

python eval_openai_compatible.py \
  --only ttl-unload-after-5min

Use a regex to select multiple cases:

python eval_openai_compatible.py \
  --only 'ttl-.*'

Limit the number of cases for a quick test:

python eval_openai_compatible.py --limit 5

Concurrency

Cases can be run concurrently:

python eval_openai_compatible.py \
  --concurrency 4

Results are still emitted in deterministic case order in the final JSON output.

The default is:

concurrency=1

Increase this when the API endpoint and model can handle multiple simultaneous requests.

Evaluation cases

Cases are defined in YAML files.

A simplified example:

- id: ttl-unload-after-5min
  question: "How do I configure a model to unload after 5 minutes?"
  tags:
    - ttl

  expect_regex:
    - "ttl"
    - "300"

  expect_tools:
    - docs__search_docs
    - docs__get_config_schema

  max_iterations: 5

The evaluator supports several assertion types.

expect_regex

Every pattern must match the final answer:

expect_regex:
  - "globalTTL"
  - "seconds"

Matching is case-insensitive.

expect_any

At least one pattern must match:

expect_any:
  - "globalTTL"
  - "models.*.ttl"

forbid_regex

A pattern must not appear in the final answer:

forbid_regex:
  - "someMadeUpOption"

expect_tools

Every listed tool must be called:

expect_tools:
  - docs__search_docs
  - docs__get_config_schema

expect_tools_any

At least one of the listed tools must be called:

expect_tools_any:
  - docs__get_doc
  - docs__list_docs

forbid_tools

The listed tools must not be called:

forbid_tools:
  - config__get_config

require_tools

Require the agent to use at least one tool:

require_tools: true

max_iterations

Limit the number of model/tool iterations for a case:

max_iterations: 6

Docs Agent tools

The evaluator provides the llama-swap Docs Agent with these tools:

Tool	Purpose
docs__search_docs	Keyword search across the documentation
docs__list_docs	List available documentation
docs__get_doc	Read a documentation page
docs__get_config_schema	Look up a configuration key in config-schema.json
config__get_config	Dummy running configuration for evaluation
sys__now	Current server date/time

The documentation tools read directly from docs/kb/**/*.md.

Configuration schema lookups read from:

config-schema.json

This keeps the evaluation environment deterministic and avoids requiring the actual llama-swap Docs Agent tool implementation to be running.

Output

Human-readable progress and failures are printed to stderr:

[  1/20] ttl-unload-after-5min -> PASS 3it 4.2s tools=['docs__search_docs', 'docs__get_config_schema']
[  2/20] routing-basic -> PASS 2it 2.8s tools=['docs__search_docs']
[  3/20] ttl-invalid-key -> FAIL 4it 5.1s tools=['docs__search_docs']

======================================================================
Results: 2/3 passed (66.7%)  1 failed  12.1s
Model: your-model  Base: http://localhost:8080  Iterations avg: 3.0

A JSON summary is written to stdout, making it possible to pipe the results into other tooling:

python eval_openai_compatible.py > results.json

The JSON includes:

• Model
• API base URL
• Number of cases
• Passed/failed counts
• Pass rate
• Per-case results
• Iteration counts
• Tool calls
• Errors
• Final answers
• Individual assertion results

Environment variables

An API key can be supplied with:

export OPENAI_API_KEY="..."

or:

python eval_openai_compatible.py \
  --api-key "..."

The API key is only sent as a Bearer token to the configured API endpoint.

Useful options

--repo-root PATH       Path to the llama-swap repository
--model MODEL          Model name
--base-url URL         OpenAI-compatible API base URL
--api-key KEY          API key (defaults to OPENAI_API_KEY)
--cases PATH           Evaluation cases directory
--group GROUP          Filter by group; repeatable
--only REGEX           Filter case IDs by regex
--max-iterations N     Default maximum agent iterations
--temperature N        Model temperature
--concurrency N        Number of cases to run concurrently
--timeout N             API request timeout in seconds
--retries N             Retries for transient API failures
--limit N               Run only the first N matching cases

Run:

python eval_openai_compatible.py --help

for the complete list.

Why this exists

LLM-based documentation agents can appear to work while still making subtle mistakes: using the wrong configuration key, skipping documentation lookup, inventing options, or giving an answer that is technically plausible but unsupported by the current llama-swap documentation.

This evaluator makes those behaviors testable.

The evaluation cases are intended to evolve alongside llama-swap's documentation and configuration surface, providing a repeatable way to check Docs Agent behavior as models, prompts, and llama-swap itself change.

Status

This is an evaluation harness for the llama-swap Docs Agent and is intended primarily for development, model evaluation, and regression testing.

The evaluator currently targets OpenAI-compatible chat-completions APIs with tool/function calling.

I'd also add a very short GitHub tagline/description:

> **Evaluation harness for the llama-swap Docs Agent using OpenAI-compatible APIs.**

And for the repo name, **`llama-swap-docs-agent-eval`** pairs nicely with this README.