llm-eval-lite

You don't always need a heavyweight eval framework — sometimes you just need to assert that the model said the right thing, returned valid JSON, and didn't leak a forbidden string. llm-eval-lite is that: composable checks + a runner + a markdown report, in pure stdlib.

Install

pip install -e .

Usage

from llmeval import Case, Contains, IsJSON, JSONHasKeys, NotContains, run_suite, to_markdown

cases = [
    Case("greeting", model_output_1, [Contains("hello"), NotContains("error")]),
    Case("structured", model_output_2, [IsJSON(), JSONHasKeys(["intent", "confidence"])]),
]
print(to_markdown(run_suite(cases)))

Checks included: Contains, NotContains, Matches (regex), IsJSON, JSONHasKeys, MaxLength. Drop in your own — any callable returning (bool, message) works. MIT licensed.