Proposal: EvalPort export for evals.json/grading.json/benchmark.json (interop with Anthropic skill-creator format)

[adhabnr-ux]

Hi — first, thanks for building this. The safety/quality/reliability/cost split (and the audit scope rules around --include-all) is a genuinely well-thought-out shape for skill evaluation, and I appreciate that skill_eval/eval_schemas.py was already written to match Anthropic skill-creator's evals.json/grading.json/benchmark.json formats rather than inventing yet another bespoke schema. That's a rare and useful design choice.

I maintain EvalPort (Apache-2.0), an open interchange format for portable LLM eval test cases, graders, suites, and results, plus a Python/TS SDK (evalport-sdk) and 37 adapter packages that translate to/from other eval tools (DeepEval, LangSmith, MLflow, Ragas, Guardrails, etc. — all under adapters/ in that repo). The goal is the same instinct visible in your eval_schemas.py docstring, generalized: instead of every eval tool inventing its own JSON shape, define one shared shape and let tools import/export it.

Why I think skill-eval is an interesting fit: your dataclasses already map almost 1:1 onto EvalPort's core objects:

skill-eval	EvalPort
EvalCase (id, prompt, expected_output, files, assertions)	TestCase
AssertionResult / GradingResult (assertion_results, pass_rate, execution_metrics)	Grader output / Result
BenchmarkReport (scores across the 4 dimensions, runs)	ResultSet
TriggerQuery / TriggerReport	TestCase + ResultSet (trigger-rate as a metric)

Because skill-eval already speaks the skill-creator vocabulary, an EvalPort adapter here would effectively give you two interop paths for the price of one: skill-eval's native format, skill-creator's format, and EvalPort as the neutral middle layer other frameworks can already read/write. Concretely, something like:

from evalport import TestCase, Result, ResultSet
from skill_eval.eval_schemas import EvalCase, BenchmarkReport

def eval_case_to_evalport(ec: EvalCase) -> TestCase:
    return TestCase(
        id=ec.id,
        input=ec.prompt,
        expected=ec.expected_output,
        metadata={"assertions": ec.assertions, "files": ec.files},
    )

def benchmark_report_to_resultset(report: BenchmarkReport) -> ResultSet:
    return ResultSet(
        suite_id=report.skill_name,
        results=[Result.from_dict(r) for r in report.runs],
        scores=report.scores,
        passed=report.passed,
    )

Comparable adapters already in the repo if it's useful to see the shape of a finished one: adapters/deepeval-openeval-adapter (assertion/metric-based grading, closest in spirit to your AssertionResult) and adapters/guardrails-openeval-adapter (safety-oriented, closest in spirit to your audit dimension).

I don't want to presume a PR is wanted (I saw the CONTRIBUTING.md note to open an issue first for anything nontrivial, which is exactly what this is), so I'm floating the idea here first. Happy to build a skill-eval <-> EvalPort adapter and send it as a PR if that's of interest, or to just leave this here as a pointer in case it's useful for future compatibility work — no pressure either way.

— Sahi, independent contributor (not affiliated with AWS)