Hi — I maintain EvalPort (OpenEval), an open, portable schema for LLM/agent evaluation results, and I think Coder Eval is a strong fit for a small conversion adapter.
What I looked at in the actual source: src/coder_eval/models/results.py and src/coder_eval/models/experiment.py define real, structured result objects, not just a scalar score — EvaluationResult and CriterionResult/CriterionResultUnion (a discriminated union covering the weighted 0.0–1.0 criteria types: file checks, run_command, llm_judge, agent_judge, classification, etc.), plus ResultSummary, SuiteRollup, and — for the A/B experiment layer — TaskResult, VariantResult, VariantAggregate, and ExperimentResult. The README also points at docs/REPORT_SCHEMA.md as a field-level reference for run.json/variant.json/task.json. That's a well-defined enough model (weighted per-criterion results rolling up into a suite result, with pass/fail thresholds) to map onto OpenEval's ResultSet — each CriterionResult as a graded per-criterion result, ResultSummary/SuiteRollup as the suite-level rollup, and the experiment layer's VariantResult/ExperimentResult as multiple result sets under one comparison.
What I'm offering: a small standalone adapter package (following the shape of the existing adapters in adapters/, e.g. the AutoGen one) with to_openeval()/from_openeval() converting Coder Eval's result models to/from a payload that validates against openeval.validate.validate_suite()/validate_result_set(). Happy to build a first draft and open a PR, or hand off pointers if someone here wants to take it on — meant as a low-pressure invitation given how well-specified docs/REPORT_SCHEMA.md already is, not a demand.
Spec: https://github.com/adhabnr-ux/evalport — happy to answer questions about the target shape.
Hi — I maintain EvalPort (OpenEval), an open, portable schema for LLM/agent evaluation results, and I think Coder Eval is a strong fit for a small conversion adapter.
What I looked at in the actual source:
src/coder_eval/models/results.pyandsrc/coder_eval/models/experiment.pydefine real, structured result objects, not just a scalar score —EvaluationResultandCriterionResult/CriterionResultUnion(a discriminated union covering the weighted 0.0–1.0 criteria types: file checks,run_command,llm_judge,agent_judge, classification, etc.), plusResultSummary,SuiteRollup, and — for the A/B experiment layer —TaskResult,VariantResult,VariantAggregate, andExperimentResult. The README also points atdocs/REPORT_SCHEMA.mdas a field-level reference forrun.json/variant.json/task.json. That's a well-defined enough model (weighted per-criterion results rolling up into a suite result, with pass/fail thresholds) to map onto OpenEval'sResultSet— eachCriterionResultas a graded per-criterion result,ResultSummary/SuiteRollupas the suite-level rollup, and the experiment layer'sVariantResult/ExperimentResultas multiple result sets under one comparison.What I'm offering: a small standalone adapter package (following the shape of the existing adapters in
adapters/, e.g. the AutoGen one) withto_openeval()/from_openeval()converting Coder Eval's result models to/from a payload that validates againstopeneval.validate.validate_suite()/validate_result_set(). Happy to build a first draft and open a PR, or hand off pointers if someone here wants to take it on — meant as a low-pressure invitation given how well-specifieddocs/REPORT_SCHEMA.mdalready is, not a demand.Spec: https://github.com/adhabnr-ux/evalport — happy to answer questions about the target shape.