[antonwolfy]

Summary

Follow-up to the review of #214 (task-format importers). It settles two points there. First, a promptfoo assert with a blank judge value, or a contains-any assert with an empty list, imports cleanly, and then eval reference refuses the whole file (review). Second, an Inspect exact target such as ... imports as a check that no answer can pass (review). Neither thread got an author reply.

On main (3a1f287), questions import gives:

Input	Imported check	validate_grading / grade
promptfoo {type: llm-rubric, value: ''}	{'type': 'judge', 'native': 'llm-rubric'}	a judge check needs a criterion or a reference
promptfoo {type: factuality, value: ' '}	{'type': 'judge', 'reference': ' ', 'native': 'factuality'}	a judge check needs a criterion or a reference
promptfoo {type: contains-any, value: []}	{'type': 'match', 'values': [], ...}	match has no values
Inspect sample target: "...", --scorer exact	{'type': 'exact', 'values': ['...']}	validates; passed is False for zzz, ... and ?

eval reference checks every question's grading block before it grades any (grading_by_id). So one of the first three asserts makes it refuse the whole file. The fourth fails safe, but it scores 0 without saying why. docs/answer-formats.md already says a blank promptfoo assert imports as unsupported, and on main that does not hold for these cases.

Changes

• task_formats/imports/promptfoo.py:
  • _other_check: a judged assert (llm-rubric, g-eval, model-graded-closedqa, factuality, model-graded-factuality) whose value is missing or blank after strip() returns no check. It then imports as unsupported, with the assert `<type>` not translated note, the same path as a blank text assert. A non-blank judge value imports as before.
  • _text_check: a contains* or icontains* assert with an empty list now imports as unsupported the same way.
• task_formats/imports/inspect_ai.py (_checks): the grader's exact check compares SQuAD-normalised text and never passes an empty target. For the exact scorer, the importer now also drops a target that squad_normalize turns into nothing (..., the). A sample left with no target is unsupported, as it already is for match. includes and the model-graded scorers are unchanged.
• Tests, in tests/test_answer_imports.py:
  • The three promptfoo asserts above, each expected unsupported with the note.
  • A file that mixes them with a good llm-rubric assert: validate_grading returns nothing and the good judge check is kept.
  • ... and ["The", "Paris"] rows under scorer="exact" in the Inspect punctuation test, renamed test_inspect_punctuation_only_target_is_unsupported_for_match_and_exact.
• docs/answer-formats.md states the empty-list, blank-judge and Inspect exact rules next to the existing blank-value rule.
• Changelog fragment (Fixed).

Risk

Only imports that eval reference could not use, or that could never pass, change:

• The three kinds of promptfoo assert now import as unsupported (listed, not counted) instead of producing a grading block that eval reference refuses.
• An Inspect sample under the exact scorer whose targets all normalise to nothing becomes unsupported instead of always failing. A sample with at least one real target keeps it (["The", "Paris"] imports as values: ["Paris"]).

Other assert types, the match, includes and model-graded Inspect scorers, and every non-blank value import as before.

Testing

On a Linux review host, at 7c79ac3 (on main 3a1f287):

• make ci-local: 28 passed, 0 failed, 7 skipped:
  • Skipped because their job condition is false on a PR: test-latest, docs deploy, pip-audit report.
  • Skipped because a tool is missing or offline: external-catalog-smoke (needs docker and network), conda-forge-deps (no micromamba), semgrep secrets (runs on the PR).
  • Skipped because ci-local cannot evaluate the step's expression: cli-surface Package the result.
  • 9302 tests passed and 4 were skipped. Changed-line coverage was 100% (6 lines measured).
  • The first two runs at this commit each failed one test, tests/test_reference_checks.py::test_the_regex_limit_is_shared_by_every_pattern_of_one_check, a wall-clock bound (elapsed < 1.5 * REGEX_TIMEOUT). It measured 2.16 s and 1.79 s while other jobs loaded the host. It passes on its own, passed in a full -n auto --cov run of this commit, and passed in the third ci-local run. This change does not touch the regex check.
• The repro above: on main it prints the outputs in the table. At this commit, all four import as unsupported, and validate_grading returns [] for each.
• The 5 new or changed tests fail on main's code and pass here.
• Mutants. Each one, applied to a copy of this commit, fails the tests that cover it:
  • Dropping the empty-list return from _text_check fails the contains-any case and the mixed-file test.
  • Dropping the blank-judge return from _other_check fails the llm-rubric case, the factuality case and the mixed-file test.
  • Testing only value is None for a judged assert fails the same three tests.
  • Dropping the exact rule fails the Inspect test.
  • Stripping punctuation for exact but keeping articles fails the Inspect test (on The).
  • Applying the SQuAD rule to every scorer fails the Inspect test (includes must keep ...).
• ruff check and mypy pass on the touched modules, as do tests/test_docs_code_refs.py, tests/test_growth_ratchet.py, tests/test_audit_test_quality.py and tests/test_import_architecture.py.
• Not run: the matrix legs on Python 3.11, 3.12 and 3.14, and the Windows test leg beyond ci-local's smoke.

Same bug class, not in this PR

• task_formats/imports/openai_evals.py: a modelgraded spec with whitespace-only criteria and no ideal imports as a judge check with criterion ' ', which validate_grading refuses. The not (criteria or reference) guard does not strip.
• eval/reference_checks.py: validate_grading does not flag a hand-written exact value that SQuAD-normalises to nothing (..., the). Such a check validates and never passes.
• promptfoo starts-with and regex with a list value import as the str() of the list ([] becomes the string '[]').

Revert

A plain revert. Nothing is stored in a new format: questions that were imported keep the checks they were written with.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 7c79ac37e7d8, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 5 of 80 turns, 0.3 of 40 minutes, 115k in / 2k out tokens.

The change makes the promptfoo importer return unsupported for judged asserts with a blank or missing value and for contains* asserts with an empty list. It also makes the Inspect importer drop exact targets that SQuAD-normalise to nothing. I checked the new conditions against as_list, blank_value, validate_grading and the exact grader in reference_checks.py. The conditions are correct and match what eval reference refuses. The new tests cover each branch. I found no defects.

No issues found.

For a human reviewer

• Inspect's own exact scorer would count an empty-normalising answer (e.g. ...) as equal to an empty-normalising target. Dropping the target follows dokimi's grader, which never passes an empty target, but it differs from the native scorer. Confirm that is intended.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.82%. Comparing base (3a1f287) to head (7c79ac3).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #266      +/-   ##
==========================================
- Coverage   92.83%   92.82%   -0.01%     
==========================================
  Files         262      262              
  Lines       33016    33020       +4     
  Branches     5510     5512       +2     
==========================================
+ Hits        30649    30651       +2     
- Misses       1571     1573       +2     
  Partials      796      796              

Files with missing lines	Coverage Δ
dokimi/dokimi/task_formats/imports/inspect_ai.py	82.14% <100.00%> (+1.02%)	⬆️
dokimi/dokimi/task_formats/imports/promptfoo.py	81.35% <100.00%> (+0.43%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.