[antonwolfy]

Summary

Follow-up to the review of #191, where report generate was found to still end in a traceback for an eval file that parses but holds no evaluations (review; no reply). On main, with --questions holding {"questions": []}:

--eval holds	main	this PR
{"evaluations": []}	exit 1, ZeroDivisionError: division by zero traceback from rate = f"{count / total_q:.0%}" (dokimi/report/generator.py:472)	exit 1, Error: --eval <path> holds no evaluations, no report written
{}	the same traceback	the same message
[]	exit 1, Error: <path> must hold a JSON object	the same message as above
[1] (a non-empty list)	exit 1, Error: <path> must hold a JSON object	unchanged

The table is for the default --format markdown. With --format json, main never crashed on an empty eval: it exited 0 and wrote a report of zero stats. This PR refuses it with the same message, because a report of no evaluations has nothing to show, and an empty eval means the run that wrote it judged nothing.

summarise_diagnoses returns total: 0 for no records, so the .get("total", 1) default never applied.

Changes

• cmd_report_generate (dokimi/commands/report.py): an eval that is {}, {"evaluations": []} or [] prints Error: --eval <path> holds no evaluations to stderr and exits 1 before the questions are read and before any report is written, for both --format markdown and --format json. This is the exit code and Error: ... style of the command's other unreadable-eval cases. A non-list evaluations (null, 5) still goes to the existing RecordError message.
• ReportGenerator._format_markdown (dokimi/report/generator.py): when the failure summary's total is 0, the Failure Analysis table's Rate cell shows n/a. report generate can no longer get there, but EvaluationPipeline._generate_report (dokimi/orchestrator/pipeline.py) also calls the generator, on its own run's eval file. Nothing in that path refuses a run that judged no answers: an empty --questions-from or cached question set passes the schema, which has no minimum item count.
• Tests (tests/test_report_cli_inputs.py): each empty shape exits 1 with exactly that message and writes no report, in both formats, a one-evaluation eval writes the report, and a direct generator test with zero evaluations shows n/a in every Rate cell. Five existing tests, two in that file and three in tests/test_questions_artifact.py, used {"evaluations": []} to get past the eval and now use a one-record eval. They test an unwritable output, unreadable, invalid or empty-question questions files, and reading a legacy questions set. Two of the test_questions_artifact.py ones would otherwise have kept passing on the new eval error instead of the error they test.
• Changelog fragment (Fixed).

Risk

report generate now refuses an empty eval file. With --format markdown that input only ever reached the traceback. With --format json it used to exit 0 and write a zero-stats report, and now exits 1: a script that writes an empty eval and asks for JSON fails where it used to pass. [] gets the new message instead of "must hold a JSON object", and it still exits 1. Pipeline reports for a run with no evaluations show n/a rates instead of crashing. Every report with at least one evaluation is unchanged.

Testing

On a Linux review host, at 619256b (on main 3a1f287):

• make ci-local: 28 passed, 0 failed, 7 skipped (agent-quality.yml:external-catalog-smoke needs docker and network; ci.yml:test-latest, docs.yml:deploy and pip-audit.yml:report are off on a PR; cli-interface.yml:cli-surface / Package the result uses an expression the local runner does not evaluate; conda-forge.yml:conda-forge-deps needs micromamba; the org-level semgrep secrets check runs on the PR). 9306 tests passed, 4 skipped; changed-line coverage 100% (5 lines measured).
• cli.py report generate with {"evaluations": []}, {} and [] as --eval, with --format markdown and with --format json: exit 1, the one Error: --eval ... holds no evaluations line on stderr, no traceback, no report file. With [1]: exit 1, must hold a JSON object as before.
• Against main's code, the three empty-shape cases and the direct n/a test fail (two with the ZeroDivisionError at generator.py:472). The one-evaluation test passes there: it pins behaviour that already worked.
• Mutants, each run against the new tests:
  • Dropping the empty-eval check in cmd_report_generate fails all three empty-shape cases.
  • Limiting the check to --format markdown fails the three --format json cases.
  • Removing the n/a guard fails the direct generator test.
  • Inverting the guard (if not total_q) fails the one-evaluation test and the direct generator test.
• Not run: a full pipeline run that judges zero answers. It needs a doc source and a model. The direct generator test stands in for it.

Revert

A plain revert. Nothing is stored and no schema changes.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md (N/A: no doc lists report generate's exit-1 cases)
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml) (the changelog fragment is exempt)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 89d4b771bb6c, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 9 of 93 turns, 0.7 of 48 minutes, 130k in / 3k out tokens.

The PR makes report generate exit 1 with a clear message on an eval file with no evaluations, and guards the Rate cell against a zero total in the generator. The generator guard is correct. The CLI guard is broader than the crash it fixes: it also rejects a --format json run that worked on main.

1 finding is on the changed lines, as comments below.

For a human reviewer

• Product call: should an eval with zero evaluations be an error for every output format, or only for the markdown report that crashed?

Not verified

I did not run the full test suite or make ci-local. I checked the JSON-format claim by calling the generator directly, not through the CLI.

[vchamarthi]

No blocking findings. The refusal and the n/a guard behave as the description says.

Checked on 619256b against its merge base, report generate with --questions holding {"questions": []}:

--eval holds	base	head
{"evaluations": []}	ZeroDivisionError, exit 1	Error: --eval <path> holds no evaluations, exit 1
{}	ZeroDivisionError, exit 1	same message, exit 1
{"evaluations": [], "meta": 1}	ZeroDivisionError, exit 1	same message, exit 1
[]	must hold a JSON object, exit 1	holds no evaluations, exit 1
[1], {"evaluations": null}	existing messages	unchanged

tests/test_report_cli_inputs.py and tests/test_questions_artifact.py pass (133 passed). The other averages in generator.py are already guarded, so generator.py:472 was the only unguarded division.

Not repeated: the --format json point from the earlier review, which the description and 619256b already answer.

Coverage: read the whole diff and probed the CLI on the shapes above. Not checked: CI results on this head, a full pipeline run that judges zero answers, and the mutation results in the description.

[vchamarthi]

question: report generate now refuses an empty eval that report eval still accepts

report.py:128 (cmd_report_generate): the new check exits 1 for {"evaluations": []}, but report eval on the same file exits 0, writes p_full.md and prints Questions: 0 ... Delta: +0.0. The reason given for the refusal (an empty eval means the run judged nothing) holds for both commands.

How to check: {"evaluations": []} as --eval: report generate exits 1 with holds no evaluations; report eval --product p --eval <file> exits 0 and writes a zero-stats report (base: the same for report eval, pre-existing).
Is the split intended? If so, a line in the PR description saying why eval keeps writing an empty report would help. If not, the same check belongs in report eval, or in #250 if that PR is where the report commands are unified.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.85%. Comparing base (3a1f287) to head (619256b).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #268      +/-   ##
==========================================
+ Coverage   92.83%   92.85%   +0.02%     
==========================================
  Files         262      262              
  Lines       33016    33020       +4     
  Branches     5510     5511       +1     
==========================================
+ Hits        30649    30660      +11     
+ Misses       1571     1567       -4     
+ Partials      796      793       -3     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/report.py	81.81% <100.00%> (+0.56%)	⬆️
dokimi/dokimi/report/generator.py	94.80% <100.00%> (+7.14%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

• Ran: report generate at 619256b with 10 --eval shapes in both --format markdown and --format json; the full suite, ruff and mypy at the head and at its merge with current main; and the PR's new tests against the base sources.
• {"evaluations": []}, {}, [] and {"evaluations": [], "x": 1} exit 1 in both formats with Error: --eval <path> holds no evaluations and write no report. [1], "", null and 0 give must hold a JSON object, and {"evaluations": null} and {"evaluations": {}} give expected a list of records, as before. This matches the description.
• On the base sources, 7 of the new tests fail: the six empty-shape cases and the n/a test. Every other test passes.
• No open findings and no commits from this review.
• Left for the author: the open question on the report eval / report generate split (inline on report.py:128) is a product decision and does not block this PR.

[xaleryb]

Approved at 619256b: every point from our review is fixed at this head or left as a named follow-up, the full suite, ruff, mypy and bandit pass on Linux, and CI is green.