[xaleryb]

Summary

• tasks inspect let one malformed file end the whole scan, so no other file under the path was reported:
  • promptfoo tests: 5 or tests: true, and a dokimi questions file with "questions": null or 5: the value was iterated without a type check (TypeError). Both now read as no items; the questions loader still reports the bad value as a problem. The fix is the expression suggested in the review on feat(harnesses): per-phase network policy for docker-* runs, off by default #218.
  • a promptfoo tests file with a non-string row key (AttributeError on startswith), and an explore task with lines: [.inf] (OverflowError, which _parse_lines did not catch). Found while sweeping task_formats/ for the same class.

Evidence

• python cli.py tasks inspect <dir> on a directory with the bad file plus a valid samples file: main -> traceback, nothing reported; this branch -> both files reported.
• Two parametrized tests, one case per crash; each case fails with its source line reverted to main. Full suite 8113 passed, 21 skipped; ruff, mypy, bandit clean.

Merge Danger

Door: two-way. Revert restores the crashes and nothing else.

Blast Radius: tasks inspect (and the explore task loader's error type for an infinite line), only on the malformed values above.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.25%. Comparing base (85b8d84) to head (403cf7a).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #227    +/-   ##
========================================
  Coverage   92.25%   92.25%            
========================================
  Files         236      239     +3     
  Lines       29582    29937   +355     
  Branches     4865     4918    +53     
========================================
+ Hits        27290    27618   +328     
- Misses       1615     1632    +17     
- Partials      677      687    +10     

Files with missing lines	Coverage Δ
dokimi/dokimi/explore/tasks.py	94.28% <100.00%> (ø)
dokimi/dokimi/task_formats/configs.py	97.74% <100.00%> (+0.02%)	⬆️
dokimi/dokimi/task_formats/datasets.py	95.63% <100.00%> (+0.01%)	⬆️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

Review: #227 — fix(tasks): one malformed file no longer stops tasks inspect

Head d68e563 · base 85b8d84 · verdict: approve, minor suggestions

Verified claims

Probe probes/hostile.py (runs inspect_path on a dir holding one bad file plus a valid ok/s.jsonl):

Case	base	head
promptfoo tests: 5	TypeError configs.py:82	reported, scan continues
promptfoo tests: true	TypeError configs.py:82	reported
questions "questions": null	TypeError datasets.py:336	reported + loader problem "questions is NoneType"
promptfoo tests row key 1:	AttributeError configs.py:127	reported
explore lines: [.inf]	OverflowError tasks.py:100	reported, problem "'lines' must contain integers"

All 4 new test cases fail on base and pass on head. tests/test_task_format_detect.py: 64 passed on head, 60 on base; the same 5 tests fail on both trees here (Windows, minimal dependencies: questions-track and CLI-output tests, symlinks). The PR doesn't cause them.

Swept for the same class of bug, all fine on both trees: tests as a dict, questions as a dict or string, lines: [.nan], references: 5, int keys in samples/OpenAI rows, CSV rows (_parse_csv stringifies keys), non-dict rows (FileSource.rows filters them out).

Findings

None at important or higher.

Suggestions (minor)

1. changed: tests: 5 now passes silently. Before, it crashed. Now the report shows items=None, problems=[] and no lint flags the file, while a bad questions value gets a problem. Consider elif tests is not None and not isinstance(tests, list): report.problems.append(f"tests is {type(tests).__name__}, expected a list or a path") in PromptfooProbe.describe (configs.py:92-98).
2. Tests: the PR body names tests: true, but only tests: 5 is tested. items in (None, 0) is loose: each case can assert its exact value (None for promptfoo, 0 for questions).
3. pre-existing: lines: [true] is accepted as line 1 (int(True)), tasks.py:100. If that's not intended, reject bools there. Follow-up issue, not this PR.

[xaleryb]

Thanks. 403cf7a takes the first two:

1. A promptfoo tests that is neither a list nor a path is now a problem on its report: tests is int, expected a list or a path (bool for tests: true), next to the existing no tests.
2. tests: true is its own case, and each case asserts its exact items (None for promptfoo, 0 for questions) and its problem. Without the new line both promptfoo cases fail.

Full suite, ruff and mypy pass on Linux. I left lines: [true] for a follow-up, as you suggested.