[antonwolfy]

Summary

Follow-up to the reviews of #213 (network policy, SWE-bench grader), #246 (review), #157 (review), #188 (review) and #187 (review). In each one, the reviewer's mutation run found a code change that no test catches. Each review point is a test gap, not a bug: the code is right, and nothing pins it.

On main today, each of these changes to the code still passes the test files that cover it:

Change to the code	Test files run	On main
proxy_env_flags: NO_PROXY / no_proxy under an allowlist get the sidecar URL instead of localhost,127.0.0.1,::1	test_network_policy.py	34 passed
policy(): no .lower() on the mode (either call, or both), so network_mode = "Public" is refused	test_network_policy.py	34 passed
_declared(): allowed_hosts = "pypi.org" taken as list(hosts), one host per letter	test_network_policy.py	34 passed
resolve(): egress = "huggingface.co" iterated the same way (same branch, network_policy.py:157)	test_network_policy.py	34 passed
parse_pytest, parse_pytest_options, parse_pytest_v2: the len(parts) guard removed, so a bare FAILED line raises IndexError	test_swebench_tasks.py	52 passed, 1 skipped
_apply_correction: method="holm" whatever correction says	test_stats_correction.py	20 passed
run_process_metrics: only the first guidance_globs entry honoured	test_repo_process_metrics.py, test_repo_run.py, test_repo_manifest.py, test_repo_builtin_graders.py	334 passed
if events is not None replaced by if True, so an agent that left no trace gets a process metrics failed (TypeError) line in trace_errors	the same four	334 passed
_usage_value_fits: None accepted for a str field, so {"usage": {"model": null}} replays	test_caching.py	65 passed
skipped_factors: factors walked in reverse, so the dry run prints the skipped-factor lines out of --factors order	test_tool_preference.py	75 passed

Changes

• tests/test_network_policy.py: test_an_allowlisted_verifier_goes_through_its_own_sidecar also asserts NO_PROXY=localhost,127.0.0.1,::1 and no_proxy=... in the verifier's docker exec. New test_a_task_network_mode_is_read_whatever_its_case checks that "Public" (a mode) resolves to public and "No-Network" (an alias) to none. New test_a_single_host_given_as_a_string_is_one_host_not_its_letters covers allowed_hosts = "pypi.org" and egress = "huggingface.co".
• tests/test_swebench_tasks.py: the bare-status-word test for parse_seaborn is now parametrized over parse_seaborn, parse_pytest, parse_pytest_options and parse_pytest_v2. Each must skip FAILED, PASSED and PASSED without raising, and still read PASSED t.py::f from the same log.
• tests/test_stats_correction.py: test_apply_correction_uses_the_requested_method uses [0.01, 0.02, 0.03], where the two methods differ. It is parametrized: BH gives [0.03, 0.03, 0.03] and Holm [0.03, 0.04, 0.04]. The arithmetic is in the comment.
• tests/test_repo_process_metrics.py: new test_a_read_that_only_the_second_guidance_glob_matches_is_guidance checks that .github/instructions/c.md counts as guidance.
• tests/test_repo_run.py: new test_an_agent_that_left_no_trace_is_not_given_a_process_metrics_error runs the fake harness's silent agent and checks that process is null with no process metrics failed line in trace_errors.
• tests/test_caching.py: {"usage": {"model": null}} is added to the wrong-shape cases, which must be a cache miss.
• tests/test_tool_preference.py: test_dry_run_names_the_groups_a_factor_is_skipped_for compares the printed skipped-factor lines as an ordered list. With --factors assertive,exclusion,long_desc, exclusion comes before long_desc, unlike FACTORS. This also replaces the two not in out checks.
• No changelog fragment: nothing under dokimi/dokimi/ changes.

Risk

None. Only tests change, and the package and its output are unchanged.

Testing

On a Linux review host, at 9e5dc10 (on main 3a1f287):

• make ci-local: 28 passed, 0 failed, 7 skipped (external-catalog-smoke, which needs docker and network; test-latest, docs.yml:deploy and pip-audit.yml:report, whose job conditions are false on a PR; the cli-surface "Package the result" step, which uses an expression ci-local cannot evaluate; conda-forge-deps, since micromamba is not installed; semgrep secrets, since semgrep is not installed). 9308 tests passed, 4 skipped; changed-line coverage 100% (0 lines of package code changed).
• The seven touched test files plus test_docs_code_refs.py, test_growth_ratchet.py, test_audit_test_quality.py and test_import_architecture.py: 664 passed, 1 skipped. ruff check is clean on the touched files.
• Mutants: each change in the table above, applied to a copy of this branch, fails its new test (15 of 15). The .lower() mutant was run three ways: the alias lookup only, the fallback name only, and both. Forcing method="bh" was also run; it fails the Holm case here, and on main the existing Holm tests already catch it. The unmutated copy passes: 377 passed, 1 skipped.
• Merged with the three other follow-up branches of this sweep on main 3a1f287: pytest -n auto 9321 passed, 4 skipped; mypy and ruff check . clean.

Revert

A plain revert. It removes tests only and leaves nothing behind.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 9e5dc10f4f34, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 11 of 80 turns, 0.8 of 40 minutes, 205k in / 3k out tokens.

Tests-only change that pins ten mutations the earlier reviews found uncaught: NO_PROXY under an allowlist, mode case folding, single-string host lists, the bare-status-word guard in the three pytest parsers, the BH/Holm choice, the second guidance glob, the no-trace agent, a null usage.model, and skipped-factor order. I checked each new test against the code it targets and found no defects. Where I could check the code, the tests would fail on the mutation they name. I could not run the suite here.

No issues found.

Not verified

I could not run pytest because the sandbox has no network to install it. The claims from reading the code:

• The silent action exists in tests/repo_eval_fixture.py.
• runner.py:306 gates _process_or_null on events is not None, so an if True mutant would append the "process metrics failed" line the new test forbids.
• The BH and Holm expectations are float-exact. I checked the products in Python, and correct_family rounds to 4 decimals anyway.
  I did not run the NO_PROXY and resolve assertions against network_policy.py.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.85%. Comparing base (3a1f287) to head (9e5dc10).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
+ Coverage   92.83%   92.85%   +0.02%     
==========================================
  Files         262      262              
  Lines       33016    33016              
  Branches     5510     5510              
==========================================
+ Hits        30649    30656       +7     
+ Misses       1571     1570       -1     
+ Partials      796      790       -6     

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.