[napetrov]

Summary

Stacked on #210 (review only the top commit). Runs SWE-bench rows through dokimi's own TaskSpec and the local docker-* harnesses; Harbor stays an export target only.

rows.jsonl ──stage()──> TaskSpec(format="swe-bench")
                         ├ environment/Dockerfile  FROM swebench/sweb.eval.x86_64.<id>  (/testbed -> /app)
                         ├ solution/               gold patch, upstream apply chain
                         └ tests/                  upstream eval script + stdlib grade.py -> reward.txt
dokimi tasks run --swe-bench rows.jsonl --tasks <id> --harnesses docker-oracle,docker-claude

• dokimi/harnesses/swebench.py: the loader. It uses prebuilt instance images, or a row's own image_name/docker_image, so SWE-smith/rebench rows carry their own image.
• dokimi/harnesses/swebench_grade.py: a stdlib port of the SWE-bench log parsers and the RESOLVED_FULL rule (MIT attribution).
• swebench_specs.json: generated from swebench==4.1.0 by scripts/gen_swebench_specs.py. dokimi does not depend on swebench.
• tasks inspect now names docker-* as the consumer when every repo in the file has a spec.

Evidence

Oracle (docker-oracle; gold should score 1, an empty patch 0):

Instance	Gold	Empty
django-11099 / 11133	1 / 1	0 / 0
pytest-5227 / 7373	1 / 1	0 / 0
sympy-20590	1	0
requests-863	0 (live-httpbin P2P tests: dataset drift)	0
requests-2317	hung on httpbin, killed	-

• docker-claude pilot (sonnet, 3 instances × 1 trial): 3/3 resolved in 84 s for $0.08.
• Local checks: ruff and mypy clean, 7951 tests passed. A docker-gated test runs when DOKIMI_SWEBENCH_DOCKER=1.

Merge Danger

• Low. The change is additive: a new --swe-bench flag and new modules, plus small edits to tasks.py, datasets.py, the docs and package-data.
• The tasks run with network_mode=public, and the harness forwards the host proxy into the verifier, so tests that use the network can change outcome.
• The harness container name dokimi-run-<task>-<pid> collides when one task runs twice in parallel in one process.
• Must merge after feat(tasks): detect eval-task formats with tasks inspect #210.

🤖 Generated with Claude Code

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi tasks run` parameters changed:
    + added option `--swe-bench`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.95522% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.71%. Comparing base (a2b787a) to head (4179814).

Files with missing lines	Patch %	Lines
dokimi/dokimi/harnesses/swebench_grade.py	82.50%	13 Missing and 15 partials ⚠️
dokimi/dokimi/harnesses/swebench.py	95.03%	3 Missing and 4 partials ⚠️
dokimi/dokimi/commands/tasks.py	89.47%	1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
- Coverage   92.74%   92.71%   -0.04%     
==========================================
  Files         254      256       +2     
  Lines       32087    32421     +334     
  Branches     5344     5422      +78     
==========================================
+ Hits        29759    30058     +299     
- Misses       1560     1576      +16     
- Partials      768      787      +19     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/docker_solver.py	97.73% <100.00%> (ø)
dokimi/dokimi/harnesses/tasks.py	96.04% <100.00%> (+0.01%)	⬆️
dokimi/dokimi/task_formats/datasets.py	96.60% <100.00%> (+0.96%)	⬆️
dokimi/dokimi/commands/tasks.py	92.47% <89.47%> (-0.35%)	⬇️
dokimi/dokimi/harnesses/swebench.py	95.03% <95.03%> (ø)
dokimi/dokimi/harnesses/swebench_grade.py	82.50% <82.50%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

Rebased on #218 (network policy). Changes in 313ed32:

• Staged rows no longer set network_mode = "public", so solve and verify run offline. On the 7-row pilot the gold patch gives the same verdict offline and with --network solve=public --network verify=public (5 pass; requests-863/2317 fail in both).
• --all skips swebench.KNOWN_BAD: requests-863/2317, whose tests fetch httpbin.org live, and the 6 oracle failures the Harbor SWE-bench adapter lists (harbor 6099b8c, adapters/swebench/README.md, Known Issues). Naming a row in --tasks still runs it.

🤖 Generated with Claude Code

[xaleryb]

• Ran: SWE-bench Verified rows through tasks run --swe-bench with docker-oracle on real sweb.eval images, plus dry runs on crafted row files.
• --tasks pylint-dev__pylint-4661 --harnesses docker-oracle (Verified, default network) -> 0/1, NOT RESOLVED: the gold patch adds appdirs to setup.cfg, and the eval script's pip install -e . cannot reach PyPI offline -> expected RESOLVED. It does resolve with --network verify=allowlist:pypi.org,files.pythonhosted.org and with verify=public. Choosing the default and updating docs/coding-harnesses.md ("the image already has the repo's dependencies") is left to the author.
• --swe-bench verified.jsonl --all -> task_runs.json has 493 results and does not record the skipped KNOWN_BAD ids or the 500 source rows -> expected the skip stamped in the artifact.
• Pushed:
  • 10b3924: an --all selection with no runnable rows exits 1 instead of writing an empty artifact.
  • 7b5c05b: path-like or duplicate instance_id is refused before anything is staged.
  • 169ceff: tests for XFAIL, the whole-log fallback, and pytest_v2 ANSI and old-style lines.

[antonwolfy]

Review at 169ceff. The SWE-bench staging and the grader follow the upstream rules closely, and the offline default for solve and verify is pinned by tests. Five things are worth fixing before merge: the unquoted values in eval.sh, the leaked verifier sidecar, the mutation survivors in the SWE-bench tests, the network-policy test gaps and the stacking. One nit inline.

I agree with @xaleryb's points on --all not recording the skipped rows and on offline pip install -e .. New evidence: every one of the 282 version entries in swebench_specs.json carries an install step, and eval.sh runs it with no -e, so a failed install shows only as an unresolved verdict. @xaleryb's question on the network default waits for the author.

What I ran, on a Linux review host at 169ceff:

• make ci-local with Docker: 30 passed, 3 failed, 6 skipped. 7,981 tests passed; changed-line coverage 95% (1,420 lines). The failures are gitleaks (the binary download returned HTTP 504; a direct gitleaks detect over the PR's 11 commits found no leaks) and pip-audit (reports filelock and setuptools advisories in the lock files, which this PR does not touch).
• Reproductions for the code findings, with the Docker CLI faked and no model or network call.
• Mutation runs: 25 mutants of the SWE-bench loader and grader, 6 of them pass the full suite; 41 mutants of network_policy.py, 9 of them pass the suite.

Findings that cannot be anchored inline:

1. The PR is stacked on #210 and #218. The description says "review only the top commit", but the branch holds 11 commits and the diff is 95 files and +6,882 lines. AGENTS.md says PRs are not stacked on each other's branches. Two of the three changelog fragments (task-format-detect, task-network-policy) belong to the other PRs and would be counted twice once those merge. The description is also stale after 313ed32: it says the tasks run with network_mode=public and the proxy is forwarded into the verifier, but stage() now sets no network_mode. Suggestion: rebase onto the merged base so the diff holds only this PR, and update the Merge Danger section.
2. tests/test_network_policy.py lets real regressions through. Mutation check: dropping the solve phase from container_phases, replacing the allowlist NO_PROXY value with the sidecar URL, removing the lowercasing of network_mode ("Public"), and handling allowed_hosts or egress given as a plain string all pass the suite. Tests to add: an oracle run with solve=allowlist:... that expects a sidecar create; an assertion on the allowlist NO_PROXY value; network_mode = "Public"; and a string allowed_hosts.

Checked, no issues

• Offline default: stage() sets no network_mode, so solve and verify resolve to none; mutants that flip either default fail the tests.
• Instance ids: path-like (../.., absolute, a/b, .hidden) and duplicate ids are refused before any directory is created.
• Policy modes: unknown modes (host, bridge, empty, boolean) fail closed; proxy credentials of the form user:pw@ are redacted in run.log; network.json holds no proxy URLs.
• Schema and docs: task_inspect.v1 is registered in SCHEMA_VERSIONS with a schema file and fixture; changelog fragments use the expected headings; no new doc page needs a mkdocs.yml entry.
• docker_solver.py: the +80/-80 is the proxy and policy code moving to network_policy.py; I found no behavior change against the base.

[antonwolfy]

I pushed a commit per nit, up to f384575, which fix the nit from my review:

• f384575: dokimi/docs/coding-harnesses.md:161: the sentence now says tasks list and tasks run read only the first two, and tasks run --swe-bench also takes a SWE-bench file.

make ci-local at f384575: 30 passed, 3 failed, 6 skipped (the 3 failures match the reviewed head).

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the five risks from my review; @xaleryb on --swe-bench --all writes a task_runs.json that does not record the skipped KNOWN_BAD ids or the source row count (confirmed: the skip appears only on stderr; the artifact holds only the rows that ran).

[antonwolfy]

I pushed a commit per nit, up to 402891b, which fix the three nits from my review:

• 581e14f: dokimi/dokimi/harnesses/swebench_grade.py:97: parse_seaborn checks len(parts) > 1 before reading parts[1] in the FAILED and PASSED branches.
• 7a03b2e: dokimi/dokimi/commands/tasks.py:132: the --all banner lists only the KNOWN_BAD ids present in the file, and is not printed when there are none.
• 402891b: dokimi/changelog.d/20261024_100000_napetrov_run-swebench.md:10: --all now skips "the 9 rows", matching KNOWN_BAD.

make ci-local at 402891b: 33 passed, 0 failed, 6 skipped.

Not approving yet. Still open: the risk from my review.

[napetrov]

Thanks for the reviews. Open items, mapped to shas (head 8d074f7):

• quoting in eval.sh, including a space in a file name: a1c8c8d, b669a77
• grader pins (bad codes, ERROR statuses, Django headers, apply fallbacks): 833d7dc
• rm -f quoting and the +++ tab strip: a96e0f8
• parse_seaborn on a bare FAILED or PASSED line: 581e14f, test in 3a23923
• --all banner: 7a03b2e, test in 3a23923
• changelog count of skipped rows: 402891b
• docs sentence on tasks run --swe-bench: f384575
• new in this push: a null or non-string text field in a row is a row error, not a traceback: 3a23923

Stacking: main is merged in, and the diff is now the 12 files of this PR. The verifier sidecar cleanup and the network-policy test gaps are #218 code, already on main and outside this diff.

Still open for the maintainers' decision and not changed in this push: the network default for verify, and whether --all records the skipped known-bad ids in task_runs.json.

Ready for another look.

[antonwolfy]

LGTM. The added-file cleanup is now pinned, the null and non-string row fields become row errors, and the --all banner names only the rows the file holds. One nit on the seaborn test.

@xaleryb's questions on the network default for verify and on --all not recording the skipped KNOWN_BAD ids wait for the author.

What I ran, on a Linux review host at 8d074f7:

• make ci-local with Docker: 33 passed, 0 failed, 6 skipped. 8,426 tests passed; changed-line coverage 94% (317 lines).
• Mutation run: 23 mutants of the loader, the grader and the --all banner against tests/test_swebench_tasks.py. The rm -f -- quoting, the \t? on both header regexes, each field of the new string check and every banner mutant fail a test. Five mutants pass the full suite, all bare-status-word guards in the grader parsers.

Checked, no issues

• Stage check: stage() raises SweBenchError for None in each of repo, base_commit, problem_statement, patch and test_patch; removing any one field from the tuple fails a test.
• Added files: patch_files returns ([], ["new file.py"]) for +++ b/new file.py<TAB>, and eval.sh gets rm -f -- 'new file.py'.

[vchamarthi]

Not approving yet: 1 item to fix before merge.

The grader and eval script match swebench 4.1.0, and the earlier quoting and test-gap findings are fixed. One fail-open path remains in the verifier: test.sh exits 0 whatever the verdict, so a failed docker cp of reward.txt scores an unresolved row as 1. One smaller mismatch is between tasks inspect and tasks run. CI is still pending at this head.

CI at send time: 0 failing, 7 pending.

[vchamarthi]

Should fix: swebench_grade.main returns 0 for an unresolved row, so the verifier falls open when reward.txt cannot be copied out; return 0 if report["resolved"] else 1.

docker_solver.py:657-668 at 31be3de: if docker cp <ctr>:/logs/verifier/reward.txt returns non-zero (not a timeout) or the file is absent, and the container is still running, _verify returns 1.0 if proc.returncode == 0 else 0.0. proc is bash -lc of test.sh, whose last command is grade.py, and main ends with return 0 regardless of report["resolved"] (swebench_grade.py:222). So a row that wrote reward.txt = 0 but whose docker cp failed once (daemon hiccup, not a timeout) is recorded as reward 1.0 and passes. With the exit code following the verdict, the same failure gives 0.0, and the normal path is unchanged because reward.txt wins when it is read. test_the_grader_script_writes_reward_and_report currently asserts main(...) == 0 for an unresolved log; it should assert 1.

[napetrov]

Fixed in ec2af13: main returns 0 if report["resolved"] else 1. test_the_grader_script_writes_reward_and_report asserts 1 for the unresolved log and 0 for a resolved one.

[vchamarthi]

Non-blocking: _swe_bench_consumers reports docker-* when every repo is in the table, but repo_spec also refuses a version the table lacks, so inspect can promise a file that tasks run rejects; check the version too.

datasets.py:121 tests r.get("repo") not in known. swebench.py repo_spec raises no SWE-bench test spec for <repo> version '<v>' when entry["versions"] lacks the row's version (also when it has no test_cmd). A row such as {repo: "pytest-dev/pytest", version: "9.9"} therefore inspects as docker-* and fails at stage(). In load_swebench with ids=None that error propagates out of the list comprehension, so one such row in a few hundred refuses the entire --all run before any task runs. The docs line 'a SWE-bench file whose repos all have a test spec' describes the inspect check, not what run needs.

[napetrov]

Fixed in ec2af13: _swe_bench_consumers also checks each row's version has a test_cmd, and lists the misses as versions_without_test_spec; the inspect test covers pytest-dev/pytest 9.9. With the stage-after-check change (below), such a row now refuses --all before anything is written.

[vlad-perevezentsev]

KNOWN_BAD misses rows whose tests need the internet. Verify runs offline by default, so the gold patch fails on them too. In requests 2.0-2.7, test_requests.py has HTTPBIN = 'http://httpbin.org/', and FAIL_TO_PASS holds tests like test_HTTP_200_OK_HEAD (requests.head(httpbin('get'))). That covers Lite psf__requests-1963, -2148, -2674 and Verified psf__requests-1766, -1921, the same reason as -1724. In Verified, sphinx-doc__sphinx-7985 (FAIL_TO_PASS) and -8269, -8475 (PASS_TO_PASS) need test_build_linkcheck.py::test_defaults, which checks google.com and sphinx-doc.org links. A PyPI allowlist does not help these rows. So --all still runs rows that no patch can pass, and the changelog line "skips the rows whose gold patch fails" does not hold. Please settle the verify network default first (see @xaleryb's proposal), then build KNOWN_BAD from a docker-oracle run over Lite and Verified under that default.

[napetrov]

Taken in ec2af13: verify now defaults to the PyPI allowlist (xaleryb's proposal), and KNOWN_BAD adds psf__requests-1766, -1921, -1963, -2148, -2674 and sphinx-doc__sphinx-7985, -8269, -8475 with your reasons (17 rows). Not done: a full docker-oracle sweep over Lite and Verified to derive the list; the changelog now says 'known to fail' rather than claiming the full set.

[vlad-perevezentsev]

The in-container agent does not get the repo's Python env. The image sets PATH=/opt/miniconda3/bin:..., and conda activate testbed is only in /root/.bashrc. The agent runs as uid 10001 with a tmpfs HOME, so its python and pytest are the base conda ones, which do not have the repo's dependencies. It has to find /opt/miniconda3/envs/testbed on its own or it cannot run the tests, which lowers docker-claude scores compared with other SWE-bench harnesses. Suggestion: put the env first on PATH in the Dockerfile, or name it in the instruction:

ENV PATH=/opt/miniconda3/envs/testbed/bin:$PATH

[napetrov]

Fixed in ec2af13: the staged Dockerfile sets ENV PATH=/opt/miniconda3/envs/testbed/bin:$PATH. Probed on sweb.eval django-11099 as uid 10001 under bash -lc: command -v python gives /opt/miniconda3/envs/testbed/bin/python (/etc/profile does not reset PATH).

[vlad-perevezentsev]

main() returns 0 for NOT RESOLVED too, so the exit code of test.sh carries no verdict. But when reward.txt cannot be pulled, _verify falls back to that exit code and records 1.0. Repro with the fake Docker CLI (DockerCLI(reward=None)) on a staged django__django-11099: reward=1.0, passed=True. Please return the verdict as the exit code, so a lost reward file reads as a fail and not as a pass:

return 0 if report["resolved"] else 1

[napetrov]

Fixed in ec2af13, same change as the thread above: the exit code is the verdict, so a lost reward.txt reads as a fail.

[vlad-perevezentsev]

_refuse_unrunnable (docker_solver.py:760) refuses task-md and terminal-bench-v1 tasks for docker-claude-host, but not swe-bench. A host agent has an open network, so it can look up the upstream fix of the public issue. Its instruction names /app and /testbed, which are container paths, and the host has no testbed env to run the tests. Please add FORMAT_SWE_BENCH to that check, so a host run is refused before it is paid for.

[napetrov]

Fixed in ec2af13: _refuse_unrunnable refuses FORMAT_SWE_BENCH for a host-side agent; test_a_host_side_agent_is_refused_a_swe_bench_task.

[vlad-perevezentsev]

A FAIL_TO_PASS test that is SKIPPED counts neither as passed nor as failed, so a run where all of them are skipped resolves, and so does a row with empty lists. grade(log_with('SKIPPED t.py::f'), {'FAIL_TO_PASS': ['t.py::f'], 'PASS_TO_PASS': []}) gives resolved: True. This matches upstream, but this run is offline, unlike upstream's, and dokimi's rule is that a check that did not run is not a pass. Could a FAIL_TO_PASS test with no PASSED/XFAIL result leave the row unresolved, or at least be listed in report.json and run.log?

[napetrov]

Changed in ec2af13: a FAIL_TO_PASS test without a PASSED/XFAIL result is a failure, and an empty FAIL_TO_PASS does not resolve. PASS_TO_PASS keeps upstream's rule (a skip counts neither way). Documented as a deliberate difference from upstream.

[vlad-perevezentsev]

The full eval log (/logs/verifier/test_output.log) and report.json stay in the container: _verify pulls only reward.txt, and then the container is removed. When a row scores 0 (an offline ConnectionError, a failed install, a test patch that did not apply), run.log shows at most 10 failed test names, so finding the cause needs a new run. Please print the end of the log before the grader runs, for example:

tail -n 200 /logs/verifier/test_output.log

[napetrov]

Fixed in ec2af13: test.sh runs tail -n 200 /logs/verifier/test_output.log after eval.sh and before the grader; pinned by test_the_verifier_prints_the_end_of_the_eval_log.

[vlad-perevezentsev]

An empty file that a test patch adds has no ---/+++ lines in git's diff, only diff --git and new file mode, so it is missing from the rm -f list. Five real rows have one: Lite django__django-13448, sphinx-doc__sphinx-10451; Verified django__django-13837 (tests/utils_tests/test_module/__main__.py), django__django-14311, sphinx-doc__sphinx-10673. If the agent creates that path, git apply stops with "already exists" and the row scores 0, while a non-empty added file is removed first. Please take the names from the diff --git a/X b/Y headers too.

[vlad-perevezentsev]

The grader reads /tests/config.json in the same container, after eval.sh has run the agent's code as root. A conftest.py that rewrites that file with empty FAIL_TO_PASS/PASS_TO_PASS lists makes the row resolve, because empty lists resolve. Upstream SWE-bench parses the log on the host, so the code under test cannot change the grading rules. Could the harness pull test_output.log and grade it on the host with swebench_grade.grade and the config from the TaskSpec? That would also keep report.json and remove the exit-code fallback.

[vlad-perevezentsev]

RUN mv /testbed /app does not rename in place on overlayfs: the checkout is in a lower layer, so mv copies it into a new layer. A quick check, RUN mv /usr/share /usr/share2 on a local image, adds a 31.5 MB layer for a 31 MB directory. So every instance image gets a full copy of the repo (with .git), and the in-container chown copies it once more. Please say this in the doc instead of "Nothing is built but that one layer".

[vlad-perevezentsev]

Comments on lines outside the diff:

• dokimi/dokimi/harnesses/swebench.py:271-278: Rows are checked and written one by one, so a bad row later in the file is refused after the earlier rows are already staged. With [django__django-11099, ../x__y-1], load_swebench raises "instance_id must match", but django__django-11099/ is already on disk. The changelog says such a file "is refused before anything is staged". Please validate every selected row (id, fields, spec, eval script) first and write the files after that.

[antonwolfy]

LGTM. I pushed a commit per nit, up to 31be3de, which fix the nit from my review:

• 31be3de: dokimi/tests/test_swebench_tasks.py:275: the seaborn test now includes a PASSED line with no test name, so the middle guard is pinned.

make ci-local at 31be3de: 33 passed, 0 failed, 6 skipped.

Not merging yet: @xaleryb on network default for verify (offline vs a PyPI allowlist) and whether --all records the skipped KNOWN_BAD ids in task_runs.json; the author said both are left for the maintainers waits for your answer, @napetrov.

[napetrov]

Pushed ec2af13 (merged main, docs table conflict resolved: main's consumer column, this PR's swe-bench row).

• verdict exit code (vchamarthi, vlad-perevezentsev): fixed.
• verify network default: PyPI allowlist (pypi.org, files.pythonhosted.org), solve offline, per @xaleryb's proposal and 5/8 vs 7/8 data; --network verify=none stays.
• KNOWN_BAD: +8 rows (httpbin requests, live-linkcheck sphinx), 17 total.
• testbed env on PATH, host agent refused, skipped/empty FAIL_TO_PASS unresolved, eval-log tail in run.log.
• outside-diff comment (rows staged one by one): load_swebench now builds and checks every selected row before writing any; test_a_bad_row_late_in_the_file_stages_nothing.
• inspect checks versions too.

Not changed: --all recording skipped KNOWN_BAD ids in task_runs.json; task_runs.v1 has no field for it, so it needs a schema change in a follow-up. The banner on stderr names them.

Local: full suite 9028 passed (growth ratchet fixed after), ruff, mypy clean.