feat(tasks): run SWE-bench rows with docker-* harnesses

dokimi/changelog.d/20261024_100000_napetrov_run-swebench.md

@@ -0,0 +1,13 @@
+### Added
+
+- **`tasks run --swe-bench <rows.jsonl>`** runs SWE-bench rows (Lite,
+  Verified) with the `docker-*` harnesses: each row runs in its prebuilt
+  `swebench/sweb.eval.*` image, `docker-oracle` applies the gold patch, and the
+  verifier runs SWE-bench's eval script and grades the log with the repo's
+  SWE-bench parser (reward 1 only when every `FAIL_TO_PASS` and `PASS_TO_PASS`
+  test passes). `tasks inspect` now names `docker-*` as the consumer of a
+  SWE-bench file whose repos and versions all have a test spec.
+  The solve runs offline and the verifier on a PyPI allowlist by default;
+  `--all` skips the 17 rows in `swebench.KNOWN_BAD` known to fail with the
+  gold patch in the prebuilt image (live-network tests, Harbor's known
+  issues). Every row of a file is checked before anything is staged.

dokimi/docs/coding-harnesses.md

@@ -78,6 +78,56 @@ refused (a few hundred bytes of nested aliases expand without bound), and
 frontmatter nested too deeply to parse is a parse error. YAML and TOML dates
 and times become ISO strings, and YAML sets sorted lists.
 
+### SWE-bench rows: `tasks run --swe-bench`
+
+`python cli.py tasks run --swe-bench <rows.jsonl> --tasks <instance_id>[,...]
+--harnesses docker-oracle,docker-claude` runs SWE-bench rows (Lite, Verified,
+or any file with the same fields) with the `docker-*` harnesses; `--all`
+runs every row. Each row is staged under `<output-dir>/swe-bench-tasks/<id>/`
+as a task the harness already runs, by `dokimi/harnesses/swebench.py`:
+
+- environment: the prebuilt SWE-bench instance image
+  (`swebench/sweb.eval.x86_64.<id with __ as _1776_>:latest`, or the row's own
+  `image_name`/`docker_image`), with the checkout moved to `/app` and `/testbed`
+  left as a link to it. Nothing is built but that one layer; the first run pulls
+  the image (about 4 GB, 1-2.5 GB of it unique per instance).
+- oracle (`docker-oracle`): the row's gold `patch`, applied as SWE-bench applies
+  a prediction (`git apply`, then `patch --fuzz=5`).
+- verifier: SWE-bench's eval script (reinstall the repo, reset and apply
+  `test_patch`, run the repo's test command on the files it touches), then
+  `swebench_grade.py`, which parses the log with the repo's SWE-bench parser:
+  reward 1 only when every `FAIL_TO_PASS` and `PASS_TO_PASS` test passes.
+  Unlike upstream, a skipped `FAIL_TO_PASS` test (or a row with none) does
+  not resolve. The end of the eval log goes to `run.log`; the grader's exit
+  code is the verdict too.
+
+The image's `testbed` conda env is first on `PATH`, so an in-container agent's
+`python` and `pytest` are the repo's. A host-side agent
+(`docker-claude-host`) is refused: the instruction names container paths and
+the host has no `testbed` env.
+
+The per-repo test commands and parsers are a table,
+`dokimi/harnesses/swebench_specs.json`, generated from the `swebench`
+package by `scripts/gen_swebench_specs.py`; dokimi does not depend on
+`swebench`. Rows from a repo outside the table (SWE-bench Multilingual,
+SWE-smith, SWE-Gym repos) are refused with the repo's name, and so is a
+version the table lacks. The solve runs offline (see
+[Network policy](#network-policy)); the verifier gets an allowlist of
+`pypi.org` and `files.pythonhosted.org`, because the eval script's install
+step fetches build requirements and the packages a gold patch adds
+(`pylint-dev__pylint-4661`, `astropy__astropy-12907` fail offline).
+`--network verify=none` runs fully offline, `--network verify=public` gives
+the verifier upstream's network. A file is checked row by row before any task
+is written.
+
+`--all` skips the rows in `swebench.KNOWN_BAD`, whose gold patch fails in the
+prebuilt image under that default: `psf__requests-863`, `-1724`, `-1766`,
+`-1921`, `-1963`, `-2148`, `-2317` and `-2674` (their tests fetch httpbin.org),
+`sphinx-doc__sphinx-7985`, `-8269` and `-8475` (`test_build_linkcheck.py`
+checks live links), and the six Harbor's SWE-bench adapter lists as failing under its
+oracle (harbor 6099b8c, `adapters/swebench/README.md`, Known Issues). Naming
+one in `--tasks` still runs it.
+
 ### Other eval formats: `tasks inspect`
 
 `python cli.py tasks inspect <path> [--json] [--out FILE]` names the eval-task format of a
@@ -108,7 +158,7 @@ real files in each format, with their sources in its `MANIFEST.yaml`.
 | `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` (with `environment/Dockerfile`) |
 | `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` (with `environment/Dockerfile`) |
 | `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | `docker-*` run it as it is; `harbor`, `benchflow` through a cached Harbor export (not a `solution.yaml` task) |
-| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | none yet |
+| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | `docker-*` via `tasks run --swe-bench` (Python SWE-bench repos) |
 | `inspect-samples` | JSON/JSONL/CSV rows with `input` and `target` (or the keys a `FieldSpec` names) | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
 | `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | `questions import` ([answer-only formats](answer-formats.md)) |
 | `openai-evals` | JSONL rows with `input` (chat messages) and `ideal` | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
@@ -145,7 +195,8 @@ sets `deploy.resources.limits.memory`, which `docker compose` honours and
 dokimi reads as the task's own value. `DOKIMI_TBENCH_V1_MEMORY_MB` changes the
 8192 default. The Harbor export writes the same value to `task.toml`.
 
-`tasks list` and `tasks run` still read only the first three.
+`tasks list` and `tasks run` still read only the first three; `tasks run --swe-bench`
+also takes a SWE-bench file.
 
 The `docker-*` harnesses mount the oracle at `/solution` and `/oracle` and the
 verifier at `/tests` and `/verifier`, because a script written for one layout

dokimi/dokimi/commands/tasks.py

@@ -122,9 +122,33 @@ def location(name: str) -> str | None:
             for h in harnesses if h != baseline and (loc := location(h)) not in (None, base)]
 
 
+def _select_swebench(args: argparse.Namespace) -> list:
+    """Stage the selected rows of a SWE-bench dataset file under the output dir."""
+    from dokimi.harnesses.swebench import KNOWN_BAD, load_swebench, read_instances
+
+    ids = None if args.all else [t.strip() for t in (args.tasks or "").split(",") if t.strip()]
+    if ids == []:
+        raise SystemExit("No tasks selected. Pass --tasks <instance_id[,...]> or --all.")
+    try:
+        if ids is None:
+            skipped = sorted(KNOWN_BAD.keys() & {str(r.get("instance_id")) for r in read_instances(Path(args.swe_bench))})
+            if skipped:
+                print(f"--swe-bench --all skips {len(skipped)} known-bad rows (name them in --tasks to run them): "
+                      + ", ".join(skipped), file=sys.stderr)
+        tasks = load_swebench(Path(args.swe_bench), Path(args.output_dir) / "swe-bench-tasks", ids)
+    except (OSError, ValueError) as exc:  # SweBenchError and bad JSON are ValueErrors
+        raise SystemExit(f"--swe-bench: {exc}") from None
+    if not tasks:  # an empty run must not exit 0 with an empty artifact
+        raise SystemExit(f"--swe-bench: no runnable rows in {args.swe_bench} "
+                         "(--all skips the known-bad rows)")
+    return tasks
+
+
 def _select_tasks(args: argparse.Namespace) -> list:
     from dokimi.harnesses import discover_tasks, load_task
 
+    if getattr(args, "swe_bench", None):
+        return _select_swebench(args)
     root = _tasks_root(args)
     try:
         if args.all:
@@ -254,6 +278,11 @@ def register(sub, positive_int) -> None:
     run_p.add_argument("--baseline-harness", default=None, dest="baseline_harness")
     run_p.add_argument("--model", default=None, help="Model ref passed to the harness")
     run_p.add_argument("--tasks-root", default=None, help=_TASKS_ROOT_HELP)
+    run_p.add_argument(
+        "--swe-bench", default=None, dest="swe_bench", metavar="FILE",
+        help=("A SWE-bench dataset (.jsonl or .json rows); --tasks names instance ids. "
+              "Each row runs in its prebuilt swebench/sweb.eval.* image (docker-* harnesses)."),
+    )
     run_p.add_argument("--output-dir", default="results/task-runs", dest="output_dir")
     run_p.add_argument("--out-json", default=None, dest="out_json")
     run_p.add_argument("--out-md", default=None, dest="out_md",

dokimi/dokimi/harnesses/docker_solver.py

@@ -83,6 +83,7 @@
 from .tasks import (  # noqa: F401  (timeouts re-exported)
     _DEFAULT_BUILD_TIMEOUT,
     _DEFAULT_VERIFY_TIMEOUT,
+    FORMAT_SWE_BENCH,
     FORMAT_TASK_MD,
     FORMAT_TBENCH_V1,
     ContainerStep,
@@ -728,15 +729,14 @@ def _describe_command(self, task: TaskSpec, model: Optional[str]) -> list[str]:
 def _refuse_unrunnable(task: TaskSpec, mode: str, agent_location: str) -> None:
     """Refuse, from the task file alone, what a docker-* agent cannot run.
 
-    - A ``task.md`` task, for a host-side agent: its instruction names absolute
-      container paths (``/root/...``, ``/app/...``) and the agent runs on the
-      host in a copy of ``/app``, so it writes to the operator's machine and the
-      verifier, in the container, finds nothing: a false fail plus a host write.
-      The in-container agent runs such a task when it works in ``/app``.
+    - A ``task.md``, terminal-bench-v1 or SWE-bench task, for a host-side
+      agent: its instruction names container paths (``/root/...``, ``/app/...``)
+      and the agent runs on the host in a copy of ``/app``: a host write plus a
+      false fail, as the verifier in the container finds nothing. The
+      in-container agent runs such a task when it works in ``/app``.
     - A declared ``environment.workdir`` outside ``/app``, for either agent:
       both edit ``/app`` only (see :func:`_require_agent_workdir` for the
       image's own ``WORKDIR``, which needs the built image).
-
     - A task without the directory a mode copies in: ``tests/`` (``verifier/``)
       for the verifier, and ``solution/`` (``oracle/``) for ``docker-oracle``.
       ``docker cp`` would fail only at that step, after the image build and,
@@ -757,7 +757,7 @@ def _refuse_unrunnable(task: TaskSpec, mode: str, agent_location: str) -> None:
                 raise ValueError(f"{task.name}: no {what} to copy to {dest}")
     if mode not in ("claude", "probe"):
         return
-    if agent_location == "host" and task.format in (FORMAT_TASK_MD, FORMAT_TBENCH_V1):
+    if agent_location == "host" and task.format in (FORMAT_TASK_MD, FORMAT_TBENCH_V1, FORMAT_SWE_BENCH):
         raise ValueError(
             f"{task.name}: a {task.format} task cannot be solved by a host-side agent "
             f"(its instruction names absolute container paths); {_IN_CONTAINER}"