feat(tasks): run terminal-bench v1 tasks natively, export to Harbor lazily

dokimi/changelog.d/20261024_100000_napetrov_run-tbench-v1.md

@@ -0,0 +1,16 @@
+### Added
+
+- **terminal-bench v1 tasks run natively** — a `task.yaml` task
+  (laude-institute/terminal-bench `original-tasks/`) loads as a `TaskSpec` and
+  runs under `docker-oracle` / `docker-claude` as terminal-bench runs it:
+  `solution.sh` at `/oracle`, `run-tests.sh` and `tests/` at `/tests`, the
+  image's user, `TEST_DIR` and the compose environment, and the pytest summary
+  as the verdict. A task one `docker run` cannot reproduce (extra services,
+  volumes, tmpfs, ports, a service command, build args or target, a host
+  environment value; `solution.yaml`; `run_tests_in_same_shell`) is refused
+  with the reason. Harbor-backed harnesses and `benchflow*` run a Harbor copy
+  written lazily into a content-keyed cache; its `test.sh` scores the same
+  pytest summary. `tasks inspect` lists `docker-*`, `harbor` and `benchflow` as
+  consumers (only `docker-*` for a `solution.yaml` task, which has no Harbor export). A v1 task gets 8 GB of memory
+  (its compose `deploy.resources.limits.memory` when set; the default is
+  `DOKIMI_TBENCH_V1_MEMORY_MB`, else 8192), where other formats default to 2 GB.

dokimi/docs/coding-harnesses.md

@@ -69,8 +69,8 @@ Jekyll or Hugo, an unclosed block) is an author's note, and the task loads
 from `task.toml` with a warning (none for a note without frontmatter, or with
 Hugo's TOML `+++` frontmatter, which is not frontmatter here).
 
-Discovery skips a lone `task.md` without frontmatter with a warning;
-`load_task` on it raises. A lone `task.md` with frontmatter is a task, whatever
+Discovery skips a lone `task.md` without frontmatter, and a `task.yaml`
+mapping with no `instruction`, with a warning; `load_task` on either raises. A lone `task.md` with frontmatter is a task, whatever
 its keys. One that opens with `---` but does not parse (a horizontal rule, an
 unclosed block, invalid YAML, a non-mapping) fails discovery of the whole
 root, as a malformed `task.toml` does; `coverage` skips it. YAML aliases are
@@ -107,7 +107,7 @@ real files in each format, with their sources in its `MANIFEST.yaml`.
 |---|---|---|---|
 | `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` (with `environment/Dockerfile`) |
 | `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` (with `environment/Dockerfile`) |
-| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | none yet |
+| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | `docker-*` run it as it is; `harbor`, `benchflow` through a cached Harbor export (not a `solution.yaml` task) |
 | `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | none yet |
 | `inspect-samples` | JSON/JSONL/CSV rows with `input` and `target` (or the keys a `FieldSpec` names) | answer-only | none yet |
 | `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | none yet |
@@ -122,7 +122,30 @@ real files in each format, with their sources in its `MANIFEST.yaml`.
 | `dokimi-explore-task` | YAML with `id`, `query` and `references[].path` | localization | explore track |
 | `git-lfs-pointer` | a git-LFS spec v1 pointer stub | unknown | none (fetch it first) |
 
-`tasks list` and `tasks run` still read only the first two.
+A terminal-bench v1 task (`task.yaml`, laude-institute/terminal-bench
+`original-tasks/`) runs as it is under `docker-*`: `solution.sh` is copied to
+`/oracle/solution.sh`, `run-tests.sh` and `tests/` to `/tests`, both run as the
+image's user with `TEST_DIR=/tests` and the compose `environment`, and the
+verdict is the pytest `short test summary info` (every listed test passed, skipped or
+xfailed), not the exit code, as terminal-bench reads it. `docker-*` refuses,
+with the reason, a task whose `docker-compose.yaml` needs more than one
+`docker run` of its default build stage (a second service, volumes, tmpfs,
+ports, a service `command` other than `sleep infinity`, `build.args` or
+`build.target`, an `environment` value read from the host), a `parser_name` other
+than `pytest`, `run_tests_in_same_shell`, or (oracle only) a `solution.yaml`.
+The Harbor-backed harnesses and `benchflow*` run a Harbor copy written on first
+use to `~/.cache/dokimi/harbor-export/<content hash>/` (mode 0700) by
+[`harbor_export.py`](../dokimi/harnesses/harbor_export.py); its `test.sh` scores
+the same pytest summary, so a task gets one verdict on every backend.
+
+Memory: terminal-bench sets no limit, so a v1 task gets
+`environment.memory_mb = 8192` (not the 2048 of other formats; tasks such as
+`predicate-pushdown-bench` are OOM-killed at 2 GB), unless its compose file
+sets `deploy.resources.limits.memory`, which `docker compose` honours and
+dokimi reads as the task's own value. `DOKIMI_TBENCH_V1_MEMORY_MB` changes the
+8192 default. The Harbor export writes the same value to `task.toml`.
+
+`tasks list` and `tasks run` still read only the first three.
 
 The `docker-*` harnesses mount the oracle at `/solution` and `/oracle` and the
 verifier at `/tests` and `/verifier`, because a script written for one layout

dokimi/docs/contributing-terminal-bench-task.md

@@ -30,6 +30,8 @@ read it. Contribute the task itself to the content repository; contribute change
 to this format here, next to the code that parses it
 ([`dokimi/harnesses/tasks.py`](../dokimi/harnesses/tasks.py),
 [`dokimi/harnesses/docker_solver.py`](../dokimi/harnesses/docker_solver.py)).
+An existing terminal-bench v1 task (`task.yaml` + `run-tests.sh`) runs without
+conversion; see the format table in [coding-harnesses.md](coding-harnesses.md).
 
 ## 1 — Pick a task
 

dokimi/dokimi/harnesses/benchflow.py

@@ -44,6 +44,7 @@
 from dokimi import process
 
 from .command import clear_run_dir
+from .harbor_export import harbor_view
 from .models import HarnessResult, OperationRecord, PreflightError
 from .tasks import TaskSpec
 
@@ -95,8 +96,7 @@
 # the LiteLLM it starts all the same.
 _BRIDGE_NO_PROXY = ("172.17.0.1", "host.docker.internal")
 _PROXY_VARS = ("http_proxy", "https_proxy", "HTTP_PROXY", "HTTPS_PROXY")
-# Paths bench and docker read from the env; see _subprocess_env.
-_PATH_VARS = ("BENCHFLOW_SKILL_CREATOR_DIR", "DOCKER_CONFIG")
+_PATH_VARS = ("BENCHFLOW_SKILL_CREATOR_DIR", "DOCKER_CONFIG")  # read from the env; see _subprocess_env
 
 
 @dataclass(frozen=True)
@@ -122,8 +122,8 @@ def run(
         timeout_sec: Optional[float] = None,
         dry_run: bool = False,
     ) -> HarnessResult:
-        # bench runs with cwd=<cell>: every path it is given must be absolute.
-        # The row records the dir as given, like a docker-* row.
+        # bench runs with cwd=<cell>: every path absolute; the row records the dir as given.
+        task = harbor_view(task)  # a terminal-bench v1 task runs from its Harbor copy
         given_dir = Path(output_dir) if output_dir else None
         output_dir = given_dir.resolve() if given_dir else None
         if output_dir:

dokimi/dokimi/harnesses/command.py

@@ -20,6 +20,7 @@
 
 from .models import HarnessResult, OperationRecord
 from .operations import load_operations
+from .harbor_export import harbor_view
 from .tasks import TaskSpec, harbor_task_problem
 
 
@@ -64,6 +65,7 @@ def run(
         dry_run: bool = False,
     ) -> HarnessResult:
         if self.kind == "harbor":
+            task = harbor_view(task)  # a terminal-bench v1 task runs from its Harbor copy
             # Harbor's own rejection of the directory exits 1, which would read
             # as a fail; refuse first, so the cell is an error (missing data).
             problem = harbor_task_problem(task, needs_solution=self.agent == "oracle")

dokimi/dokimi/harnesses/docker_solver.py

@@ -75,13 +75,17 @@
 from .container_agent import AGENT_BIN, DOCKER_STEP_TIMEOUT, FILES_DIR, ContainerSandbox, Docker
 from .models import HarnessResult, OperationRecord
 from .network_policy import (DEFAULTS, NetworkPlan, PhasePolicy, build_args, enter_phase, needs_sandbox,
-                             proxy_env_flags, resolve, resolve_recorded, skill_needs, start_args, redact as _redact)
+                             resolve_recorded, skill_needs, start_args, redact as _redact)
 from .skill_treatment import Treatment, build_plugin, check_skill, skill_listing, treatment_notice
 from .solver_stream import record_solver, run_solver
+from .task_env import _exec_env, unresolved_env  # noqa: F401  (re-exported)
+from .tbench_v1 import reward as _pytest_reward
 from .tasks import (  # noqa: F401  (timeouts re-exported)
     _DEFAULT_BUILD_TIMEOUT,
     _DEFAULT_VERIFY_TIMEOUT,
     FORMAT_TASK_MD,
+    FORMAT_TBENCH_V1,
+    ContainerStep,
     TaskSpec,
     _task_build_timeout,
     _task_verify_timeout,
@@ -298,7 +302,7 @@ def _build_image(self, task: TaskSpec, image: str, log, net: Optional[PhasePolic
         built from a copy of ``environment/`` without that directory — the
         same rule BenchFlow applies to its no-skill runs.
         """
-        env_dir = task.path / "environment"
+        env_dir = task.build_context
         flags = build_args(net or DEFAULTS["build"])
         # Honor the task's own build_timeout_sec (some images need >10 min);
         # _DEFAULT_BUILD_TIMEOUT is only the fallback when the task omits it.
@@ -318,7 +322,7 @@ def _build_image(self, task: TaskSpec, image: str, log, net: Optional[PhasePolic
                 (context / "skills").mkdir()
             self._docker(
                 ["build", "-q", *flags, "-t", image,
-                 "-f", str(context / "Dockerfile"), str(context)],
+                 "-f", str(context / task.dockerfile), str(context)],
                 log, timeout=timeout,
             )
 
@@ -347,8 +351,9 @@ def _start_container(self, task: TaskSpec, image: str, container: str, log,
         lifetime = int(max(1800.0, (task.timeout_sec or _DEFAULT_SOLVE_TIMEOUT)
                            + _task_verify_timeout(task) + 600))
         self._docker(["rm", "-f", container], log, check=False)
+        env_args = [a for k, v in task.container_env.items() for a in ("-e", f"{k}={v}")]
         self._docker(
-            ["run", "-d", "--name", container, *net_args,
+            ["run", "-d", "--name", container, *net_args, *env_args,
              "--cpus", cpus, "--memory", memory, image, "sleep", str(lifetime)],
             log,
         )
@@ -367,10 +372,10 @@ def _solve_oracle(self, task: TaskSpec, container: str, log, net: Optional[Phase
         ``/solution`` (Harbor) and ``/oracle`` (BenchFlow), because a script
         written for one layout refers to its siblings by that layout's path.
         """
-        _copy_dir_into(task.solution_dir, container, ("/solution", "/oracle"), log, self._docker)
+        step = task.oracle_step
+        _copy_into(step, container, log, self._docker)
         self._docker(
-            ["exec", "-u", "root", *_exec_env(task, "oracle", net), container, "bash", "-lc",
-             "chmod +x /solution/solve.sh && /solution/solve.sh"],
+            ["exec", *_user(step), *_exec_env(task, "oracle", net), container, "bash", "-lc", step.command],
             log, check=False, timeout=timeout,
         )
 
@@ -627,16 +632,21 @@ def _verify(self, task: TaskSpec, container: str, output_dir: Path, log,
         SkillsBench ``test.sh`` runs ``pytest /verifier/test_outputs.py`` even
         after ``bench tasks export`` has renamed the directory to ``tests/``.
         """
-        _copy_dir_into(task.tests_dir, container, ("/tests", "/verifier"), log, self._docker)
+        step = task.verifier_step
+        _copy_into(step, container, log, self._docker)
         # After an in-container agent the verifier runs its /app code as root: no
         # proxy credentials then (the sidecar kept them from the agent; see :func:`_exec_env`).
         agent_code = self.mode == "claude" and self.agent_location == "container"
         proc = self._docker(
-            ["exec", "-u", "root", *_exec_env(task, "verifier", net, credentials=not agent_code),
-             container, "bash", "-lc",
-             "mkdir -p /logs/verifier && chmod +x /tests/test.sh && /tests/test.sh"],
+            ["exec", *_user(step), *_exec_env(task, "verifier", net, credentials=not agent_code),
+             container, "bash", "-lc", step.command],
             log, check=False, timeout=_task_verify_timeout(task),
         )
+        if step.parser != "reward-file":  # terminal-bench: the pytest summary is the verdict
+            verdict, tests = _pytest_reward((proc.stdout or "") + (proc.stderr or ""))
+            (output_dir / "verifier_tests.json").write_text(json.dumps(
+                {"parser": step.parser, "exit_code": proc.returncode, "tests": tests}, indent=2), encoding="utf-8")
+            return verdict
         reward_file = output_dir / "reward.txt"
         pulled = self._docker(
             ["cp", f"{container}:/logs/verifier/reward.txt", str(reward_file)],
@@ -736,16 +746,20 @@ def _refuse_unrunnable(task: TaskSpec, mode: str, agent_location: str) -> None:
     that never runs. It copies neither directory. Oracle mode is refused only
     for a missing directory.
     """
+    problem = task.local_docker_problem(oracle=mode == "oracle")
+    if problem:
+        raise ValueError(f"{task.name}: a docker-* harness cannot run it: {problem}")
     if mode != "probe":
-        needed = [("tests", task.tests_dir)] + ([("solution", task.solution_dir)] if mode == "oracle" else [])
-        for role, directory in needed:
-            if not directory.is_dir():
-                raise ValueError(f"{task.name}: no {directory.name}/ directory to copy to /{role}")
+        steps = [task.verifier_step] + ([task.oracle_step] if mode == "oracle" else [])
+        for src, dest in (c for step in steps for c in step.copies):
+            if not src.exists():
+                what = f"{src.name}/ directory" if src in (task.solution_dir, task.tests_dir) else src.name
+                raise ValueError(f"{task.name}: no {what} to copy to {dest}")
     if mode not in ("claude", "probe"):
         return
-    if agent_location == "host" and task.format == FORMAT_TASK_MD:
+    if agent_location == "host" and task.format in (FORMAT_TASK_MD, FORMAT_TBENCH_V1):
         raise ValueError(
-            f"{task.name}: a task.md task cannot be solved by a host-side agent "
+            f"{task.name}: a {task.format} task cannot be solved by a host-side agent "
             f"(its instruction names absolute container paths); {_IN_CONTAINER}"
         )
     declared = _declared_workdir(task)
@@ -819,59 +833,18 @@ def _task_workdir(task: TaskSpec, container: str, log, docker: "Docker") -> str:
     return found if found != "/" else "/app"
 
 
-def _copy_dir_into(src: Path, container: str, dests: tuple[str, ...], log, docker: "Docker") -> None:
-    """Copy *src*'s contents to each dest: ``docker cp src c:dest`` nests ``src``
-    inside a dest the image already has (``RUN mkdir -p /tests/...``), so the
+def _copy_into(step: ContainerStep, container: str, log, docker: "Docker") -> None:
+    """Copy a step's files in. A directory's contents go to its dest: ``docker cp src c:dest``
+    nests ``src`` inside a dest the image already has (``RUN mkdir -p /tests/...``), so the
     entry script would be missing. Same as BenchFlow: ``mkdir -p``, then ``src/.``."""
-    docker(["exec", "-u", "root", container, "mkdir", "-p", *dests], log)
-    for dest in dests:
-        docker(["cp", f"{src}/.", f"{container}:{dest}"], log)
-
-
-def _exec_env(task: TaskSpec, table: str, net: Optional[PhasePolicy] = None, *,
-              credentials: bool = True) -> list[str]:
-    """``docker exec -e`` flags for the oracle or verifier step.
-
-    First the proxy variables of the step's network policy (:func:`.network_policy.proxy_env_flags`),
-    on the exec, not on ``docker run``, so they reach only that step: a public verifier behind
-    a corporate proxy that installs pytest at verify time (52 of 101 SkillsBench ``test.sh``)
-    otherwise fails with reward 0. With ``credentials=False`` (the verifier after an
-    in-container agent runs the agent's code) their ``user:pw@`` is dropped. Without *net*,
-    the task's own plan.
-
-    Then the task's ``[verifier.env]`` / ``[oracle.env]``: only literal values
-    are forwarded; BenchFlow's ``${VAR}`` host references are left to
-    BenchFlow, so a secret is never read from our environment on a task
-    file's say-so.
-    """
-    if net is None:
-        plan = resolve(task)
-        net = plan.verify if table == "verifier" else plan.solve
-    flags = proxy_env_flags(net, credentials=credentials)
-    source = task.verifier if table == "verifier" else task.oracle
-    env = source.get("env")
-    if not isinstance(env, dict):
-        return flags
-    for key, value in env.items():
-        if isinstance(value, bool):
-            flags += ["-e", f"{key}={'true' if value else 'false'}"]  # YAML true, not Python True
-        elif isinstance(value, (str, int, float)) and "${" not in str(value):
-            flags += ["-e", f"{key}={value}"]
-    return flags
-
-
-def unresolved_env(task: TaskSpec, table: str) -> list[str]:
-    """Names in ``[verifier.env]`` / ``[oracle.env]`` that :func:`_exec_env`
-    does not forward: a ``${VAR}`` host reference (with or without a default),
-    or a value that is not a scalar (list, table, null)."""
-    source = task.verifier if table == "verifier" else task.oracle
-    env = source.get("env")
-    if not isinstance(env, dict):
-        return []
-    return sorted(
-        str(k) for k, v in env.items()
-        if not isinstance(v, (str, int, float)) or (isinstance(v, str) and "${" in v)
-    )
+    dirs = [d if src.is_dir() else posixpath.dirname(d) for src, d in step.copies]
+    docker(["exec", "-u", "root", container, "mkdir", "-p", *dict.fromkeys(dirs)], log)
+    for src, dest in step.copies:
+        docker(["cp", f"{src}/." if src.is_dir() else str(src), f"{container}:{dest}"], log)
+
+
+def _user(step: ContainerStep) -> list[str]:
+    return ["-u", step.user] if step.user else []
 
 
 def _ignore_top_level_skills(env_dir: Path):