Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
### Changed

- **`docker-*` task containers are hardened like `repos` containers** — the
task container starts with `--init`, `--cap-drop ALL`, `--security-opt no-new-privileges`
and `--pids-limit 32768`, and gets back only `CHOWN`, `DAC_OVERRIDE` and `FOWNER`
(the harness's own root steps on the agent's tree) and `SETUID` and `SETGID`
(task scripts that drop privileges, such as apt's `_apt` sandbox). A setuid binary
in a task image can no longer raise its privileges. Every other capability Docker
grants by default is dropped, so a solve or verifier step that needs one now fails:
killing another user's process (`KILL`), raw sockets (`NET_RAW`), `mknod`
(`MKNOD`), `chroot` (`SYS_CHROOT`). Setting the setgid bit on a file whose group
root is not in (`FSETID`) exits 0 and leaves the bit off. A port below 1024 still
binds. Before, the container ran with Docker's default capabilities and no process
limit. `docker-*` results from before this change may not be comparable for a task
whose step needs a dropped capability: re-run it to compare.
10 changes: 10 additions & 0 deletions dokimi/docs/coding-harnesses.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,16 @@ The result is in `metrics.isolation_probe` and `isolation_probe.json`:
- **Memory.** The agent's CLI and its tmpfs HOME run inside the task's
cgroup, so they count against `environment.memory_mb` (about 100 MB for a
short session); the host launch ran outside it.
- **Capabilities.** Every `docker-*` task container, `docker-oracle` included,
starts with `--init`, `--cap-drop ALL`, `no-new-privileges` and `--pids-limit 32768`,
and gets back only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID` and `SETGID`
(`TASK_CONTAINER_HARDENING` in `harnesses/container_agent.py`). Every other
capability Docker grants by default is dropped, so a step that needs one
fails: signalling another user's process (`KILL`), a raw socket (`NET_RAW`),
`mknod` (`MKNOD`), `chroot` (`SYS_CHROOT`). Without `FSETID`, `chmod g+s` on
a file whose group root is not in exits 0 and leaves the bit off. A port
below 1024 still binds: Docker sets the container's
`ip_unprivileged_port_start` to 0.
- **Network.** The container sits on a per-run `--internal` Docker network,
which has no route out. The agent reaches the LLM through an allowlisting
CONNECT proxy (`harnesses/egress_proxy.py`, stdlib only). It runs in a
Expand Down
9 changes: 9 additions & 0 deletions dokimi/docs/contributing-terminal-bench-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,15 @@ correctness baseline for the *task*, not a tutorial. Every task needs one,
because the `docker-oracle` harness is how a task is proven runnable without
spending model tokens.

The solve and the verifier run in a container without Linux capabilities
beyond `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID` and `SETGID`, so a setuid
binary cannot raise its privileges and the process count is capped at 32768.
A step that signals another user's process, opens a raw socket, or runs `mknod`
or `chroot` fails; `docker-oracle` will show it. One loss is silent: setting the
setgid bit on a file whose group root is not in exits 0 and leaves the bit off.
A port below 1024 still binds. See
[What the agent can reach](coding-harnesses.md#what-the-agent-can-reach).
Comment on lines +186 to +193

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list of what now fails is not right. Ports below 1024 still work: Docker (20.10+) sets ip_unprivileged_port_start to 0 inside each container network namespace, so the host value does not matter (checked on Docker 29 with the host at 1024: root and nobody both bind port 80 with these flags). On the other side, more steps fail than listed: chroot (SYS_CHROOT), mknod (MKNOD), setcap in a package postinst (SETFCAP), setting setgid bits on files of another group (FSETID). Please drop the NET_BIND_SERVICE case and list the real ones, here, in coding-harnesses.md:296-299 and in the changelog fragment.


## 8 — Local verification

```bash
Expand Down
13 changes: 13 additions & 0 deletions dokimi/dokimi/harnesses/container_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
from typing import Callable, Iterable, Optional

from dokimi import process
from dokimi.repos.container import PIDS_LIMIT

from . import egress_proxy
from .agent_sandbox import (
Expand Down Expand Up @@ -83,6 +84,18 @@
_FILE_VARS = ("NODE_EXTRA_CA_CERTS", "SSL_CERT_FILE", "REQUESTS_CA_BUNDLE", "AWS_CA_BUNDLE",
"GOOGLE_APPLICATION_CREDENTIALS", "AWS_SHARED_CREDENTIALS_FILE", "AWS_CONFIG_FILE")
PROXY_PORT = 3128
# The task container's hardening flags are the ones repos/container.py sets, --init included
# (PID 1 reaps the orphans a step leaves behind). Unlike there, root here gets back the few
# capabilities its own steps need: CHOWN and FOWNER (chown/chmod of the agent's tree) and
# DAC_OVERRIDE (root writing the agent's files). SETUID/SETGID are for task scripts that drop
# privileges (apt's _apt sandbox, su); the harness's own `exec -u` steps need neither.
# KILL, NET_BIND_SERVICE and the rest stay dropped.
TASK_CONTAINER_HARDENING = (
"--init", "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
"--pids-limit", str(PIDS_LIMIT),
"--cap-add", "CHOWN", "--cap-add", "DAC_OVERRIDE", "--cap-add", "FOWNER",
"--cap-add", "SETUID", "--cap-add", "SETGID",
)
_PROBE_TIMEOUT_SEC = 120.0
DOCKER_STEP_TIMEOUT = 600.0 # a docker step without a budget of its own (cp, exec, inspect, ...)
# The sidecar needs only a Python 3 interpreter. Pinned by digest (a tag moves, and
Expand Down
11 changes: 5 additions & 6 deletions dokimi/dokimi/harnesses/docker_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@
)
from .claude_json import _last_json_object, _parse_claude_json # noqa: F401
from .command import _parse_reward_text
from .container_agent import AGENT_BIN, DOCKER_STEP_TIMEOUT, FILES_DIR, ContainerSandbox, Docker
from .container_agent import (AGENT_BIN, DOCKER_STEP_TIMEOUT, FILES_DIR, TASK_CONTAINER_HARDENING,
ContainerSandbox, Docker)
from .models import HarnessResult, OperationRecord
from .network_policy import (DEFAULTS, NetworkPlan, PhasePolicy, build_args, enter_phase, needs_sandbox,
resolve_recorded, skill_needs, start_args, redact as _redact)
Expand Down Expand Up @@ -352,11 +353,9 @@ def _start_container(self, task: TaskSpec, image: str, container: str, log,
+ _task_verify_timeout(task) + 600))
self._docker(["rm", "-f", container], log, check=False)
env_args = [a for k, v in task.container_env.items() for a in ("-e", f"{k}={v}")]
self._docker(
["run", "-d", "--name", container, *net_args, *env_args,
"--cpus", cpus, "--memory", memory, image, "sleep", str(lifetime)],
log,
)
run_args = ["run", "-d", "--name", container, *TASK_CONTAINER_HARDENING, *net_args,
*env_args, "--cpus", cpus, "--memory", memory, image, "sleep", str(lifetime)]
Comment on lines +356 to +357

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

repos/container.py pairs --pids-limit with --init, but here PID 1 is sleep, which never reaps. Every process that outlives its docker exec (nohup ... &, a daemon a test starts) becomes a zombie when it exits and still counts against the new limit (checked: 5 orphans give 5 zombies in pids.current). Please add --init to TASK_CONTAINER_HARDENING too.

self._docker(run_args, log)
return enter_phase(plan.solve if inside else DEFAULTS["solve"], container, self._docker, log,
bridged="bridge" in net_args)

Expand Down
2 changes: 1 addition & 1 deletion dokimi/scripts/growth_baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"files": {
"dokimi/commands/novelty.py": {"lines": 984, "rationale": "estimate-only projection (expected_usd, arm scaling, extrapolation flag) in novelty CLI, on top of main's W-series and #173 lines; file split tracked separately"},
"dokimi/harnesses/benchflow.py": {"lines": 869},
"dokimi/harnesses/docker_solver.py": {"lines": 900, "rationale": "pre-existing when the gate was introduced"},
"dokimi/harnesses/docker_solver.py": {"lines": 899, "rationale": "pre-existing when the gate was introduced"},
"dokimi/llm.py": {"lines": 1118, "rationale": "grew on main in #180 (replay a stored retry) after #93 seeded this baseline; split it in its own PR"},
"dokimi/report/model_compare.py": {"lines": 1209, "rationale": "grew on main in #73 (judge-validity --context stored) before the ratchet landed; split it in its own PR"},
"tests/test_arm_runner.py": {"lines": 1176, "rationale": "pre-existing when the gate was introduced"},
Expand Down
20 changes: 20 additions & 0 deletions dokimi/tests/test_docker_solver_harness.py
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,26 @@ def test_start_container_malformed_memory_falls_back_instead_of_crashing(tmp_pat
assert run_call[run_call.index("--memory") + 1] == "2g"


def test_start_container_drops_capabilities_and_bounds_pids(tmp_path, monkeypatch):
# Regression (#229 P2-1a): the task container had no cap drop, no-new-privileges or
# pids limit, unlike repos/container.py. Only the named capabilities the phase steps
# need (chown, DAC override, file owner, setuid/setgid for task scripts) are added back.
# The flags must sit before the image: after it, docker hands them to `sleep`.
task = load_task(_fixture_task(tmp_path))
run_call = next(c for c in _start(task, "container-6") if c[0] == "run")
image_at = run_call.index("img:latest")
assert run_call[run_call.index("--cap-drop") + 1] == "ALL"
assert run_call[run_call.index("--security-opt") + 1] == "no-new-privileges"
assert run_call[run_call.index("--pids-limit") + 1] == "32768"
added = {run_call[i + 1] for i, arg in enumerate(run_call) if arg == "--cap-add"}
assert added == {"CHOWN", "DAC_OVERRIDE", "FOWNER", "SETUID", "SETGID"}
for flag in ("--init", "--cap-drop", "--security-opt", "--pids-limit", "--cap-add"):
assert flag in run_call
assert all(i < image_at for i, arg in enumerate(run_call) if arg == flag)
Comment thread
antonwolfy marked this conversation as resolved.
Resolved
assert "--privileged" not in run_call
assert [run_call[i + 1] for i, arg in enumerate(run_call) if arg == "--security-opt"] == ["no-new-privileges"]

Comment thread
antonwolfy marked this conversation as resolved.
Resolved

def test_solve_oracle_copies_the_whole_solution_directory(tmp_path, monkeypatch):
# Regression test: a real run against uxlfoundation/skills'
# onetbb-histogram-local-aggregation failed (reward 0) because solve.sh
Expand Down
Loading