[napetrov]

Summary

P2-1a of #229. The docker-* task container (docker_solver._start_container) now starts with the same hardening flags repos/container.py sets, plus five named capabilities that root needs for the harness's own steps.

docker run -d --name <c>
  --cap-drop ALL                       (was: Docker default caps)
  --security-opt no-new-privileges     (was: absent)
  --pids-limit 32768                   (was: absent)
  --cap-add CHOWN --cap-add FOWNER     chown/chmod of the agent tree
  --cap-add DAC_OVERRIDE               root writes into agent-owned files
  --cap-add SETUID --cap-add SETGID    apt sandbox, setpriv, runuser
  ...

Files:

• harnesses/container_agent.py: TASK_CONTAINER_HARDENING (the flag tuple, with the reason for each add-back).
• harnesses/docker_solver.py: the run call uses it (900 → 899 lines, growth baseline lowered).
• tests/test_docker_solver_harness.py: test_start_container_drops_capabilities_and_bounds_pids.
• docs/coding-harnesses.md, docs/contributing-terminal-bench-task.md: a note on the dropped capabilities for task authors.
• changelog fragment.

No shared policy builder: repos/container.py and metrics/example_runner.py are untouched.

Evidence

Failing test on the old code, passing on the new (tests/test_docker_solver_harness.py):

test_start_container_drops_capabilities_and_bounds_pids
  before: ValueError: '--cap-drop' is not in list
  after:  passed

Full suite on the pre-rebase tree: 8708 passed, 4 skipped. After the final commit, test_docker_solver_harness.py, test_docker_task_md.py and test_container_agent.py pass (207 passed, 1 skipped, then 141 passed, 1 skipped after a comment-only change), and make ci-local CI_LOCAL_ARGS='--offline --jobs static,typecheck' passes on the rebased HEAD.

Smoke runs: a small image, docker run with the old flags and with the new ones, --cpus 8, --rm, run-specific names, every container stopped. Each probe runs as the harness does (root steps via exec -u root, agent steps as uid 10001):

• Old flags: every check passes except no-new-privs, which the old flags do not set.
• Drop ALL alone: apt, setpriv/runuser, chown, agent write into a bind mount, root write into it, chmod, mkdir and killing an agent process all fail.
• Full new set: every check passes except killing the agent process from root (KILL is not added back, on purpose).
• New set minus one capability at a time: each of CHOWN, DAC_OVERRIDE, FOWNER, SETUID and SETGID breaks the steps it covers, so each add-back is needed.
• bind-port-80 passed in every mode: the test host has ip_unprivileged_port_start=0, so NET_BIND_SERVICE is not exercised here.

End to end: dokimi tasks run --harnesses docker-oracle on a small task passed (1/1, reward 1.0) with the hardened docker run in the run log.

Merge Danger

Door: two-way. Revert restores the old container.

Runs can now fail where a task needs a dropped capability. This is the behaviour change to check before merge:

• KILL: a solve or verifier that signals another user's process (kill as root on uid-10001 processes) fails.
• NET_BIND_SERVICE: binding a port below 1024 fails on hosts where net.ipv4.ip_unprivileged_port_start is 1024.
• NET_RAW: ping, raw sockets fail.
• A process limit of 32768 applies to every task container.
• setuid binaries in a task image can no longer raise privileges (no-new-privileges), so a task that relies on sudo fails.

Blast Radius: every docker-* task container (docker-oracle, docker-claude*, docker-probe*, SWE-bench rows from #213 once they land). The egress sidecar container (container_agent.py, docker create near L186) is still unhardened; that is a follow-up, not in this PR.

Numbers do not change: this PR does not touch scoring, token or cost accounting.

Refs #229 (P2-1a).

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.83%. Comparing base (a2b787a) to head (b1340c2).
⚠️ Report is 14 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #248      +/-   ##
==========================================
+ Coverage   92.74%   92.83%   +0.08%     
==========================================
  Files         254      262       +8     
  Lines       32087    33019     +932     
  Branches     5344     5510     +166     
==========================================
+ Hits        29759    30652     +893     
- Misses       1560     1571      +11     
- Partials      768      796      +28     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/container_agent.py	97.89% <100.00%> (+0.01%)	⬆️
dokimi/dokimi/harnesses/docker_solver.py	97.73% <100.00%> (+<0.01%)	⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

The change is right: TASK_CONTAINER_HARDENING matches the repos/container.py flags (--cap-drop ALL, no-new-privileges, --pids-limit 32768), _start_container is the only place a task container is started, and no harness step needs a capability that is now dropped. One thing to fix before merge: the new test does not pin where the flags sit in the docker run argv.

What I ran, on Linux at 890089a:

• make ci-local with Docker: 33 passed, 0 failed, 6 skipped (offline-only or not applicable on a PR), including the docker-oracle smoke and changed-line coverage.
• Mutation runs of the new test against copies of docker_solver.py and container_agent.py: dropping any single cap, --cap-drop ALL, no-new-privileges or --pids-limit is caught; moving the flags is not (see inline).
• The harness's reap step (kill -9 -1 as the agent uid), chown/chmod as root, setpriv and su with exactly these flags on a small image: all work; pkill of another user's process as root fails, as the PR says.

Checked, no issues

• Flags: the set matches repos/container.py apart from the five added capabilities; CapEff inside the container is 0xcb, which is CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID.
• Growth baseline: docker_solver.py is 899 lines, as growth_baseline.json says.
• Docs: the #what-the-agent-can-reach anchor exists and the fragment follows the changelog.d format.

[vlad-perevezentsev]

The list of what now fails is not right. Ports below 1024 still work: Docker (20.10+) sets ip_unprivileged_port_start to 0 inside each container network namespace, so the host value does not matter (checked on Docker 29 with the host at 1024: root and nobody both bind port 80 with these flags). On the other side, more steps fail than listed: chroot (SYS_CHROOT), mknod (MKNOD), setcap in a package postinst (SETFCAP), setting setgid bits on files of another group (FSETID). Please drop the NET_BIND_SERVICE case and list the real ones, here, in coding-harnesses.md:296-299 and in the changelog fragment.

[vlad-perevezentsev]

This changes the environment every docker-* cell runs in, so a task that needs a dropped capability goes from pass to reward 0, and nothing in the run artifacts says which setup a run used. Please say in the fragment that docker-* results from before this change may not be comparable for such tasks. Were all catalog tasks' oracles re-run with these flags, or only the #229 smoke task?

[vlad-perevezentsev]

repos/container.py pairs --pids-limit with --init, but here PID 1 is sleep, which never reaps. Every process that outlives its docker exec (nohup ... &, a daemon a test starts) becomes a zombie when it exits and still counts against the new limit (checked: 5 orphans give 5 zombies in pids.current). Please add --init to TASK_CONTAINER_HARDENING too.

[vlad-perevezentsev]

The harness itself does not run apt or setpriv (setpriv is not used anywhere in the repo), and its own docker exec -u needs no capability inside the container. SETUID/SETGID are there for task scripts (apt dropping to _apt, su). Please fix the comment and the changelog line so the next person who tightens this knows what really needs them.

[vlad-perevezentsev]

The comment says these are the flags repos/container.py sets, but 32768 is a second copy of repos.container.PIDS_LIMIT. Consider one shared constant. The test checks >= 4096 while the docs promise 32768; please assert the documented value and that the flags come before the image name.

[xaleryb]

• Ran: the task container with the TASK_CONTAINER_HARDENING flags on an --internal network, on a host whose net.ipv4.ip_unprivileged_port_start is 1024; then the full suite, lint, types and mkdocs build --strict on the PR merged into main.
• Low-port bind: root binds port 80 and uid 10001 binds port 81 without error. The changelog and both docs say this fails because NET_BIND_SERVICE is dropped. It cannot fail here: Docker sets ip_unprivileged_port_start to 0 inside the container's network namespace.
• Not documented: mknod and chroot went from working to failing. chmod 2755 on a file whose group root is not in now exits 0 and silently drops the setgid bit (2755 -> 755).
• Pushed: 41355f8, which corrects the changelog fragment and the two docs to name the losses that really happen (KILL, NET_RAW, MKNOD, SYS_CHROOT, and FSETID, which fails silently) and to say a low-port bind still works.
• Left for the author: the PR body's Merge Danger list and the ip_unprivileged_port_start=0 sentence about the test host. The container's value is 0; the host's can be 1024.

[antonwolfy]

Review at 41355f8. The change since my last review is docs only (changelog fragment and two docs pages), and it fixes the low-port claim. The flags in the code are right. My earlier thread on the test is still open, and one unfixed point from @vlad-perevezentsev blocks.

Still open from my earlier review:

• dokimi/tests/test_docker_solver_harness.py:243: the test does not pin where the flags sit in the run argv, and the pids assert is still >= 4096. With the flags moved after "sleep", str(lifetime) the new test alone still passes (1 passed); "99999999999" and "4096" for the limit also pass. Only an unrelated assert, run[-2:] == ["sleep", "1800"] in test_docker_task_md.py, catches the move. A --privileged or seccomp=unconfined added to the tuple also passes. Keep the suggestion from the thread: index checks against the image name and == "32768".

I agree with @vlad-perevezentsev on the missing --init (PID 1 is sleep, which never reaps) and on the apt/setpriv comment at container_agent.py:87-89. New evidence: no harness code calls setpriv, runuser, apt or su (the root steps are mkdir, chown -R, chmod and rm -rf), and the same comment still lists NET_BIND_SERVICE as dropped.

@vlad-perevezentsev's question on whether the catalog tasks' oracles were re-run with these flags, and the note that earlier docker-* results may not be comparable, wait for the author. So do @xaleryb's points on the PR body's Merge Danger list and the ip_unprivileged_port_start=0 sentence.

What I ran, on Linux at 41355f8:

• make ci-local with Docker: 31 passed, 2 failed, 6 skipped; 8,980 tests passed, 2 skipped. Both FAILs are the gitleaks install and scan steps (Install gitleaks exit 22, Scan commits exit 127), where the proxy returned 504 on the download; a gitleaks run over the PR's two commits found no leaks. mypy, deptry, the schema compatibility check, the changelog check and the docs checks pass.
• 19 mutants of TASK_CONTAINER_HARDENING and _start_container against the six docker-related test files: dropping or changing any cap, --cap-drop ALL, no-new-privileges or --pids-limit is caught; flag order after the image name, a huge pids value, and extra flags are not.

Checked, no issues

• Call sites: _start_container at docker_solver.py:356 is the only docker run for the task container.
• Growth baseline: docker_solver.py is 899 lines, as growth_baseline.json says.
• Docs: mkdocs build --strict passes and the #what-the-agent-can-reach anchor exists.
• Low ports: the corrected docs and changelog no longer claim a port below 1024 fails.

[xaleryb]

Approved at 41355f8. Every earlier point re-run at this head passes, no must-fix or mid item is open, and CI is green.

[antonwolfy]

LGTM. The two commits since my last review take everything I asked for: --init is in TASK_CONTAINER_HARDENING, the limit comes from repos.container.PIDS_LIMIT, the comment no longer claims the harness runs apt or setpriv, and the test pins the flags' position and the exact limit. One nit on the test.

What I ran, on a Linux review host at c690563:

• make ci-local with Docker: 33 passed, 0 failed, 6 skipped; 8,980 tests passed, 2 skipped; changed-line coverage, mypy, deptry, the schema check, the changelog check, mkdocs build and the docker-oracle smoke pass.
• Mutants of TASK_CONTAINER_HARDENING and _start_container against the three docker test files: dropping or changing --init, --cap-drop ALL, no-new-privileges, any cap-add, the pids value, and moving the flags after the image or after sleep are all caught. An extra --privileged or --security-opt seccomp=unconfined is not (see inline).

@vlad-perevezentsev's question on whether the catalog tasks' oracles were re-run with these flags, and @xaleryb's points on the PR body's Merge Danger list and the ip_unprivileged_port_start=0 sentence, wait for the author.

Checked, no issues

• Flags: the tuple matches repos/container.py (--init, --cap-drop ALL, no-new-privileges, --pids-limit) plus the five cap-adds; _start_container places it before the image.
• Docs: the changelog, coding-harnesses.md and the contributing page name the same flags and capabilities as the code, and the changelog now says earlier docker-* results may not be comparable.
• Growth baseline: docker_solver.py is 899 lines, as growth_baseline.json says. The new repos import creates no cycle.