[napetrov]

Stacked on #218 (feat/task-network-policy, itself on #210); review/merge after both. (AGENTS.md says PRs are not stacked; stacked here on request.)

Summary

TaskSpec stays canonical; Harbor is only an export target, written lazily.

 load_task(dir)
   task_format: harbor | task-md
+               | terminal-bench-v1 (task.yaml)  -> tbench_v1.read
 docker-oracle / docker-claude
-  build ./environment, copy solution/ tests/, reward.txt
+  build task.build_context (-f task.dockerfile), -e container_env
+  oracle_step / verifier_step:
+    tb-v1: solution.sh -> /oracle, run-tests.sh + tests/ -> /tests, image user
+    verdict = pytest "short test summary info" (not exit code), as terminal-bench
+  refuse with reason: extra services/volumes/tmpfs/ports, solution.yaml,
+    run_tests_in_same_shell, non-pytest parser
 harbor* / benchflow*
+  harbor_view(task): content-keyed export in $TMPDIR/dokimi-harbor-export/<sha16>/
+    (follows harbor.mappers.terminal_bench; test.sh scores the same pytest summary)
 tasks inspect: tb-v1 consumers none-yet -> docker-*, harbor, benchflow
+memory: tb-v1 environment.memory_mb = compose deploy.resources.limits.memory
+        else 8192 ($DOKIMI_TBENCH_V1_MEMORY_MB), not the 2048 default; exported to task.toml
+network (#218): environment.network_mode public -> oracle + verify public, agent solve default

dokimi/harnesses/
├── tbench_v1.py      # task.yaml/compose reader, refusals, PytestParser port
├── harbor_export.py  # lazy Harbor layout + awk verdict
└── task_env.py       # env helpers moved out of docker_solver.py (growth cap)

Evidence

• Before: tasks run --harnesses docker-oracle on a terminal-bench original-tasks/ dir: no task.toml/task.md, not loadable; tasks inspect consumers none-yet.
  After: tests/test_tbench_v1.py (37 unit + 1 docker test gated by DOKIMI_DOCKER_TESTS=1): load/defaults, refusals, pytest-summary verdict incl. exit-code mismatch both ways, awk == Python verdict parity, export layout, cache keying, harbor harness runs the export. Full suite 7877 passed; make ci-local --offline 0 failed.
• Real docker-oracle over upstream terminal-bench (d28711d) sample: hello-world 1/1; first 5 of 25 sampled: 2 pass, 3 fail from task rot/limits (pinned apt version gone, OOM at the 2g default memory, QEMU guest test) – table in follow-up comment.
• Memory: at 8 GB the two 2 GB OOMs clear: predicate-pushdown-bench passes; add-benchmark-lm-eval-harness runs its solution to the end, then fails at test collection (transformers has no AutoModelForVision2Seq; unpinned dependency, upstream rot).
• Corpus survey (241 tasks): 215 runnable natively, 26 refused with reason (8 solution.yaml, 2 same-shell, rest multi-service/compose keys) – those go through the Harbor export.

Scoring: pytest summary vs exit code

docker-oracle with the real solution.sh and with a no-op one (true), 60 sampled tasks (the earlier 25 + 35 random, seed 212), 8 CPUs / 8 GB per container: 118 runs verified (qemu-startup: image build fails, apt). The summary verdict and exit == 0 agree on all 118. A static scan of all 241 run-tests.sh finds the scripts where the exit code is not pytest's:

task	run	summary	exit code	why
play-lord	no-op (ran)	0 (4 FAILED)	0 -> pass	pytest ... then if [ $? -eq 0 ]; then echo <summary>; fi: an if with a false test and no else exits 0
multistep-definite-integral	static (run_tests_in_same_shell, refused by docker-*)	follows the tests	always 0	pytest ... | tee, then echo "Pytest end."
stable-parallel-kmeans	both	agree	agree	saves $? and exit $exit_code

• terminal-bench v1 scores the summary only: harness.py _is_resolved (L536-542, all parsed tests PASSED; no summary means unresolved), and PytestParser maps SKIPPED/XFAIL to passed and XPASS/ERROR to failed; the exit code is never read (d28711d).
• Harbor's mapper scores the exit code: mappers/terminal_bench.py REWARD_LOGGING_SUFFIX (L37-44, _EXIT_CODE=$? -> reward 1/0), added to run-tests.sh by copy_test_script_with_reward_logging (L357-361, L480) (harbor 6099b8c). It does not check run_tests_in_same_shell or parser_name.
• Recommendation: keep the summary. It is what terminal-bench and its published scores use. The exit code matched on every sampled run, but on play-lord Harbor's rule passes a no-op (and on multistep-definite-integral it passes anything). Harbor parity would give those tasks a free pass.

Merge Danger

Door: two-way

New format path; harbor/task-md code paths only touched by refactors (build context, step objects) covered by existing tests.

Blast Radius: docker-harnesses

Regressions would show as changed docker build/cp argv for existing Harbor/task-md tasks.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.96482% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.24%. Comparing base (f5cca76) to head (a43b174).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/harnesses/harbor_export.py	91.22%	4 Missing and 6 partials ⚠️
dokimi/dokimi/harnesses/tbench_v1.py	93.42%	8 Missing and 2 partials ⚠️
dokimi/dokimi/harnesses/tasks.py	90.62%	4 Missing and 2 partials ⚠️
dokimi/dokimi/task_formats/dirs.py	80.00%	1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #212      +/-   ##
==========================================
+ Coverage   92.19%   92.24%   +0.04%     
==========================================
  Files         235      239       +4     
  Lines       29387    29934     +547     
  Branches     4836     4917      +81     
==========================================
+ Hits        27094    27613     +519     
- Misses       1615     1633      +18     
- Partials      678      688      +10     

Files with missing lines	Coverage Δ
dokimi/dokimi/harnesses/benchflow.py	95.43% <100.00%> (+0.02%)	⬆️
dokimi/dokimi/harnesses/command.py	97.29% <100.00%> (+0.02%)	⬆️
dokimi/dokimi/harnesses/docker_solver.py	97.73% <100.00%> (+0.01%)	⬆️
dokimi/dokimi/harnesses/task_env.py	100.00% <100.00%> (ø)
dokimi/dokimi/task_formats/dirs.py	91.15% <80.00%> (-1.09%)	⬇️
dokimi/dokimi/harnesses/tasks.py	96.03% <90.62%> (-0.65%)	⬇️
dokimi/dokimi/harnesses/harbor_export.py	91.22% <91.22%> (ø)
dokimi/dokimi/harnesses/tbench_v1.py	93.42% <93.42%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

The CLI surface now matches the base branch: no user-facing CLI change.

[napetrov]

docker-oracle on 25 sampled upstream tasks: 20/25 pass. Failures: 2 OOM under the 2g default memory (lm-eval-harness, predicate-pushdown-bench), 2 upstream rot (build-pmars pinned apt version gone, cron-broken-network example.com content), 1 task-side (build-tcc-qemu). docker-claude pilot with haiku: 3/3 pass, $0.006.

[xaleryb]

• Ran: the full suite, lint and types, the tb-v1 docker-oracle test on real docker, and a Harbor export of each case.
• A passing tb-v1 task whose run-tests.sh writes ERROR: pip's dependency resolver ... to stderr -> docker-oracle reward 0, with the pip line counted as a failed test -> reward 1, which is what the export's test.sh gives.
• DOKIMI_TBENCH_V1_MEMORY_MB=16384 after a first export -> the cached task.toml still says 8192 -> 16384.
• A directory with a task.toml at $TMPDIR/dokimi-harbor-export/<key>/<task>/ -> it is run and scored as the export -> a per-user cache.
• A solution.yaml task -> tasks inspect lists harbor and benchflow, every such cell errors, and a .staging-* dir is left behind -> no harbor/benchflow consumer and no leftover dir.
• Compose build.dockerfile: Dockerfile.x -> the export has no environment/Dockerfile -> environment/Dockerfile.

Pushed:

• 4f61d6a runs run-tests.sh with 2>&1.
• 586d805 adds the resolved task.toml to the cache key.
• 39b2d04 moves the cache to ~/.cache/dokimi/harbor-export, mode 0700.
• 4c92aa5 stops listing harbor/benchflow for a solution.yaml task and cleans up staging in a finally.
• cd1d348 writes the export to environment/Dockerfile and adds tests that pin the context, COPY and workdir branches.

Left for the author: whether agent runs of solution.yaml tasks should export without solution/.

[antonwolfy]

Review at cd1d348. The native tb-v1 path is sound: the pytest-summary verdict, the refusals, the memory default and the per-user cache all behave as documented. The problems are at the edges, mostly in the Harbor export. The PR shape and one test gap are below, the rest are inline.

What I ran, on a Linux review host at cd1d348:

• make ci-local (with Docker): 32 passed, 1 failed, 6 skipped. The failure is pip-audit on the lock files, which this PR does not touch. Changed-line coverage gate passes.
• Reproductions for the code findings, with no model or network involved.
• A mutation run on the new tests: 138 mutants, 47 of which pass the full tb-v1, task-md and network-policy tests. The gaps are in the inline comments below.

Findings that cannot be anchored inline:

1. The PR is stacked on #218 and #210. AGENTS.md (Commits and pull requests) says: "When a change depends on an open PR, wait for it to merge or put both in one PR; PRs are not stacked on each other's branches." The diff against main is 96 files and 13 commits, and carries task_formats/*, network_policy.py, task_inspect.v1.json, the corpus fixtures and three changelog fragments. This PR cannot be reviewed or merged on its own. Rebase onto main after #218 lands, or fold the three into one PR.

Also untested (outside the inline anchors): the refusal of a host-side agent on a tb-v1 task at docker_solver.py:760. Narrowing that condition to (FORMAT_TASK_MD,) passes the full suite. Add _refuse_unrunnable(tb1_task, "claude", "host") expecting ValueError.

@xaleryb's question on whether agent runs of solution.yaml tasks should export without solution/ waits for the author.

Checked, no issues

• Schema versioning: task_inspect.v1 is a new kind registered in SCHEMA_VERSIONS with a fixture; no existing schema changed. No with_docs/without_docs keys involved.
• Docs against code: the 8192 default, compose memory limit, ~/.cache/dokimi/harbor-export mode 0700, tasks inspect consumers and --network parsing match the code.
• Refactor: the Harbor and task-md docker build/cp/exec argv is unchanged apart from the existence check.
• Export concurrency: the atomic replace handles a lost race, and staging is removed in finally.

[antonwolfy]

I pushed a commit per nit, up to 33cff32, which fix the nit from my review:

• 33cff32: dokimi/docs/coding-harnesses.md:129: the verdict rule now reads "every listed test passed, skipped or xfailed", matching _PASSING in tbench_v1.py.

make ci-local at 33cff32: 32 passed, 1 failed (pip-audit, as at the reviewed head), 6 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the bug and the seven risks from my review.

[xaleryb]

Merged main after #210 landed (5df4fce): conflicts merged three-way against e4b4e80, the commit this branch was built on. The PR's own diff against main is line for line the same as before (22 files); the two consumer-table rows for harbor and task-md take main's "(with environment/Dockerfile)". Full suite, ruff, mypy and bandit pass on Linux at 8c08dc3, and each fix above was checked by mutating it back (15 of 15 fail).

This branch still carries #218's first two commits (ec91e10, bab9fa9), so #218 should land first.

Still for the author: whether an agent run of a solution.yaml task exports without solution/. Not raised before: _append_dockerfile writes ENV k=<json.dumps(v)>, and Docker does not read \uXXXX escapes, so a non-ASCII compose env value reaches the exported Dockerfile as a literal escape.

[xaleryb]

Approved at 8c08dc3: the review points are fixed here or are edge cases left to the author (named in the thread), the full suite passes on Linux, and CI is green.

[xaleryb]

Merged main after #218 landed (a43b174): this branch carried #218's first two commits, so each conflict was merged three-way against bab9fa9. #218's files are now identical to main and drop out of this diff. Full suite, ruff, mypy and bandit pass on Linux.

[xaleryb]

Approved at a43b174: re-merged with main after #218 landed (#218's files now identical to main), the review points are fixed or left to the author as named in the thread, the full suite passes on Linux, and CI is green.