feat(harnesses): per-phase network policy for docker-* runs, off by default by napetrov · Pull Request #218 · intel/dokimi

napetrov · 2026-10-24T18:39:54Z

Stacked on #210 (feat/task-format-detect); review/merge after it. #213 (SWE-bench) is rebased on this branch.

Summary

Every docker-* cell gets a network per phase, recorded with where it came from:

phase    default   modes: none | allowlist:host[:port],... (egress sidecar) | public (bridge + host proxy)
build    public    docker build --network host|none
solve    none      oracle / agent (an in-container agent still reaches its LLM via the sidecar)
verify   none      tests/test.sh

sources, later wins
  DEFAULTS
  task.toml  [environment] network_mode/allowed_hosts | allow_internet  -> verify, oracle solve
             [agent] / [verifier] network_mode                         -> that phase
             environment.egress                                         -> agent solve
  CLI        dokimi tasks run --network PHASE=MODE[:hosts]  (repeatable, all=every phase)

 DockerSolverHarness.run
+  plan = network_policy.resolve(task, self.network, agent=...)
   _build_image(plan.build)
   _start_container(...)            # --network none | bridge | internal+sidecar
+    enter_phase(plan.solve)        # leave/join bridge before solve.sh runs
   _solve_oracle(plan.solve)
+  _verify_phase(plan.verify)       # stop solve sidecar, join/leave bridge, own -verify sidecar for an allowlist
+  metrics.network = {build, solve, verify: {mode, hosts, source}, verify_egress, skill}
+  <cell>/network.json              # errored cells too

Skills that need the network declare it in SKILL.md as metadata.network: "allowlist:pypi.org,..." (reason in compatibility, the Agent Skills spec field). It is recorded as metrics.network.skill.missing, not granted. Widening only the skill arm would put its baseline on another network.

Evidence

Unit tests: tests/test_network_policy.py (22) cover resolution order, Harbor fields, bad --network specs, and docker argv order (disconnect < solve.sh < connect < test.sh). They also check that proxy env reaches only the public step, that the verifier gets its own sidecar, and that an errored cell keeps network.json. Full suite: 7954 passed. make ci-local: all green except pip-audit (filelock / setuptools CVEs in the lock, also on main).

Real runs with docker-oracle on this host (full tables in scratch/netpolicy/RESULTS.md):

probe from inside the container	default (none)	`verify=public`	`verify=allowlist:pypi.org`
interfaces	lo	eth0, lo	eth0 (internal), lo
DNS example.com / docker gateway 172.17.0.1:22	fail / fail	ok / open	fail / fail
https pypi.org / example.com via proxy env	fail / fail	200 / 200	200 / 403

corpus	none -> open, verdict changes
Harbor examples (12)	4 flip 0 -> 1 (hello-world, hello-alpine, hello-skills, reward-kit-example): test.sh installs uv/pytest at verify time
terminal-bench v1 (12, with #212)	12 flip: run-tests.sh does apt + uv + pip install pytest
SWE-bench pilot (7, with #213)	0 flip: gold passes offline. requests-863/2317 fail in both (live httpbin)

Allowlist on real tasks: reward-kit-example and hello-alpine pass with verify=allowlist:<7-8 HTTPS hosts> and fail with pypi.org alone. Each time, verify_egress.denied named the missing hosts.

Merge Danger

Door: two-way

Behaviour change: a task with [environment] network_mode = "public" now also gives the oracle the network, and a verifier that nothing grants a network runs fully offline (--network none, where before it was an internal network). Undeclared Harbor tasks that install their test deps at verify time score 0 until run with --network verify=public. That was already true before; Harbor's own default is public.

Blast Radius: docker-*

Only docker-* harnesses; --network with any other harness is refused. Plain HTTP (apt over :80) through an allowlist still fails behind an upstream proxy (an existing egress_proxy rule).

🤖 Generated with Claude Code

@task

A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…efault build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

github-actions · 2026-10-24T18:49:31Z

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi tasks run` parameters changed:
    + added option `--network`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

codecov-commenter · 2026-10-24T18:52:55Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.10744% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.23%. Comparing base (f5cca76) to head (fe10dd1).

Files with missing lines	Patch %	Lines
dokimi/dokimi/harnesses/network_policy.py	95.73%	4 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #218      +/-   ##
==========================================
+ Coverage   92.19%   92.23%   +0.04%     
==========================================
  Files         235      236       +1     
  Lines       29387    29582     +195     
  Branches     4836     4865      +29     
==========================================
+ Hits        27094    27286     +192     
- Misses       1615     1618       +3     
  Partials      678      678

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/tasks.py	`92.81% <100.00%> (+0.04%)`	⬆️
dokimi/dokimi/harnesses/container_agent.py	`97.81% <100.00%> (+1.92%)`	⬆️
dokimi/dokimi/harnesses/docker_solver.py	`97.75% <100.00%> (+0.04%)`	⬆️
dokimi/dokimi/harnesses/runner.py	`99.17% <100.00%> (+0.04%)`	⬆️
dokimi/dokimi/harnesses/network_policy.py	`95.73% <95.73%> (ø)`

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… solve default Rebased onto intel#218. The corpus manifest's terminal-bench hello-world now names its consumers (docker-*, harbor, benchflow), not none-yet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…s allowlisted A verify allowlist opens the sandbox for its sidecar, and _solve_claude and _solve_probe took any sandbox to mean an in-container agent: docker-claude-host and docker-probe-host then ran in the task container. Dispatch on agent_location, and run the agent preflight only for an in-container agent. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

…f erroring the arm metadata.network is only recorded, never granted, and the Agent Skills spec leaves metadata a free string map: a value such as "pypi.org" errored every skill-arm cell while its baseline ran. Record it with the parse error and missing: null. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

…rify_egress network.json was written before the build, and an allowlisted verifier records its sidecar decisions in metrics.network afterwards, so the cell file never had them. Write it in the outer finally, errored cells included, and say in the docs that egress.jsonl covers the solve sidecar only. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

… solve Replacing the branch in resolve() with a no-op left the whole suite green, so agents on Harbor allowlist tasks could silently lose the declared hosts. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

xaleryb · 2026-10-25T06:59:46Z

Ran: the full test suite, lint, types and security scan on the head plus the commits below. Also ran real docker runs of docker-probe-host and docker-oracle with an allowlisted verifier, and a CLI dry-run of a skill arm.
Pushed on top of bab9fa9:
- 099813a: a [verifier] allowlist on a -host arm opened the sandbox, so docker-claude-host and docker-probe-host ran their agent inside the task container. The probe verdict flipped from 0 to 1, and claude-host failed the container credential preflight. Dispatch now goes by agent_location.
- c76e48f: a skill whose metadata.network is "pypi.org" errored every skill-arm cell while its baseline ran. The value is now recorded with error and missing: null.
- 9e340b1: network.json was written before the verifier ran, so it never had verify_egress. It is now written in the outer finally, and the docs say egress.jsonl covers the solve sidecar only.
- 0661bce: pins that an [environment] allowlist reaches the agent's solve. With that branch disabled, the whole suite still passed.
For the author: with an allowlisted verifier, a -host arm still starts the solve sidecar, so the cell gets an empty egress.jsonl even though the agent ran on the host. Starting the sidecar only for the verify phase would avoid this.

antonwolfy

Review at 0661bce. The network policy itself is well put together: precedence, the connect/disconnect order around the oracle and verifier, and network.json on errored cells are pinned by tests. The problems are at the edges, and in what the PR carries along. Two bugs in tasks inspect and a sidecar leak are worth fixing before merge.

What I ran, on a Linux review host at 0661bce:

make ci-local: 30 passed, 3 failed, 6 skipped. The test suite and changed-line coverage pass. The two gitleaks jobs fail on installing the tool, and a direct gitleaks detect over the PR's commits finds no leaks. pip-audit fails on the lock files, which this PR does not touch.
Reproductions for the code findings, with docker faked (no real containers) and no network.
69 mutants of network_policy.py, docker_solver.py, container_agent.py and runner.py run against test_network_policy.py and the neighbouring docker tests: 15 survive. Survivors include the oracle's solve=allowlist opening no sidecar, the oracle exec losing its proxy env under --network solve=public, the verify sidecar never being stopped, and solve_enforced hard-coded to True.

Findings that cannot be anchored inline:

The PR is stacked on #210. The description says so, and the diff carries all of task_formats/, tasks inspect, task_inspect.v1 and about 3,500 lines of fixtures next to the network policy. AGENTS.md says PRs are not stacked on each other's branches. The copy is also older than #210's head: it lacks @xaleryb's 6cdba63 (a FIFO in a tree scan hangs tasks inspect here) and bc96c3c (unreadable files reported as unknown). Rebase onto the base once #210 lands, so this PR shows only the network policy.
A task value that resolve() rejects leaves the cell with no network.json and no metrics.network. docker_solver.py:141 calls self._plan(task) before output_dir exists and before the finally that writes network.json. With [environment] network_mode = "host" the result is returncode 1, metrics = {error: ValueError..., operation_count: 1}, and no files under the output root. The commit message says errored cells keep network.json; this path does not. Move the plan resolution inside the try, or write a minimal network.json with the error.
docs/contributing-terminal-bench-task.md:118 and :120 (outside this diff) now contradict the code. Line 118 says anything but "public" "means --network none" and "the image build always gets host network". Line 120 says allow_internet is Harbor-only. docker-* now reads allow_internet, [agent], [verifier], allowlist, and --network build=none. A task author following that page gets the wrong network. Update both rows.

@xaleryb's point that a -host arm with an allowlisted verifier still starts the solve sidecar, and so writes an empty egress.jsonl, waits for the author.

On the tests: add one for an oracle with solve=allowlist and one for the verify sidecar's stop and cleanup. Both mutants pass the full docker test set.

Checked, no issues

Phase order: stop sidecar < disconnect < solve.sh < connect < start verify sidecar < test.sh < rm container < network rm holds in the faked docker call log.
Credentials: with HTTPS_PROXY=http://uu:SECRETPW@... set, an oracle run with solve=public verify=public has no SECRETPW in any output file. run.log shows ***@.
Skill metadata: a malformed or list-valued metadata.network is recorded as error and does not error the arm.
Schemas: task_inspect.v1 is a new kind registered in SCHEMA_VERSIONS, with a fixture. metrics.network fits the free-form metrics object. Docs build and the nav check pass.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy

I pushed a commit per nit, up to f85231e, which fix the nit from my review:

f85231e: dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md:14: the Changed entry now says an unknown network_mode, or allowed_hosts without allowlist, errors the run instead of running offline.

make ci-local at f85231e: 32 passed, 1 failed, 6 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the two bugs and the four risks from my review.

Main now has intel#210 as squash f5cca76, which this branch carried as its early commits f4ce883..e4b4e80; each add/add conflict was three-way merged against e4b4e80, so the branch adds exactly its own network-policy changes on top of main. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

verifier_sidecar() returned no handle when _start_proxy raised after docker start, so neither _verify_phase nor sandbox.close() removed <proxy>-verify: the started container kept the upstream proxy env on the run's network and the network rm failed silently. Remove it before re-raising, as ContainerSandbox.open() does for the solve sidecar. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

… refused run() resolved the plan before the cell dir existed and outside the try whose finally writes network.json, so a task with an unknown network_mode errored with no network record at all. A refused plan now leaves network.json with the reason (not on a dry run) and still raises. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

The contributor guide still said anything but "public" means --network none, that the build always gets host network, and that allow_internet is Harbor-only; docker-* now reads no-network/allowlist/public, [agent]/[verifier] overrides, allow_internet and --network build=none. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

…sidecar teardown and solve_enforced Four mutants of the network policy passed the whole docker test set: an oracle under solve=allowlist opening no sidecar, the oracle exec losing its proxy env under solve=public, the verify sidecar never being removed, and solve_enforced hard-coded to True. Each now fails a test. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>

xaleryb · 2026-10-26T02:32:42Z

Merged main after #210 landed (c992d73) instead of rebasing, to keep your branch's history: the add/add conflicts were merged three-way against e4b4e80, and the diff against main is now the network policy only (10 files), identical to the branch's own changes. CI had skipped this branch while it conflicted, so this push is its first run on main.

On top:

243ec2f: a verifier sidecar that fails to start is removed.
c96ce8b: a refused network plan leaves its reason in network.json (moved into network_policy.resolve_recorded so docker_solver.py keeps its line and complexity baselines). The errored cell's result still has no metrics.network.
ae25e3e: the environment.network_mode / allow_internet rows in contributing-terminal-bench-task.md now say what the code does, each checked against network_policy.py.
fe10dd1: tests for the oracle under solve=allowlist and solve=public, the verify sidecar's teardown and solve_enforced on -host arms. Six mutants that survived at the merge commit fail now.

Full suite, ruff, mypy and bandit pass on Linux at fe10dd1. #212 and #213 carry this branch's first two commits, so this one should land before them.

Still for the author: stop_proxy raising in the finally; a -host arm with an allowlisted verifier still starting the solve sidecar and writing an empty egress.jsonl.

xaleryb

Approved at fe10dd1: the review points are fixed here or are edge cases left to the author (named in the thread), the full suite passes on Linux, and CI is green.

Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#218 landed as a squash; this branch carried its first two commits (ec91e10, bab9fa9), so each conflict was merged three-way against bab9fa9.

@task

…azily (#212) * feat(tasks): detect eval-task formats with `tasks inspect` A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): task_inspect.v1 report, config-named datasets, LFS pointers `tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(tasks): real-format corpus for tasks inspect, with its manifest tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(harnesses): per-phase network policy for docker-* runs, off by default build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(harnesses): write network.json in every docker-* cell dir, errored cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): run terminal-bench v1 tasks natively, export to Harbor lazily A task.yaml task loads as a TaskSpec and runs under docker-oracle and docker-claude as terminal-bench's harness runs it (solution.sh at /oracle, run-tests.sh + tests/ at /tests, image user, TEST_DIR and compose env, pytest-summary verdict). Tasks one docker run cannot reproduce are refused with the reason. Harbor-backed harnesses and benchflow run a content-keyed Harbor export whose test.sh applies the same pytest-summary rule. Stacked on #210. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): give terminal-bench v1 tasks 8 GB of memory, per task from compose terminal-bench sets no memory limit; dokimi's 2 GB default OOM-killed add-benchmark-lm-eval-harness and predicate-pushdown-bench. A v1 task now reads compose deploy.resources.limits.memory when set, else 8192 MB (DOKIMI_TBENCH_V1_MEMORY_MB overrides), and the Harbor export writes it to task.toml. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(tasks): tb-v1 on the network policy: public oracle/verify, agent solve default Rebased onto #218. The corpus manifest's terminal-bench hello-world now names its consumers (docker-*, harbor, benchflow), not none-yet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(tasks): score a tb-v1 verifier on its one ordered output stream docker-* parsed stdout followed by all of stderr, so a stderr line such as pip's "ERROR: pip's dependency resolver ..." landed after the pytest summary and failed a passing task. Run run-tests.sh with 2>&1, as terminal-bench's pane and the Harbor export's test.sh read it. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): key the Harbor export on the task.toml it writes The cache key hashed only the task's files, so a cached export kept the memory_mb of its first run after DOKIMI_TBENCH_V1_MEMORY_MB changed, while docker-* used the new value. Hash the resolved task.toml too. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): keep the Harbor export cache per user, not in shared /tmp The cache sat at a predictable $TMPDIR path, created group-writable, and any dir there with a task.toml was run and scored as the export. Default to ~/.cache/dokimi/harbor-export, created 0700. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): no harbor/benchflow consumer for a solution.yaml tb-v1 task export() refuses solution.yaml on every run, yet tasks inspect listed harbor and benchflow, and each refusal left a .staging-* dir in the cache. Report the refusal as a problem, clean staging in a finally, and say tasks list/run read three formats now. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): export a named tb-v1 Dockerfile as environment/Dockerfile Every Harbor environment builds environment/Dockerfile, so a task whose compose names build.dockerfile: Dockerfile.x exported with nothing to build. Rename it, refuse a clash, and pin the subdir-context, kept COPY tests/solution.sh and working_dir branches with tests. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs: count xfailed tests as passing in the tb-v1 verdict rule Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(tasks): write non-BMP task metadata into the exported task.toml as TOML reads it json.dumps escaped a character above U+FFFF as a surrogate pair, which tomllib rejects, so every Harbor and BenchFlow run of such a task failed on parse. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): key the tb-v1 pytest results by node id, as the exported test.sh reads them Keyed by the name after the file, a later PASSED b.py::test_x hid an earlier FAILED a.py::test_x: docker-* scored 1.0 where the Harbor export's test.sh scored 0. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): run the exported tb-v1 test.sh against a fake run-tests.sh Only its awk was run before: dropping the exit code, the tee, the 2>&1, TEST_DIR or the reward file name passed the suite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): skip a task.yaml without an instruction when discovering tasks A non-task directory with a task.yaml and a Dockerfile made tasks list and --all fail for the whole root; it is skipped with a warning now, as a task.md without frontmatter is, and load_task on it still raises. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): refuse a tb-v1 service command, build arg or target, or host environment on docker-* One docker run of the default stage, idling, with literal values did none of these, so such a task ran and its failure read as the task's, not a refusal. The idle sleep infinity and TEST_DIR stay accepted; the top-level-key and extra-volume refusals are now pinned too. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin that the benchflow harness runs a tb-v1 task from its Harbor export Dropping harbor_view from BenchflowHarness.run passed the suite; bench would then get a task.yaml directory with no task.toml. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin the host-side agent refusal for a tb-v1 task Narrowing the refusal to task.md tasks passed the suite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com>

…erdict and is retried (#234) * fix(harnesses): a verifier sidecar removal that times out keeps the verdict and is retried In the #218 review the verifier sidecar's teardown had a second gap, left to the author and not changed before the merge: `_verify_phase` stops the `-verify` sidecar in a `finally` with `stop_proxy`, which raises `TimeoutExpired` when `docker rm` times out. That replaced the reward the verifier had just computed with a TIMEOUT cell (exit 124, reward 0), and `ContainerSandbox.close()` could not retry, because it did not know the verifier sidecar, which then stayed on the network that `close()` removes. `verifier_sidecar()` now records the sidecar on its sandbox. The verify phase stops it with the new `stop_proxy_quietly`, which logs a timed-out removal instead of raising, and `close()` uses it for both sidecars, so a removal that timed out is retried before `network rm`. The solve sidecar's stop before the verifier still raises: the verifier must not start while it may still run. `docker_solver.py` keeps its 900 lines. Each of 5 mutants of the fix fails the new test. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(harnesses): teardown also covers a verifier sidecar that failed to start, or a crashed run left From vlad-perevezentsev's review of #234: - `verifier_sidecar()` recorded the sidecar on its sandbox only after a successful start. If the start failed and the removal in its `except` timed out, `close()` did not know the sidecar, which stayed on the run's network. It is recorded before the start now; a removal there that works marks it stopped, so `close()` does not touch it again. - `open()` cleared a crashed run's container and solve sidecar, but not its `-verify` sidecar, so a leftover one still held the network and `network create` failed the next cell with the same id. It is cleared with the others. The test that counts the solve sidecar's removals on a timeout now leaves the verifier sidecar's leftover removal out of its count. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> --------- Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

napetrov added 5 commits October 23, 2026 22:18

feat(harnesses): write network.json in every docker-* cell dir, error…

bab9fa9

…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov mentioned this pull request Oct 24, 2026

feat(tasks): run SWE-bench rows with docker-* harnesses #213

Merged

github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Oct 24, 2026

napetrov mentioned this pull request Oct 24, 2026

feat(tasks): run terminal-bench v1 tasks natively, export to Harbor lazily #212

Merged

napetrov marked this pull request as ready for review October 25, 2026 02:30

napetrov requested review from antonwolfy, jharlow-intel, vchamarthi and xaleryb as code owners October 25, 2026 02:30

xaleryb added 4 commits October 24, 2026 23:43

napetrov mentioned this pull request Oct 25, 2026

CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94

Open

antonwolfy reviewed Oct 25, 2026

View reviewed changes

Comment thread dokimi/dokimi/harnesses/docker_solver.py

Comment thread dokimi/dokimi/task_formats/configs.py

Comment thread dokimi/dokimi/task_formats/datasets.py

Comment thread dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md Outdated

Fix the changelog entry for the network_mode validation change

f85231e

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

antonwolfy reviewed Oct 25, 2026

View reviewed changes

xaleryb added 5 commits October 25, 2026 19:20

xaleryb approved these changes Oct 26, 2026

View reviewed changes

xaleryb merged commit 85b8d84 into intel:main Oct 26, 2026
20 checks passed

xaleryb mentioned this pull request Oct 26, 2026

fix(tasks): one malformed file no longer stops tasks inspect #227

Merged

antonwolfy mentioned this pull request Oct 26, 2026

fix(harnesses): a verifier sidecar removal that times out keeps the verdict and is retried #234

Merged

5 tasks done

Navigation Menu

Repository navigation

feat(harnesses): per-phase network policy for docker-* runs, off by default - #218

feat(harnesses): per-phase network policy for docker-* runs, off by default#218
xaleryb merged 15 commits into
intel:mainfrom
napetrov:feat/task-network-policy

napetrov commented Oct 24, 2026

Uh oh!

github-actions Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 24, 2026 •

edited

Loading

Uh oh!

xaleryb commented Oct 25, 2026

Uh oh!

antonwolfy left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antonwolfy left a comment

Uh oh!

xaleryb commented Oct 26, 2026

Uh oh!

xaleryb left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

napetrov commented Oct 24, 2026

Summary

Evidence

Merge Danger

Uh oh!

github-actions Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI interface change detected

Uh oh!

codecov-commenter commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xaleryb commented Oct 25, 2026

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

xaleryb commented Oct 26, 2026

Uh oh!

xaleryb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 24, 2026 •

edited

Loading

codecov-commenter commented Oct 24, 2026 •

edited

Loading