Repository navigation
feat(tasks): detect eval-task formats with tasks inspect - #210
Merged
Merged
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi tasks inspect` parameters changed:
+ added option `--out`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #210 +/- ##
==========================================
+ Coverage 92.07% 92.20% +0.13%
==========================================
Files 233 235 +2
Lines 29090 29387 +297
Branches 4787 4836 +49
==========================================
+ Hits 26784 27096 +312
+ Misses 1620 1614 -6
+ Partials 686 677 -9
🚀 New features to boost your workflow:
|
napetrov
marked this pull request as ready for review
October 24, 2026 15:57
napetrov
requested review from
antonwolfy,
jharlow-intel,
vchamarthi and
xaleryb
as code owners
October 24, 2026 15:57
This was referenced Oct 24, 2026
`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 24, 2026
…azily A task.yaml task loads as a TaskSpec and runs under docker-oracle and docker-claude as terminal-bench's harness runs it (solution.sh at /oracle, run-tests.sh + tests/ at /tests, image user, TEST_DIR and compose env, pytest-summary verdict). Tasks one docker run cannot reproduce are refused with the reason. Harbor-backed harnesses and benchflow run a content-keyed Harbor export whose test.sh applies the same pytest-summary rule. Stacked on intel#210. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A config naming a path with a NUL byte raised ValueError in links._key, and a skill-reach key YAML loads as bool or int (on:, yes:, 1:) raised TypeError in the key lint; either one killed the whole `tasks inspect`. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Probes copy YAML values into details unchanged. --json hid it with default=str, but --out (save_artifact) raised TypeError and wrote nothing, and a YAML .nan went out as bare NaN in both modes. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…iles The 64 MiB guard trusted st_size, which is 0 for a FIFO or device: a FIFO left in a scanned tree hung the scan, and /dev/zero was read until memory ran out. The scan now takes regular files only and the read is bounded. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
docker-* always builds environment/Dockerfile, so a harbor or task-md task without it errors in every cell, yet inspect listed docker-* as a consumer whenever tests/ or verifier/ existed. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Removing the symlink skip, the dot-directory skip, RecursionError from the parse catch or the file-cap flag left every task-format test green; two of those turn into crashes on real trees. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Ran the suite, lint, type and security checks on Linux,
Pushed: |
Contributor
|
CodeFactor is red here and on the PRs built on this branch (#212, #213, #214) because of 3 findings, all in the corpus added by e4b4e80: |
…e inspect does Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Pushed bc96c3c for the second point above: a tree scan now reports a file it could not read or parse as One consequence to be aware of: any unclaimed file with a read problem now shows up, including an unrelated YAML file that uses aliases (refused as before, now visible in the tree output). Still open from the first comment: single-file ambiguity in |
… files Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Pushed 492020e: Re-scoping our first point: single-file |
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…this PR Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Correction to my last comment: CodeFactor did not apply |
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#211 landed with a copy of this branch's first commit (f4ce883) plus its own reach consumers. Each conflicting file was merged three-way against f4ce883, so main's additions (BY_TRIGGER_SET and its consumers) and this branch's later commits are both kept.
…ntel#211 Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
Merged main after #211 landed (ddf1858, d277faa). #211 carried a copy of this branch's first commit (f4ce883) plus its own That made trigger sets consumed by |
…iles solution.sh: shebang above the canary line; Dockerfile: pip3 --no-cache-dir; verifier/test.sh: test pytest's exit code directly. Detection reads only which files exist, and MANIFEST.yaml notes each change, as its header asks. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Contributor
|
CodeFactor is green since c804980: the three findings in the copied corpus files are fixed, and Order from here:
|
antonwolfy
reviewed
Oct 25, 2026
antonwolfy
left a comment
antonwolfy
left a comment
Contributor
There was a problem hiding this comment.
Review at c804980. The detection model and the safety guards hold up: YAML aliases are refused, .py files are never imported, and FIFOs, symlinked children and size caps are handled in tree scans. Four things are worth fixing before merge; they are inline.
What I ran, on a Linux review host at c804980:
make ci-local: 26 passed, 3 failed. 8089 tests passed, and the changed-line coverage gate passed.gitleaksfailed at the download step, andpip-auditfails on the setuptools and filelock advisories in the lock files, which this PR does not touch. Neither is caused by the PR.- Reproductions for the code findings, with no network access.
- A mutation run of 59 mutants over the new code against
test_task_format_detect.pyandtest_task_format_corpus.py: 21 survived. The ones that matter are in the last inline comment.
Checked, no issues
- Docs and schema: the format, kind and consumer table in
coding-harnesses.mdmatches the probes, and thetask_inspect.v1enums match the constants.task_inspectis a new entry inSCHEMA_VERSIONS, so no existing version is edited in place. - Scope claim: no file under
harnesses/changes, and a test pinsdiscover_tasksas unchanged. - Repo rules: the changelog fragment is present, no generated artifacts are committed (
large.parquetis a synthetic LFS pointer), and the corpus verifiers are kept out of collection. - Tree-scan reports: a file that cannot be read inside a tree is reported as
unknownwith the read error, not dropped.
_FIELD_SPEC's [^)]* let each unclosed FieldSpec( run to the end of the segment, and search retried at every following one: 40000 of them after an @task file's dataset call took 43 s. The arguments are now bounded at 1000 characters and a run stops at the next FieldSpec(, so each start scans at most up to the next one. The bound alone (the review's [^)]{0,500}) is linear too but still took 17 s on a file at the 64 MiB read cap; with the stop it takes 1.2 s. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
An Excel or Sheets export of an Inspect samples.csv or a promptfoo tests.csv starts with a BOM, which csv.DictReader kept in the first header (\ufeffinput), so the file was unknown and a BOM on __expected hid the promptfoo marker. Both consumers drop it: Inspect's csv_dataset reads utf-8-sig and promptfoo parses CSV with bom: true. The BOM is stripped for CSV only, not by opening every file as utf-8-sig. A JSON or JSONL file with a BOM stays a 'not valid JSON: Unexpected UTF-8 BOM' problem, because that is what its readers do: read_json_input (catalog trigger-set --trigger-set), load_questions and Inspect's json_dataset all decode plain utf-8, so stripping it here would advertise a consumer that refuses the file. YAML needs nothing: the loader skips a leading BOM. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
TaskMdProbe.fingerprint reads task.md (_is_benchflow_task_md, _has_frontmatter) and walk called detect outside any try, so one mode-000 task.md ended inspect_path with a PermissionError and no report for any other task. walk now catches OSError around detect, the one place that covers every directory probe, and reports the directory as unknown with 'cannot read: ...', as an unreadable file in a tree is. This also covers a directory whose entries cannot be stat'ed (a mode-000 directory holding a task.toml), which failed the same way before this PR in task_format. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
_docker_runnable needs environment/Dockerfile and a tests/ or verifier/ directory, but only the Dockerfile half was tested: returning just the Dockerfile check passed every test. A Harbor and a task-md task with both verifier directories removed must not be claimed for docker-*. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
- the samples of a model-graded OpenAI evals spec are read without the ideal lint (graded = False passed every test); - promptfoo tests: file://tests.csv labels a CSV that has no __expected column (the only fixture CSV is claimed with or without the reference); - a scan does not enter node_modules/ or venv/ (only .git and .venv were tested, and those are skipped as dot directories anyway). Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
xaleryb
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 26, 2026
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#210 landed as a squash (f5cca76), so its files conflicted add/add here; they were merged three-way against e4b4e80, the intel#210 commit this branch was built on, keeping both sides' consumers in the docs table, the schema enum and model.py.
xaleryb
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 26, 2026
Main now has intel#210 as squash f5cca76, which this branch carried as its early commits f4ce883..e4b4e80; each add/add conflict was three-way merged against e4b4e80, so the branch adds exactly its own network-policy changes on top of main. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
xaleryb
added a commit
that referenced
this pull request
Oct 26, 2026
…azily (#212) * feat(tasks): detect eval-task formats with `tasks inspect` A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): task_inspect.v1 report, config-named datasets, LFS pointers `tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(tasks): real-format corpus for tasks inspect, with its manifest tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(harnesses): per-phase network policy for docker-* runs, off by default build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(harnesses): write network.json in every docker-* cell dir, errored cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): run terminal-bench v1 tasks natively, export to Harbor lazily A task.yaml task loads as a TaskSpec and runs under docker-oracle and docker-claude as terminal-bench's harness runs it (solution.sh at /oracle, run-tests.sh + tests/ at /tests, image user, TEST_DIR and compose env, pytest-summary verdict). Tasks one docker run cannot reproduce are refused with the reason. Harbor-backed harnesses and benchflow run a content-keyed Harbor export whose test.sh applies the same pytest-summary rule. Stacked on #210. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(tasks): give terminal-bench v1 tasks 8 GB of memory, per task from compose terminal-bench sets no memory limit; dokimi's 2 GB default OOM-killed add-benchmark-lm-eval-harness and predicate-pushdown-bench. A v1 task now reads compose deploy.resources.limits.memory when set, else 8192 MB (DOKIMI_TBENCH_V1_MEMORY_MB overrides), and the Harbor export writes it to task.toml. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(tasks): tb-v1 on the network policy: public oracle/verify, agent solve default Rebased onto #218. The corpus manifest's terminal-bench hello-world now names its consumers (docker-*, harbor, benchflow), not none-yet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(tasks): score a tb-v1 verifier on its one ordered output stream docker-* parsed stdout followed by all of stderr, so a stderr line such as pip's "ERROR: pip's dependency resolver ..." landed after the pytest summary and failed a passing task. Run run-tests.sh with 2>&1, as terminal-bench's pane and the Harbor export's test.sh read it. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): key the Harbor export on the task.toml it writes The cache key hashed only the task's files, so a cached export kept the memory_mb of its first run after DOKIMI_TBENCH_V1_MEMORY_MB changed, while docker-* used the new value. Hash the resolved task.toml too. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): keep the Harbor export cache per user, not in shared /tmp The cache sat at a predictable $TMPDIR path, created group-writable, and any dir there with a task.toml was run and scored as the export. Default to ~/.cache/dokimi/harbor-export, created 0700. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): no harbor/benchflow consumer for a solution.yaml tb-v1 task export() refuses solution.yaml on every run, yet tasks inspect listed harbor and benchflow, and each refusal left a .staging-* dir in the cache. Report the refusal as a problem, clean staging in a finally, and say tasks list/run read three formats now. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): export a named tb-v1 Dockerfile as environment/Dockerfile Every Harbor environment builds environment/Dockerfile, so a task whose compose names build.dockerfile: Dockerfile.x exported with nothing to build. Rename it, refuse a clash, and pin the subdir-context, kept COPY tests/solution.sh and working_dir branches with tests. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs: count xfailed tests as passing in the tb-v1 verdict rule Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(tasks): write non-BMP task metadata into the exported task.toml as TOML reads it json.dumps escaped a character above U+FFFF as a surrogate pair, which tomllib rejects, so every Harbor and BenchFlow run of such a task failed on parse. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): key the tb-v1 pytest results by node id, as the exported test.sh reads them Keyed by the name after the file, a later PASSED b.py::test_x hid an earlier FAILED a.py::test_x: docker-* scored 1.0 where the Harbor export's test.sh scored 0. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): run the exported tb-v1 test.sh against a fake run-tests.sh Only its awk was run before: dropping the exit code, the tee, the 2>&1, TEST_DIR or the reward file name passed the suite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): skip a task.yaml without an instruction when discovering tasks A non-task directory with a task.yaml and a Dockerfile made tasks list and --all fail for the whole root; it is skipped with a warning now, as a task.md without frontmatter is, and load_task on it still raises. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(tasks): refuse a tb-v1 service command, build arg or target, or host environment on docker-* One docker run of the default stage, idling, with literal values did none of these, so such a task ran and its failure read as the task's, not a refusal. The idle sleep infinity and TEST_DIR stay accepted; the top-level-key and extra-volume refusals are now pinned too. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin that the benchflow harness runs a tb-v1 task from its Harbor export Dropping harbor_view from BenchflowHarness.run passed the suite; bench would then get a task.yaml directory with no task.toml. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(tasks): pin the host-side agent refusal for a tb-v1 task Narrowing the refusal to task.md tasks passed the suite. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
dokimi tasks inspect <path> [--json]. It works out which eval-task format a file, task directory or tree is in, then reports the kind, the item count, which parts are present or missing (instruction / environment / oracle / verifier / metadata), lint problems, and which dokimi backends can consume it today. This PR only detects formats. It doesn't run or convert any new format, andtasks list/tasks run/catalogbehave as before.harbor_task_problemis None), docker-*sandbox=)expected_skill)load_questionsaccepts it)I took field names from upstream source, not from memory: SWE-bench
swebench/types.py, Inspectdataset/_dataset.py(Sample/FieldSpec), terminal-benchoriginal-tasks/*(241 task.yaml keys tallied), skill-creatorreferences/schemas.mdandSKILL.md, skill-reachmodels.py/queries.py(including the legacyid->query_idmigration and the out_of_scope rule), the promptfoo examples (149 configs tallied), and openai/evalsdocs/build-eval.md.Evidence
tasks inspectdoesn't exist, and nothing reads the trigger sets that dokimi writes.After: a run over real upstream samples:
tests/test_task_format_detect.py: 37 tests. They cover every format, ambiguity, unknown, malformed JSON/JSONL/YAML/TOML/YAML-alias inputs (each becomes a problem, not a crash), an Inspect file that would write a marker if imported (the marker is never written), a mixed tree, and a regression check thatdiscover_tasksis unchanged.make ci-local: tests (7840 passed), mypy, ruff, growth, import contract, coverage, mkdocs and cli-surface all pass. pip-audit fails on the setuptools GHSA-xmj3-9gf7-h52w advisory; that failure is on main too and isn't caused by this PR.Merge Danger
Door: two-way. It adds a new command and a new package and changes no existing behaviour or persisted format. The
--jsonoutput isn't a versioned artifact yet.Blast Radius: additive
🤖 Generated with Claude Code