[napetrov]

Summary

PR2 of the code-index work: eval tasks generate --from graph (D11 on #94). Built on the SCIP reader (#339) and repo inspect --index (#349). It writes enumeration tasks whose gold comes from a SCIP index. No LLM or docker is involved.

eval tasks generate SRC@<rev> --from graph --index index.scip --out <tasks>
  CodeIndex.load              # #339 reader; new: symbols()
  detect_prefix               # index paths -> tree paths (as repo inspect --index)
  references: functions defined in non-test, non-generated tree files (C++ overloads = one target)
    gold = files (or path::Class.method with enclosing ranges) referencing it,
           outside the defining files; any ref outside the tree -> drop
  derives: classes with >=2 direct subclasses (index.bases), all in the tree
    accepted = a second defining file (.pyi), classes a scip-clang reduction dropped
  _choose(seed): ~half ambiguous names ("39 symbols named `run`")
  write <tasks>/graph_<kind>_<name>_<hash8>/{task.json, gold.json}   # kept unless --force
grader dokimi:graph_enum: last ```json fence -> F1 vs gold; exclude/accepted; Class::m() == Class.m

Diff by area:

+dokimi/commands/tasks_generate.py   # CLI leaf eval tasks generate
+dokimi/repos/graph_tasks.py         # plan / write
+dokimi/repos/graders/graph.py       # dokimi:graph_enum (stdlib, runs in the grading container)
 dokimi/repos/code_index/            # + CodeIndex.symbols(), descriptor_names()
 dokimi/repos/environment/structure.py  # two helpers made public, no behaviour change
 docs/repo-evaluation-contract.md    # §1.2, §4.5 row, new §4.7, §9 example

--from history (fail-to-pass in docker) comes in a separate PR. --from replay stays repo tasks replay until the M2/M3 tombstone.

Evidence

Generated from real indexes, then checked with eval check (null fails, oracle reaches 1.0):

Repo	Index	Granularity	Candidates (references / derives)	Tasks	eval check
dokimi	scip-python, 617/617 docs	functions	454 / 5	12 in 4 s, 227 MB peak	12/12 OK
oneDAL	scip-clang, 3051/3051 docs	files	669 / 160	12 in 12 s, 346 MB peak	12/12 OK

On oneDAL, merging overloads took the candidate count from 770 to 669; graph_references_set_* covers 3 overloads.

• New tests in tests/test_eval_tasks_generate_graph.py (40 tests) cover:
  • generator rules: refs outside the tree, test/generated definitions, protocol names, size gate, subdirectory index, innermost function, same-name methods in one file, C++ overloads, derives (cap, .pyi, scip-clang reduction), seeded choice, ids per symbol, kept/force;
  • grader: oracle 1.0, parse error, exclude/accepted, path normalisation (a repo/ directory kept), Class::m() and bare names, last fence, malformed gold, script mode under python3 -I;
  • CLI: end-to-end on a git repo with an encoded SCIP file, plus the exit 2 and exit 1 cases.
• make ci-local: 0 failed, 30 passed, 7 skipped.
• Local self-review (finder, design lens, verifier) found 3 blocking issues: id collisions across reruns, C++ overloads, and method names within a file. All three are fixed, with tests.

Merge Danger

Door: one-way for the new public names: the --references/--derives flags (allowlisted with a reason), the grader name dokimi:graph_enum with its metrics, and the task.json/gold.json fields and id scheme of generated tasks. Tasks written now keep these names. The code is two-way. No existing format changes; CodeIndex.bases gains a keyword minimal=True.

Blast Radius: additive

Existing commands are unchanged. The structure.py renames are internal.

Part of #41

🤖 Generated with Claude Code

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi eval tasks generate`
    + argument `source`
    + option `--cache-dir`
    + option `--derives`
    + option `--dry-run`
    + option `--force`
    + option `--from`
    + option `--help/-h`
    + option `--index`
    + option `--max-turns`
    + option `--out`
    + option `--references`
    + option `--seed`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[exai-bot]

OpenCodeReview: Commit 6f77801 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `6f77801`: 8 still running, 3 passing, 1 skipped

Check	Status
Build docs (strict)	still running
CLI interface diff	still running
CodeQL (actions)	still running
CodeQL (python)	still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com...	still running
Windows smoke test (py3.13)	still running
mypy	still running
self / repo validate	still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (246593a): 1 medium</summary>

Open threads from the bot: 0

</details>

[Copilot]

🟡 Changes recommended

Namespace stripping conflates distinct subclasses, while scope lookup misattributes class-body and same-line method references, producing incorrect gold answers and scoring.

3 open issues

• Preserve namespaces to distinguish same-named subclasses · New
• Exclude references inside nested class bodies from function gold · New
• Use span containment to resolve same-line method scopes · New

What changed in this PR

3 new findings.

This PR extends dokimi’s repository evaluation flow with eval tasks generate --from graph, building enumeration tasks from a SCIP index of a pinned revision without an LLM or Docker.

Changes:

• Adds seeded task generation for function references and direct subclasses.
• Introduces the dokimi:graph_enum grader with F1 scoring and answer normalization.
• Registers the CLI command and updates documentation, interface snapshots, and tests.

File	Description
dokimi/​tests/​test_repo_inspect_structure.py	Updates tests for public helper names.
dokimi/​tests/​test_eval_tasks_generate_graph.py	Tests generation, grading, and CLI behavior.
dokimi/​tests/​data/​cli_flags_allowlist.txt	Allows task-count flags.
dokimi/​dokimi/​repos/​graph_tasks.py	Plans and writes index-derived tasks.
dokimi/​dokimi/​repos/​graders/​main.py	Registers graph grading and oracle execution.
dokimi/​dokimi/​repos/​graders/​graph.py	Implements enumeration scoring and normalization.
dokimi/​dokimi/​repos/​graders/​__init__.py	Registers the grader and its gold file.
dokimi/​dokimi/​repos/​environment/​structure.py	Exposes two shared filtering helpers.
dokimi/​dokimi/​repos/​code_index/​symbols.py	Exposes descriptor names and kinds.
dokimi/​dokimi/​repos/​code_index/​index.py	Adds symbol enumeration and unreduced bases.
dokimi/​dokimi/​commands/​tasks.py	Wires task generation into the CLI.
dokimi/​dokimi/​commands/​tasks_generate.py	Implements generation options and command handling.
dokimi/​dokimi/​cli_routes.py	Marks generation as a free operation.
dokimi/​docs/​repo-evaluation-contract.md	Documents task generation and grading semantics.
dokimi/​docs/​reference/​cli-interface.json	Records the new command’s interface.
dokimi/​docs/​architecture.md	Describes the graph-task module’s role.
dokimi/​changelog.d/​20261104_120000_napetrov_eval-tasks-generate-graph.md	Announces graph task generation and grading.

_{Review effort: Balanced}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.21514% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.79%. Comparing base (94ad4db) to head (6f77801).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/graph_tasks.py	96.05%	5 Missing and 7 partials ⚠️
dokimi/dokimi/commands/tasks_generate.py	91.25%	6 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #399      +/-   ##
==========================================
+ Coverage   94.76%   94.79%   +0.03%     
==========================================
  Files         352      357       +5     
  Lines       46407    47071     +664     
  Branches     7949     8056     +107     
==========================================
+ Hits        43978    44623     +645     
- Misses       1539     1552      +13     
- Partials      890      896       +6     

Files with missing lines	Coverage Δ
dokimi/dokimi/cli_routes.py	100.00% <ø> (ø)
dokimi/dokimi/commands/tasks.py	93.75% <100.00%> (+0.05%)	⬆️
dokimi/dokimi/repos/code_index/index.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/code_index/symbols.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/environment/structure.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/graders/__init__.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/graders/graph.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/graders/main.py	82.60% <100.00%> (+0.51%)	⬆️
dokimi/dokimi/commands/tasks_generate.py	91.25% <91.25%> (ø)
dokimi/dokimi/repos/graph_tasks.py	96.05% <96.05%> (ø)

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[napetrov]

a: 142e6af namespaces. b: 142e6af class bodies. c: 142e6af same-line ties.

a: 142e6af namespaces. b: 142e6af signatures. c: 142e6af blank gold. d: 142e6af contract.

[napetrov]

Bot review run marker with no finding text, so nothing to answer. No change.

Bot review run marker with no finding text, so nothing to answer. No change.

The CLI-surface change is intended: eval tasks generate is the new command, and its interface snapshot docs/reference/cli-interface.json is updated in this PR. No change.

Coverage-tool notice, not a review finding. No change.

[napetrov]

DCO fix: the branch was rebased with Signed-off-by added to every commit. Tree unchanged.

• 4ba6b117 -> 1dd1568c (exclude-file grader error)
• 6501bfc6 -> 533b5820 (function gold needs a range on every function)
• merge 702951164 -> f4b16f39; earlier commits rewritten under new shas, content identical.

[antonwolfy]

Review at 533b582. The generator and the grader hold up: every earlier bot point (namespaces, class bodies, same-line scopes, signature references, blank and excluded gold, partial ranges) is fixed and has a test. What is left is thin test coverage on lines that decide the gold and the score, and one doc gap. Two risks and two nits inline.

What I ran, on a Linux review host at 533b582:

• make ci-local: 29 passed, 1 failed, 7 skipped; 15,759 tests; changed-line coverage 97%. The one red step is test / Run tests: a 1 s timing test (test_unbalanced_quote_in_hook_completes_quickly) and two 60 s timeouts in test_audit_test_quality.py hit on a loaded run; all three pass on rerun and none touch this PR's files.
• tests/test_eval_tasks_generate_graph.py and the CLI snapshot tests pass; changed-line coverage 97% (462 lines).
• Mutation runs of graph_tasks.py (58 mutants, 25 pass the suite) and graders/graph.py (49 mutants, 12 real survivors), and CLI edge cases (bad revision, non-git source, unreadable index, --dry-run, --force).

Checked, no issues

• Scope lookup: nested functions, same-line functions and class bodies inside a function give the expected innermost function.
• C++ overloads: one target, defining files are the union, order is deterministic.
• Grader: F1, parse errors, the oracle reaching 1.0 for Class::m() and ./path forms, and malformed gold are all right.
• Repo rules: snapshot, allowlist, changelog fragment and docs pages are consistent; no nav change needed.
• Commits: all non-merge commits carry Signed-off-by.

[antonwolfy]

I pushed a commit per nit, up to 28e5586, which fix the two nits from my review:

• 0bbbd3b: dokimi/docs/repo-evaluation-contract.md:1053: the exit-1 sentence now also covers a source that cannot be resolved at the revision.
• 28e5586: dokimi/dokimi/repos/graph_tasks.py:392: the shared-name hint now also shows the Class.name form.

make ci-local at 28e5586: 30 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the two risks from my review.

[napetrov]

Fixed in 94cab68: the two nits are in 0bbbd3b and 28e5586.

Both open risks are fixed; see the top comment.

@antonwolfy: thanks for the two nit commits. The open risks, each with its fix:

• span edges and target rule: b1f0cd7, pinned by new tests (your mutants fail them).
• exclude and path normalisation: 9026459, pinned by new tests.
  Not pinned: the tie rule in _innermost (> to >=). No fixture has two scopes that start at one position, so the test would pin an accident. Say if you want it anyway.
  Ready for another look.

[vlad-perevezentsev]

Review at 9f1bfdc. Both risks from my last review are fixed: the span edges, the target rule, the path normalisation and the exclude list are now pinned by tests. One scoring gap is left in the grader's bare-name rule, and a few smaller points; they are in the line comments.

On the _innermost tie (> to >=): agreed, no test is needed. Two function or class definitions that start at one position do not occur in the indexes this reads, so a test would pin an accident.

What I ran, at 9f1bfdc:

• make ci-local --offline: 19 passed, 0 failed, 13 skipped (network and Docker steps); 15,724 tests; changed-line coverage 97% (462 lines).
• The mutants from my last review, again on tests/test_eval_tasks_generate_graph.py: all 11 are now caught; only the tie mutant survives, as said above.
• The grader on hand-made gold for the line comments: a bare a.py::run for gold a.py::Impl.run gives F1 1.0; a bare x.h::Child with gold ns1.Child and accepted ns2.Child gives F1 0.5.
• The free CI smoke (docker-oracle, docker-probe) with the changed graders/main.py: both pass.

Checked, no issues

• Earlier bot points: namespaces, class bodies, same-line scopes, signature references, blank gold, gold in an exclude file and partly ranged indexes are fixed, each with a test.
• Grader in the container: main.py loads graph by name under -I, as for the other graders, and the test runs it that way.
• Repo rules: SPDX headers on the new files, changelog fragment (Added), CLI snapshot and flag allowlist are in order; no new docs page, so no nav change.

[napetrov]

Thanks: each line comment is fixed with a test; see the threads.

[xaleryb]

• Ran on Python 3.13, the head merged into current main plus the commits below: the full suite, ruff, mypy, bandit, the growth and comment-block checks, the CLI snapshot and the benchmark gate. Each new test fails on the pre-fix source.
• Pushed:
  • b87ee0b: derives tasks write distinct. Before it, a bare Child naming a non-subclass in a gold file scored gold Outer.Child / ns1.Child as a hit (f1 1.0, expected a false positive).
  • 24659db: a test for gold run next to a method Impl.run in the same file. Nothing failed when _distinct compared whole names, and then an all-wrong answer scored f1 1.0.
  • 8771dde: the scip-clang files prompt counts a declaration. A header prototype is gold, so an answer that followed the old prompt lost recall.
  • 6f77801: the run warns when the index lacks tree files of its languages.
• Left for the author: an index without tests/test_core.py exits 0, and the gold leaves out that file's reference. Recommendation: exit 2 by default, as for an index that maps to no tree file (§4.7), with an opt-in flag for a partial index.

[vlad-perevezentsev]

Review at 6f77801. All five points from my last review are fixed, each with a test: a bare a.py::run next to gold a.py::Impl.run is now a false positive, and Impl.run and Impl::run() still score 1.0. Two points are left, in the line comments.

Agreed with @xaleryb: an index that lacks a tree file still writes tasks whose gold misses that file. In the line comment I suggest dropping the affected targets rather than exiting 2.

What I ran, at 6f77801:

• make ci-local --offline: 18 passed, 1 failed, 13 skipped; 15,739 tests; changed-line coverage 97% (502 lines). The failure is schema_compat: the base drifted after #387 added run_id and dokimi_version on main, and this PR changes no schema.
• On the PR merged into current main: test_eval_tasks_generate_graph.py, test_cli_flags.py, test_repo_inspect_structure.py, test_docs_code_refs.py: 208 passed.
• The grader on generated tasks: distinct with a module-level run, the over-qualified cli.Impl.run, and an index without tests/test_core.py (the task is still written, without that file in its gold).

Checked, no issues

• Bare names: distinct is written for function and class answers and checked by _read_gold, and ranking by shared trailing parts resolves only when one item is the longest match.
• @xaleryb's commits: distinct for derives, the run / Impl.run test and the declaration wording for scip-clang file answers match the code and the contract.
• Docs: §4.7 and the generate help now match the code on the granularity fallback, the grader errors and the exit codes.

[vlad-perevezentsev]

A tree file the index lacks still gets tasks written with a gold that misses it.

As @xaleryb noted: with tests/test_core.py in the tree but not in the index, run is still a task with gold src/api.py, src/cli.py, and the command exits 0 after the warning. The prompt says test files count, so a right answer that names tests/test_core.py scores a false positive.

Suggestion: drop a target whose name appears as a word in an unindexed file, as a reference outside the tree already drops it, and keep the warning. Exit 2 for the whole index would also refuse most scip-clang indexes, which leave out files that compile_commands.json does not build.

[vlad-perevezentsev]

The shared-name hint now says "Two answers" when only one of them is an answer.

Since distinct joined the check, gold src/a.py::Impl.run next to a module-level run in that file (not a referrer) gives "Two answers share a name in one file". An agent that trusts it may add src/a.py::run, which is a false positive (F1 0.89 instead of 1.0).

Suggestion: "A name repeats within one file: write each answer with its namespace or class (ns::Name, Class.name)."

[antonwolfy]

Review at 6f77801. Both risks from my earlier review stay fixed, and the new distinct mechanism holds up. The merge waits for two points from @vlad-perevezentsev that are still open at this head. I have no new findings of my own.

Waiting on the author:

• @vlad-perevezentsev's point on tasks_generate.py:77: a tree file the index lacks still gets tasks with a gold that misses it; 6f77801 only warns and exits 0 (repro: src/u9.py in the tree, not in the index). @xaleryb's choice (drop the targets, or exit 2 with an opt-in) is unanswered.
• @vlad-perevezentsev's point on graph_tasks.py:438: the shared-name hint misleads when distinct brings in a function that is not an answer (gold src/a.py::Impl.run with a module-level src/a.py::run: the answer src/a.py::run scores F1 0.5). The code is unchanged and the author has not replied.

What I ran, on a Linux review host at 6f77801:

• make ci-local: 29 passed, 1 failed, 7 skipped; 15,789 tests. The red step is test / Run tests: test_unbalanced_quote_in_hook_completes_quickly took 1.49 s against a 1 s limit, and two tests in test_audit_test_quality.py hit the 60 s timeout on a loaded run. None touch this PR's files. The changed-line coverage step passes.
• Reproductions for the unindexed-file and hint points above, with the LLM transport mocked, and a mutation run of 23 mutants over the distinct code in the grader and the generator: 20 caught, 3 survive. The three only change what is written to gold.json or are equivalent, so I do not raise them.

Checked, no issues

• Span edges and path normalisation: the two risks from my earlier review stay pinned by tests.
• _resolve ranking: max/min and tie-handling mutants are caught; cli.Impl.run names Impl.run over run.
• Warning edge cases: extensionless files, .PY, dotfiles, a sibling root such as sub2/, and an empty index give no false warning and no crash.
• Docs: §4.7 and the generate help match the code for the distinct field, one target per symbol on scip-python, the declaration wording and the exit codes.
• Commits: all 21 non-merge commits carry Signed-off-by.