Repository navigation
feat(eval): eval coverage against a frozen need graph (#331) - #340
Conversation
…nst a frozen need graph (intel#331) need_graph.v1 and eval_coverage.v1 schemas; evals name graph nodes in 'needs' (evals.json rows or task metadata). Pure function of files, no model call; exit 1 under --min-coverage / --min-rare-critical, 2 on unknown ids. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…IC=]X, strict needs Novelty's NeedNode keeps confirmed_by and NeedGraph keeps provenance, so both readers accept the same need_graph.v1 file. Duplicate node ids, a needs value that is not a list of ids, an unreadable task, a non-list evals and an empty suite exit 2. Problems are structured; the report records the weights; the digest covers only identity fields. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi eval coverage`
+ argument `evals`
+ option `--graph`
+ option `--help/-h`
+ option `--min`
+ option `--out`
~ `dokimi runs list` parameters changed:
~ changed option `--kind`: choices: ["'answers'", "'arms'", "'cli_routes'", "'eval'", "'explore_runs'", "'guidance_reach'", "'matrix_rollup'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'repo_budget'", "'repo_environment'", "'repo_readiness'", "'result'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'cli_routes'", "'eval'", "'eval_coverage'", "'explore_runs'", "'guidance_reach'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'repo_budget'", "'repo_environment'", "'repo_readiness'", "'result'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]Updated on every push; the |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects allow output paths to overwrite consumed input files, accept null needs labels without error, and produce incorrect threshold results from rounded metrics.
3 open issues
What changed in this PR
3 new findings.
This PR adds deterministic eval coverage reporting against a frozen need graph, partially delivering AC-J07-007 without model calls.
Changes:
- Computes weighted coverage, critical gaps and per-eval relevance, with configurable coverage floors.
- Adds graph/report schemas and artifact integration; preserves graph confirmation and provenance metadata.
- Adds tests, CLI documentation and journey evidence for the new command.
| File | Description |
|---|---|
dokimi/tests/test_runs_show.py |
Adds display expectations for both artifact kinds. |
dokimi/tests/test_runs_scan.py |
Adds artifact discovery expectations. |
dokimi/tests/test_import_architecture.py |
Registers the coverage module’s architectural layer. |
dokimi/tests/test_eval_coverage.py |
Tests reports, validation, determinism and CLI behavior. |
dokimi/tests/outcomes/journeys.yaml |
Records partial delivery and enforced outcome evidence. |
dokimi/tests/fixtures/artifacts/need_graph.v1.json |
Adds a frozen graph fixture. |
dokimi/tests/fixtures/artifacts/eval_coverage.v1.json |
Adds a coverage report fixture. |
dokimi/tests/data/cli_flags_allowlist.txt |
Allows the graph and coverage-floor flags. |
dokimi/dokimi/schemas/need_graph.v1.json |
Defines the frozen graph format. |
dokimi/dokimi/schemas/eval_coverage.v1.json |
Defines the coverage report format. |
dokimi/dokimi/runs/kinds.py |
Registers artifact subjects and summary counts. |
dokimi/dokimi/novelty/needs.py |
Preserves confirmation and provenance metadata. |
dokimi/dokimi/eval_coverage.py |
Implements label reading, coverage calculations and Markdown output. |
dokimi/dokimi/commands/evaluate.py |
Registers the coverage subcommand. |
dokimi/dokimi/commands/eval_coverage.py |
Adds CLI arguments, output handling and coverage gates. |
dokimi/dokimi/cli_routes.py |
Classifies coverage as a free command. |
dokimi/dokimi/artifacts.py |
Registers both schema versions. |
dokimi/docs/user-journeys/journeys.md |
Documents partial coverage-gap delivery. |
dokimi/docs/reference/cli-interface.json |
Updates the CLI interface snapshot. |
dokimi/data/skills/dokimi-design-evals/references/coverage-report.md |
Explains how generated coverage supplements the manual report. |
dokimi/changelog.d/20261031_180000_napetrov_eval-coverage.md |
Announces the command and graph metadata changes. |
Review effort: Balanced
| if args.out and Path(args.out).resolve() == Path(args.graph).resolve(): | ||
| print("Error: --out would overwrite --graph", file=sys.stderr) | ||
| sys.exit(EXIT_USAGE) |
There was a problem hiding this comment.
Fixed in 91d982b: --out is refused when it would overwrite any file the run reads, not only the graph. That covers the evals.json, task.toml/task.md, terminal-bench task.yaml and docker-compose.yaml, and a directory's discovered files. Red tests first: test_out_never_overwrites_an_input_file and test_out_never_overwrites_a_terminal_bench_task_file.
| cov = report["coverage"] | ||
| return [f"{metric} coverage {cov[metric]:.2f} < {floor:.2f}" for metric, floor in floors | ||
| if cov[metric] is not None and cov[metric] < floor] |
There was a problem hiding this comment.
Fixed in 91d982b: --min is judged on the unrounded share (exact_shares), so 6 of 7 passes 0.85714 and 2 of 3 fails 0.66669. The report keeps its rounded display. Test: test_a_floor_is_judged_on_the_unrounded_share.
| if value is None: | ||
| return LabelledEval(eid, source, None) |
There was a problem hiding this comment.
Fixed in 91d982b: an explicit needs: null is malformed in both readers and exits 2; a missing needs key stays unlabelled. Tests: test_needs_that_is_not_a_list_of_ids_is_a_problem (null case) and test_a_null_task_needs_is_malformed_and_a_missing_key_is_unlabelled.
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 21 selected item(s). Review details
|
| unreadable input, a graph with duplicate node ids, or a `needs` label that is not a list of | ||
| ids the graph has. Novelty's need graph now keeps `confirmed_by` and `provenance`. Building the graph, the user |
There was a problem hiding this comment.
Nit (accuracy): needs: null is not a list, but it does not trigger exit 2. For example, {"evals": [{"id": "e", "needs": null}]} with a valid graph and no floor exits 0 and reports unlabelled: _labelled() treats None as a missing label. Either document that explicit null is also unlabelled, or distinguish missing keys from null in the reader and add a regression test so this exit-code promise holds.
There was a problem hiding this comment.
Fixed in 91d982b: an explicit needs: null is malformed in both readers and exits 2; a missing needs key stays unlabelled. Tests: test_needs_that_is_not_a_list_of_ids_is_a_problem (null case) and test_a_null_task_needs_is_malformed_and_a_missing_key_is_unlabelled.
| if not evals: | ||
| print(f"Error: no evals.json rows or tasks under {root}", file=sys.stderr) | ||
| sys.exit(EXIT_USAGE) | ||
| if args.out and Path(args.out).resolve() == Path(args.graph).resolve(): |
There was a problem hiding this comment.
Must-fix — input clobbering: dokimi eval coverage evals.json --graph graph.json --out evals.json passes this guard and replaces the source suite with the coverage artifact. save_artifact uses os.replace without an overwrite safeguard. The same issue affects directory inputs when --out names their evals.json, evals/evals.json, or a task configuration file read by read_labels. Reject output paths resolving to any input file, not just the graph, and add regression tests asserting exit 2 and unchanged source contents.
There was a problem hiding this comment.
Fixed in 91d982b: --out is refused when it would overwrite any file the run reads, not only the graph. That covers the evals.json, task.toml/task.md, terminal-bench task.yaml and docker-compose.yaml, and a directory's discovered files. Red tests first: test_out_never_overwrites_an_input_file and test_out_never_overwrites_a_terminal_bench_task_file.
| if value is None: | ||
| return LabelledEval(eid, source, None) |
There was a problem hiding this comment.
Must-fix: explicit "needs": null is accepted as unlabelled rather than reported as malformed. For example, {"evals":[{"id":"e","needs":null}]} produces no problems and the CLI exits 0, although a present needs value must be a list of strings. Both the JSON-row and task-metadata paths use .get("needs"), conflating a missing key with an explicit null. Use a missing-key sentinel (or pass key presence separately), reject explicit null, and add a regression test alongside the malformed-label cases.
There was a problem hiding this comment.
Fixed in 91d982b: an explicit needs: null is malformed in both readers and exits 2; a missing needs key stays unlabelled. Tests: test_needs_that_is_not_a_list_of_ids_is_a_problem (null case) and test_a_null_task_needs_is_malformed_and_a_missing_key_is_unlabelled.
|
|
||
|
|
||
| def _share(nodes: list[dict], covered: set[str]) -> float | None: | ||
| return round(sum(n["id"] in covered for n in nodes) / len(nodes), 4) if nodes else None |
There was a problem hiding this comment.
Must-fix: rounding coverage before threshold enforcement can give the wrong exit code. With three core nodes and two covered, core_rare_critical becomes 0.6667, so --min core_rare_critical=0.66669 passes even though actual coverage (2/3) is below the floor. Rounding can also incorrectly fail an above-floor suite. The CLI's _below() compares these report values directly. Preserve full precision for threshold comparisons, handling both _share() and the weighted calculation, and add rounding-boundary tests.
There was a problem hiding this comment.
Fixed in 91d982b: --min is judged on the unrounded share (exact_shares), so 6 of 7 passes 0.85714 and 2 of 3 fails 0.66669. The report keeps its rounded display. Test: test_a_floor_is_judged_on_the_unrounded_share.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #340 +/- ##
==========================================
+ Coverage 94.05% 94.11% +0.06%
==========================================
Files 297 299 +2
Lines 38018 38226 +208
Branches 6424 6455 +31
==========================================
+ Hits 35756 35975 +219
+ Misses 1452 1442 -10
+ Partials 810 809 -1
🚀 New features to boost your workflow:
|
… floors, needs null is malformed --out is refused when it would overwrite the graph or any file a read consumes, including a tbench task.yaml and docker-compose.yaml. A --min floor is judged on the unrounded share, so 6 of 7 passes 0.85714 and 2 of 3 fails 0.66669. An explicit needs: null is malformed (exit 2); a missing needs key stays unlabelled. Red tests first. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| sys.exit(EXIT_USAGE) | ||
| report = coverage_report(graph, evals, str(root)) | ||
| if args.out: | ||
| report = save_artifact("eval_coverage", report, Path(args.out)) |
There was a problem hiding this comment.
Must-fix: output filesystem failures are unhandled here, and the CLI entry point does not catch OSError. With valid inputs, python cli.py eval coverage evals.json --graph graph.json --out . reaches os.replace in save_artifact and raises a directory-related OSError, printing a traceback rather than a controlled CLI error. Permission-denied destinations fail similarly. Catch OSError around the save, print a concise error and exit with the appropriate error code; add a regression test using an existing directory as --out.
There was a problem hiding this comment.
Fixed in 3fa238b: a failing --out write (a directory, no permission) now prints Error: cannot write <path>: <reason> and exits 2, no traceback. Regression test: test_out_on_a_directory_is_a_controlled_error.
| lines += [f"| {e['id']} | {e['source']} | {', '.join(e['needs']) or '-'} | {e['relevance']} |" | ||
| for e in report["evals"]] |
There was a problem hiding this comment.
Must-fix: escape user-provided values before inserting them into Markdown table cells. A valid row such as {"id": "save|load", "needs": ["n1"]} adds an extra column, shifting Needs and Relevance under the wrong headings; an ID containing a newline splits the row. Graph node IDs and task-directory names can also contain these characters. Escape pipes and normalize CR/LF in IDs, sources and the joined needs cell, and add a rendering regression test.
There was a problem hiding this comment.
Fixed in 3fa238b: table cells escape | and flatten CR/LF (_cell in eval_coverage.py), so save|load and a newline in an id stay in their columns. Regression test: test_markdown_table_cells_escape_pipes_and_newlines.
…ipes and newlines; mypy - _below reads the share into a local, so mypy narrows it (the CI mypy failure). - A failing --out write (a directory, no permission) is a usage error, not a traceback. - Markdown table cells escape "|" and flatten line breaks, so an id cannot add a column or split a row. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
||
|
|
||
| # every task file a task reader may parse: Harbor / BenchFlow (task.toml, task.md), terminal-bench v1 (task.yaml, compose) | ||
| _TASK_FILES = ("task.toml", "task.md", tbench_v1.CONFIG, tbench_v1.COMPOSE) |
There was a problem hiding this comment.
Must-fix: the overwrite guard omits instruction.md, allowing the report to destroy a supported task's instructions. With tasks/t1/task.toml containing [metadata]\nneeds = ["n1"] and an existing tasks/t1/instruction.md, eval coverage tasks --graph graph.json --out tasks/t1/instruction.md passes this guard and save_artifact() replaces the instruction file with JSON. Although the metadata-only reader does not read instructions, load_task() uses this file when executing the suite. Include existing instruction.md files in the protected paths and add an overwrite regression test alongside the configuration-file cases.
Suggestion:
| _TASK_FILES = ("task.toml", "task.md", tbench_v1.CONFIG, tbench_v1.COMPOSE) | |
| _TASK_FILES = ("task.toml", "task.md", "instruction.md", tbench_v1.CONFIG, tbench_v1.COMPOSE) |
There was a problem hiding this comment.
Fixed in db9ef3e. The overwrite guard now lists instruction.md, which load_task runs, so --out refuses it with exit 2 (red before the fix).
Test: test_out_never_overwrites_an_input_file[instruction].
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 3fa238b. The labels, relevance classes, the unrounded --min gate and the overwrite guard hold up. Three things are worth fixing before merge: a stray task.md aborts the whole run, a --min floor on an empty metric passes silently, and the gate tests miss the cases that decide the exit code.
What I ran, on a Linux review host at 3fa238b:
make ci-local: 30 passed, 0 failed, 7 skipped; 12,007 tests; the changed-line coverage step passes.- Reproductions for the two code findings through
python -m dokimi eval coverage, with no model involved. - A mutation run: 105 mutants, 37 survived the new tests. The ones that decide an exit code or a reported figure are in the inline comment on the gate tests; I re-ran three of them (
<to<=, default metric, no.resolve()in the guard) and all three pass the full file.
Checked, no issues
- Schemas:
need_graph.v1andeval_coverage.v1are new kinds,schema_compatagainst the base reports OK, and asave_graphfile validates as written. - Docs: the changelog exit codes,
--min [METRIC=]Xforms and relevance classes match the code;cli-interface.jsonmatches the parser. - Commits: all four carry
Signed-off-by. - Round trip:
confirmed_byandprovenancesurvivesave_graphthenload_graph.
Bot points I checked that no longer apply at this head (9)
- Copilot's point on
--outoverwriting an eval suite does not apply at this head: the guard now resolves--outagainst the graph and every file the run reads, and refuses with exit 2. - Copilot's point on comparing rounded values with
--mindoes not apply: the floor is judged onexact_shares, computed from unrounded counts and weights. - Copilot's point on
needs: nulldoes not apply: an explicit null ismalformed_needsin both readers and exits 2. - Exai-bot's point on
--outclobbering an input does not apply at this head: the guard coversevals.json, the graph and every task file the run reads. - Exai-bot's point on
needs: nulldoes not apply: an explicit null is malformed in both readers and exits 2, and a missing key isunlabelled. - Exai-bot's point on rounding before the
--mincheck does not apply: the gate uses the unrounded shares fromexact_shares. - Exai-bot's point on an unhandled
OSErrorfrom--outdoes not apply: the save is wrapped, printsError: cannot write ...and exits 2. - Exai-bot's point on Markdown table cells does not apply:
_cellescapes pipes and flattens line breaks. - Exai-bot's point on
instruction.mddoes not apply as a defect of this guard: the run never readsinstruction.md, and the guard protects the files the run reads.--outnaming any other existing file replaces it, as it does in every command that takes--out.
…k.md notes --out could replace a task's instruction.md, which load_task runs, because the overwrite guard listed task.toml and task.md but not instruction.md. A task root with a planning-notes task.md (no frontmatter) aborted the report: the directory counted as a task and read_task_config raised. Both now follow discover_tasks: instruction.md is a protected input, and a lone task.md without frontmatter is not a task. Tests: the instruction.md overwrite (red before), the notes directory (red before), a floor equal to the share, a bare --min on the weighted figure, a hand-computed exact_shares case, and a relative --out to an input. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| if not evals: | ||
| print(f"Error: no evals.json rows or tasks under {root}", file=sys.stderr) | ||
| sys.exit(EXIT_USAGE) | ||
| if args.out and Path(args.out).resolve() in {Path(args.graph).resolve(), *input_files(root)}: |
There was a problem hiding this comment.
Must-fix: output-path resolution can fail before reaching the controlled write-error handler. On supported Python 3.11/3.12, create a self-referencing symlink with ln -s loop loop, then run valid inputs with --out loop: Path.resolve() raises RuntimeError, which neither this command nor the CLI entry point catches, producing a traceback instead of exit 2. Resolution may also raise OSError. Wrap the overwrite guard's path resolution/input enumeration in error handling for both exceptions and add a symlink-loop regression test.
There was a problem hiding this comment.
Fixed in daa0a1e. The output-path guard now catches OSError and RuntimeError from resolve() and exits 2 with Error: cannot resolve <path>.
Regression test test_out_that_is_a_symlink_loop_exits_2_not_a_traceback: red on Python 3.11 (RuntimeError: Symlink loop), green on 3.11 and 3.14.
A floor on a coverage figure with no node behind it (n/a) used to pass silently. It now exits 1 with "<metric> coverage n/a < X", as the changelog and the --min help say. The changelog also records the known gap: a terminal-bench v1 task.yaml has its metadata.needs ignored. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Path.resolve() raises RuntimeError on a symlink loop on Python 3.11 and 3.12, and the output-path guard let it escape. The guard now treats OSError and RuntimeError as an unresolvable --out and exits 2. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| if not root.exists(): | ||
| print(f"Error: {root} does not exist", file=sys.stderr) | ||
| sys.exit(EXIT_USAGE) |
There was a problem hiding this comment.
Must-fix: root.exists() is outside the input-error handler. On supported Python 3.11–3.13, checking a path beneath a directory without search permission raises PermissionError; the CLI's outer handler does not catch OSError, so this prints a traceback and exits 1 rather than the promised input-error exit 2. For example, as a non-root user, create locked/evals.json, run chmod 000 locked, then invoke eval coverage locked/evals.json --graph graph.json. Move the existence check into the existing input try block and add a regression test (mocking Path.exists to raise PermissionError also works).
There was a problem hiding this comment.
Fixed in 095d2fe: the existence check now sits inside the input error handler, so an unreadable parent exits 2 with the error, not a traceback.
Test: test_an_unreadable_evals_directory_exits_2_not_a_traceback (mocked Path.exists).
| sys.exit(EXIT_USAGE) | ||
| if args.out: | ||
| try: | ||
| clobbers = Path(args.out).resolve() in {Path(args.graph).resolve(), *input_files(root)} |
There was a problem hiding this comment.
Must-fix: the guard checks only the report destination, but save_artifact also deletes <out>.rejected after a successful save. For example, with a valid graph at cov.json.rejected, eval coverage skill --graph cov.json.rejected --out cov.json passes this check and then deletes the graph. An evals input at that sibling path is equally vulnerable. Include the resolved cleanup path in the input-collision check (or disable that cleanup for this command), and add a regression asserting the input remains intact.
There was a problem hiding this comment.
Fixed in 095d2fe: the --out guard also checks <out>.rejected, which save_artifact removes after a valid save. An input at that path is refused.
Test: test_out_whose_rejected_sibling_is_the_graph_is_refused (the graph is left intact).
… <out>.rejected An unreadable parent directory made the existence check raise PermissionError outside the input error handler, so it printed a traceback. The check now sits inside that handler. save_artifact removes <out>.rejected after a valid save, so an input at that path was deleted. The --out guard now treats it as an output too. The guard moves to a helper, which keeps the command under the complexity ratchet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The delta closes all three threads: a task.md without frontmatter is skipped as discover_tasks does, a --min floor on an n/a metric now exits 1, and the gate tests pin the cases I listed. One nit on the symlink-loop test.
What I ran, on a Linux review host at 095d2fe:
make ci-local: 30 passed, 0 failed, 7 skipped; 12,017 tests; changed-line coverage 98%- Mutation run on the new code (14 mutants): dropping the
<out>.rejectedsibling, the n/a branch,instruction.mdin the protected files, the frontmatter rule, or movingroot.exists()out of thetryeach fail a test. The survivors are theRuntimeErrorentry in the--outexcept tuple anderrors="replace"in_is_task. - Probes: a
task.mdwithout frontmatter next to atask.tomlwith--outon it exits 2 and leaves the file intact; relative--outto<graph>.rejectedexits 2; a non-UTF-8task.mdexits 0.
Checked, no issues
- Commits: all 8 non-merge commits carry
Signed-off-by. - Docs: the changelog now states the n/a exit 1 and the terminal-bench
task.yamlgap;--helpfor--minmatches the code.
Bot points I checked that no longer apply at this head (12)
- Copilot's point on
--outoverwriting an eval suite does not apply: the guard resolves--outagainst the graph and every file the run reads, and exits 2. - Copilot's point on comparing rounded values with
--mindoes not apply: the floor is judged onexact_shares, from unrounded counts. - Copilot's point on
needs: nulldoes not apply: an explicit null is malformed in both readers and exits 2. - exai-bot's point on
--outclobbering an input does not apply: the guard coversevals.json, the graph and every task file. - exai-bot's point on
needs: nulldoes not apply: null is malformed, a missing key isunlabelled. - exai-bot's point on rounding before the
--mincheck does not apply: the gate usesexact_shares. - exai-bot's point on an unhandled
OSErrorfrom--outdoes not apply: the save is wrapped and exits 2. - exai-bot's point on Markdown table cells does not apply:
_cellescapes pipes and flattens line breaks. - exai-bot's point on
instruction.mddoes not apply:instruction.mdis now in the protected task files and--outon it exits 2. - exai-bot's point on a symlink-loop
--outdoes not apply:resolve()errors are caught and exit 2. - exai-bot's point on
root.exists()outside the handler does not apply: it is inside thetryand aPermissionErrorexits 2. - exai-bot's point on the
<out>.rejectedcleanup does not apply: the guard checks that sibling too, and an input there is refused.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to 0f218cd, which fix the nit from my review:
0f218cd:dokimi/tests/test_eval_coverage.py:396: the symlink-loop test now patchesPath.resolveto raiseRuntimeError("Symlink loop")for the--outpath and asserts exit 2 andcannot resolvein stderr, so it reaches theexceptbranch on every Python.
make ci-local at 0f218cd: 30 passed, 0 failed, 7 skipped.
Bring in main at ad734f5 (intel#339, a SCIP code index reader; intel#340, eval coverage against a frozen need graph). Both sides register new artifact kinds at the end of the same lists, which are in insertion order, not sorted. Each conflict keeps both sides, main's entries first: - dokimi/dokimi/artifacts.py: the module docstring names need_graph and eval_coverage and then repo_hits_judged; SCHEMA_VERSIONS lists need_graph.v1 and eval_coverage.v1, then repo_hits_judged.v1. - dokimi/dokimi/runs/kinds.py: FIELDS keeps the need_graph and eval_coverage rows, then the repo_hits_judged row. - dokimi/tests/test_runs_scan.py: EXPECTED keeps the need_graph.v1 and eval_coverage.v1 rows, then the repo_hits_judged.v1 row. The auto-merged docs/reference/cli-interface.json is what scripts/cli_surface.py dump writes for the merged tree, so it needs no change. cli_routes.py, tests/data/cli_flags_allowlist.txt, tests/test_runs_show.py and the contract keep the entries of both sides. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Bring in main at ad734f5 (intel#339, a SCIP code index reader; intel#340, eval coverage against a frozen need graph). No conflicts. CI's schema_compat compares the branch with main, which gained eval_coverage.v1 and need_graph.v1 after this branch's base. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Summary
Adds
dokimi eval coverage. Given a frozen need graph and theneedslabels on an eval set, it reports how much of a subject's problem space the evals cover and how relevant each eval is. It makes no model call, so the same files always give the same report (#331, AC-J07-007 → Partial).need_graph.v1: the file novelty'ssave_graphwrites is valid as is. Nodes carrypath_classandconfirmed_by(userorhypothesis).eval_coverage.v1: weighted coverage, the weights it used, core+rare_critical and rare_critical shares, gaps, per-eval relevance and structured problems.relevant_core,relevant_peripheral,out_of_scope,orphan(needs: []) orunlabelled(noneedskey).needsvalue that is not a list of ids;NeedNode.confirmed_byandNeedGraph.provenancenow survive a save/load round trip, so one need-graph format has one owner.--graph,--outand the repeatable--min [METRIC=]X, which follows the--timeout [PHASE=]SECpattern.Evidence
tests/test_eval_coverage.py: 18 tests, including an outcome-marked gaps/relevance test, determinism, digest stability and a check that both graph readers agree on the fixture.make ci-localgives rc=0.confirmed_by. The per-metric--min-*flags broke the flag-registry pattern.Merge Danger
Low. This PR adds new files and one new command. The only change to existing behaviour is that novelty's
as_dictnow writesconfirmed_byon every node (plusprovenancewhen it is set). Nothing in the repo reads saved need graphs yet.Follow-ups (#331):
needsthroughtask_formats(this also covers the tbenchtask.yamlcase);--outcurrently exits 1, the same code as "coverage below floor".🤖 Generated with Claude Code