eval coverage: coverage of a skill's problem space and relevance of each eval, reproducible (need graph + optional interview) #331

What this is for

design-evals (J7) separates four claims about an eval set: valid, correctly graded, relevant, covering. dokimi checks the first two. Nothing measures the last two. The coverage report in data/skills/dokimi-design-evals/references/coverage-report.md is written by hand. python -m dokimi.coverage gives only a product × awareness/work task count. AC-J07-007 (#97) asks for a report that lists the areas an eval set has no task for. #94 already reserves eval coverage for it.

This issue proposes how to measure eval coverage of a skill's problem space and the relevance of each eval to that space, so that the same inputs always give the same numbers. It also reports a pilot on the 10 intel/skills skills that ship evals/evals.json.

Model

Problem space = the need graph from dokimi/novelty/needs.py. build_need_graph(name, description) never sees the artifact body. This is the property that matters here: the skill cannot define both the problem and its own relevance to it. Nodes carry path classes core, branch, rare_critical, supporting and out_of_scope, with weights from PATH_WEIGHTS. The out_of_scope nodes are the negative controls.

flowchart LR
  D[name + description] --> G[need graph, k draws]
  G --> F[frozen need_graph.json<br/>digest, model, prompt version]
  I[optional user interview] -->|confirm / add / drop / relabel| F
  E[evals] --> M[eval → node map<br/>metadata.needs, or LLM draft]
  F --> M
  M --> R[coverage + relevance report<br/>pure function of the files]

1. Freeze the graph. Generate it once with k draws, write need_graph.json (graph, digest, model, prompt version, draws) next to the suite, and commit it. Later runs read the file and do not regenerate it.
2. Optional interview. Show the user the graph and ask four questions: which nodes never come up in their work, which needs are missing, which nodes are mislabelled, and which nodes are rare_critical. Each node gets confirmed_by: user | hypothesis. Without an interview every node stays hypothesis, and the report says so. This is the "hypothesis for the user to confirm" step that design-evals already describes, made into an input.
3. Map evals to nodes. Use one of two layers:
   • Deterministic: metadata.needs: [node ids] on a task, or needs on an evals.json row. These are read the same way coverage.py reads tags.
   • Draft: an LLM proposes needs for an unlabelled eval. Proposals go through DOKIMI_RESPONSE_CACHE on a fixed draw, are written as draft, and become deterministic once a person accepts them.
4. Report. The report is computed from the files only, with no model call:
   • weighted coverage over in-scope nodes, plus coverage of core + rare_critical and of rare_critical alone. draft mappings and draft scenarios are counted separately and never add to coverage;
   • the uncovered core and rare_critical nodes, listed as gaps;
   • the relevance class of each eval: relevant_core, relevant_peripheral, out_of_scope (maps only to out-of-scope nodes) or orphan (maps to nothing);
   • a node × design-evals axis matrix (happy path, branches, recovery, ambiguity, constraints, negative, neighbouring), which replaces the hand-written table;
   • optional: join with a novelty run, so nodes that are still_missing or degraded and have no eval come first.

CLI (fits #94 v2.1)

dokimi eval coverage DIR --subject skill:PATH [--graph need_graph.json] [--json]
    # free: reads the frozen graph and the needs labels; exit 1 if a --min-* floor fails
dokimi eval coverage graph --subject skill:PATH --draws 3 --out need_graph.json --max-usd 1 --dry-run
    # costed: builds the graph; prints the spread over the draws
dokimi eval coverage interview need_graph.json [--answers answers.yaml]
    # interactive, or replays a recorded answers file; writes confirmed_by
dokimi eval coverage suggest DIR --graph need_graph.json --out needs.draft.json --max-usd 1
    # costed: drafts eval → node labels; never edits the tasks

Flags: --min-coverage F, --min-rare-critical F (gate floors, off by default), and the shared --model, --max-usd, --dry-run and --json from cli_flags.py. Output follows the §2.7 JSON shape, with next pointing to suggest / interview when there are gaps or hypotheses.

Pilot (2026-10-31, intel/skills e04536c, claude-sonnet-5-5, $1.33 total)

10 skills, 3 evals each, 3 graph draws per skill. Script and raw output: available on request (scratch, not committed).

Tasks

• T1 Schema for need_graph.v1.json: graph, digest, model, prompt version, draws, per-node confirmed_by, and the spread of coverage over the draws.
• T2 needs labels: read metadata.needs (Harbor/terminal-bench task.toml/task.md) and needs on evals.json rows; validate the ids against the graph.
• T3 eval coverage DIR: the free report (coverage, gaps, relevance per eval, axis matrix), JSON + Markdown, --min-* floors, exit codes per CLI exit codes: audit every command against the 0/1/2 rule (#94 A2) #275.
• T4 eval coverage graph: k draws with draw indices, cache, --max-usd, write the frozen graph with the spread.
• T5 eval coverage suggest: draft labels under a stricter mapper. It must name one primary node per eval, and secondary nodes only when the eval's check fails if that need goes wrong. Measure nodes per eval and the near-neighbour control before and after.
• T6 eval coverage interview: interactive questions plus a replayable answers.yaml, so the interview is reproducible too.
• T7 Negative-control self-test: plant far and near foreign evals and require orphan / out_of_scope for the far ones. This becomes a fixture test (no model calls, recorded replies).
• T8 Join with novelty still_missing / degraded nodes (optional section).
• T9 design-evals skill: replace "write it by hand" in coverage-report.md with the command, and keep the rule that draft covers nothing.
• T10 Pilot with real interviews on 2–3 intel/skills skill owners; compare the interview-confirmed graph with the description-only graph.

Open questions

• A graph built from the description inherits a narrow description. Should eval coverage graph also accept user-provided sources (issues, traces) as extra inputs, recorded in the graph, while still never reading the skill body?
• Gate on coverage at all, or report only? The proposal: report only until T5 lowers the nodes-per-eval inflation.

Related: #97 (AC-J07-007, mutation controls), #94 (eval coverage in the v2.1 map), #121 (user loop), #98 (novelty joined to task effect), #220 (auto-improvement measurement gates).