CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94

v2.1, 2026-10-25. v2 (10-24) rewrote the 10-19 proposal. It put the evaluation loop from #121 (plan → validate → run → compare → gate → report) at the top level, with the subject and eval groups below it. v2.1 adds three things:

• every point of @antonwolfy's review, now mapped in §8;
• the six persona walkthroughs (§7), folded into §2 and §3;
• a full flag audit with an enforcement plan (§2.4, stage F).

Each decision in §6 now has its context, the options and a recommendation. v1's code audit and experiments E1–E3 (cited below) still hold and are in this issue's edit history (the 10-19 and 10-24 versions).

Why

dokimi has 23 top-level commands, plus the product alias. They hold 68 leaf operations: 64 on 10-19, then repo inspect, repo budget, tasks inspect (#210) and catalog formats (#211). Open PRs add about 7 more. The v1 diagnosis still holds:

• The names describe where code was added, not what a user wants. run is a docs lint. compare diffs lint snapshots. evaluate and eval sit side by side.
• One operation lives in many places. Execution is spread over 8 groups, comparison over 5 and reports over 4.
• The flags are getting worse, not better. On main @ 66cd3d83, the 68 leaves declare 222 distinct option strings (full audit in §2.4):
  • Output is spelled 11 ways. --out (16 leaves) is a file in 13 and a directory in 3. The others are --output 15, --out-json 10, --out-md 8, --output-dir 7, --out-dir 5, --json 10, --format json, --results-dir, --items-out and --no-json.
  • "Exit 1 on a failed check" is spelled five ways, --strict in 7 leaves with 7 different meanings among them.
  • Spend caps are --budget, --max-usd, --max-cost-usd and --max-batch-usd, while --budgets means token fractions.
  • Some names carry different meanings. --max-tokens is retrieval tokens in answers/benchmark but an output ceiling in arms/novelty/tools. --model is the model under test, the extractor or the adjudicator, depending on the leaf. --judge is a model in one place and a boolean in arms run.
  • Open PRs add -o, --answers-out/--eval-out, --trials and a third JSON convention.

Two things changed since v1, and they change the answer:

1. User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 defined the user loop: plan → validate → run → compare → gate → record, with plan.v1/result.v1 (feat(runner): plan.v1 / result.v1 and engine-computed completeness #161), a policy gate (J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99), bundled skills (feat(skills): draft dokimi-design-evals skill #162) and a GitHub Action. v1 had an experiment_plan.v1 and runs gate for the same ideas. Two plan schemas and two gate commands for one concept is the conflict to resolve first.
2. The CLI is already an API.
   • Bundled skills name commands, and a test fails if a named command does not parse (feat(skills): draft dokimi-design-evals skill #162).
   • journeys.yaml resolves its invokes through build_parser() (docs, tests: user-journey catalog (J1–J15) and criterion evidence trace #102).
   • task_inspect.v1 records consumer command names in an artifact (feat(tasks): detect eval-task formats with tasks inspect #210).
   • repo-eval.yml inputs map to flags (ci: reusable repo-eval workflow (validate, scan, tasks check) with dokimi pinned #113), and the workflow parses repo validate --json fields.
   • VISION.md: "Agents are first-class users… The CLI and the library are the interface."

So the question is no longer only "which group does each leaf go in". It is which API an agent, a CI workflow and a person can all drive through the journeys J1–J15.

1. Design goals, taken from VISION.md and the journeys

2. The API

2.1 Shape

dokimi
  ── the loop (any subject; what CI, skills and the Action call) ──────────────
  assess     <path>                 what is this, what is free to check, what is unknown; never spends
  plan       <executor> [flags]     write plan.v1: cells (arm × model × harness) × evals × reps, budget, analysis, policy
  validate   <plan>                 every pre-spend check: schema, inputs, judge validity, comparability, estimate vs cap
  run        <executor> [flags]     plan, then execute into a run directory; or: run --plan FILE; resumes by --name
  compare    <run|baseline:NAME>…   paired comparison; method, unit and contrasts are declared and recorded
  gate       <run|comparison>       apply policy.v1, print a verdict, set the exit code
  report     <run|comparison>…      render saved results (md | json | html | pr-comment | record | badge); no model calls
  runs       list · show · explain · resume · replay · export · import · rm · gc · baseline {save, list, rm}

  ── subjects: cheap and specific diagnostics (J14 direct operations) ─────────
  skill      check · routing · novelty {analyze, atoms, probe, items} · shorten
  repo       fetch · inspect · scan · lint · budget · reach      (+ run · compare · report as shortcuts, §5 L6)
  catalog    check · formats · list · show · intents · routing · trigger-probe
  docs       check · compare

  ── evals: building them and trusting them (J7) ──────────────────────────────
  eval       inspect · import · check · coverage · split {lock, status}
             personas {discover, validate} · questions {generate, analyze, refine, review}
             triggers {generate} · tasks {list}
             judge {validate, wording, agreement}
             score {pointwise, pairwise, panel, checklist, grounding, ragas, reference}

  ── analysis and data ────────────────────────────────────────────────────────
  trace      inspect · consistency
  tools      preference
  data       export · import · schema {list, show} · models {list}

That is 15 roots in four sections. dokimi --help prints the sections in this order; argparse does not group subcommands by itself, so a small help formatter does it. Every leaf's one-line help ends in [free] or [paid].

Why verbs at the top and nouns below:

• The loop is the same for every subject. A skill A/B, an AGENTS.md A/B and a model comparison are all plan, validate, run, compare, gate. If the verbs sat under each noun (repo run, arms run, tasks run), every subject would need its own copy of resume, budget, comparison and gating. That is the code today, with 18 places that pair, diff or test results.
• Diagnostics are not the same across subjects. Novelty needs an atom inventory, reach needs a harness profile, repo scan needs a tree. They belong to the subject group (G7). They stay specialised, but they share the envelope and the flag vocabulary.
• CI, skills and the Action read the loop verbs, not the nouns. User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 says the skills and the Action "never compute a verdict. They call dokimi gate --policy …". With the loop at the top level, that sentence is a literal command.

The positional is the subject, and the subject is what the result is keyed by.

• One skill: skill VERB SKILL, for example skill routing ext/foo --against skills/. This gives one record per skill.
• Several skills, or a property between skills: catalog VERB ROOT, for example catalog routing skills/ (all-vs-all) or catalog trigger-probe skills/. This gives a matrix or an index, plus a record per skill.
• Flags never change what is being evaluated, so v2's --all-vs-all and --catalog flags go away.
• --arm names variants of the subject. It never names the subject.

2.2 plan and run: one plan schema, executors as positionals

dokimi plan  <executor> [executor flags] [shared flags] --out plan.yaml   # write only
dokimi run   <executor> [executor flags] [shared flags] [--name NAME]     # plan + execute
dokimi run   --plan plan.yaml [--name NAME]                               # execute a saved plan
dokimi validate plan.yaml                                                 # check + estimate, no spend

• Executors:
  • arms: the question/answer track. Questions come from --questions FILE, or from the subject's eval set.
  • answers
  • tasks: Harbor, BenchFlow, terminal-bench, SWE-bench.
  • repo: container graders, .agent-eval/.
  • localization: explore.
  • catalog: a composite. Each cell carries its own executor (tasks, routing, trigger-probe). --preset static|live|full picks the cells, and static resolves to free checks only.
  • Recipes that become plans once there is one answering stack: pipeline (evaluate), benchmark, subject.
• plan <x> and run <x> take the same flags. They are built from one argparse parent per executor; E1 showed this can be done without touching handlers. run <x> writes the plan into the run directory before the first call. So every run has a replayable plan.v1, including runs started from flags (G4).
• One schema: plan.v1 from feat(runner): plan.v1 / result.v1 and engine-computed completeness #161. experiment_plan.v1 from v1 is dropped. feat(runner): plan.v1 / result.v1 and engine-computed completeness #161 is extended additively. The existing fields (subjects, suites, models, harnesses, grader, varied_variable, cells, budget) keep their meaning. New:
  • executor, and cells[].executor for composite plans.
  • cells[] as explicit (arm, harness, model, bin) tuples. This is the --cell grammar in §2.4. Repeated --model/--harness still give the full cross.
  • arms: {name: ref}, using the arm grammar in §2.5.
  • budget: {max_usd, max_usd_per_cell, max_cells}.
  • analysis: see §2.10.
  • judges[]: model, role, aggregation, and validity: RUN with a threshold.
  • policy: a path, or an inline policy.v1.
  • splits: source, dev, val or sealed per task set. source is J15's originating task, kept out of scoring. The sealed state lives in the eval split record (§2.3), not in the plan.
  • env: network, delegation, work dir, cache dirs, image, timeouts.
  • dokimi_version, plan_digest: both required.
• The executor is a positional, not --type. E2 settles this: 95 distinct flags across the 10 execution commands, 58 of them unique to one command. The flags a user sees are the executor's own flags plus the shared vocabulary (§2.4).
• Resume.
  • run --plan resumes by default and skips finished cells, as repos/runner.py does today. --force re-runs them.
  • run <x> --name NAME puts the run at RUNS_ROOT/NAME/. Re-running the same command with the same name resumes it. A different plan digest under the same name is an input error, not a silent fork.
  • Resume refuses cells written by another dokimi_version or plan_digest, unless --force is given.
• validate is one place for every check before spending.
  • The plan schema, and that the inputs exist (today benchmark preflight).
  • The eval self-checks (eval check: null, oracle, isolation; later the mutation controls of J7: draft tasks, negative controls and coverage gaps (AC-J07-004, AC-J07-006, AC-J07-007) #97). These take hours on heavy builds, so they are opt-in (--self-check) or reused from an earlier result (--reuse-check check.json).
  • Judge validity: a judged metric whose judge has no validity run, or one calibrated on another model or prompt digest, gives invalid.
  • Comparability against --baseline, when one is given.
  • Sealed splits: refused once burned.
  • The cost estimate: expected_calls, and expected_usd as an upper bound, judge included, as in feat(novelty): estimate-only projects the calls and dollars a run will spend #166. An estimate above the cap gives fail with reason: over_budget, before the first paid call.
  • run <x> --dry-run is the same thing without writing a plan file.

2.3 Run directories, records and baselines

• One directory per run under the runs root: results/ by default, overridden by DOKIMI_RUNS_DIR. A run directory holds:
  • run.json: the envelope (R1)
  • plan.v1.yaml
  • result.v1.json, written even when the run is interrupted (SIGTERM, cap), with status: incomplete
  • the executor's own artifacts
  • Paths inside artifacts are relative to the run directory.
• runs list scans the root and reads schema_version and run_id (--subject, --verdict, --kind filter it). An index.jsonl is only a cache that can be rebuilt (as @antonwolfy proposed). Files written before the envelope existed are listed too.
• Diagnostics write run directories too (skill novelty analyze, repo reach, repo scan, tools preference). So runs list shows the whole history, not only run. --out-dir puts the run directory somewhere else; --out FILE also writes the primary artifact to FILE.
• Reuse instead of paying twice. A command that consumes an earlier result takes --from RUN, for example skill shorten --from $NOVELTY_RUN or skill novelty probe --from $RUN. --cache-dir is a shared flag.
• A baseline is a named pointer to a run, like a git tag. runs baseline save NAME RUN creates one, and compare RUN baseline:NAME resolves it. Any artifact kind can be a baseline.
  • Names can be namespaced per subject, for example skill/a@main.
  • --expect-previous RUN_ID makes the save a compare-and-swap, so two nightlies cannot race on the pointer.
• Baselines survive ephemeral CI runners.
  • runs export NAME|RUN > bundle.tgz and runs import bundle.tgz move a run with its plan and result. The bundle is content-addressed by plan_digest plus the result sha, and import verifies hash and schema. A PR can poison a cached artifact, so import never trusts it blindly.
  • baseline: also accepts a file or a URL, for example a committed baselines/main.json. Storage can then be a git branch, actions/cache, OCI or S3 without dokimi knowing which (User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 Q1 and Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138 Q2 stay open; the API does not depend on them).
• Records for the Action, the catalog and a site or badge (User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 record, Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138 P1.6) are result.v1 plus the envelope.
  • The key is skill_sha (the tree hash of the skill directory, not the repo commit), task_set_hash, the resolved model_id, the harness and its version, and dokimi_version.
  • report RUN --per-subject --format record|badge|site-json --out-dir records/ writes one record per subject. A run over two skills gives two records.
  • A skill that is evaluated upstream gets the record state evaluated_upstream, not not_run.
  • dokimi writes files only; the workflow commits them (Skill catalog integration (intel/skills): per-skill evals in catalog CI, results, promotion to validated #138 P2.12).
• Stateful eval-set records. eval split lock DIR --sealed records a digest. Any run that scores a sealed split marks it burned, and validate refuses to use it again (Skill auto-improvement: measurement gates (A) and improve command (B) #220). eval split status prints the state.
• Housekeeping. Repo batches are gigabytes, so runs rm RUN and runs gc --keep-baselines --older-than 30d exist.

2.4 One flag vocabulary, enforced (dokimi/cli_flags.py; this is #4 wave 5)

Audit. These are all 222 option strings on main @ 66cd3d83, grouped by concept. The generator (scripts/cli_flag_audit.py) ships with F1, and the per-flag listing goes into that PR.

The remaining ~60 flags are specific to one method (for example --max-atoms, --no-reversal, --typed-threads, --salt, --large-bytes). They stay local to their leaf but follow the naming patterns below.

Patterns for local flags.

• A cap is --max-THING.
• "Process only the first N" is --limit N.
• A boolean is --x / --no-x (BooleanOptionalAction).
• A role model is --ROLE-model P/M.
• A per-phase setting is --SETTING [PHASE=]VALUE.
• No local flag may reuse a stem that the registry owns: out, model, judge, usd, budget, token, reps, json, format, strict.

The contract. Every paid leaf takes --dry-run, --max-usd and --json, and its one-line help is labelled [paid]. Every leaf takes --json. The same concept has the same spelling in four places:

• the flag;
• the plan.v1 field;
• the dokimi.api keyword (max_usd);
• the environment variable (DOKIMI_MAX_USD) and the repo-eval.yml / Action input.

There is one precedence rule: the most specific source wins, except for caps, where the lowest cap wins. So a CLI --max-usd can lower plan.budget.max_usd but never raise it.

How the code enforces it (stage F, §5):

1. One registry. dokimi/cli_flags.py holds a FlagSpec per concept: canonical name, type and parser (parse_model_ref, parse_arm, parse_cell, parse_phase_map, parse_usd), nargs, metavar, help, default policy, plan_field, api_kwarg, env, and banned synonyms. Leaves call cli_flags.add(parser, "max_usd"). The created action is tagged with its concept.
2. A contract test walks build_parser(). It fails when:
   • a registry name or a banned synonym was added with a raw add_argument;
   • one name has two types, nargs, choices or concepts across leaves;
   • a banned synonym is anything other than a tombstoned flag (which exits 2 and prints the new spelling);
   • a local flag is not listed in dokimi/cli_flags_local.toml (leaf, flag, one-line reason), or uses a stem the registry owns;
   • a [paid] leaf lacks --dry-run/--max-usd/--json.
3. Parity tests.
   • Every plan.v1 field maps to a registry concept or is marked file_only.
   • The dokimi.api keywords equal the registry api_kwargs.
   • The repo-eval.yml and Action inputs equal the registry names; the test parses the workflow YAML.
   • Every DOKIMI_* environment variable read in the code is in the registry.
4. A snapshot in CI. docs/reference/cli-interface.json (routes, flags, --json schema names) is generated, and cli-interface CI (ci: report user-facing CLI surface changes on every PR #88) fails when it is stale. So every flag change or JSON shape change appears as a diff in the PR and needs a changelog.d fragment. repo-eval.yml reads d["ok"], so a JSON shape change is breaking in the same way a rename is.
5. The docs follow. test_docs_code_refs, the feat(skills): draft dokimi-design-evals skill #162 skills test and the journeys.yaml resolver reject banned spellings in docs, skills, journeys and workflows.
6. A ratchet, not a big bang. The first PR records today's 222 strings as the allowlist, so the test only fails on new drift. Each later PR moves one concept into the registry and deletes its allowlist rows. The test then flips to strict.

2.5 One arm grammar

--arm none                      no added context
--arm skill:PATH[@REV]          a skill bundle, optionally at a git revision   --arm skill-catalog:DIR
--arm guidance:none|raw|rev:REV                                                --arm patch:FILE
--arm docs:SOURCE               context7 | local:PATH | url:URL | mcp:…
--arm placebo:FILE|auto         same-length off-topic control; auto = generated, length-matched, digest recorded
--arm selfgen:CREATOR_DIR       a skill generated by a named skill-creator (J15); its model and cost count toward --max-usd
--arm NAME=REF+REF              combination, named (#137 `guidance:none+skill:…`, #41 composite arms)

The executor decides which refs it supports and refuses the others with a message that lists them. The harness (claude-code, codex, copilot, benchflow:<agent>, docker-oracle) is --harness or a --cell key, separate from the arm. A harness that needs a model of its own family is paired explicitly: --cell harness=codex,model=openai/gpt-x --cell harness=claude-code,model=anthropic/….

2.6 One verdict vocabulary and one exit-code table

• Exit 2 stays the input error. argparse hard-codes it, and cli.py already maps InputError to it. User loop: eval engine contracts (plan/validate/compare/gate/record), 3 skills, GitHub Action #121 proposed 2 for inconclusive, which collides, so inconclusive moves to 3. This matches feat(runner): plan.v1 / result.v1 and engine-computed completeness #161's cell statuses.
• Every command that does work follows the table, not only the commands that issue verdicts. run, resume and replay exit 5 on an infra failure and 3 on a budget stop, even though they issue no pass/fail. compare exits 4 on invalid, so a pipeline stops before gate. An uncaught exception is caught at the top of cli.py and exits 5, so a crash never reads as a regression.
• reason says why.
  • With inconclusive: interval_crosses, sample_cap, budget_stop, incomplete.
  • With invalid: mismatch:FIELD, uncalibrated_judge, burned_split, no_cells.
  • With fail from validate: over_budget.
  • With infra_error: kind: transient|config.
• Several subjects (a catalog run): the exit code is the worst subject verdict, and subjects[] lists each one.
• --json output always carries verdict and status, and exits 0 only for pass. Today a capped judge-validity run prints RESULT: PASS (inconclusive …) and exits 0 (AC-J11-002 gap); under this table it exits 3.

2.7 Machine output: one JSON shape, with next steps

Every command with --json prints exactly one document on stdout, on every exit code including 2 and 5. Logs go to stderr.

{
  "schema_version": "cli_output.v1",
  "kind": "comparison.v1",
  "run_id": "…",
  "status": "complete | partial | budget_stopped | cancelled | failed",
  "verdict": "pass | fail | inconclusive | invalid | infra_error | null",
  "reason": "interval_crosses",
  "error": {"code": "moved | input | no_credentials | infra", "message": "…", "retryable": false, "field": "--arms"},
  "summary": {"contrast": "skill-none", "effect": 0.12, "ci": [0.03, 0.21], "n_units": 40, "unit": "task",
              "method": "cluster-bootstrap", "models": ["anthropic/claude-…@resolved"], "dokimi_version": "0.2.0"},
  "subjects": [{"subject": "skill:skills/a", "verdict": "pass"}],
  "artifacts": {"run_dir": "results/…", "result": "result.v1.json", "report": "report.md"},
  "cost": {"spent_usd": 0.42, "cap_usd": 5.0, "remaining_usd": 4.58, "estimate_kind": "upper_bound", "calls": 118, "priced": true},
  "limits": [{"code": "property_of_pair", "text": "novelty is a property of (skill, model); says nothing about task outcomes"}],
  "next": [{"argv": ["dokimi", "validate", "results/…/plan.v1.yaml"], "why": "estimate before spend",
            "spends": false, "est_usd": 0, "kind": "sequence | alternative"}]
}

• verdict: null means "this command issues no verdict", for example report. A run that measured nothing has status, not a null verdict.
• next is ordered. next[].argv is an array, so paths with spaces survive. est_usd comes from validate's estimator, never from a guess inside a leaf. It is printed in human output too.
• assess hands over its ladder in next, cheapest first: free checks, then a novelty --dry-run, then a run with none, the skill and placebo:auto.
• data schema list|show KIND prints any artifact schema, including cli_output.v1, policy.v1 and the route table cli_routes, so an agent can check what it parses. data models list --json lists the known models, the aliases (sonnet → a pinned ID) and the prices.
• Moved commands are tombstones, not hidden aliases (§5). For example, dokimi novelty analyze … prints moved: dokimi skill novelty analyze … with the argv rewritten, plus error.code: moved and next[0].argv under --json, and exits 2. Tombstones never forward on their own, because forwarding would skip the --dry-run a user would make first. Names that change meaning (run, compare, report) also detect their old argument shapes: run --spec, compare a.json b.json with lint snapshots, report eval FILE. They answer with the tombstone instead of a generic parse error. A name freed in one release is not reused in that same release (§5).

2.8 The library mirrors the loop

The CLI is a thin layer over dokimi.api:

• plan(executor, **settings) -> Plan
• validate(plan) -> Validation
• run(plan) -> Run
• compare(*runs, analysis=None) -> Comparison
• gate(x, policy) -> Verdict
• report(x, fmt) -> str

The keyword names are the registry's api_kwargs (§2.4). Handlers stop passing argparse.Namespace around (#4 wave 5: "a typed run config instead of an argparse.Namespace hand-off", for example subjects.py:138). Tests call dokimi.api instead of build_parser().parse_args([...]) (151 call sites today), so the next rename does not touch them.

2.9 CI and the Action are the loop, too

Rules for CI:

• --max-usd is required when CI=true.
• --out-dir has a stable layout for upload-artifact.
• The concurrency key is the repo plus plan_digest.
• --seed is recorded. LLM runs stay nondeterministic, so gate decides on intervals, never on an exact replay.
• The inputs of repo-eval.yml and the proposed skill-eval.yml are registry names (§2.4), and the workflow inputs are versioned as their own contract.

2.10 Statistics are declared, not chosen silently

The code has four comparison methods today:

• McNemar in task_runs_report.py; binary_stats.py:91 warns that it assumes independent trials;
• a task-level sign-flip test in repos/compare.py;
• Holm/BH with --pool-replicates in model_compare;
• judge-score bootstraps.

A compare that dispatches on schema_version alone would pick among them without saying so. With 5 reps, McNemar on trials counts n five times over. So:

• plan.v1.analysis declares:

  • contrasts, for example skill-none and skill-placebo, each confirmatory or exploratory;
  • unit, default task (the cluster);
  • method (auto, mcnemar, sign-flip, cluster-bootstrap, wilcoxon);
  • correction (holm, bh, none) and family (per-model or all);
  • metric, and pass_k (k, and whether it is per task or per trial);
  • stopping (fixed n, or a sequential rule with alpha spending);
  • blocking and randomization (seeded run order, judge position order, judge blinding to arm names).

• compare reads the plan's analysis by default. --method, --unit, --contrast, --baseline-arm, --correction and --metric override it, and an override marks the result exploratory. Under auto, the method chosen is printed. Trial-level pairing with reps > 1 is refused unless --unit trial is given.

• comparison.v1 records the resolved method, unit, n_units, whether reps were pooled, the family size, raw and adjusted p, the CI method, and each judge with its validity run.

• policy.v1 is versioned now (J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99). Each rule has:

  • a contrast id;
  • a statistic (the CI lower bound, not the point estimate);
  • a margin (superiority or non-inferiority);
  • min_n_units;
  • how inconclusive maps to block or warn.

  "Beat a placebo" is a rule (require: skill > placebo). There is a built-in --policy helps: skill > none and skill > placebo, on the lower bound. data schema show policy prints the schema.

• Adaptive stopping is not optional stopping. A budget stop without a declared sequential rule is inconclusive (reason: budget_stop), never a pass.

3. The journeys under this API

Commands marked new need new code; the rest are moves of existing commands. The journey pages show the short path (about six commands, no plan file); the reference shows everything.

Every journey uses the same loop verbs. What differs between journeys is the subject group and the executor, as the vision says: "Novelty, routing, repository scans, tool preference and whatever comes next are ways to get sharper or cheaper answers to the same question."

4. Full mapping

No path moves twice. The New column is final for every row. v1's low-level forms (runs compare plugins, experiment recipe evaluate) are gone. The recipes run pipeline|benchmark|subject are permanent names: their implementation changes once there is one answering stack (X), but their names do not. This settles @antonwolfy's M4 concern.

5. Plan

Conventions from #4 still apply: one purpose per PR; renames, refactors and number-changing changes are never mixed; every PR that touches cli.py or a flag adds a changelog.d fragment.

Migration policy: tombstones for one minor release. v0.1.0 is a GitHub release with no PyPI upload, and no consumer outside this repo was found. The one pinned caller, oneDAL #3818, pins repo-eval.yml by SHA, so it is unaffected until it bumps the pin. @antonwolfy proposed renaming with no deprecation before 0.1.0, but 0.1.0 has since shipped. So each moved path parses its old argv, prints the new command with the argv rewritten, and exits 2 (§2.7). That still gives agents and scripts a message they can act on. Tombstones are removed one minor release later. A name is not reused in the release that frees it: run, compare and report get their loop meaning only in the release after the docs lint, the snapshot diff and the old reports have moved out.

Order:

1. A1, A3 and F1a now.
2. A2 as the full audit.
3. M1–M3 after A1/F1a.
4. F1b concept by concept, as Architecture cleanup: fix live bugs, then deepen the LLM, eval-record, novelty, arms and process modules #4 wave 5.
5. R1–R3; R4 after wave 4.
6. L1 with feat(runner): plan.v1 / result.v1 and engine-computed completeness #161; L2–L6.
7. N.

6. Decisions

Settled by the v1 review (please object here if not):

Open for this round. Each has the context, the options, and a recommendation. The body above is written as if every recommendation is accepted, so a "no" here names the sections to revert. Votes from the persona walkthrough (§7) are in brackets.

D1. Where the loop verbs live.
Context: #121, the skills and the Action already spell dokimi plan|validate|compare|gate. v1 put them under experiment/runs. Every journey in §3 passes through them.
Options: (a) top-level roots; (b) an experiment group; (c) top level, but each verb lands only when a real dispatcher exists behind it.
Recommendation: (c), as in §5 L. [6/6 for top level. The repo maintainer and the researcher make it conditional on real dispatchers.]

D2. One plan schema, and how big.
Context: v1 proposed experiment_plan.v1 beside #161's plan.v1. The personas found v2's five added fields too few. Catalog plans need a per-cell executor. Repo runs need (harness, model, bin) cells and two caps. Statistics need a declared analysis. CI needs a version and a digest.
Options: (a) extend plan.v1 with the five fields only; (b) keep two schemas; (c) extend plan.v1 with everything in §2.2.
Recommendation: (c). This needs #161's author and reviewer to agree. [6/6 for one schema; 4 personas found (a) too small.]

D3. Exit codes.
Context: argparse uses 2 for usage errors, so #121's 0–4 cannot hold. CI must tell "regression" from "crashed" from "not enough data".
Options: (a) 0 pass, 1 fail, 2 input, 3 inconclusive, 4 invalid, 5 infra_error, on verdict commands only; (b) the same table on every command that runs work, with crashes mapped to 5 and reason codes; (c) #121's 0–4.
Recommendation: (b), as in §2.6. Under (a) a crash exits 1 and blames the PR, and run exits 0 on a Docker failure. [6/6 for 3/4/5; the CI engineer and the coding agent required (b).]

D4. Tombstones or working aliases.
Context: there are no installed users, and the one external pin is a SHA. run, compare and report change meaning, so a tombstone cannot catch an old call to them.
Options: (a) tombstones for moved paths; (b) aliases for one release; (c) tombstones, plus detection of the old argument shapes, error.code: moved in JSON, and no reuse of a freed name in the same release.
Recommendation: (c), as in §2.7 and §5. The repo shortcuts of L6 are the one deliberate exception, per Q4. [6/6 for tombstones.]

D5. assess as a root.
Context: J1 is the only journey with no entry today, and agents and new users reach for it first. It is new code. The maintainer persona sees little value for users who already know their subject.
Options: (a) root, after L2; (b) root, early, free-only, with a cheapest-first next ladder; the costed draft plan after L2; (c) defer, ship runs explain first.
Recommendation: (b). [5 yes, 3 of them "later"; 1 defer.]

D6. catalog as a root, and what it owns.
Context: #138 treats a skill collection as a subject. v2 gave catalog only list/check, and hid the catalog-wide operations behind flags elsewhere.
Options: (a) a root with list/check only; (b) data catalog; (c) a root, with the rule that the positional is what the result is keyed by: catalog routing, catalog trigger-probe, plan catalog, and skill routing SKILL --against ROOT.
Recommendation: (c), as in §2.1. [5 yes, 1 no view.]

D7. cli_output.v1 on every leaf.
Context: "agents are first-class users", and today each command prints its own JSON shape. Under v2's envelope, the coding agent could not answer "does it help?" from stdout.
Options: (a) v2's envelope; (b) minimal, without next; (c) the envelope in §2.7, with status, reason, error, summary, subjects[], cost.remaining_usd, next[] as argv, and emitted on every exit code.
Recommendation: (c). [6/6 yes; the maintainer warns that est_usd must come from one estimator, which §2.7 now requires.]

D8. Statistics in compare and gate.
Context: there are four methods in the code, and a dispatcher that picks among them silently. With 5 reps, McNemar on trials counts n five times over.
Options: (a) dispatch silently; (b) flags only; (c) declared in plan.v1.analysis, recorded in comparison.v1, the task as the default unit, trial-level pairing refused with reps, overrides flagged exploratory, and policy.v1 versioned now.
Recommendation: (c), as in §2.10.

D9. Baselines on ephemeral runners.
Context: a baseline pointer under results/ disappears with the runner, and two nightlies can race on it.
Options: (a) leave storage to the user; (b) runs export/import bundles that are content-addressed and verified, baseline: as a file or URL, and save --expect-previous; (c) a dokimi-hosted store.
Recommendation: (b), as in §2.3. Where records live stays open (#121 Q1).

D10. How strictly flags are enforced.
Context: the audit in §2.4 shows 222 option strings. Three names (--max-tokens, --out, --model) carry different meanings, and --judge has two types. v2 said only that CI "can check" the vocabulary. Without enforcement, every new command adds a spelling: open PRs already add -o, --trials and a third JSON convention.
Options:

• (a) review only: a written vocabulary and reviewers catch drift;
• (b) a lint that warns on unknown flags;
• (c) a registry plus a contract test, in ratchet mode first (today's state allowed, new drift fails), then strict once F1b lands; with parity tests across the flag, plan.v1, dokimi.api, the env vars and the workflow inputs, and the snapshot diff in ci: report user-facing CLI surface changes on every PR #88's CI;
• (d) strict from the first PR, migrating all 222 strings at once.

Recommendation: (c). (a) has already failed: the drift happened under review. (d) mixes renames with each other, against #4's one-purpose rule, and blocks every open PR at once.

7. Persona walkthrough (2026-10-25)

Six agents, each playing one user, wrote out their journey with the v2 commands: skill author (J2/J15), repo maintainer (J3/J4), cold coding agent (J1/J14), CI engineer (J12/J13), eval researcher (J7/J8) and catalog maintainer (#138). They agreed on the shape (all six voted for D1–D4 and D7). Every one of them got stuck where a loop verb was named but the flags or fields that make it answer something were not. Each finding below is now in the body:

Kept open for M1 review: the researcher's rename of eval score/eval judge to eval score --mode … --judge … (several judges = a panel) and eval grader {check, validate, agreement, wording}, covering deterministic and LLM graders under one noun. It is a good fit but not a blocker, and it changes 10 leaf names.

Pushed back on: the maintainer suggested dropping next from most leaves, because the costs would drift. §2.7 keeps next but requires every estimate to come from validate's estimator.

8. @antonwolfy's review: where each point went

Definition of done

• Every operation in §4 is reachable at its new path. Old paths are tombstones for one minor release, then removed. repo run/compare/report are shortcuts over the same handlers.
• dokimi --help shows the 15 roots in four sections, with [free]/[paid] labels, and the generated CLI reference matches.
• Every run writes a run directory with run.json, plan.v1 and result.v1, including interrupted runs. runs list|show|replay|export|import work over every run kind. compare, gate and report dispatch on schema_version, and comparison.v1 records the method and unit.
• The flag contract test is strict: every shared concept comes from cli_flags.py, every local flag is in the reviewed list, one name has one meaning, and the parity tests (plan.v1, dokimi.api, env, workflow inputs) pass. cli-interface.json is current.
• --json emits cli_output.v1 on every leaf and every exit code.
• Every command that runs work follows the §2.6 table, no crash exits 1, and no inconclusive exits 0.
• The J1–J15 docs pages use only the new commands and show the short path, and test_docs_code_refs plus the journeys.yaml resolver pass.