Repository navigation
Conversation
M1) The evaluation commands now live under `dokimi eval`: personas, questions, the eval leaves of tasks, catalog and repo, and the `eval` leaves grouped into score, judge, questions, personas and triggers. The command functions are unchanged; the parsers and their registration move. `questions panel-review` is now `questions review`. Every old path is a tombstone: it prints `moved: dokimi <new argv>` to stderr and exits 2 without running. The table is in cli_routes.TOMBSTONES and tests/test_cli_eval_moves.py covers all 21 moves. Refs intel#94, intel#276 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): - removed command `dokimi catalog trigger-set`
- option `--catalog`
- option `--format`
- option `--help/-h`
- option `--model`
- option `--out`
- option `--provider`
- option `--questions`
- option `--rewrite`
- option `--strict`
- option `--target`
- option `--trigger-set`
- removed command `dokimi eval checklist`
- option `--answers`
- option `--arms`
- option `--baseline-arm`
- option `--concurrency`
- option `--help/-h`
- option `--judge-model`
- option `--judge-provider`
- option `--judge-temperature`
- option `--output`
- option `--questions`
- removed command `dokimi eval cross-judge`
- option `--answers`
- option `--baseline-arm`
- option `--both`
- option `--checklist`
- option `--concurrency`
- option `--grounded`
- option `--help/-h`
- option `--judge-temperature`
- option `--judges`
- option `--no-stored`
- option `--output`
- option `--pairwise`
- option `--product`
- option `--questions`
- option `--require-validated`
- option `--strict`
- option `--validity`
- removed command `dokimi eval grounding`
- option `--answers`
- option `--help/-h`
- option `--output`
- option `--threshold`
- removed command `dokimi eval judge-validity`
- option `--answers`
- option `--bases`
- option `--concurrency`
- option `--context`
- option `--escalate-model`
- option `--escalate-provider`
- option `--escalate-ties`
- option `--help/-h`
- option `--judge-model`
- option `--judge-provider`
- option `--judge-temperature`
- option `--max-bases`
- option `--min-accuracy`
- option `--mode`
- option `--output`
- option `--product`
- option `--scorer`
- option `--seed`
- option `--sequential`
- option `--strict`
- option `--typed-max-len`
- option `--typed-threads`
- option `--variants`
- removed command `dokimi eval judge-wording`
- option `--answers`
- option `--bases`
- option `--concurrency`
- option `--help/-h`
- option `--judge-model`
- option `--judge-provider`
- option `--judge-temperature`
- option `--min-accuracy`
- option `--output`
- option `--product`
- option `--redraws`
- option `--seed`
- option `--strict`
- option `--variants`
- option `--wordings`
- removed command `dokimi eval pairwise`
- option `--answers`
- option `--arms`
- option `--baseline-arm`
- option `--concurrency`
- option `--escalate-model`
- option `--escalate-provider`
- option `--escalate-ties`
- option `--help/-h`
- option `--judge-model`
- option `--judge-provider`
- option `--judge-temperature`
- option `--output`
- option `--product`
- removed command `dokimi eval panel-score`
- option `--answers`
- option `--concurrency`
- option `--help/-h`
- option `--judge-configs`
- option `--judge-temperature`
- option `--limit`
- option `--model`
- option `--output`
- option `--product`
- option `--provider`
- option `--roles`
- removed command `dokimi eval ragas`
- option `--answers`
- option `--help/-h`
- option `--ragas-model`
- option `--ragas-provider`
- removed command `dokimi eval reference`
- option `--answers`
- option `--concurrency`
- option `--help/-h`
- option `--judge-model`
- option `--judge-provider`
- option `--no-judge`
- option `--output/-o`
- option `--questions`
- removed command `dokimi personas`
- option `--help/-h`
- removed command `dokimi personas approve`
- option `--file`
- option `--help/-h`
- removed command `dokimi personas discover`
- option `--count`
- option `--description`
- option `--github-token`
- option `--help/-h`
- option `--model`
- option `--output`
- option `--product`
- option `--provider`
- option `--repo`
- option `--save-analysis`
- removed command `dokimi questions`
- option `--help/-h`
- removed command `dokimi questions analyze`
- option `--concurrency`
- option `--help/-h`
- option `--model`
- option `--output`
- option `--product`
- option `--provider`
- option `--questions`
- removed command `dokimi questions generate`
- option `--context7-id`
- option `--count`
- option `--doc-source`
- option `--embeddings`
- option `--help/-h`
- option `--model`
- option `--output`
- option `--personas`
- option `--product`
- option `--provider`
- option `--top-k`
- option `--topics`
- option `--validate`
- removed command `dokimi questions import`
- argument `path`
- option `--format`
- option `--grader`
- option `--help/-h`
- option `--output/-o`
- option `--scorer`
- option `--task`
- removed command `dokimi questions panel-review`
- option `--concurrency`
- option `--help/-h`
- option `--limit`
- option `--model`
- option `--output`
- option `--product`
- option `--provider`
- option `--questions`
- option `--reviewers`
- removed command `dokimi questions refine`
- option `--dry-run`
- option `--fill-gaps`
- option `--fill-model`
- option `--fill-provider`
- option `--help/-h`
- option `--output`
- option `--product`
- option `--questions`
- option `--sim-threshold`
- option `--target-advanced`
- option `--target-beginner`
- option `--target-intermediate`
- removed command `dokimi repo tasks check`
- argument `source`
- option `--allow-unsupported`
- option `--arm`
- option `--build-network`
- option `--build-timeout`
- option `--cache`
- option `--cache-dir`
- option `--cache-root`
- option `--docker`
- option `--grade-timeout`
- option `--help/-h`
- option `--image`
- option `--jobs/-j`
- option `--manifest`
- option `--manifest-file`
- option `--out`
- option `--rebuild-image`
- option `--source-dir`
- option `--task`
- option `--warm`
- removed command `dokimi tasks inspect`
- argument `path`
- option `--help/-h`
- option `--json`
- option `--out`
- removed command `dokimi tasks list`
- option `--help/-h`
- option `--tasks-root`
+ added command `dokimi eval check`
+ argument `source`
+ option `--allow-unsupported`
+ option `--arm`
+ option `--build-network`
+ option `--build-timeout`
+ option `--cache`
+ option `--cache-dir`
+ option `--cache-root`
+ option `--docker`
+ option `--grade-timeout`
+ option `--help/-h`
+ option `--image`
+ option `--jobs/-j`
+ option `--manifest`
+ option `--manifest-file`
+ option `--out`
+ option `--rebuild-image`
+ option `--source-dir`
+ option `--task`
+ option `--warm`
+ added command `dokimi eval import`
+ argument `path`
+ option `--format`
+ option `--grader`
+ option `--help/-h`
+ option `--output/-o`
+ option `--scorer`
+ option `--task`
+ added command `dokimi eval inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
+ option `--out`
+ added command `dokimi eval judge`
+ option `--help/-h`
+ added command `dokimi eval judge agreement`
+ option `--answers`
+ option `--baseline-arm`
+ option `--both`
+ option `--checklist`
+ option `--concurrency`
+ option `--grounded`
+ option `--help/-h`
+ option `--judge-temperature`
+ option `--judges`
+ option `--no-stored`
+ option `--output`
+ option `--pairwise`
+ option `--product`
+ option `--questions`
+ option `--require-validated`
+ option `--strict`
+ option `--validity`
+ added command `dokimi eval judge validate`
+ option `--answers`
+ option `--bases`
+ option `--concurrency`
+ option `--context`
+ option `--escalate-model`
+ option `--escalate-provider`
+ option `--escalate-ties`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge-temperature`
+ option `--max-bases`
+ option `--min-accuracy`
+ option `--mode`
+ option `--output`
+ option `--product`
+ option `--scorer`
+ option `--seed`
+ option `--sequential`
+ option `--strict`
+ option `--typed-max-len`
+ option `--typed-threads`
+ option `--variants`
+ added command `dokimi eval judge wording`
+ option `--answers`
+ option `--bases`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge-temperature`
+ option `--min-accuracy`
+ option `--output`
+ option `--product`
+ option `--redraws`
+ option `--seed`
+ option `--strict`
+ option `--variants`
+ option `--wordings`
+ added command `dokimi eval personas`
+ option `--help/-h`
+ added command `dokimi eval personas discover`
+ option `--count`
+ option `--description`
+ option `--github-token`
+ option `--help/-h`
+ option `--model`
+ option `--output`
+ option `--product`
+ option `--provider`
+ option `--repo`
+ option `--save-analysis`
+ added command `dokimi eval personas validate`
+ option `--file`
+ option `--help/-h`
+ added command `dokimi eval questions`
+ option `--help/-h`
+ added command `dokimi eval questions analyze`
+ option `--concurrency`
+ option `--help/-h`
+ option `--model`
+ option `--output`
+ option `--product`
+ option `--provider`
+ option `--questions`
+ added command `dokimi eval questions generate`
+ option `--context7-id`
+ option `--count`
+ option `--doc-source`
+ option `--embeddings`
+ option `--help/-h`
+ option `--model`
+ option `--output`
+ option `--personas`
+ option `--product`
+ option `--provider`
+ option `--top-k`
+ option `--topics`
+ option `--validate`
+ added command `dokimi eval questions refine`
+ option `--dry-run`
+ option `--fill-gaps`
+ option `--fill-model`
+ option `--fill-provider`
+ option `--help/-h`
+ option `--output`
+ option `--product`
+ option `--questions`
+ option `--sim-threshold`
+ option `--target-advanced`
+ option `--target-beginner`
+ option `--target-intermediate`
+ added command `dokimi eval questions review`
+ option `--concurrency`
+ option `--help/-h`
+ option `--limit`
+ option `--model`
+ option `--output`
+ option `--product`
+ option `--provider`
+ option `--questions`
+ option `--reviewers`
+ added command `dokimi eval score checklist`
+ option `--answers`
+ option `--arms`
+ option `--baseline-arm`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge-temperature`
+ option `--output`
+ option `--questions`
+ added command `dokimi eval score grounding`
+ option `--answers`
+ option `--help/-h`
+ option `--output`
+ option `--threshold`
+ added command `dokimi eval score pairwise`
+ option `--answers`
+ option `--arms`
+ option `--baseline-arm`
+ option `--concurrency`
+ option `--escalate-model`
+ option `--escalate-provider`
+ option `--escalate-ties`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge-temperature`
+ option `--output`
+ option `--product`
+ added command `dokimi eval score panel`
+ option `--answers`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-configs`
+ option `--judge-temperature`
+ option `--limit`
+ option `--model`
+ option `--output`
+ option `--product`
+ option `--provider`
+ option `--roles`
+ added command `dokimi eval score pointwise`
+ option `--answers`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge-temperature`
+ option `--output`
+ option `--product`
+ option `--run-id`
+ added command `dokimi eval score ragas`
+ option `--answers`
+ option `--help/-h`
+ option `--ragas-model`
+ option `--ragas-provider`
+ added command `dokimi eval score reference`
+ option `--answers`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--no-judge`
+ option `--output/-o`
+ option `--questions`
+ added command `dokimi eval tasks`
+ option `--help/-h`
+ added command `dokimi eval tasks list`
+ option `--help/-h`
+ option `--tasks-root`
+ added command `dokimi eval triggers`
+ option `--help/-h`
+ added command `dokimi eval triggers generate`
+ option `--catalog`
+ option `--format`
+ option `--help/-h`
+ option `--model`
+ option `--out`
+ option `--provider`
+ option `--questions`
+ option `--rewrite`
+ option `--strict`
+ option `--target`
+ option `--trigger-set`
~ `dokimi eval score` subcommand required: None -> True
~ `dokimi eval score` parameters changed:
- removed option `--answers`
- removed option `--concurrency`
- removed option `--judge-model`
- removed option `--judge-provider`
- removed option `--judge-temperature`
- removed option `--output`
- removed option `--product`
- removed option `--run-id`Updated on every push; the |
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 116 selected item(s). Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #325 +/- ##
=======================================
Coverage 94.05% 94.05%
=======================================
Files 297 297
Lines 38018 38033 +15
Branches 6424 6426 +2
=======================================
+ Hits 35756 35771 +15
- Misses 1452 1453 +1
+ Partials 810 809 -1
🚀 New features to boost your workflow:
|
…m1-eval Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/benchmarking-and-comparison.md
The top-level CLI table still named personas and questions as live roots. Both are tombstones now (they exit 2 with a moved line), so list them as eval personas and eval questions. Thread: PRRT_kwDOVBg7g86yOm97 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
eval tasks list --tasks-root <missing> is refused with exit 2 and a "not a directory" line, not a silent success. The empty-listing trap applies only to an existing empty directory. Thread: PRRT_kwDOVBg7g86yOm9z Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
eval score is a live group, so the moved-and-exit-2 rule does not cover bare eval score: it exits 2 with a usage error, and eval score --help lists the leaves. Thread: PRRT_kwDOVBg7g86yOm9x Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The stale bare eval score reference in the methodology (judge-independence warning), benchmarking and runs docs now names eval score pointwise. Surface lists that named the tombstoned questions and personas roots now name eval questions and eval personas. Thread: PRRT_kwDOVBg7g86yOm94 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The new "see eval inspect" line in import_path had no test (Codecov patch: 1 line missing). A file no adapter recognises now has a test that names the error and the path. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| hits, so on those queries it is stricter than the `skill-catalog:` arm; `skill-creator --target <skill>` (a catalog skill) writes an | ||
| [anthropics/skills skill-creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator) | ||
| eval set; `questions` writes `questions.v2` with `expected_skill` for the arm below. | ||
| eval set; `eval questions` writes `questions.v2` with `expected_skill` for the arm below. |
There was a problem hiding this comment.
Nit: questions here is an output-format value, not the retired CLI root. eval triggers generate still accepts --format questions (alongside reach and skill-creator); --format 'eval questions' is invalid, and the eval questions group does not perform this export. Keep the format name unchanged, preferably spelling it as --format questions.
Suggestion:
| eval set; `eval questions` writes `questions.v2` with `expected_skill` for the arm below. | |
| eval set; `--format questions` writes `questions.v2` with `expected_skill` for the arm below. |
There was a problem hiding this comment.
Right, and thanks for the catch. eval questions has only the generate, analyze, refine and review leaves (dokimi eval questions --help), so it does not write questions.v2 with expected_skill. The export is eval triggers generate --format questions: --format {reach,skill-creator,questions} is in its help, and catalog.py _write_trigger_set calls save_questions_artifact(ts.to_questions(...)) for it.
Fixed in d58f125 with the line spelled as you suggested, plus the full command name so the flag cannot be read as belonging to the bullet's skill-creator item.
…s group `eval questions` has generate, analyze, refine and review leaves only; the questions.v2 export with expected_skill is `eval triggers generate --format questions`. Thread: PRRT_kwDOVBg7g86yT_9o Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at d58f125. The move itself is right. Each of the 21 old paths prints moved: dokimi <new argv> and exits 2 without running. Each new path is a leaf with the same options as the old one. No code, test, bundled skill or workflow still calls an old path. What is left is wording (help text, the changelog, a few docs) and one routing case: a typo after eval score. Both are in the line comments.
What I ran, at d58f125:
make ci-local --offline: 0 failed, 19 passed, 13 skipped (the network and Docker steps).- The CLI snapshot of the base against this head: the only leaves removed are the 21 old paths, and the only leaves added are their 21 new paths. For each pair the options are the same (names, defaults, choices, required). No other leaf changed.
- A mutation run: I changed the tombstone exit code from 2 to 97. Only the 50 tombstone tests in
test_cli_eval_moves.pyandtest_cli_routes.pyfail; 12,002 pass. So no other test still calls an old path and passes only because that path exits 2. - The old and new paths by hand:
eval scorebare, with--help, with--answers a.jsonand with a typo;personas --help;repo tasks check .. I also parsed therepo-eval.ymlstep's argv (eval check . --manifest --task --out --warm --allow-unsupported): it reachescmd_repo_tasks_checkwith the same flags.
Checked, no issues
- Tombstones: they follow #94 D4 and its migration policy: name the new command, run nothing, exit 2.
eval scorefollowed by a leaf, bare, or with only--helpis not caught by its own tombstone. task_inspect.v1labels:tasks inspectand this schema are not in 0.1.0, so no released output has the oldconsumerslabels. The old enum values are kept only so thatschema_compatagainst the base passes, and nothing in dokimi reads the labels back.- Dispatch: no code reads the subcommand
destnames that changed, and every new group requires a leaf. - The bot's threads: all five are fixed at this head, including the open one on
--format questions(evaluating-treatments.md:553).
| # A group that stays under its own name (`eval score` -> `eval score pointwise`) keeps its | ||
| # help and its usage error when it has no leaf; any other argument is an old call. | ||
| if tomb.new[:n] == _canonical(tomb.old) and all(a in ("-h", "--help") for a in argv[n:]): | ||
| continue |
There was a problem hiding this comment.
A typo after eval score is reported as a moved command.
dokimi eval score pointwse --answers a prints moved: dokimi eval score pointwise pointwse --answers a and exits 2. eval score data.json does the same. The old eval score took only options, so a plain word right after eval score is never an old call, and the suggested argv is not a command either. argparse's own invalid choice: 'pointwse' (choose from pointwise, ragas, ...) is the useful message here.
Suggestion: skip a group that stays in place unless the next argument is an option:
if tomb.new[:n] == _canonical(tomb.old) and (
all(a in ("-h", "--help") for a in argv[n:]) or not argv[n].startswith("-")
):
continueI tried this. test_cli_eval_moves.py and test_cli_routes.py pass, except test_an_old_path_exits_2_and_names_the_new_argv[eval score], which feeds eval score data.json. Give that test an option tail (--answers a.json) instead.
There was a problem hiding this comment.
Fixed in 7dd04cf (head is now 8d4104d). eval score pointwse gets argparse's invalid-choice message, not the moved line, and an option after the group is still an old call. I took your condition and added a red-then-green test (test_a_typo_after_a_group_that_stays_is_not_an_old_call: red on the parent tree). The moved-path test for eval score now feeds --answers a.json, as you suggested.
|
|
||
| | Before | After | | ||
| |---|---| | ||
| | `tasks inspect` | `eval inspect` | |
There was a problem hiding this comment.
33 other fragments in changelog.d/ still name the old paths, and they go into the same release notes as this one.
The PR description calls them history, but they have not been released yet. scriv collect renders every file in changelog.d/ into the next release and only then deletes it; 0.1.0 is already in CHANGELOG.md. So 0.2.0 will list "Added: tasks inspect <path>" (20261023_120000_napetrov_task-format-detect.md) and, in the same notes, say that tasks inspect moved. tasks inspect, questions import and eval reference are not in 0.1.0 at all. For someone upgrading, they are new commands announced under names that already exit 2.
Suggestion: rename the paths in the unreleased fragments using the table above. At least do it in the eight that mention tasks inspect, questions import or eval reference.
There was a problem hiding this comment.
Holding this one for a decision, so I have not changed the 33 fragments in this PR. The fragments are other authors' unreleased entries, and renaming them is wider than this PR. Options: (a) rename the old paths in all 33 in this PR, using the table in the eval-group entry; (b) rename only the eight that name tasks inspect, questions import or eval reference, which leaves the notes partly old; (c) leave them and fix them before the next release. My recommendation is (a), because otherwise 0.2.0 announces new commands under names that exit 2. I will not touch those files until the call is made.
| def register(sub, positive_int) -> None: | ||
| """Add the `repo` subcommand group.""" | ||
| def register(sub, positive_int, eval_sub) -> None: | ||
| """Add the `repo` subcommand group. Its check moved to `eval check` (#94 M1).""" |
There was a problem hiding this comment.
The repo help still lists the "task self-check", which moved to eval check.
dokimi --help and dokimi repo --help show "... budget, task self-check, agent runs and their report" (line 429), but repo tasks check now only prints moved: dokimi eval check.
Suggestion: drop "task self-check" from that help string.
| def register(sub, positive_int) -> None: | ||
| """Add the `eval` subcommand group.""" | ||
| def register(sub, positive_int): | ||
| """Add the `eval` subcommand group and its `score` and `judge` groups; return the eval subparsers. |
There was a problem hiding this comment.
The eval help no longer says what the group holds.
dokimi --help shows eval Evaluate answers [paid] (line 471). The README's CLI block says "Score answers, run RAGAS, or use a multi-judge panel" (README.md:128). eval now also holds personas, questions, import, inspect, check, tasks and triggers. So someone looking for where questions or repo tasks check went gets no hint from the top-level help.
Suggestion: something like help="Build and check eval sets, score answers, calibrate judges", and the same in the README line.
| scores degraded answers lower. | ||
|
|
||
|  | ||
|  |
There was a problem hiding this comment.
The picture still shows the old commands.
The alt text now says dokimi eval check and dokimi eval judge validate. The image assets/readme/eval-checks.png itself still shows $ dokimi repo tasks check and $ dokimi eval judge-validity, and both of these now exit 2.
Suggestion: regenerate the image with the new names.
There was a problem hiding this comment.
Holding this one for a decision. The PNG is a static picture with no generator in the repo, and it still shows dokimi repo tasks check and dokimi eval judge-validity, which both exit 2 now. Options: (a) recapture the picture in this PR; (b) replace it with a text list of the three commands until someone recaptures it; (c) leave it as a follow-up after merge. My recommendation is (b), because the alt text already names the new commands and a text list is correct today. I will not change the image until the call is made.
| | `catalog scan --root <missing>` | `--root is not a directory: <path>` | 1 | | ||
| | `tasks run --tasks-root <missing>` | `FileNotFoundError: Task directory not found: demo-widget` | 1 | | ||
| | `tasks list --tasks-root <missing>` | prints nothing | 0 | | ||
| | `eval tasks list --tasks-root <missing>` | `--tasks-root is not a directory: <path>` | 2 | |
There was a problem hiding this comment.
Nit: The two rows above have the same problem the bot found in this one.
On this head, catalog scan --root /nonexistent exits 2, not 1. tasks run --tasks-root /nonexistent prints --tasks-root is not a directory: /nonexistent and also exits 2.
Suggestion:
| `catalog scan --root <missing>` | `--root is not a directory: <path>` | 2 |
| `tasks run --tasks-root <missing>` | `--tasks-root is not a directory: <path>` | 2 |
There was a problem hiding this comment.
Fixed in 713482a: both rows now say exit 2 with the --tasks-root is not a directory message. I checked on head: catalog scan --root /nonexistent exits 2, and tasks run --tasks-root /nonexistent with a named task or with --all prints that message and exits 2.
| # Builds the image from the manifest's Dockerfile (the tree without its | ||
| # tasks), then grades the null and the oracle workspace in it. | ||
| - name: repo tasks check | ||
| - name: eval check |
There was a problem hiding this comment.
Nit: The step now runs eval check, but the header comment still says the job runs repo tasks check (line 11).
The Summary step also prints ### repo tasks check headings (lines 411 and 415). If those headings follow the job name on purpose, keep them. The header comment describes the command, so it should name eval check.
There was a problem hiding this comment.
Fixed in 06d3c01: the header line now says eval check. I kept the ### repo tasks check headings in the Summary step on purpose. They follow the job name repo tasks check ${{ matrix.task }}, which stays as it is.
| A path that is itself a live leaf under the old prefix is not a tombstone: ``eval score | ||
| pointwise`` is live even though ``eval score`` moved there. |
There was a problem hiding this comment.
Nit: Three docstrings here no longer match the code.
Leaf (line 39) gives ("eval", "score") as its leaf example, but that is now a group. Tombstone (line 48) says it always answers with new and exits 2, which is not true for a bare eval score or for eval score --help. _is_route (line 265) says a tombstone on a group shadows every route under it, but moved_argv now skips live leaves. These docstrings explain the tricky part for whoever writes the M2/M3 entries.
Suggestion: use ("eval", "score", "pointwise") as the example, and state the exception for a group that stays in place in the other two.
There was a problem hiding this comment.
Fixed in 7dd04cf. The Leaf example is ("eval", "score", "pointwise"). The Tombstone docstring names the group that stays in place. The _is_route docstring says that moved_argv skips live leaves.
| tasks_p = repo_sub.add_parser("tasks", help="Review tasks built from past reviews (no LLM)") | ||
| tasks_sub = tasks_p.add_subparsers(dest="repo_tasks_cmd", required=True) | ||
| check_p = tasks_sub.add_parser( | ||
| # eval check (#94 M1): the self-check of `eval check`, moved under eval |
There was a problem hiding this comment.
Nit: A few old names are left, one of them in user output.
evaluate.py:146and:198passcontext="panel-score"/"eval-score", soeval score panelprints⚠ Evaluator independence warning (panel-score): ...to stderr.questions.v2.json:225still saysquestions generate --validate.- This comment says "the self-check of
eval check, moved under eval", which is circular: it moved fromrepo tasks check. Line 26 still says "shared bytasks checkandrun". - The module docstring at
evaluate.py:4still says "score, ragas, panel-score". test_cli_routes.py:267still says "no command moves in this release".
There was a problem hiding this comment.
Fixed in 8387f45 and 7dd04cf. The independence warning now names its command (eval score panel, eval score pointwise). The schema description says eval questions generate --validate. The _context docstring, the eval check comment, the evaluate.py module docstring and the test comment at test_cli_routes.py:267 name the current commands.
| # Build image for dokimi's own repository-mode evals: CI's Python, the lock, every CI tool, | ||
| # and a warm uv cache so `make dev` inside a run works without downloading anything. | ||
| # Pinned by digest, like the tools below, so a rerun grades in the same image. To move it: | ||
| # `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `repo tasks check`. | ||
| # pinned by digest. To move the pin: `docker buildx imagetools inspect python:3.11-slim-bookworm`, then `eval check`. |
There was a problem hiding this comment.
Nit: Shortening this comment dropped the reason for the warm uv cache.
The old text said the cache is there so make dev inside a run downloads nothing. Lines 18-21 now warm it with no word on why, so someone bumping the image may drop it.
Suggestion: put a one-line comment above line 18, which keeps the two-line rule:
# A warm uv cache, so `make dev` inside a run downloads nothing.There was a problem hiding this comment.
Fixed in 8d4104d with the one-line comment above the ENV line, worded as you suggested.
`eval score` stays a group, so a plain word after it is a mistyped leaf. It now gets argparse's invalid-choice message, not a "moved" line that suggests a command that does not exist. An option after it is still an old call. Also fixes the docstrings that described the old group as a leaf. Thread: PRRT_kwDOVBg7g86yVdbI Thread: PRRT_kwDOVBg7g86yVdbX Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
repo's help no longer lists the task self-check (it is eval check). The eval help and its README line say what the group holds now. catalog scan and tasks run exit 2 for a missing root, as the table says. Thread: PRRT_kwDOVBg7g86yVdbM Thread: PRRT_kwDOVBg7g86yVdbP Thread: PRRT_kwDOVBg7g86yVdbT Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The independence warning names its command (eval score panel, eval score pointwise), not the retired context names. The schema text and the docstrings name the commands the code now has. Thread: PRRT_kwDOVBg7g86yVdbP Thread: PRRT_kwDOVBg7g86yVdba Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g86yVdbW Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g86yVdbb Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain to block approval of the CLI restructuring.
No open issues 🎉
What changed in this PR
0 new findings; nothing to fix.
This implements M1 of the CLI restructuring in #94 and #276, consolidating evaluation commands under dokimi eval while preserving their handlers and arguments.
Changes:
- Moves evaluation commands into nested groups and adds migration messages for retired paths, without executing them.
- Updates consumer labels while retaining compatibility with existing
task_inspect.v1artifacts. - Migrates tests, workflow invocations, documentation, and bundled guidance to the new command paths.
| File | Description |
|---|---|
dokimi/tests/test_typed_scorer.py |
Uses the new judge-validation path. |
dokimi/tests/test_trigger_set_readers.py |
Updates trigger commands and consumer expectations. |
dokimi/tests/test_tasks_command.py |
Migrates task listing and inspection calls. |
dokimi/tests/test_task_format_detect.py |
Updates inspection calls and import labels. |
dokimi/tests/test_task_format_corpus.py |
Updates the inspection command in its docstring. |
dokimi/tests/test_skill_selection.py |
Migrates trigger-generation test calls. |
dokimi/tests/test_session_runner.py |
Uses nested scoring and judge commands. |
dokimi/tests/test_repo_skill_arms.py |
Exercises skill-arm checks through eval check. |
dokimi/tests/test_repo_runner.py |
Migrates self-check test helpers. |
dokimi/tests/test_repo_run.py |
Updates self-check and timeout test invocations. |
dokimi/tests/test_repo_replay.py |
Checks replay tasks through eval check. |
dokimi/tests/test_repo_progress.py |
Updates the progress-test docstring. |
dokimi/tests/test_repo_host.py |
Migrates host-check calls and help checks. |
dokimi/tests/test_repo_eval_task_selection.py |
Updates workflow step and argument expectations. |
dokimi/tests/test_repo_builtin_graders.py |
Migrates built-in grader self-check calls. |
dokimi/tests/test_reference_checks.py |
Updates command names in comments and docstrings. |
dokimi/tests/test_record_readers.py |
Migrates reader tests and command-index handling. |
dokimi/tests/test_ragas_runtime.py |
Updates the RAGAS command in documentation. |
dokimi/tests/test_questions_output_unwritable.py |
Uses nested question commands and review. |
dokimi/tests/test_questions_generate_inputs.py |
Migrates question-generation input tests. |
dokimi/tests/test_questions_generate_config.py |
Uses the nested generation command. |
dokimi/tests/test_questions_artifact.py |
Migrates refinement and trigger-generation calls. |
dokimi/tests/test_provider_call_budgets.py |
Updates pointwise and panel command paths. |
dokimi/tests/test_printed_hints_parse.py |
Exercises hints from relocated persona commands. |
dokimi/tests/test_personas_discover_output.py |
Migrates question-analysis output testing. |
dokimi/tests/test_personas_approve.py |
Uses eval personas validate. |
dokimi/tests/test_panel_properties.py |
Migrates panel-scoring integration calls. |
dokimi/tests/test_openai_compatible_route.py |
Uses the judge-agreement path. |
dokimi/tests/test_model_defaults.py |
Updates default-model command cases. |
dokimi/tests/test_llm_cli.py |
Migrates provider-validation command cases. |
dokimi/tests/test_judge_validity.py |
Updates validation and agreement invocations. |
dokimi/tests/test_judge_temperature_entry_points.py |
Migrates temperature-forwarding test commands. |
dokimi/tests/test_judge_sequential_design.py |
Updates sequential validation invocation. |
dokimi/tests/test_judge_provider_default.py |
Updates provider-default route expectations. |
dokimi/tests/test_judge_provider_choices.py |
Updates provider-option command paths. |
dokimi/tests/test_judge_not_measured.py |
Migrates judge verdict and gate tests. |
dokimi/tests/test_journeys_trace.py |
Uses eval check in abbreviation tracing. |
dokimi/tests/test_exit_codes_input_errors.py |
Migrates persona input-error cases. |
dokimi/tests/test_exit_codes_boundaries.py |
Migrates trigger and refinement exit-code tests. |
dokimi/tests/test_eval_questions_file_errors.py |
Updates commands for unreadable-question tests. |
dokimi/tests/test_eval_output_write_errors.py |
Migrates scoring and judge output-error cases. |
dokimi/tests/test_eval_grounding_output_errors.py |
Updates grounding and validation paths. |
dokimi/tests/test_empty_question_rule.py |
Migrates empty-question refusal tests. |
dokimi/tests/test_cross_judge_judge_errors.py |
Uses the judge-agreement command. |
dokimi/tests/test_cli_routes.py |
Updates route, cost, help, and tombstone expectations. |
dokimi/tests/test_cli_eval_moves.py |
Adds migration, argument-preservation, and live-route tests. |
dokimi/tests/test_catalog_command.py |
Migrates trigger-generation error testing. |
dokimi/tests/test_bundled_skills.py |
Updates the calibration-command matcher. |
dokimi/tests/test_answer_imports.py |
Migrates import/scoring calls and checks the inspection hint. |
dokimi/tests/outcomes/journeys.yaml |
Updates journey commands and documentation anchors. |
dokimi/tests/outcomes/journeys.py |
Updates the surface-validation docstring example. |
dokimi/tests/fixtures/task_formats/MANIFEST.yaml |
Updates expected format-consumer labels. |
dokimi/SECURITY.md |
Updates the optional RAGAS command reference. |
dokimi/README.md |
Updates the CLI overview and help examples. |
dokimi/pyproject.toml |
Updates command references in comments only. |
dokimi/dokimi/treatments/arms.py |
Updates the trigger-generation docstring reference. |
dokimi/dokimi/task_formats/model.py |
Emits the new import and trigger consumer labels. |
dokimi/dokimi/task_formats/imports/promptfoo.py |
Updates reference-scoring comments. |
dokimi/dokimi/task_formats/imports/native_logs.py |
Updates scoring references in documentation. |
dokimi/dokimi/task_formats/imports/common.py |
Updates the checklist command comment. |
dokimi/dokimi/task_formats/imports/__init__.py |
Updates import documentation and the inspection error hint. |
dokimi/dokimi/task_formats/__init__.py |
Updates the public inspection-command docstring. |
dokimi/dokimi/schemas/task_inspect.v1.json |
Adds new consumer labels while retaining old values. |
dokimi/dokimi/schemas/questions.v2.json |
Updates command references in schema descriptions. |
dokimi/dokimi/repos/selfcheck.py |
Updates the self-check command docstring. |
dokimi/dokimi/repos/host.py |
Updates host-probe command documentation. |
dokimi/dokimi/questions/ragas_seed.py |
Updates the topic-input command example. |
dokimi/dokimi/eval/reference_run.py |
Updates the reference-scoring module docstring. |
dokimi/dokimi/eval/question_text.py |
Updates the validation exit-code reference. |
dokimi/dokimi/eval/pairwise.py |
Updates the judge-calibration command example. |
dokimi/dokimi/eval/judge_context.py |
Updates command names in warning messages. |
dokimi/dokimi/eval/cross_judge.py |
Updates documentation and report command references. |
dokimi/dokimi/commands/tasks.py |
Registers task listing and inspection under eval. |
dokimi/dokimi/commands/repo.py |
Passes evaluation subparsers to repository registration. |
dokimi/dokimi/commands/repo_run.py |
Moves self-check registration to eval check. |
dokimi/dokimi/commands/questions.py |
Relocates questions/import and renames panel-review. |
dokimi/dokimi/commands/personas.py |
Relocates persona commands and updates next-step hints. |
dokimi/dokimi/commands/judge_reliability.py |
Splits scoring and calibration routes into nested groups. |
dokimi/dokimi/commands/evaluate.py |
Creates score/judge groups and returns evaluation subparsers. |
dokimi/dokimi/commands/catalog.py |
Registers trigger generation under eval. |
dokimi/dokimi/commands/arms.py |
Updates trigger-conversion guidance and errors. |
dokimi/dokimi/commands/answers.py |
Updates the imported-evaluation comment. |
dokimi/dokimi/commands/answer_formats.py |
Relocates import and reference-scoring registration. |
dokimi/dokimi/cli.py |
Wires relocated commands into the shared evaluation group. |
dokimi/dokimi/cli_routes.py |
Updates routes and adds tombstones with live-group exceptions. |
dokimi/dokimi/caching.py |
Updates the reference-scoring docstring reference. |
dokimi/docs/user-journeys/journeys.md |
Migrates self-check and calibration instructions. |
dokimi/docs/user-journeys/flow-map.md |
Updates the repository journey label. |
dokimi/docs/subscription-backends.md |
Updates the pointwise scoring example. |
dokimi/docs/runs.md |
Updates commands associated with output locations. |
dokimi/docs/reviewing.md |
Updates the judge-agreement provider example. |
dokimi/docs/repo-evaluation-mergeability.md |
Updates the free self-check command reference. |
dokimi/docs/repo-evaluation-contract.md |
Migrates self-check commands and workflow documentation. |
dokimi/docs/README.md |
Updates command references and image alternative text. |
dokimi/docs/quickstart.md |
Migrates walkthrough commands and exit-code examples. |
dokimi/docs/methodology.md |
Updates scoring and question-processing command references. |
dokimi/docs/external-content.md |
Updates task commands and missing-root behavior documentation. |
dokimi/docs/evaluating-treatments.md |
Migrates calibration, agreement, and trigger examples. |
dokimi/docs/difficulty-rubric.md |
Updates the trigger-set command reference. |
dokimi/docs/contributing-terminal-bench-task.md |
Migrates task-listing instructions. |
dokimi/docs/coding-harnesses.md |
Updates inspection, listing, and format-consumer commands. |
dokimi/docs/architecture.md |
Updates repository self-check and entry-point descriptions. |
dokimi/docs/answer-formats.md |
Migrates import, inspection, and reference-scoring examples. |
dokimi/docs/adding-doc-source.md |
Updates the question-generation command reference. |
dokimi/data/skills/dokimi-design-evals/SKILL.md |
Updates bundled judge-calibration guidance. |
dokimi/data/skills/dokimi-design-evals/references/verifier-self-test.md |
Migrates listing, self-check, and calibration instructions. |
dokimi/data/README.md |
Updates artifact-generation command examples. |
dokimi/CONTRIBUTING.md |
Updates the optional RAGAS command reference. |
dokimi/changelog.d/20261030_160000_napetrov_eval-group-m1.md |
Documents command moves and upgrade behavior. |
.github/workflows/repo-eval.yml |
Runs eval check while preserving the job name. |
.github/workflows/ci.yml |
Updates RAGAS-related comments only. |
.agent-eval/README.md |
Migrates repository self-check instructions. |
.agent-eval/Dockerfile |
Updates and shortens comments; build behavior is unchanged. |
Review effort: Balanced
The repo-eval header now gives the reason in two lines and points at the contract doc for the jobs and the pin rule. The moved_argv comment is shortened to the same limit. Thread: PRRT_kwDOVBg7g86yVdbW Thread: PRRT_kwDOVBg7g86yVdbX Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Summary
Stage M1 of #94 (CLI API v2, tracking #276): the evaluation commands move under
dokimi eval, handler-neutral.personasandquestionsroots go away; the evaluation leaves oftasks,catalogandrepomove undereval, grouped intoeval score,eval judge,eval questions,eval personasandeval triggers.Every old path is a tombstone: it prints
moved: dokimi <new argv>to stderr, exits 2 and runs nothing. Arguments carry over unchanged.repo run,repo report,repo compareandrepo tasks replaystay where they are.Changes beyond the moves:
task_inspect.v1consumers enum keeps the old labels and adds the new ones (removing enum values is breaking underschema_compat).questions panel-reviewparser renamed toreviewto match the route table.repo-eval.yml: the step runsdokimi eval check; the job namerepo tasks check ${{ matrix.task }}is unchanged in case it is a required check.89809a25(includes feat(repo): Codex and Copilot CLI harness profiles, native review mode #116 and fix(repo): scan accepts an Action without @ref that a workflow uses #306). Two docs files that feat(repo): Codex and Copilot CLI harness profiles, native review mode #116 touched were updated for the new names.docs/reference/cli-interface.jsonregenerated (make cli-snapshot).Evidence
tests/test_cli_eval_moves.py: for each of the 21 old paths and the two retired roots, exit 2 withmoved: dokimi <new argv>, arguments preserved; each new path is a live leaf; thepersonasandquestionsroots are gone.tests/test_cli_routes.py: tombstone table is clean (tombstone_problems() == []); paid leaves match the route table.89809a25).make ci-localwithTMPDIR=/tmpon the final tree before the last rebase: 0 failed, 30 passed, 7 skipped (pytest inside: 11790 passed, 6 skipped). After the rebase, the diff is content-identical except index hashes and hunk offsets; the targeted tests (test_record_readers,test_judge_reliability,test_cli_eval_moves,test_cli_routes,test_repo_l0_cases,test_repo_l0_versions) pass: 572 passed. The PR's CI run is the full check of the rebased head.changelog.d/20261030_160000_napetrov_eval-group-m1.md:### Changedwith the before-to-after table and upgrade notes.Merge Danger
eval scoreis now a group (its old pointwise behaviour moved toeval score pointwise);moved_argvskips a tombstone when argv continues into a live leaf, andtombstone_problemsallows a group to stay under its own name.task_inspectoutput values change.consumersnow emitseval triggers generateandeval importwhere it saidcatalog trigger-setandquestions import. The schema enum keeps both, so the schema check passes, but a JSON consumer that matches the old strings must switch. Reviewers: confirm this is wanted rather than keeping the old emitted labels.repo-evaljob name is unchanged on purpose; check whether it is a required status before renaming it later.Held or out of scope (flagged, not changed here)
repo validate,explore list, native-import naming.catalog trigger-set --formatto--dialect, and thepython cli.pyhints inbaseline.pyare out of scope..agent-eval/tasks/review-pr-conventions/oracle_answer.mdandreview.patch(historical fixtures), andCHANGELOG.mdand existingchangelog.d/fragments (history).Refs #94, #276
🤖 Generated with Claude Code