[vlad-perevezentsev]

Summary

arms run, arms matrix-run and subjects run pass an arms run its configuration as an argparse.Namespace. matrix-run copies the Namespace and mutates it for each cell. subjects run builds one by hand from FORWARDED_RUN_CONFIG_FIELDS. _build_arms_output reads most fields with getattr(args, ..., default), so when a caller drops a field, the run looks the same as if the user had chosen the default.

This PR replaces that hand-off with one frozen dataclass and moves the run out of commands/. It is the second half of the #4 Wave 5 item, as described in this #4 comment. No flag changes; stdout, exit codes and arms.v1 stay the same.

• dokimi/eval/arms_run.py (new; no existing eval/* file changes):
  • ArmsRunConfig: 24 fields, named after the flags' current dest. Each default is what arms run runs with when the flag is left out, so the getattr defaults are gone.
  • run_arms is the body of _build_arms_output, split to stay under the complexity cap.
  • cell_config builds a matrix cell's config with dataclasses.replace(...).
  • run_matrix(config, cells, specs, questions, on_cell) checks every cell before the first paid call, then runs the cells in turn and hands each finished one to on_cell.
  • The run helpers move here under public names (probe_sampling is also used by judge_reliability).
• commands/arms.py keeps argument parsing, output paths, saving, printing and exit codes. run_config_from_args is the one place where flags become a config. The commands' run_matrix writes the cells, the plugin deltas and the rollup.
• subjects run builds the config directly. FORWARDED_RUN_CONFIG_FIELDS, forwarded_run_config() and the regex test over getattr(args, ...) are removed.
• Errors inside the run are now exceptions. LibraryIdError replaces a SystemExit raised inside _build_arms_output, and MatrixCellError names the failing cell. Both commands print the same messages and exit 1, as before.

There is one order change in matrix-run: the check that cell names give distinct file names now runs before the harness check, so both cheap checks come before the paid sampling probe.

Commits, each passing the tests on its own: (1) move the shared checks to the new module; (2) ArmsRunConfig + run_arms for arms run and matrix-run; (3) subjects run builds the config; (4) the matrix loop moves to the module.

#229, "the arms Namespace" (not assessed yet). Three duplicates went away:

• the field list FORWARDED_RUN_CONFIG_FIELDS, now the dataclass fields;
• the 9 getattr defaults, now the dataclass defaults, which a test checks against the arms run flags;
• the per-cell Namespace copy, now dataclasses.replace.

The remaining duplicates are the flag definitions themselves (#94 F1a/F1b). F1b can now rename fields to the registry's api_kwarg in one place.

Depends on #264, overlaps #238

• test(arms): pin what arms run, matrix-run and subjects run hand the arms runner #264 (tests only) pins what each entry point hands ArmRunner. Its tests are not on this branch, so CI here does not run them until test(arms): pin what arms run, matrix-run and subjects run hand the arms runner #264 merges. Run locally against every commit of this branch, they pass with no edit (11 passed). After test(arms): pin what arms run, matrix-run and subjects run hand the arms runner #264 merges, this branch is rebased and one commit rewrites that file's docstring, which describes the hand-off removed here. Its tests stay unchanged.
• fix(cli): run, compare, arms, tasks, explore and repo exit with a message on bad input or output #238 edits cmd_arms_matrix_run. Merging it gives 4 mechanical conflict hunks in commands/arms.py; I rebase after it merges.

Evidence

• New tests/test_arms_run_config.py (5 tests):

  • the config defaults equal the arms run flag defaults;
  • a doc arm with no library id prints Error: ... and exits 1, for arms run and matrix-run;
  • a failing cell is named in the error, and the cells before it are kept;
  • run_matrix refuses a bad cell before any provider call.

  The three CLI tests also pass on main.

• 9 other test files change only their imports and monkeypatch targets.

• Mutation sweep: 19 single edits (fields dropped on the way to the runner, defaults changed, cells not applied, errors misrouted). Each one fails a test.

• pytest with test(arms): pin what arms run, matrix-run and subjects run hand the arms runner #264's file in place: 9260 passed, 57 skipped. make ci-local reports one failure, "Fragment for package changes", which needs the skip-changelog label.

• Growth: commands/arms.py goes from 772 to 513 lines. The _build_arms_output (13) and _validate_expected_skills (11) baseline entries are removed.

Merge Danger

Door: two-way. This is an internal refactor; the CLI, stdout, exit codes and artifacts do not change.

Blast Radius: the three commands that run arms, guarded by #264's parity tests.

Overlap: #238 (see above). #255 changes --provider choices in lines this PR does not touch.

Not in this PR:

• Flags (CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 F1b), including the flags that subjects run and matrix-run lack today: adding them changes behaviour.
• Typed plugins/judge_panel: they stay comma-separated strings, because tuples would change how a cell's plugin list is split.

Checklist

• Tests added and pytest passes
• ruff, mypy, and schema checks pass (make ci-local)
• Docs: evaluating-treatments.md depends_on and the architecture note name the new module. No changelog fragment (internal refactor): needs skip-changelog
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header

Refs #4
Refs #229

[xaleryb]

Ran: the full suite, lint and types on the head; the head merged into current main; old-vs-new CLI runs of arms run, matrix-run and subjects run.

• Merge with main breaks a test. Input: merge the head into current main. Observed: tests/test_answering_stack_parity.py::test_both_stacks_match_pinned_values fails with ImportError: cannot import name '_resolve_doc_library_id' from 'dokimi.commands.arms'. Expected: green. Fix: rebase, then import resolve_doc_library_id from dokimi.eval.arms_run at :265/:275. No alias.
• Input: ArmsRunConfig(product=..., judge=True, judge_provider='openai'). Observed: judge claude-sonnet-5-5 on openai, while arms run --judge --judge-provider openai resolves gpt-6.1-sol. With openrouter the config runs and the CLI refuses. Expected: what the docstring promises. Proposal: resolve via default_judge_model(judge_provider), using either a None default or the JUDGE_MODEL_FOR_PROVIDER sentinel in __post_init__, and add a non-default provider case to the defaults test.
• 4d7ca1d: pins that matrix-run exits 2 on a blank question. Before it, nothing failed if that branch was dropped.
• Left: rebase; skip-changelog label or a fragment, so the Tests jobs run.

[antonwolfy]

The refactor holds up, with two open points from @xaleryb that I confirm and that should be fixed before merge. I found nothing new.

I agree with @xaleryb that the head merged into current main fails test_answering_stack_parity.py, and that ArmsRunConfig keeps the Claude judge for a non-default judge_provider. New evidence: ArmsRunConfig(product="p", judge=True, judge_provider="openai").judge_model is claude-sonnet-5-5; the CLI paths are fine only because cli.py resolves the sentinel first.

What I ran, on a Linux review host at 4d7ca1d:

• make ci-local: 25 passed, 3 failed, 7 skipped. 9302 tests passed; changed-line coverage 95% (232 lines measured).
• Fragment for package changes fails because the PR changes dokimi/dokimi/ with no changelog fragment and no skip-changelog label, as @xaleryb noted; either one turns it green. The other two failures are not the PR's: pip-audit lists filelock advisories in the lock files, which this PR does not touch. test_unbalanced_quote_in_hook_completes_quickly took 1.04s against a 1s limit, and passes on its own (508 passed in test_repo_environment.py).
• Mutation run: 119 single-point mutants of arms_run.py, commands/arms.py and commands/subjects.py (fields dropped on the way to ArmRunner, run_config_from_args fields, defaults, cell_config, exit codes, error routing). The related test files kill all 119.

Checked, no issues

• Parity: every flag the base passed to ArmRunner, Judge and JudgePanel still reaches them; exit codes and messages for LibraryIdError, MatrixCellError and EmptyQuestionError match the base.
• Description: ArmsRunConfig has 24 fields named after the arms run flag dests, and the defaults match the flags apart from the judge model above.
• Growth baseline: the removed entries and the new line counts match; test_growth_ratchet.py passes.
• Leftovers: no references to the removed helpers remain in code, tests or docs. All five commits carry Signed-off-by.