[napetrov]

Part of #4.

Summary

dokimi arms run can now ask each question as a multi-turn session, with the history kept and the token usage recorded per turn.

 dokimi arms run ... --session-turns 3 [--turn-reminder TEXT]
   question: "How do I parallelize a loop?"            # turn 0: the plain single-turn prompt
   follow_ups: ["Now with a reduction?", "What does it cost?"]
   turn 0  send [user]                                  -> a1   usage: in/out/cache/cost
   turn 1  send [user, asst, user]                      -> a2   usage: in/out/cache/cost
   turn 2  send [user, asst, user, asst, user]          -> a3   usage: in/out/cache/cost
+  arm record: answer = a3, token_usage / metrics = sum over turns,
+              session.turns = per-turn metrics, stopped_reason
+  <arm>+reminder variant (only with --turn-reminder): same arm, reminder text
+              appended to every user turn, not to the assistant history

Off by default. Without --session-turns, the runner and its output are unchanged.

• Sessions are measurement only. --judge is refused with --session-turns, because the judge would score the last answer against the first question.
• Agentic arms (mcp:, skill-agent:, skill-catalog:) are refused in a session with session_unsupported.
• A question with fewer than N-1 follow_ups fails before any answer is paid for. Every record of a run has the same number of turns.
• A turn that raises ends the session and keeps the completed turns. An empty reply ends it with stopped_reason: empty_reply.
• The new logic is in dokimi/eval/session.py, so arm_runner.py grows by one hook (_answer_session_arm) and the output shape gains a session block only in session mode.

Evidence

Tests (new file tests/test_session_runner.py, 13 tests) use the fake_llm adapter only. Nothing reaches a provider.

Fail-on-old: dokimi/eval/arm_runner.py and dokimi/commands/arms.py replaced with the intel/main versions, the rest of the branch kept:

FAILED test_the_cli_refuses_judge_with_session_before_any_answer
FAILED test_an_empty_reply_ends_the_session
FAILED test_session_options_are_validated_up_front
FAILED test_a_failing_turn_keeps_the_completed_turns
FAILED test_reminder_arm_repeats_the_reminder_on_every_user_turn
FAILED test_per_turn_usage_with_cache_tokens_is_recorded
FAILED test_an_agentic_arm_is_refused_in_a_session
FAILED test_a_question_short_of_follow_ups_fails_before_any_answer_is_paid
FAILED test_sessions_are_not_judged
FAILED test_skill_context_is_in_the_opening_turn_only
FAILED test_the_artifact_names_the_session_settings
FAILED test_session_sends_the_full_history_each_turn
12 failed, 1 passed in 1.62s   (test_plain_runs_are_unchanged_by_the_session_fields passes on both: it is the guard)

Local checks on the rebased head (rebased onto intel/main caae46a):

uv run ruff check .                              All checks passed!
uv run mypy dokimi                               Success: no issues found in 223 source files
uv run python -m dokimi.schema_check             exit 0
uv run python -m dokimi.config_check             exit 0
uv run pytest -n auto -q                         7081 passed, 22 skipped

Merge Danger

Door: two-way. The flags are opt-in and additive. Reverting the commit restores the old runner.

Blast Radius: low. The code runs only when --session-turns is passed.

Behaviour changes for callers:

• Two new arms run flags. --session-turns N (N >= 2) and --turn-reminder TEXT. --turn-reminder without --session-turns is an error.
• follow_ups is a new optional question field. Questions without it run as before. A session run needs N-1 entries on every question.
• Session records are not judged. --judge with --session-turns fails up front, before any call is made.
• Cost reading. token_usage and metrics of a session arm are the sums over all turns, so a session arm's cost is not comparable with a single-turn arm's cost for the same question.
• Matrix and subjects parsers do not expose the new flags. They forward the two fields only when present, so no existing config changes meaning.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated: docs/benchmarking-and-comparison.md (new "Multi-turn sessions" section)
• changelog fragment added (changelog.d/)
• No ad-hoc run artifacts committed
• New code and test files carry the SPDX license header

🤖 Generated with Claude Code

[antonwolfy]

Review at c2ccd81. The session design is sound: the history is copied per call, the reminder stays out of the assistant turns, and the judge and agentic refusals fire before any answer is paid for. Four things are worth fixing before merge, all inline: the cost of a failed session, the empty-reply path, the pairing commands ignoring session runs, and untested CLI wiring and options.

What I ran, on a Linux review host at c2ccd81:

• make ci-local: the test job passes (7082 passed, 21 skipped) and so do the changed-line coverage, changelog, schema-check, schema-compat, package-smoke and gitleaks jobs. The ruff, mypy, zizmor, pre-commit, bandit, deptry, mkdocs and pip-audit jobs did not run in my venv: make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head.
• Reproductions for the code findings, with the LLM transport mocked (fake_llm).
• A mutation run: 66 mutants of the new code against tests/test_session_runner.py, 8 of which pass the full file and are findings, plus 1 equivalent mutant. The 13 tests fail on the base modules except the guard test, as the description says.

Checked, no issues

• Schema versioning: arms.v1.json has no additionalProperties: false on the record or arm, so session, session_turns and the arm session block are additive. A success run and a mid-session failure both validate.
• Matrix and subjects: those parsers do not define the new flags, so forwarded_run_config forwards nothing and getattr(..., None) applies.
• Usage sums: n_calls, cache read and write and cost add correctly over turns; cost_summary reads a successful session like any other arm.
• Docs and fragment: the python cli.py arms run form matches the other examples on that page; the fragment name and format match the others. No new page, so no nav: change.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.57534% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.06%. Comparing base (94e3eb1) to head (893dd1c).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/eval/arm_runner.py	93.42%	4 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #174      +/-   ##
==========================================
+ Coverage   91.97%   92.06%   +0.08%     
==========================================
  Files         223      224       +1     
  Lines       27709    27911     +202     
  Branches     4553     4592      +39     
==========================================
+ Hits        25486    25695     +209     
+ Misses       1574     1572       -2     
+ Partials      649      644       -5     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/arms.py	89.21% <100.00%> (+0.17%)	⬆️
dokimi/dokimi/eval/matrix_rollup.py	82.00% <ø> (ø)
dokimi/dokimi/eval/plugin_delta.py	87.12% <100.00%> (+1.52%)	⬆️
dokimi/dokimi/eval/question_text.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/session.py	100.00% <100.00%> (ø)
dokimi/dokimi/eval/arm_runner.py	94.46% <93.42%> (+0.75%)	⬆️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 41fc167. The four fixes from my earlier review are in and hold up: the error arms keep their spend, an empty reply is an errored arm, run_config records the session settings, and the CLI wiring and per-call options are tested. One bug comes from the spend fix itself, and two test gaps are left. Two nits on the docs.

What I ran, on a Linux review host at 41fc167:

• make ci-local: 27 passed, 1 failed, 7 skipped. The test job passes (7114 passed, 1 skipped) and so do changed-line coverage, mypy, deptry, schema-compat, package-smoke, windows-smoke, mkdocs and gitleaks. pip-audit fails on gitpython 3.2.0 (CVE-2026-114145, fix in 3.2.1), which comes from the lock files this PR does not touch.
• Reproductions for the code findings, with the LLM transport mocked (fake_llm).
• A mutation run: 35 mutants of the new code against tests/test_session_runner.py, 4 of which pass the full file (3 are findings below, 1 is a harmless raw_answer duplicate).

Checked, no issues

• Empty reply: empty_reply: turn N is the index of the empty turn, and its spend is kept. A failing or empty first turn gives a valid arms.v1 artifact and the report renders it.
• Old artifacts: plugin_delta, matrix_rollup and model_compare.replicate_key treat a missing session_turns and turn_reminder as equal to None, so two old runs still pair.
• Reminder arm: --turn-reminder gives [baseline, baseline+reminder], validates against the schema and renders in the report.

[antonwolfy]

I pushed fixes for the two doc nits from my review, one commit each:

• 5ddf3c5: only plugin-delta refuses to pair a session run with a single-turn run; the matrix rollup does not pair them.
• 1a20b18: the turn numbers in generation_failed: turn N and empty_reply: turn N count from 0.

GitHub's Tests now fail at 1a20b18 on every Python version, in test_growth_ratchet.py. commands/arms.py::_build_arms_output has complexity 14 against its baseline of 13, and eval/plugin_delta.py::_validate_pair has 16 against 14. The two commits change only benchmarking-and-comparison.md. The baselines came to main with the growth ratchet (#93) after this PR's earlier green run, and the PR's changes to those two functions take them past it. Splitting the two functions makes Tests green again.

Still open: the bug and the two risks from my review.

[napetrov]

00204bf also takes the growth ratchet failure on the merge with main (the red Tests): the session check moved out of _build_arms_output (_refuse_judged_session), out of ArmRunner.judge (_refuse_session_records) and out of _validate_pair (_validate_session_config), with no behaviour change; check_growth.py is clean and no baseline was raised. main merged in c13a75f (no conflicts).

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi arms run` parameters changed:
    + added option `--session-turns`
    + added option `--turn-reminder`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[xaleryb]

• Ran: the suite merged into current main, plus session and judge commands on a session artifact and a 2-turn session with a skill of 12k+ chars on a Claude model.
• Session opening for a static-context arm (skill:) is one plain string. No session request carries cache_control, so turn 1 reads 0 cache tokens where the plain run reads the skill prefix. Expected: turn 0 is built like the single-turn prompt.
• eval cross-judge/pairwise/checklist --answers <session artifact> score each session's last answer against its first question and exit 0. Expected: refused, as arms run --judge is.
• The follow_ups list/blank checks and the whitespace-reply stop survive mutation.

Pushed:

• 1f64880: the opening keeps the plain run's cache breakpoint, so later turns read the prefix from cache.
• e0b6488: tests that pin the follow_ups checks and the whitespace-reply case.
• 893dd1c: require_question_text refuses session records, so every judge path refuses them; docs and fragment updated.

Left for the author: resolve the three addressed threads.

[xaleryb]

Approved at 893dd1c. Every earlier point re-run at this head passes, no must-fix or mid item is open, and CI is green.