[napetrov]

Part of #41 (follow-up to D4, #157).

Summary

repo report and repo compare now show the result.json process counts. #157 (merged) adds those counts. The code reads process defensively, so an absent or null process is skipped and older batches report and compare as they do today.

 dokimi/repos/
+├── process_counts.py   # process_values / process_means / process_order / process_line
 ├── report.py           # row["process"] = per-arm means; one "process ..." line under the row
 └── compare.py          # measures += process:<key>, counter:<name> (descriptive, not in Holm)

• Measure names: process:{tool_calls,commands,failed_commands,denied_commands,guidance_read} (guidance_read is the number of paths read), plus counter:<name> for each manifest process_metrics counter.
• In compare, a counter becomes a measure when both arms report it for a shared task. If only one arm reports it (an older batch, or a counter only one arm added), it is not compared, and one warning per (treatment, model) names it. Values are per-task means over the cells that have the counter, with a t interval and a raw p, the same way as cost and turns.
• Docs: contract §9 (the report and compare paragraphs), architecture.md, and a scriv fragment.

Evidence

• Before: repo compare showed only gate/score/cost_usd/turns/metric rows.
  After: output on a synthetic batch:

  process:tool_calls           3     14.000     10.000  -4.000 [-4.000, -4.000]      0.250       -  0/3/0
  process:failed_commands      3      2.000      0.000  -2.000 [-2.000, -2.000]      0.250       -  0/3/0
  process:guidance_read        3      0.000      1.000  +1.000 [+1.000, +1.000]      0.250       -  3/0/0
  counter:bazel                3      1.000      3.000  +2.000 [+2.000, +2.000]      0.250       -  3/0/0

• tests/test_repo_process_counts.py (synthetic result.json with and without process): measure order and deltas, p_holm None, null/absent skipped (not zero), non-counts ignored, report means and the single line. Full pytest -q: 7072 passed; ruff check . and mypy are clean. No function's radon CC went up.

Merge Danger

Door: two-way

The change only adds new keys and rows, and nothing is written to disk.

Blast Radius: narrow

A consumer that expects a fixed measure list in repo compare --json will see extra rows when cells carry process. There is a one-line import conflict with #153 in compare.py; it does not conflict with #155 or #157.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.80952% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 92.78%. Comparing base (a2b787a) to head (57ae156).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/compare.py	97.82%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #177      +/-   ##
==========================================
+ Coverage   92.74%   92.78%   +0.03%     
==========================================
  Files         254      257       +3     
  Lines       32087    32480     +393     
  Branches     5344     5434      +90     
==========================================
+ Hits        29759    30135     +376     
+ Misses       1560     1559       -1     
- Partials      768      786      +18     

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/process_counts.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/report.py	97.33% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/repos/compare.py	99.42% <97.82%> (-0.27%)	⬇️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at c301aa6. The measure plumbing is clean: Holm is untouched, null and absent process are skipped, not zeroed, and the report JSON only gains a key. One bug at the edge, where only one arm carries process, and one test gap. One nit on the docs.

What I ran, on a Linux review host at c301aa6:

• make ci-local: 20 passed, 8 failed, 7 skipped. The 8 failures (ruff, zizmor, pre-commit, bandit, mypy, deptry, mkdocs, pip-audit) exited 127, tools missing from my venv: make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head. The test step passed: 7073 passed, 21 skipped. Changed-line coverage 100% (48 lines).
• A reproduction of the unpaired finding, with no model or network access.
• A mutation run on the new tests: 34 mutants, 10 of which pass the 4 new tests (see the inline comment on the test).

Checked, no issues

• Holm: PRIMARY is still gate and score, and the process rows keep p_holm None.
• Scoping: only arms named in the compared selectors add process measures; there is no name clash with metric:, score or cost_usd.
• Values: bool, NaN, strings and a null process are skipped; guidance_read counts the list length.
• Report: summarize only adds a process key per row; a row without it prints no extra line.

[xaleryb]

• Run: the full suite, ruff and mypy on the head merged with current main, plus repo compare on a baseline batch and a treatment batch whose counters differ.
• Open: the treatment cells report counters {bazel, icpx_missing}, the baseline cells report only {bazel}. repo compare gives no counter:icpx_missing row, no unpaired entry and no warning. Expected: contract §9, the changelog fragment and the PR body say each count a compared cell reports becomes a measure. Either reword them to "each count both arms report for a shared task", or emit one warning per treatment and model that lists the measures it skipped. Per-task unpaired entries would bring the flood back.
• Pushed f12b327: it parametrizes the one-arm flood test over both arms. Until now only the older-baseline case was pinned, and a guard that checks only the baseline passed every test.
• Left for the author: the choice above, and merging after feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders #157.

[antonwolfy]

Review at f12b327. The one-arm flood is fixed and the guard now covers both arms. One test gap is left, on the first-rep case.

What I ran, on a Linux review host at f12b327:

• make ci-local: 27 passed, 2 failed, 7 skipped. The changed-line coverage step passes. pip-audit fails on the lock files, which this PR does not touch. test_no_file_or_function_grew_past_its_baseline fails on compare.py::_provenance_warnings (complexity 11, no baseline entry); that function is not in this diff and the same test fails on the PR's base.
• A mutation run on the new tests: 28 mutants, 6 of which pass the whole new test file. The one that matters is inline.
• Reproductions of the one-arm and partial-count cases, with the LLM transport mocked.

I agree with @xaleryb that repo compare silently drops a count only the treatment (or only the baseline) reports, while contract §9, the changelog fragment and the compare.py docstring say each count a compared cell reports becomes a measure. The choice between rewording and a warning waits for the author, as does merging after #157.

Checked, no issues

• Holm: process rows keep p_holm None; PRIMARY is unchanged.
• Values: a null or absent process is left out of the per-task mean, not counted as zero; multi-model batches are handled per model.
• Repo rules: no schema or persisted-format change, no new doc page, changelog fragment present.

[vchamarthi]

No blockers. 2 non-blocking comments.

Earlier findings (one-arm unpaired flood, first-rep null, arm scoping) are fixed; two merge-order points remain before this can land.

• Question: Is a rebase onto main planned? GitHub reports this PR as CONFLICTING (10 commits behind), so it cannot merge as is.

  gh pr view 177 at 4905eca: mergeStateStatus DIRTY, mergeable CONFLICTING; compare main...4905eca behind_by 10. The _compare_model split into _compare_measure touches the area other recent compare.py changes also edit. (dokimi/dokimi/repos/compare.py:128)

CI at send time: 0 failing, 0 pending.

[vchamarthi]

Question: Should this be held until #157 merges? Until then main writes no process field, so report and compare show nothing, and the §4.1 reference in the contract resolves to a section that does not define it.

gh pr view 157: state OPEN. grep process on main's compare.py finds no process handling; the author and antonwolfy already agreed on merge-after-#157 in the thread on repo-evaluation-contract.md:1008.

[napetrov]

No hold needed any more: #157 merged at 2026-10-27T03:17Z, so main writes process and the §4.1 reference resolves. This branch is merged with current main (dd3114f).

[vchamarthi]

No blockers.

The earlier conflict with main is resolved by the merge at 3e2d31f; the one open point is still the merge order with #157.

CI at send time: 0 failing, 8 pending.

[napetrov]

@xaleryb on your open point (a count only one arm reports, e.g. counter:icpx_missing, gave no row, no unpaired entry and no warning): fixed in dd3114f with the warning option. Each (treatment, model) now gets one warning listing the counts it skipped. There are still no per-task unpaired entries, so the flood stays fixed. test_process_on_one_arm_only_does_not_flood_unpaired (both arms, from your f12b327) now asserts that warning. The compare docstring, contract §9 and the changelog fragment say so.

Merge order with #157: #157 is merged. dd3114f also merges current main. make ci-local: 0 failed, 28 passed, 7 skipped (docker, conda and semgrep steps).

[antonwolfy]

LGTM. The one-arm warning does what the thread with @xaleryb asked for: one line per treatment and model, no per-task unpaired entries, and both arms are guarded. Merging main and #157 closes the contract reference. Two nits inline.

What I ran, on Linux at dd3114f:

• make ci-local: 28 passed, 0 failed, 7 skipped (docker, conda, semgrep and the jobs that do not run on a PR). Changed-line coverage passes.
• A mutation run on the new tests: 58 mutants, 42 killed. The ones that hide something are in the second inline comment.
• Reproductions with several models and treatments, the --model filter and a baseline pooled from an old and a new batch, with the LLM transport mocked.

Checked, no issues

• Warning scope: with 2 models and 2 treatments there is one warning per (treatment, model) where exactly one arm reports a count. --model limits the warnings to the selected models. They show up in the --json warnings and as warning: lines in the text.
• Holm and refactor: process rows keep p_holm None. The _compare_measure split leaves the gate, score, cost and turns rows unchanged.
• Repo rules: contract §4.1 now defines process and §2 process_metrics. No schema covers the repo compare or repo report output, so no version bump. compare.py stays under its length limit and the growth ratchet passes.

[xaleryb]

• Re-checked the head merged with current main: the full suite, ruff, mypy, bandit, and repo compare with treatment counters {bazel, icpx_missing} against baseline counters {bazel}. It now gives no icpx_missing row, no unpaired entries, and one warning guidance:raw m-1: only one arm reports counter:icpx_missing for any shared task; not compared. The earlier point is fixed.
• Pushed 57ae156: a test for that same input, with both arms carrying process and one counter on one arm only. The existing flood tests drop process from a whole arm, so a guard that checks whether an arm has any count, instead of this count, passed every test. That guard brings the per-task unpaired flood back and drops the warning.
• Left for the author: nothing.

[xaleryb]

Approved at 57ae156. Every earlier point re-run at this head passes, no must-fix or mid item is open, and CI is green.