feat(repo): compare records the start context, filters tasks, and flags inconclusive rows

dokimi/changelog.d/20261029_150000_antonwolfy_compare-reading.md

@@ -0,0 +1,26 @@
+### Added
+
+- **`repo run` records each cell's start context, and `repo compare` compares it** — `result.json`
+  `agent.first_prompt_tokens` is what the first main-thread model call was sent (input plus cache
+  creation plus cache read tokens, one call per message id, as `context trace` reads it): start-up and
+  the task. It is `null` for a trace without that call and for the `text` format. `repo compare` adds a
+  descriptive `first_prompt_tokens` row; a batch written before the field has no such row.
+- **`repo compare --task ID`** — repeatable: compares only those task ids. An id no finished cell of the
+  batches has exits 1 and names it; the tasks left out are not listed as unpaired, nor their unfinished
+  cells as missing (a cell's task is the one its `meta.json` records), and the `--json`
+  summary (`task_filter`) and the text header (`tasks: N of M (--task)`, M the tasks with a finished
+  cell) record the selection. Without `--arm`, an arm with no finished cell among the selection is
+  not compared, and both name it (`task_filter.dropped_arms`, `not compared, no finished cell among
+  the --task selection: <arms>`).
+- **Reading notes under a `repo compare` comparison** — a gate or score row of 2+ tasks whose interval
+  includes 0 says it is inconclusive and gives `mde`, an approximate, uncorrected t-based estimate of
+  the effect a 5% one-sample t test detects with 80% power at that spread, a planning figure and not
+  the power of the sign-flip test or of the Holm-corrected decision (`mde` is in every `--json` row). With 5 or fewer changed tasks the sign-flip p cannot go
+  below 0.05 (its floor is `2 / 2^k` for k changed tasks), so `mde` is `null` and the note says how
+  low p can go instead. The same holds where Holm's correction keeps a gate or score row's
+  `p_holm_floor` (its `p_holm` with its p at that floor, the other rows of the family as observed) at
+  0.05 or above, as with two such rows of six changed tasks each (`p_holm` 0.0625 at best); the note
+  then says `with Holm over N rows, p_holm cannot go below ...`. A row whose interval and p disagree says so and why
+  (`interval_p_disagree`); for gate and score it compares the interval with Holm's `p_holm`, quotes
+  it and says it is the one to read, and a descriptive row uses its raw p; descriptive rows with
+  p below 0.05 are named as uncorrected and outside Holm's family. No delta, interval or p changes.

dokimi/docs/architecture.md

@@ -337,7 +337,10 @@ next release.
   share, score, cost, turns, chosen `grade.json` metrics, `result.json`
   `process` counts), with an interval
   over tasks (bootstrap for gate and score, t for the rest), a sign-flip
-  permutation p and Holm's correction.
+  permutation p and Holm's correction. `dokimi/repos/compare_tasks.py` holds
+  what its `--task` selection keeps; `dokimi/repos/compare_reach.py` how small
+  an effect a row can find (`mde`, an uncorrected t-based estimate, and the
+  lowest `p_holm` Holm's family leaves it).
 - `dokimi/repos/selfcheck.py` — `repo tasks check`: on each task
   `grade` must fail the untouched workspace and pass it after `oracle`, and an
   isolation probe must not see the tasks or the source from the agent's

dokimi/docs/context-window.md

@@ -79,3 +79,7 @@ point `CLAUDE_CONFIG_DIR` at an empty directory to leave it out. The
 `surface` field of `--json` (the tool, skill and agent counts, and the names of
 the MCP servers and loaded plugins) shows what start-up offered, so two runs
 that differ there are not the same condition.
+
+`repo run` records the size of each cell's **first prompt**, in tokens, as
+`result.json` `agent.first_prompt_tokens`, and `repo compare` compares it between arms as
+the descriptive measure `first_prompt_tokens`.

dokimi/docs/reference/cli-interface.json

@@ -10263,6 +10263,22 @@
               "position": null,
               "required": false,
               "type": null
+            },
+            {
+              "action": "_AppendAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--task",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": null
             }
           ],
           "path": "dokimi repo compare",

dokimi/docs/repo-evaluation-contract.md

@@ -419,6 +419,12 @@ reads the upstream base (`base_sha`) and the workspace start commit
 Code's `modelUsage`), `main` when the harness reported the parent's only, and
 `null` when it reported none. `subagent_calls` counts the parent's and
 subagents' `Agent`/`Task` calls; `null` when the harness wrote no result record.
+`first_prompt_tokens` is what the first main-thread model call was sent
+(`input + cache_creation + cache_read` tokens of that call, counted once per
+message id, as `context trace` reads it): everything start-up loaded plus the
+task, so two arms that differ only in guidance differ there by what start-up
+loaded of it. It is `null` when the trace has no main-thread call with
+`usage`, and for a trace format that carries none (`text`).
 
 `result.json` `process` holds what the framework counts in the events, the
 parent's and subagents' together, so a grader need not:
@@ -1334,15 +1340,58 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re
 - `repo compare` reduces cells to one value per (task, arm, model) first:
   repetitions share a task's difficulty and are not paired across arms, so the
   task is the unit. Per treatment arm, model and measure (gate share, score,
-  cost, turns, each `--metric` from `grade.json`, and each `process` count a
+  cost, turns, `first_prompt_tokens`, each `--metric` from `grade.json`, and each `process` count a
   compared cell reports as `process:<key>` or `counter:<name>`) it gives the mean per-task
   delta against the baseline with a 95% interval over tasks (a bootstrap for
-  gate and score, a t interval for cost, turns, metrics and process counts, `-` for a single
+  gate and score, a t interval for the descriptive measures, `-` for a single
   task), the tasks up, down and tied, and a two-sided sign-flip permutation p (exact up to 16
   changed tasks; at one cell per task and arm it equals McNemar's exact test).
-  A `process` count only one arm reports for the shared tasks of a
-  (treatment, model) is not compared; a warning names it. Holm's correction runs over the gate and score comparisons of one call; the
-  rest are descriptive. A gate or score comparison carries `headroom`:
+  A `process` count or `first_prompt_tokens` only one arm reports for the shared tasks of a
+  (treatment, model) is not compared; a warning names it. When neither arm
+  reports it (a batch run before `repo run` recorded it), the row is absent.
+  Holm's correction runs over the gate and score comparisons of one call; the
+  rest are descriptive. Three notes under a comparison keep a row from being
+  misread, and change no delta, interval or p. A gate or score row of 2+ tasks whose
+  interval includes 0 is inconclusive, not evidence of no effect: the note
+  gives `mde`, a planning figure: an approximate, uncorrected t-based estimate,
+  `(t(0.975, n-1) + t(0.80, n-1)) * sd / sqrt(n)` over the per-task deltas, of
+  the mean delta a one-sample t test at the uncorrected two-sided 5% level
+  detects with 80% power at that spread, the t quantiles at `n-1` degrees of
+  freedom (every row has `mde` in `--json`; `null` for one task or deltas with
+  no spread, and then no note). It is not the power of the sign-flip test nor
+  of the Holm-corrected decision; a Holm-calibrated figure would depend on the
+  other rows' observed p, which a figure for planning the next run's task count
+  cannot know. The sign-flip p over k changed (non-tied)
+  tasks cannot go below `2 / 2^k`, so with 5 or fewer no effect of any size
+  reaches p < 0.05: `mde` is `null` there too, and the note says how low p can
+  go instead (`with 3 changed tasks the sign-flip p cannot go below 0.25: no
+  effect can reach p < 0.05 here`). Holm's correction can lift that floor: a
+  gate or score row's `p_holm_floor` is its `p_holm` with its p put at that
+  floor and the other p of the family as observed (in every `--json` row;
+  `null` on a descriptive row or one with every task tied), and where it is not below
+  0.05, `mde` is `null` and the note names Holm instead (`with Holm over 2
+  rows, p_holm cannot go below 0.0625 here: no effect can reach p_holm <
+  0.05`); a family of one row is the raw rule. `p_holm_floor` decides only
+  whether `mde` is given, not its size. A row whose interval and p disagree (the
+  interval excludes 0 while p is 0.05 or more, or includes 0 while p is below
+  0.05; for gate and score that p is `p_holm`, the one that decides them, and the
+  note quotes it) has `interval_p_disagree` `true` and a note: they come from two
+  methods, and for gate and score `p_holm` is the one to read, since a percentile
+  bootstrap runs narrow over few tasks; the inconclusive note gives way to
+  it. Descriptive rows with p below 0.05 are named on one line: their p is
+  uncorrected and outside Holm's family. `--task ID` (repeatable) compares only
+  those tasks: an id no finished cell of the batches has is refused, the tasks
+  left out are not listed as unpaired nor their cells as missing (a missing
+  cell's task is the one its `meta.json` records, else, for a cell only
+  `plan.json` names, the longest task id of the batch's finished cells that its
+  name starts with followed by `__`; a cell of neither is missing only without
+  `--task`), the `--json` summary records the
+  selection as `task_filter` (`tasks`, and `of`, the number of tasks of the
+  batches that have a finished cell, the ids `--task` can name, and
+  `dropped_arms`: without `--arm`, the arms with no finished cell among the
+  selection, which drop out of the comparison, `[]` with `--arm`; `null`
+  without `--task`), and the text header says `tasks: N of M (--task)` and,
+  under it, `not compared, no finished cell among the --task selection: <arms>`. A gate or score comparison carries `headroom`:
   `ceiling` (both arms at 1.0 or the task's `score.max` on every paired
   task), `floor` (both at 0 on every one), `near_ceiling` (the baseline at
   the top on all but at most one of 2+ tasks and the treatment nowhere below

dokimi/dokimi/commands/repo_run.py

@@ -411,7 +411,7 @@ def cmd_repo_compare(args: argparse.Namespace) -> None:
     arms = [a for spec in args.arm for a in spec.split(",") if a] if args.arm else None
     try:
         summary = compare(dirs, args.baseline, arms, args.model, tuple(args.metric or ()),
-                          pass_k=args.pass_k, path_stability=args.path_stability)
+                          pass_k=args.pass_k, path_stability=args.path_stability, tasks=args.task)
     except (CompareError, CellFileError) as exc:
         print(f"✗ {exc}", file=sys.stderr)
         sys.exit(1)
@@ -556,6 +556,8 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
                            help="Treatment arm (ARM or BATCH/ARM); repeatable or comma-separated "
                            "(default: every other arm)")
     compare_p.add_argument("--model", action="append", default=None, help="Only this model; repeatable")
+    compare_p.add_argument("--task", action="append", default=None,
+                           help="Only this task id; repeatable (default: every task of the batches)")
     compare_p.add_argument("--metric", action="append", default=None,
                            help="Also compare this numeric or boolean grade.json metric; repeatable")
     compare_p.add_argument("--pass-k", type=positive_int, default=None, dest="pass_k", metavar="K",