feat(repo): replay grades record each credited hit; repo report --hits lists them

dokimi/changelog.d/20261029_120000_antonwolfy_replay-hit-pairs.md

@@ -0,0 +1,4 @@
+### Added
+
+- **Replay hits you can check** — a `dokimi:review_replay` grade now records `details.matches`: per credited thread, the thread (id, anchor, gold comment) and the finding that hit it (its index, location, severity, message) with their distance in lines, so a hit can be read from `grade.json` alone. The metrics are unchanged. `dokimi repo report --hits` lists those pairs by task, arm, model and rep, as the cell's `result.json` names them even when a match carries a key of the same name (`--json` for records), and counts the cells graded before `matches` existed.
+- **Parse errors in `repo report`** — a row notes how many of its cells had an agent answer whose findings JSON did not parse (`grade.json` `parse_error`): scored 0 on a score-only row, counted as failed on a gate row, and not counted for a cell already counted as a grader failure. `--json` gives `parse_errors` per row, so a lower recall from answers that did not parse can be told from one from worse reviews.

dokimi/docs/reference/cli-interface.json

@@ -10792,6 +10792,22 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreTrueAction",
+              "choices": null,
+              "const": true,
+              "default": false,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": true,
+              "key": "--hits",
+              "kind": "option",
+              "nargs": 0,
+              "position": null,
+              "required": false,
+              "type": null
+            },
             {
               "action": "_StoreTrueAction",
               "choices": null,

dokimi/docs/repo-evaluation-contract.md

@@ -616,7 +616,7 @@ failed the task, and on a task with `score` that is a failed run with score
 | `grader` | data next to `task.json` (key that renames it) | `pass` | `metrics` | `details` |
 |---|---|---|---|---|
 | `dokimi:review_seeds` | `manifest.json` (`manifest`); oracle answer `oracle.json` (`oracle_answer`), a list of findings | `null`; for a manifest with no seeds (a clean-diff control), `true` when no finding is `blocker` or `major` | `n_findings`, `n_severe`, `n_decoys`, `unmatched`, `recall` (`null` with no seeds) | `seeds_hit`, `decoys_flagged`, `recall_by_class` (`"hit/total"`) |
-| `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `recall_by_kind`, `decoys_flagged` (always `[]`) |
+| `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `matches`, `recall_by_kind`, `decoys_flagged` (always `[]`) |
 
 `manifest.json` is `{"seeds": [{id, class, file, lines, kw}], "decoys":
 [{id, file, lines, kw}]}`; `file` `"*"` matches any file, and `kw` is a
@@ -637,6 +637,12 @@ review thread of a merged PR, anchored on a line of the reviewed revision.
 - A finding on a thread's file within `window` lines of its anchor can hit
   it. Matching is one-to-one and nearest first: one finding credits at most
   one thread, and one thread at most one finding.
+- `matches` holds one object per credited pair, in `threads_hit` order: the
+  thread's `thread` (id), `thread_file`, `thread_line` and `comment`, the
+  finding's `finding` (its index among the answer's objects), `file`
+  (repository-relative), `line` (as read for matching), `severity` and
+  `message`, and their `distance` in lines; `comment`, `severity` and
+  `message` are cut to 200 characters. `repo report --hits` lists them (§9).
 - `recall_addressed` counts only threads with `addressed: true` (the code
   changed in response), `recall_by_kind` groups by `kind`, and
   `precision_proxy` is the share of findings within the window of any
@@ -1330,7 +1336,19 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re
   When cells carry `result.json` `process` (§4.1), a line under the row gives
   the mean of each count over the cells that have it (`guidance_read` counted
   as paths; `counter:<name>` for `process_metrics`); a `null` or absent
-  `process` is left out, not counted as zero.
+  `process` is left out, not counted as zero. A row whose cells include
+  agent answers with no findings JSON that parsed (`grade.json`
+  `parse_error`, §4.5) notes how many, `(scored 0)` on a score-only row and
+  `(counted as failed)` on a gate row; a cell whose `result.json` says
+  `grader_failure` is counted under `grader_fail` instead. `--json` gives
+  `parse_errors` on every row. `repo report --hits` prints, instead of the table, each credited pair of
+  the `dokimi:review_replay` cells (`details.matches`, §4.5) by task, arm,
+  model and rep: the thread and its comment, then the finding that hit it.
+  `--hits --json` gives `{batch, replay_cells, without_matches, hits}`, one
+  record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the
+  match's fields; the first five come from the cell's `result.json`, and a
+  match key of the same name does not replace them). A cell graded before `matches` was recorded is counted
+  (`N cell(s) graded without hit details`), not listed.
 - `repo compare` reduces cells to one value per (task, arm, model) first:
   repetitions share a task's difficulty and are not paired across arms, so the
   task is the unit. Per treatment arm, model and measure (gate share, score,

dokimi/dokimi/commands/repo_run.py

@@ -383,14 +383,18 @@ def _harness_bin(name: str) -> Path:
 
 
 def cmd_repo_report(args: argparse.Namespace) -> None:
-    """Per (task, arm, model): gated success, grader failures, missing cells, cost, turns."""
-    from dokimi.repos.report import CellFileError, format_report, summarize
+    """Per (task, arm, model): gated success, grader failures, missing cells, cost, turns; or the replay hits."""
+    from dokimi.repos.report import CellFileError, format_hits, format_report, load_hits, summarize
 
     batch_dir = Path(args.batch_dir)
     if not batch_dir.is_dir():
         print(f"✗ not a directory: {batch_dir}", file=sys.stderr)
         sys.exit(1)
     try:
+        if args.hits:
+            found = load_hits(batch_dir)
+            print(json.dumps(found, indent=2) if args.json else format_hits(found))
+            return
         summary = summarize(batch_dir)
     except CellFileError as exc:
         print(f"✗ {exc}", file=sys.stderr)
@@ -545,6 +549,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
     report_p = repo_sub.add_parser("report", help="Per (task, arm, model) table over a batch directory")
     report_p.add_argument("batch_dir", help="<out>/<batch> of `repo run`")
     report_p.add_argument("--json", action="store_true", help="Print the summary as JSON")
+    report_p.add_argument("--hits", action="store_true",
+                          help="Instead of the table: each (thread, finding) pair that credited a "
+                          "dokimi:review_replay thread, by task, arm, model and rep (grade.json details.matches)")
     report_p.set_defaults(func=cmd_repo_report)
 
     compare_p = repo_sub.add_parser(

dokimi/dokimi/repos/graders/review.py

@@ -41,7 +41,12 @@
 to one and nearest first, so one finding credits at most one thread and one
 thread at most one finding. Location is the only criterion (no judge,
 deterministic), so ``pass`` is null: humans miss problems and raise
-non-defects, and neither recall nor precision is a verdict.
+non-defects, and neither recall nor precision is a verdict. ``matches`` lists
+each credited pair in ``threads_hit`` order, so a grade can be checked without
+the task: the thread (``thread``, ``thread_file``, ``thread_line``,
+``comment``), the finding (``finding``, its index among the answer's objects;
+``file`` repository-relative and ``line`` as matched; ``severity``,
+``message``) and their ``distance``; strings are cut to 200 characters.
 ``precision_proxy`` is the share of findings within the window of any thread,
 a lower bound on precision. ``recall`` equals ``recall_all`` and
 ``decoys_flagged`` is ``[]``, so a seeds-style report applies unchanged.
@@ -58,7 +63,9 @@
 import json
 import re
 
-VERSION = "1"  # bump when a result for the same inputs changes; grade.json records it
+# Bump when a result for the same inputs changes (a metric or ``pass``); grade.json records it. A field added
+# to ``details`` that leaves the result as it was (``matches``) does not: a reader tells it by its presence.
+VERSION = "1"
 SEVERE = ("blocker", "major")
 
 
@@ -216,18 +223,33 @@ def _ratio(a, b):
 
 
 def _replay_pairs(gold, findings, window):
-    """``(pairs, near)``: every (distance, finding index, thread id) within ``window``, and how many
-    findings are near any thread."""
+    """``(pairs, near)``: every (distance, finding index, thread id, thread) within ``window``, and how
+    many findings are near any thread."""
     pairs, near = [], 0
     for i, f in enumerate(findings):
         path, _, line = finding_fields(f)
-        close = [(abs(line - g["line"]), i, g["id"]) for g in gold
+        close = [(abs(line - g["line"]), i, g["id"], g) for g in gold
                  if same_file(g, path) and abs(line - g["line"]) <= window]
         pairs += close
         near += bool(close)
     return pairs, near
 
 
+CUT = 200  # characters of a comment or message a match keeps
+
+
+def _cut(value):
+    return "" if value is None else str(value)[:CUT]
+
+
+def _match(g, i, f, distance):
+    """One credited (thread, finding) pair of ``details.matches``: enough to check it without the task."""
+    path, _, line = finding_fields(f)
+    return {"thread": g["id"], "thread_file": g["file"], "thread_line": g["line"], "comment": _cut(g.get("comment")),
+            "finding": i, "file": path, "line": line, "severity": _cut(f.get("severity")),
+            "message": _cut(f.get("message")), "distance": distance}
+
+
 def grade_replay(task, task_dir, answer):
     """``review_replay``: score-only, ``recall`` over the human threads."""
     gold = _read_json(task_dir, task.get("gold", "gold.json"))
@@ -237,16 +259,17 @@ def grade_replay(task, task_dir, answer):
         return {"pass": False, "parse_error": True, "recall": zero, "recall_all": zero,
                 "recall_addressed": _ratio(0, sum(bool(g.get("addressed")) for g in gold))}
     pairs, near = _replay_pairs(gold, findings, task.get("window", 5))
-    hit, used = set(), set()
+    hit, used, credited = set(), set(), {}
     # nearest first: one finding per thread, one thread per finding; ids in _id_key order, as oneDAL's min()
-    for _, i, gid in sorted(pairs, key=lambda p: (p[0], p[1], _id_key(p[2]))):
+    for dist, i, gid, g in sorted(pairs, key=lambda p: (p[0], p[1], _id_key(p[2]))):
         if i not in used and gid not in hit:
             used.add(i)
             hit.add(gid)
+            credited[gid] = _match(g, i, findings[i], dist)
     addressed = [g for g in gold if g.get("addressed")]
     recall_all = _ratio(len(hit), len(gold))
     return {"pass": None, "n_findings": len(findings), "n_severe": n_severe(findings),
-            "threads_hit": _ids(hit), "n_threads": len(gold),
+            "threads_hit": _ids(hit), "matches": [credited[gid] for gid in _ids(hit)], "n_threads": len(gold),
             "recall_all": recall_all, "recall": recall_all,
             "recall_addressed": _ratio(sum(g["id"] in hit for g in addressed), len(addressed)),
             "recall_by_kind": _per_group(gold, "kind", hit),

dokimi/dokimi/repos/grading.py

@@ -226,18 +226,25 @@ def replaced_files(run_dir: Path) -> list[str]:
     return out
 
 
-def read_grade(run_dir: Path) -> tuple[dict[str, Any] | None, list[str]]:
-    path = run_dir / "grade.json"
+def read_grade_json(path: Path) -> tuple[Any, list[str]]:
+    """The JSON value of a cell's ``grade.json``, read as the host must: a link, FIFO or other non-regular file
+    is not read, and a file nested too deep to parse is no grade. ``(None, problems)`` when it is not one."""
     try:
         if not stat.S_ISREG(os.lstat(path).st_mode):
             return None, ["grade.json is not a regular file"]
         # O_NOFOLLOW: a link swapped in after the lstat is not followed either
         with os.fdopen(os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)), "rb") as f:
-            doc = json.loads(f.read().decode("utf-8"))
+            return json.loads(f.read().decode("utf-8")), []
     except FileNotFoundError:
         return None, ["no grade.json written"]
-    except (OSError, ValueError) as exc:
+    except (OSError, ValueError, RecursionError) as exc:
         return None, [f"grade.json is not JSON: {exc}"]
+
+
+def read_grade(run_dir: Path) -> tuple[dict[str, Any] | None, list[str]]:
+    doc, problems = read_grade_json(run_dir / "grade.json")
+    if problems:
+        return None, problems
     problems = check_grade(doc)
     return (None, [f"grade.json is not {GRADE_SCHEMA}: {p}" for p in problems]) if problems else (doc, [])