fix(report): report eval, generate and pipeline use one paired arm summary (#229 P1-3)

dokimi/changelog.d/20261026_130000_napetrov_report-eval-generate-agree.md

@@ -0,0 +1,28 @@
+### Changed
+
+- **`report generate` and the pipeline summary average each arm over the complete pairs, as `report eval` does** — the
+  Overall table, the topic and source breakdowns, and the `stats` block of `--format json` used
+  to average the context arm and the baseline arm over every question scored in each, so the two
+  means could cover different questions and their difference was not the delta printed beside
+  them. The `pipeline` run did the same for the summary it prints and stores in
+  `steps.evaluation.summary`. All of them now cover only the questions scored in both arms, and
+  the delta is the difference of those means. For an eval file with an unpaired question, the
+  context and baseline numbers change; for a file the judge writes, the delta does not. The report
+  says how many questions are unpaired (`paired_questions` and `unpaired_questions` in the JSON,
+  and a note plus the Count cell in the Markdown). To see the new numbers, run `report generate`
+  on an existing eval file again.
+  The top and bottom performer tables of `report generate` rank the complete pairs too. The
+  dashboard and the `report model-compare` run summary still average each arm over its own
+  scored questions, so for a file with an unpaired question their context and baseline means
+  differ from the ones `report generate` prints.
+
+### Fixed
+
+- **`report eval` computes the delta from unrounded means** — the Delta was the difference of the
+  two arm means after each had been rounded to one decimal, so it could be 0.1 away from the
+  difference of the unrounded means. It is now the unrounded difference rounded once for display,
+  so it can move by 0.1 in the Summary, Breakdown and Difficulty tables. The arm means themselves
+  print as before. Where questions are unpaired, the Count cell now shows them, and a note says the
+  means are over the complete pairs.
+  The win rate and the degradation rate in Key Observations divide by the complete pairs, not by
+  every question, so they can rise for a file with an unpaired question.

dokimi/docs/quickstart.md

@@ -161,7 +161,9 @@ python cli.py report generate \
 
 `reports/oneTBB.md` contains summary stats (min/max/avg context-arm/baseline/delta),
 top-N best and worst answers, top improvements and degradations, topic and
-persona clustering, and recommendations. The `reports/` directory is
+persona clustering, and recommendations. The summary stats cover only the questions
+scored in both arms; questions scored in one arm are counted and left out of them, and
+`report eval` computes the same numbers from the same file. The `reports/` directory is
 git-ignored.
 
 ### Step 6 — Dashboard (optional)

dokimi/dokimi/eval/stats.py

@@ -19,6 +19,10 @@
     Holm-Bonferroni (default) or Benjamini-Hochberg adjusted p-values.
 paired_vectors(evaluations, arm_a, arm_b)
     Per-question paired score vectors for two arms, complete cases only.
+paired_summary(evaluations)
+    Both arms and their delta over the complete pairs, with the unpaired count.
+    Called by the eval report, the report generator and the pipeline summary. The
+    dashboard and the model-compare summary still average each arm separately.
 mean_ci(values, ...)
     Student-t CI for the mean, coherent with ``p_ttest``; bootstrap fallback.
 point_biserial(flags, totals)
@@ -45,9 +49,10 @@
 
 import math
 import random
+from dataclasses import dataclass
 from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
 
-from dokimi.records import RecordLike, records
+from dokimi.records import BASELINE, CONTEXT, RecordLike, records
 
 try:
     from scipy import stats as _sp_stats
@@ -333,6 +338,64 @@ def paired_vectors(
     return a_vals, b_vals
 
 
+@dataclass(frozen=True)
+class PairedSummary:
+    """Both arms of one question set over its complete pairs, and the delta between them.
+
+    Every number is over the questions scored in both arms. The ``unpaired`` ones
+    (scored in one arm, or neither) are left out of all of them and counted here,
+    so no report can print a context mean over questions its baseline mean lacks.
+    The values are unrounded: round only when rendering.
+    """
+
+    total: int
+    paired: int
+    unpaired: int
+    with_mean: float
+    without_mean: float
+    delta_mean: float
+    with_min: float
+    with_max: float
+    without_min: float
+    without_max: float
+    delta_min: float
+    delta_max: float
+    improvements: int
+    degradations: int
+    neutral: int
+
+
+def paired_summary(evaluations: Iterable[RecordLike]) -> PairedSummary:
+    """The arms and their delta over the complete pairs of ``evaluations``.
+
+    The delta is the difference of the two paired means, which is also the mean of the
+    per-question deltas (``context - baseline``). Averaging each arm over *every* scored
+    question instead gives two means over different questions, whose difference is not
+    the delta the page prints beside them. An empty input gives zeros throughout.
+    """
+    recs = records(evaluations, source="evaluations", kind="evaluations")
+    with_s, without_s = paired_vectors(recs, CONTEXT, BASELINE)
+    deltas = [w - wo for w, wo in zip(with_s, without_s, strict=True)]
+    paired = len(deltas)
+    return PairedSummary(
+        total=len(recs),
+        paired=paired,
+        unpaired=len(recs) - paired,
+        with_mean=_mean(with_s) if paired else 0.0,
+        without_mean=_mean(without_s) if paired else 0.0,
+        delta_mean=_mean(deltas) if paired else 0.0,
+        with_min=min(with_s, default=0.0),
+        with_max=max(with_s, default=0.0),
+        without_min=min(without_s, default=0.0),
+        without_max=max(without_s, default=0.0),
+        delta_min=min(deltas, default=0.0),
+        delta_max=max(deltas, default=0.0),
+        improvements=sum(1 for d in deltas if d > 0),
+        degradations=sum(1 for d in deltas if d < 0),
+        neutral=sum(1 for d in deltas if d == 0),
+    )
+
+
 def point_biserial(flags: Sequence[bool], totals: Sequence[float]) -> Dict[str, Any]:
     """Discrimination of one binary item against respondents' total scores.
 

dokimi/dokimi/orchestrator/pipeline.py

@@ -10,8 +10,8 @@
 from pathlib import Path
 
 from dokimi.defaults import DEFAULT_ANSWER_MODEL, DEFAULT_ANSWER_PROVIDER, DEFAULT_JUDGE_MODEL, DEFAULT_JUDGE_PROVIDER, DEFAULT_RETRIEVAL_TOP_K
+from dokimi.eval.stats import paired_summary
 from dokimi.mcp.factory import create_doc_source_client
-from dokimi.records import BASELINE, CONTEXT, records as read_records
 
 logger = logging.getLogger(__name__)
 
@@ -380,17 +380,14 @@ def run(self, concurrency: int = 5) -> Dict[str, Any]:
             "path": str(self.eval_path)
         }
 
-        # Compute summary stats
-        scored = read_records(evaluations, source="evaluations", kind="evaluations")
-        with_scores = [s for s in (r.score(CONTEXT) for r in scored) if s is not None]
-        without_scores = [s for s in (r.score(BASELINE) for r in scored) if s is not None]
-        deltas = [r.delta for r in scored if r.delta is not None]
-
-        if with_scores and without_scores and deltas:
+        # Summary stats over the complete pairs, the numbers the reports print for the same file.
+        summary = paired_summary(evaluations)
+
+        if summary.paired:
             results["steps"]["evaluation"]["summary"] = {
-                "with_avg": round(sum(with_scores) / len(with_scores), 1),
-                "without_avg": round(sum(without_scores) / len(without_scores), 1),
-                "delta_avg": round(sum(deltas) / len(deltas), 1)
+                "with_avg": round(summary.with_mean, 1),
+                "without_avg": round(summary.without_mean, 1),
+                "delta_avg": round(summary.delta_mean, 1)
             }
             print(f"✓ Evaluated {len(evaluations)} answers: "
                   f"context={results['steps']['evaluation']['summary']['with_avg']}, "

dokimi/dokimi/report/eval_report.py

@@ -35,10 +35,6 @@
     _TZ = None
 
 
-def avg(lst):
-    return round(sum(lst) / len(lst), 1) if lst else 0.0
-
-
 def significance_test(evals):
     """Paired t-test, Wilcoxon, and Cohen's d_z on (with_docs - without_docs) pairs.
 
@@ -91,22 +87,41 @@ def _difficulty(e: Record, default):
 
 
 def stats(qs):
-    valid = [e for e in qs if has_scores(e)]
-    with_s = [e.score(CONTEXT) for e in valid]
-    without_s = [e.score(BASELINE) for e in valid]
-    deltas = [d for e in valid if (d := e.delta) is not None]
+    """The arms and delta over the complete pairs (:func:`eval.stats.paired_summary`).
+
+    ``count`` is every question in ``qs``, ``unpaired`` the ones left out of the means.
+    Rounded here, once, for display: the delta is the difference of the unrounded means.
+    """
+    s = eval_stats.paired_summary(qs)
     return {
-        "count": len(qs),
-        "valid": len(valid),
-        "with_avg": avg(with_s),
-        "without_avg": avg(without_s),
-        "delta_avg": round(avg(with_s) - avg(without_s), 1),
-        "improvements": sum(1 for d in deltas if d > 0),
-        "degradations": sum(1 for d in deltas if d < 0),
-        "neutral": sum(1 for d in deltas if d == 0),
+        "count": s.total,
+        "valid": s.paired,
+        "unpaired": s.unpaired,
+        "with_avg": round(s.with_mean, 1),
+        "without_avg": round(s.without_mean, 1),
+        "delta_avg": round(s.delta_mean, 1),
+        "improvements": s.improvements,
+        "degradations": s.degradations,
+        "neutral": s.neutral,
     }
 
 
+def _count_cell(s: dict) -> str:
+    """The question count, with the unpaired questions the arm means leave out, if any."""
+    return str(s["count"]) if not s["unpaired"] else f'{s["count"]} ({s["unpaired"]} unpaired)'
+
+
+def _unpaired_note(s: dict) -> list[str]:
+    """A line saying what the Summary means leave out; nothing when every question is paired."""
+    if not s["unpaired"]:
+        return []
+    return [
+        f'_Arm means and delta are over the {s["valid"]} complete pairs; '
+        f'{s["unpaired"]} unpaired (scored in one arm only) left out._',
+        "",
+    ]
+
+
 def fmt_delta(d):
     if d is None:
         return "N/A"
@@ -267,11 +282,12 @@ def generate_report(eval_path: str, out_path: str, qa_json_out: str | None = Non
         "## Summary",
         "| Set | Count | Context arm | Baseline | Delta |",
         "|---|---:|---:|---:|---:|",
-        f'| **All** | {all_stats["count"]} | {all_stats["with_avg"]} | {all_stats["without_avg"]} | **{all_stats["delta_avg"]:+}** |',
-        f'| Generated (dynamic) | {dyn_stats["count"]} | {dyn_stats["with_avg"]} | {dyn_stats["without_avg"]} | **{dyn_stats["delta_avg"]:+}** |',
-        f'| Golden (static) | {static_stats["count"]} | {static_stats["with_avg"]} | {static_stats["without_avg"]} | **{static_stats["delta_avg"]:+}** |',
+        f'| **All** | {_count_cell(all_stats)} | {all_stats["with_avg"]} | {all_stats["without_avg"]} | **{all_stats["delta_avg"]:+}** |',
+        f'| Generated (dynamic) | {_count_cell(dyn_stats)} | {dyn_stats["with_avg"]} | {dyn_stats["without_avg"]} | **{dyn_stats["delta_avg"]:+}** |',
+        f'| Golden (static) | {_count_cell(static_stats)} | {static_stats["with_avg"]} | {static_stats["without_avg"]} | **{static_stats["delta_avg"]:+}** |',
         "",
     ]
+    lines += _unpaired_note(all_stats)
 
     # Statistical significance
     sig = significance_test(evals)
@@ -322,7 +338,7 @@ def _val(key):
         ("**Total**", all_stats),
     ]:
         lines.append(
-            f'| {label} | {s["count"]} | {s["with_avg"]} | {s["without_avg"]} '
+            f'| {label} | {_count_cell(s)} | {s["with_avg"]} | {s["without_avg"]} '
             f'| **{s["delta_avg"]:+}** | {s["improvements"]} | {s["degradations"]} | {s["neutral"]} |'
         )
     lines.append("")
@@ -337,7 +353,7 @@ def _val(key):
         if d in diff_groups:
             s = stats(diff_groups[d])
             lines.append(
-                f'| {d} | {s["count"]} | {s["with_avg"]} | {s["without_avg"]} '
+                f'| {d} | {_count_cell(s)} | {s["with_avg"]} | {s["without_avg"]} '
                 f'| **{s["delta_avg"]:+}** | {s["improvements"]} | {s["degradations"]} |'
             )
     lines.append("")
@@ -454,10 +470,10 @@ def _val(key):
         f'- **Dynamic vs Static agreement:** {dyn_stats["delta_avg"]:+} vs {static_stats["delta_avg"]:+} — '
         + ("good consistency, benchmark is stable." if abs(dyn_stats["delta_avg"] - static_stats["delta_avg"]) < 3
            else "notable gap — investigate question quality."),
-        f'- **Win rate:** {round(all_stats["improvements"]/all_stats["count"]*100) if all_stats["count"] else 0}% questions improved with context '
-        f'({all_stats["improvements"]}/{all_stats["count"]}).',
+        f'- **Win rate:** {round(all_stats["improvements"]/all_stats["valid"]*100) if all_stats["valid"] else 0}% questions improved with context '
+        f'({all_stats["improvements"]}/{all_stats["valid"]}).',
         f'- **Degradations:** {all_stats["degradations"]} questions '
-        f'({round(all_stats["degradations"]/all_stats["count"]*100) if all_stats["count"] else 0}%).',
+        f'({round(all_stats["degradations"]/all_stats["valid"]*100) if all_stats["valid"] else 0}%).',
         "",
     ]