[antonwolfy]

Summary

Settles #292, where repo compare takes a row's 95% interval from one method and its p from another, and the
two disagree near 0.05 (issue). The open questions (Holm, ties
under a shift, no mid-p, no flag, speed) are settled in its author's comment on the issue, and this PR follows
that comment.

On main, gate and score rows get a percentile bootstrap (bootstrap_ci), the other rows get a t interval
(mean_ci), and p always comes from the sign-flip test. Two rows from a replay of a dpnp PR (26
review_replay tasks):

row	delta	interval on main	p (sign-flip)	interval here
metric:n_findings, 26 tasks	+0.885	t: [-0.053, +1.822] (includes 0)	0.0498	[+0.059, +1.818] (excludes 0)
score, 6 tasks, 3 up and 3 tied	+0.139	bootstrap: [+0.028, +0.250] (excludes 0)	0.25	[+0.000, +0.333] (includes 0)

The replay's own n_findings deltas can't be recovered, so the first row uses 26 integer deltas that reproduce it
(the issue's row read +0.89, t [-0.091, +1.860], p 0.0497). The score row is the issue's own (deltas 0, 0, 0.333,
0.167, 0, 0.333). On main the second row carries interval_p_disagree and the "two methods" note. Here it reads
as inconclusive, with the floor note (with 3 changed tasks the sign-flip p cannot go below 0.25).

Changes

• New dokimi/dokimi/repos/compare_interval.py:
  • ShiftTest.p(theta) is the sign-flip p of d - theta over every task. A shifted delta within EPS of 0
    is a tie, so the tasks tied at 0 count once theta moves off 0. At theta = 0 it equals sign_flip_p, exact
    and Monte Carlo, None included when every delta ties, up to rounding at large deltas where magnitudes
    repeat (below); the row's own p places 0 either way. Exact, the observed sum it compares
    against is the least of its math.fsum and the product's sums for the observed signs and their mirror, so
    the observed pattern counts itself however large the deltas.
  • The observed sum and the mean the bisection starts from come from math.fsum, not sum. Python's sum
    of floats is compensated from 3.12 on and adds left to right on 3.11; math.fsum rounds exactly on every
    version, so a row's ends are the same on every Python.
  • Up to EXACT_MAX (16) non-zero shifted deltas the test is exact: one cached 2^k x k sign matrix per k,
    and one matrix-vector product per theta.
  • Above that it is Monte Carlo: one 20000 x n sign matrix, drawn once and reused at every theta (common
    random numbers). Its columns for the changed tasks are the very draws sign_flip_p makes: numpy's
    MT19937 is seeded with random.Random(seed)'s state, so it gives the same doubles. So the printed p
    does not change.
  • sign_flip_interval bisects each end to 1e-7 and returns the last shift not rejected. The bracket runs
    from the mean to a far bound past every delta and 0. The far bound is 1 past the extreme below 1, and
    twice the extreme from 1 on, so a delta of 1e16 or more cannot round the bound back onto itself. The
    ends are step points of p(theta); there is no mid-p.
  • An end is None (open) when p at the far bound is still >= 0.05, which is the case for every row of 5
    or fewer tasks. From 6 tasks on both ends are bounded, whatever the ties and however large the deltas
    (up to SCORE_BOUND, 1e150).
  • The row's own p decides whether 0 is inside: the bisection bracket starts at 0. So the interval excludes 0
    exactly when p < 0.05. That includes a row of more than 16 tasks with 16 or fewer changed, where p is
    exact and the interval is Monte Carlo.
  • Rounding to 4 places never moves an end onto or across 0. The ends are rounded with exact=True too (the
    ledger's call), since bisection only knows them within 1e-7; rounded=False returns the bisection points.
  • EXACT_MAX, N_RESAMPLES and EPS move here. compare.py imports them, so compare.EXACT_MAX and the
    ledger's from .compare import EPS still work.
• repos/compare.py (774 to 765 lines, kept small and local):
  • paired_summary uses the new interval for every measure; it no longer calls bootstrap_ci / mean_ci.
  • delta gains "method": "sign_flip_inversion". delta.mean is rounded to 4 places as before, and a mean
    that rounds onto 0 from below is now 0.0, not -0.0. Rounded, it stays inside the row's rounded ends
    (compare_interval.within); with exact=True (the ledger's call) it keeps full precision.
  • _holm and _inconclusive_note treat a None end as open. The inconclusive note keeps its "2+ tasks"
    rule through n_tasks.
  • The disagreement note reads the interval [...] excludes 0 but p_holm X is ≥ 0.05: the interval is uncorrected; p_holm decides this row. interval_p_disagree can now be true only on a gate or score row
    whose p was lifted across 0.05 by Holm's correction.
  • format_compare prints an open end as -inf / +inf, so a one-task row now reads [-inf, +inf], not -.
  • mde, p_holm_floor and the floor note are unchanged.
• repos/ledger.py: format_ledger prints the interval with the same formatter, so a null end (5 or fewer
  paired cases) prints as open and cannot crash. _entry copies delta.method into each rule result (null
  for a result with no pairs). No schema changes: there is no repo_compare schema, and
  repo_ledger.v1.json describes the ledger file, not delta.
• Review follow-ups:
  • The far bound was the extreme ± 1.0. That left the upper end of [1e16] * 6 open ((1e16, None)):
    1e16 + 1.0 == 1e16, so every shifted delta tied there. The lower end of [-1e16] * 6 was open the
    same way. Fixed in 9592980.
  • In c1f2923, delta.mean no longer reads -0.0 in --json and -0.000 in the table.
  • An interval whose ends lie strictly between 0 and 0.0005 printed as [+0.000, +0.000], and the Holm
    note then said that interval excludes 0. In 89e2d21 the interval formatter keeps 3 decimals, but a
    non-zero end that would read 0.000 prints at 2 significant digits ([+0.0002, +0.0002]), in
    repo compare and the ledger text. An end of exactly 0 still prints +0.000; the mean column stays at
    3 decimals.
  • From deltas of about 1e6 on, the exact ShiftTest.p summed the observed deltas in Python and each
    pattern in the matrix product, and the two rounded apart by more than EPS: the observed pattern stopped
    counting and the bisection went to another step point. In 02bf4c6 the observed sum is the least of the
    Python sum and the product's sums for the observed signs and their mirror.
  • The ledger calls compare(..., exact=True), which returned the raw bisection points as ends
    (0.5000000223517418), and its rule results dropped delta.method. In 7a667bd the ends are rounded to 4
    places in both cases, as on main, and delta.method is copied.
  • min_detectable said its t quantiles are those of mean_ci's interval; in 00a5894 it says they are those
    of the planning figure.
  • The Tests (py3.11) check failed: the upper end of [1e16] * 5 + [2e16] was 2e16 + 4, not 2e16.
    02bf4c6's threshold took the Python sum, and 3.11's left-to-right sum rounded the observed sum at the
    shift 2e16 + 4 below the product's, so a pattern 8 below it counted too and p rose from 2/64 to 4/64.
    In 4e871a3 the observed sum and the mean come from math.fsum.
  • Six deltas of 4e-5 gave delta.mean 0.0 with lo = hi = 0.0001: the ends are kept off 0 at
    ±0.0001, and the mean was rounded on its own. In cd41468 the rounded mean is held inside the rounded
    ends, so that row reads mean 0.0001 (-0.0001 for -4e-5). A mean whose interval holds 0 rounds as
    before. The text's mean column stays at 3 decimals (+0.000).
  • The ShiftTest docstring said p at 0 is sign_flip_p, and the module docstring that the Monte Carlo
    p(0) is the printed p. At large deltas whose magnitudes repeat (1e9 + 0.1, 2e9 + 0.2), sign patterns
    that tie the observed sum in exact arithmetic round to one side of it in the matrix product and to the
    other in sum(), so p(0) can differ. In 05cb808 both docstrings say so, and that the row's own p, not
    p(0), places 0.
  • The Tests (py3.11) check failed on 05cb808's 1e9 row: the test pinned its printed p at 0.0625, but
    sign_flip_p sums with sum(), and on those deltas 3.11's left-to-right sum counts one tie pattern
    fewer, so p reads 0.046875 and the lower end 0.0001. The row agrees with its p on both Pythons; only the
    pin depended on the version. In fc93796 the test runs that row under a compensated sum and under
    left-to-right addition, and pins each one's p and lower end.
• Tests:
  • New tests/test_repo_compare_interval.py covers:
    • both issue rows;
    • a 9-task row scaled by 1e7, whose ends and p at 401 shifts match the unscaled row and sign_flip_p;
    • open ends at 1-5 tasks, ties bounding a 6-task row, and bounded ends at ±1e16 and ±1e150, under the
      builtin sum and under left-to-right addition (3.11's);
    • the same unrounded ends for 40 rows at 1e6 to 1e100 under the builtin sum, a compensated sum and
      left-to-right addition;
    • 20 tasks with 5 or 6 changed;
    • p(0) == sign_flip_p: exact, Monte Carlo, a near-tie at the 16/17 boundary, and None when every
      delta ties;
    • a 7-task row at 1e9 where p(0) can read 0.046875: under a compensated sum its printed p is 0.0625 and
      its interval reaches 0, under left-to-right addition (3.11's) its p is 0.046875 and its lower end 0.0001;
    • agreement over 170 random delta sets: exact and Monte Carlo, ties, more than 16 tasks with 16 or
      fewer changed, exact=True and rounded;
    • ends at step points, and Monte Carlo ends that stay the same across calls with the draw cache
      cleared;
    • rounding, including a mean rounded onto 0, and a mean of ±4e-5 held inside ends of ±0.0001;
    • open-end text, small non-zero ends in the text, and method / null in JSON.
  • Updated: the tests in test_repo_compare.py and test_repo_compare_reading.py that pinned bootstrap/t
    ends or the old note wording now pin the new numbers and wording. test_repo_compare_reading.py also
    pins a Holm family of two rows with ends of +0.0002, in the table and in the note.
  • test_repo_ledger.py pins [+1.000, +1.000] and [-inf, +inf] in the ledger text, and a rule's
    delta of {mean 0.875, lo 0.5, hi 1.0, method sign_flip_inversion}.
• Docs:
  • docs/repo-evaluation-contract.md §9, the repo compare bullet: the method, exact/Monte Carlo, ties
    under a shift and the open end, step points and no mid-p, how a small non-zero end prints, the reworded
    Holm note, delta.method.
  • docs/architecture.md: the module map.
  • The compare.py module and paired_summary docstrings, and the compare_interval.py module docstring.
• Research register: research/records/decision/D018.md (separate commit). It records the method for both
  tracks. The arms track (dokimi/eval/stats.py) is not changed here; it follows in its own issue.
• Changelog fragment (Changed).

Risk

• Every repo compare interval changes value: gate and score rows lose the bootstrap, the other rows the t
  interval. Rows of 2 to 5 tasks had bounded ends; now they have null ends in --json and print
  [-inf, +inf].
• A reader of --json that formats delta.lo / delta.hi as numbers whenever n_tasks > 1 must handle
  null. Inside the package those readers are format_compare, _reading_notes, _inconclusive_note, _holm
  and format_ledger, all handled here. Each row names its method, so output from before and after this
  change can be told apart. The release before this change still computes the old intervals.
• An interval end can now sit exactly on a delta or on 0 (the step points of an exact test), e.g. [+0.000,
  +0.333] above.
• A non-zero interval end that 3 decimals would print as 0.000 now prints at 2 significant digits
  (+0.0002), so its text is wider than 3 decimals. --json is unchanged.
• The ledger's --json rule results gain delta.method (null without pairs); their ends stay at 4 places.
• Printed p, p_holm, mde and p_holm_floor do not change. delta.mean changes in two cases only: a mean
  that rounds onto 0 from below was -0.0 on main (its round) and is 0.0 here; and in repo compare, a
  non-zero mean that rounds onto 0 next to an end kept off 0 was 0.0 and is that end, ±0.0001.
• Overlap: open PR feat(repo): repo compare flags replay tasks a guidance revision may have seen #392 (the compare flags of repo compare: flag replay tasks a guidance revision may have seen #290) and open PR feat(repo): repo compare --metric judge:recall_judged reads judged recall #395 also edit repos/compare.py. This
  diff there is local: the imports, the module docstring's interval sentence and third note, _holm's one
  line, paired_summary, the format_compare interval cell, _reading_notes' first branch and
  _inconclusive_note's guard.
• Cost: one paired_summary row, median of 5 runs. The interval itself took 4 ms at 16 changed tasks (2^16
  patterns), 2 ms at 26 tasks (20 changed, Monte Carlo) and 13 ms at 200 tasks. The whole row took 52, 22 and
  161 ms, most of it in sign_flip_p, which is unchanged. Sign matrices: up to about 17 MB cached for the
  exact range, and 32 MB per Monte Carlo matrix at 200 tasks, with two kept. One interval makes about 52
  p(theta) calls. Summing the moved deltas as a Python list with math.fsum takes under 1% of the interval's
  time.

Testing

On a Linux review host, at fc93796 (on main 8d54c93):

• make ci-local: 30 passed, 0 failed, 7 skipped. The skips: external-catalog-smoke needs docker;
  test-latest, docs deploy and pip-audit report have job conditions that are false on a PR; cli-surface
  "Package the result" has an unknown expression; conda-forge-deps has no micromamba; semgrep secrets is
  not on PATH. 15681 tests passed, 64 skipped. Changed-line coverage was 100% (88 lines measured).

• The two issue rows: on main the 26-task row's t interval includes 0 against p 0.0498, and the score row's
  bootstrap interval excludes 0 against p 0.25. Here both agree with p (table above). They are pinned in
  test_the_issue_n_findings_row_now_excludes_zero_as_its_p_says and
  test_the_issue_score_row_now_includes_zero_as_its_p_says.

• The review follow-ups, at e609c31 and here:

  input	at e609c31	here
  sign_flip_interval([1e16] * 6, 0.03125)	(1e16, None)	(1e16, 1e16)
  sign_flip_interval([-1e16] * 6, 0.03125)	(None, -1e16)	(-1e16, -1e16)
  [1e16] * 5 + [2e16]	(1e16, None)	(1e16, 2e16)
  delta.mean of deltas [-1e-5, 0, 0]	-0.0 (prints -0.000)	0.0 (prints +0.000)

  test_an_end_is_bounded_at_deltas_too_large_to_move_by_one and
  test_a_mean_rounded_to_zero_is_zero_not_negative_zero fail at e609c31 and pass here.

• The small-end follow-up: six score deltas of +0.0002 for each of two treatments (p 0.03125, p_holm 0.0625)
  printed [+0.000, +0.000] at 7013897, in the table and in the Holm note that says it excludes 0. Here they
  print [+0.0002, +0.0002]. test_a_small_interval_against_p_holm_prints_its_ends_off_zero and
  test_an_end_three_decimals_would_print_as_zero_keeps_its_sign_and_value fail at 7013897 and pass here.
  The first also fails on main, which prints the same [+0.000, +0.000].

• Large deltas, at 89e2d21 and here: from deltas of about 1e6 on, the product's sum for the observed signs
  fell below the Python-summed observed sum minus EPS at 136 to 622 of 3000 random shifts per scale. Against
  a test that sums each pattern with Python's sum, as sign_flip_p does, on 6-10 task rows at 1, 1e6, 3e6,
  1e7 and 1e9: p now matches at all 2000 random shifts, and every end is within 1.4e-6 of that test's. A
  relative tolerance, EPS * max(1, sum|x|), was tried and not taken: it moved ends by up to 0.06 at 1e7.
  A tolerance of the product's worst-case rounding was tried too: it put the lower end of
  [1e16] * 5 + [2e16] at 9999999999999994. test_large_deltas_keep_the_ends_and_the_p_of_the_shifted_deltas
  fails at 89e2d21: the 9-task row scaled by 1e7 has ends 667388 and 7566582, about 720 off.

• The ledger: a rule over gate deltas [1] * 7 + [0] read lo 0.5000000223517418, hi 0.9999999925494194 and
  no method at 89e2d21; here lo 0.5, hi 1.0, method sign_flip_inversion.
  test_a_rule_delta_has_ends_at_four_places_and_names_its_method fails at 89e2d21 and passes here.

• Python 3.11's sum, at 7a667bd and here: with compare_interval's sum replaced by left-to-right
  addition (functools.reduce(operator.add, xs, 0.0)), [1e16] * 5 + [2e16] read (1e16, 2.0000000000000004e16)
  at 7a667bd, the failing check's value; here it reads (1e16, 2e16). On 238 random rows of 6 to 24 tasks at
  scales 1 to 1e100, the rounded ends at 7a667bd changed with the summation in 67 rows, and here in none.
  Here they equal 7a667bd's ends under 3.13's sum in every row, and p at 0 equals sign_flip_p in every row
  under both summations. test_an_end_is_bounded_at_deltas_too_large_to_move_by_one[left_to_right] and
  test_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_right fail at 7a667bd and pass here.

• The tiny-mean follow-up, at 4e871a3 and here: six deltas of 4e-5 read delta.mean 0.0, lo 0.0001, hi 0.0001 at 4e871a3 (-4e-5 the mirror); here the mean is 0.0001 (-0.0001). Five of 4e-5 and one of
  -1e-5, whose interval holds 0, keep mean 0.0. test_a_tiny_mean_stays_inside_its_interval_kept_off_zero
  fails at 4e871a3 and passes here; main has no compare_interval to run it on.

• p(0) at 1e9, at 4e871a3: on 6-10 task rows, 500 per shape, p(0) equals sign_flip_p on every row of
  uniform deltas, quarter fractions or tenths, and differs on 169 to 296 rows drawn from a few repeated
  magnitudes (1e9 + 0.1, 2e9 + 0.2, 3e9 + 0.3, 1e9 + 1/3), under 3.13's sum and left-to-right
  addition. 281 of 300 Monte Carlo rows of 17-22 such deltas differ too. The printed p and the side of 0 do not
  depend on it: on the first three such rows drawn where p(0) and sign_flip_p fall on either side of 0.05, the interval
  follows the printed p. test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart pins one.

• The 1e9 row on Python 3.11, at 05cb808 and here: with compare's sum replaced by left-to-right addition,
  the row reads p 0.046875 and ends (0.0001, 1500000000.2667), the failing check's p; under 3.13's sum it
  reads p 0.0625 and (0.0, 1500000000.2667). interval_p_disagree is false in both. The test now pins both
  readings on every Python: [compensated] and [left_to_right] pass here.

• A finer bisection tolerance, below EPS, was tried and not taken. It moves rounded ends into the tie zone
  around 0: the issue's score row would go from (0.0, 0.333) to (-0.0001, 0.333), and six tied deltas
  from (0.0, 0.0) to (-0.0001, 0.0001).

• Mutants, each on a copy of the commit, with the tests that pin it run there. All fail but one:

  • the row's p no longer placing 0;
  • ties dropped under a shift;
  • Monte Carlo draws not shared with sign_flip_p;
  • exact enumeration disabled;
  • tie zeroing of shifted deltas dropped;
  • no open end;
  • bisection returning the rejected point;
  • rounding allowed onto 0;
  • an open end printed as -, in compare_interval and in the ledger;
  • delta.method dropped;
  • the seed not passed to the interval;
  • the agreement flag inverted;
  • Holm no longer setting interval_p_disagree;
  • the old note wording;
  • an open end read as excluding 0 by the inconclusive note;
  • the inconclusive note given to a one-task row;
  • the old bootstrap/t interval restored;
  • the far bound back to the extreme ± 1.0, both ends and the upper end alone;
  • the "no float left" break dropped (the bisection then spins at 1e16 until pytest-timeout ends it);
  • ShiftTest.p returning 1.0 instead of None on an all-tied row, and with that guard removed;
  • the + 0.0 on delta.mean removed;
  • a small non-zero end printed at 3 decimals again, and an end of exactly 0 printed at 2 significant
    digits;
  • the exact observed sum back to the Python sum alone;
  • the ends left unrounded under exact=True;
  • delta.method not copied into the ledger's rule results;
  • the observed sum back on sum (both the left-to-right leg and the three-summation test fail);
  • the bisection's mean back on sum (the three-summation test fails, one end off by an ulp);
  • delta.mean no longer held inside its ends, and held by only its lower or only its upper end (the
    tiny-mean test fails for both signs, then for -4e-5, then for +4e-5);
  • 0 placed by ShiftTest.p(0) instead of the row's p (the 1e9 row's lower end reads 0.0001, not 0.0);
  • 0 always counted inside the interval (the 1e9 row's left-to-right case reads lower end 0.0, not 0.0001).

  The one test that passes under a mutant is test_the_ends_are_step_points_of_p with exact enumeration
  disabled, because Monte Carlo ends are step points too. That mutant is caught by
  test_p_at_zero_is_the_sign_flip_p_exact_and_monte_carlo.

• Not run: the matrix legs on Python 3.11, 3.12 and 3.14 (make ci-local runs one Python). 3.11's summation
  is simulated in the tests above, not run on a 3.11 interpreter.

Revert

git revert of the PR's commits restores the bootstrap/t intervals. No stored artifact or schema version
changes. JSON written in between carries delta.method, which the old code neither writes nor reads.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s).

CI at fc93796: 19 passing, 2 skipped.

This PR has reached its review limit, so only a re-request starts a review.

Review details

• Commit reviewed: fc93796 against merge base 8d54c93
• Coverage: 12 files in 3 groups
• Engine: OpenCodeReview v1.13.5-exai.3.d7112371, us.openai.gpt-6.1-sol
• Usage: 299k tokens (189k from cache), 47s
• Tool calls: 19 (11 file_read, 4 code_search, 2 file_read_diff, 1 ci_status, 1 pr_threads)
• Comparison sign-flip intervals and comparison behavior: 6 files
• Ledger integration and tests: 2 files
• Comparison design, contract, and release documentation: 4 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.76%. Comparing base (8d54c93) to head (fc93796).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #409   +/-   ##
=======================================
  Coverage   94.75%   94.76%           
=======================================
  Files         352      353    +1     
  Lines       46507    46479   -28     
  Branches     7970     7956   -14     
=======================================
- Hits        44070    44046   -24     
+ Misses       1544     1542    -2     
+ Partials      893      891    -2     

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/compare.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/compare_interval.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/compare_reach.py	100.00% <ø> (ø)
dokimi/dokimi/repos/ledger.py	98.46% <100.00%> (+<0.01%)	⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

• np.random.RandomState is a legacy API, and manually constructing/setting the internal MT19937… · New
• moved.tolist() allocates a Python list on every p(theta) call, and p(theta) is called… · New
• Rounding can produce -0.0 for delta.mean (e.g., a tiny negative mean rounded to 4 places).… · New

What changed in this PR

Aligns repo compare confidence intervals with the sign-flip p-value by inverting the sign-flip test to produce a consistent 95% interval across all measures, addressing interval/p disagreements near α=0.05 (issue #292).

Changes:

• Introduces compare_interval.py implementing sign-flip test inversion (exact up to 16 non-zero shifted deltas, Monte Carlo above) and shared formatting/helpers.
• Updates compare.py and ledger formatting to use the new interval everywhere, including open-ended intervals (None → ±inf in text).
• Adds/updates tests and documentation to reflect the new interval semantics and note wording.

File	Description
dokimi/​dokimi/​repos/​compare_interval.py	New implementation of sign-flip inversion interval, formatting, and helpers.
dokimi/​dokimi/​repos/​compare.py	Switches all measures to the new interval; updates Holm/disagreement logic and output formatting.
dokimi/​dokimi/​repos/​ledger.py	Uses shared interval formatter to safely print open-ended intervals.
dokimi/​tests/​test_repo_compare_interval.py	New targeted tests for inversion behavior, determinism, rounding, and JSON formatting.
dokimi/​tests/​test_repo_compare.py	Updates tests to reflect new interval method and output.
dokimi/​tests/​test_repo_compare_reading.py	Updates note/disagreement expectations and interval endpoints.
dokimi/​tests/​test_repo_ledger.py	Adds coverage for open intervals in ledger text and JSON null ends.
dokimi/​docs/​repo-evaluation-contract.md	Documents the new interval definition and open-end behavior.
dokimi/​docs/​architecture.md	Updates module map to include the new interval module.
dokimi/​research/​records/​decision/​D018.md	Records the decision and rationale for switching interval methodology.
dokimi/​changelog.d/​20261105_120000_antonwolfy_compare-sign-flip-interval.md	Changelog entry describing behavioral changes and compatibility notes.

_{Review effort: Lite}

[vlad-perevezentsev]

Review at 7013897. The change does what it says: on every row the interval now excludes 0 exactly when p is below 0.05, and the printed p, p_holm, mde and p_holm_floor stay as they were. The far bound fix (9592980) and the -0.0 fix (c1f2923) are right. What is left is small: the tie tolerance at large values and a few leftovers, in the line comments.

Agreed with @exai-bot on ends below 0.0005 printed as +0.000; on the [1e-8] * 6 ends I disagree and replied in its thread.

What I ran, at 7013897:

• make ci-local --offline: 0 failed, 19 passed, 13 skipped (network, Docker and other-Python jobs).
• Coverage of the 95% interval for the true mean delta, 300 simulated rows each, against the old bootstrap and t intervals: gate +0.3/-0.1 over 12 tasks 0.957 (bootstrap 0.890, t 0.970); gate +0.5 over 16 tasks 0.907 (0.977, 0.907); skewed continuous 0.937 (0.907, 0.933); normal 0.973 (0.953, 0.970). So the new interval covers about as well as the old ones did.
• Large values: [±1e16] * 6 and [1e16, 2e16, 3e16] * 2 now give bounded ends; the interval against the same test with sign_flip_p's sums, 150 random rows per scale, differs from 3e6 on.

Checked, no issues

• 0 and p: the row's p places 0 in _end, and _rounded keeps an end off 0, so interval_p_disagree fires only where Holm lifts p, as the docs say.
• Monte Carlo draws: the first columns of _draws are sign_flip_p's draws for the same seed, and the tests pin p(0) against it, ties near EPS included.
• Open ends: at 5 or fewer tasks the floor 2/32 is above 0.05, so both ends are null and print as ±inf in repo compare and the ledger text.
• Docs and changelog: the contract, the architecture page, D018 and the fragment say the same thing; the fragment says the ends can now be null from 2 to 5 tasks.

[antonwolfy]

Review rounds done at 4e871a3: 4 checks still running (CLI interface diff, CodeQL (actions), CodeQL (python), Windows smoke test (py3.13)), and codecov reports no changed line uncovered.

Round	Commits	Concerns
1	70138979, c1f2923b, 95929805	fixed: dokimi/dokimi/repos/compare_interval.py:98 (exai-bot), dokimi/dokimi/repos/compare.py:569 (copilot-pull-request-reviewer), codecov; declined: dokimi/dokimi/repos/compare_interval.py:61 (copilot-pull-request-reviewer), dokimi/dokimi/repos/compare_interval.py:86 (copilot-pull-request-reviewer)
2	89e2d21a	fixed: dokimi/dokimi/repos/compare_interval.py:137 (exai-bot); declined: dokimi/dokimi/repos/compare_interval.py:110 (exai-bot)
3	7a667bd8, 00a58945, 02bf4c67	fixed: dokimi/dokimi/repos/compare_interval.py:82 (@vlad-perevezentsev), dokimi/dokimi/repos/compare_interval.py:100 (@vlad-perevezentsev), dokimi/dokimi/repos/compare.py:555 (@vlad-perevezentsev)
4	4e871a3b	fixed: ci:Tests (py3.11)

Declined, with the reason in each thread:

• dokimi/dokimi/repos/compare_interval.py:61 (copilot-pull-request-reviewer): NumPy's own docs say the opposite about which API is stable.
• dokimi/dokimi/repos/compare_interval.py:86 (copilot-pull-request-reviewer): I measured it, and the list costs almost nothing.
• dokimi/dokimi/repos/compare_interval.py:110 (exai-bot): The trigger reproduces, but the shifts it leaves out are only the EPS tie zone, and resolving that zone makes published ends worse.

@vlad-perevezentsev: the rounds above answer your points; please take another look.

[napetrov]

COMMENT at 4e871a3: the inversion does what #292 asks; earlier threads (T1-T9) are resolved on head. Two nits and one question inline, none blocking.

#	Finding	Provenance
1	mean outside own interval, tiny deltas	changed (probed)
2	p(0) vs sign_flip_p at ~1e9 scale	introduced (probed)
3	ledger -0.000	pre-existing (read)

Coverage: plain + edge pass with their probes; not run: full pytest (no scipy here), end-to-end repo compare/repo ledger. CI still running at review time.

[napetrov]

question: ledger text can still print delta -0.000

The ledger calls compare(..., exact=True), and compare.py:570 only drops -0.0 on the rounded path, so a mean like -1e-5 reaches {d['mean']:+.3f} → -0.000. Same on base (read, not run). In scope for the -0.000 fix, or leave it?

[vlad-perevezentsev]

I would leave it. With exact=True the mean is not rounded, so -1e-5 is a real negative value, not -0.0; -0.000 then keeps its sign, as repo compare prints -0.0004 since c1f2923. The fix there was only for a mean rounded onto -0.0.

[antonwolfy]

The ledger calls compare(..., exact=True), where delta.mean is the unrounded mean its verdicts read (compare.py:181). A mean of -1e-5 there is a real negative value, not a -0.0 produced by rounding, so -0.000 keeps its sign. repo compare prints -0.0004 the same way. c1f2923 only maps a mean rounded onto -0.0 to 0.0. No change.

[vlad-perevezentsev]

Review at 4e871a3, unchanged since my last round. Everything I raised is fixed, and the change does what #292 asks; one nit from another reviewer is still open.

Agreed with @napetrov on the mean printed outside its own interval for tiny deltas; on the p(0) mismatch at 1e9 I could not reproduce it and asked for a row in its thread; on the ledger's -0.000 I would leave it and replied in its thread.

What I ran, at 4e871a3:

• The three new points: six deltas of 4e-5 give mean 0.0 and lo = hi = 0.0001; 900 random 6-10 task rows at 1e9, in three shapes, give ShiftTest(d).p(0.0) == sign_flip_p(d, exact=True) every time.
• make ci-local --offline from my last round still holds: the head did not change.

[napetrov]

Approving at 4e871a3. My three threads are nits/question, fine to fold in later. CI was still finishing when I approved.

[antonwolfy]

(commented) COMMENT at 4e871a3: the inversion does what #292 asks; earlier threads (T1-T9) are resolved on head. Two nits and one question inline, none blocking.

@napetrov (#409 (review)): Finding 1 (mean outside its own interval) is fixed in cd41468. Finding 2 (p(0) against sign_flip_p at 1e9) is a docstring correction plus a test, in 05cb808. Finding 3 (ledger -0.000) is declined, with the reasons in its thread.

[antonwolfy]

(commented) Review at 4e871a3, unchanged since my last round. Everything I raised is fixed, and the change

@vlad-perevezentsev (#409 (review)): The open nit, the mean printed outside its interval, is fixed in cd41468. The p(0) mismatch reproduces only when magnitudes repeat: the example row and the counts are in its thread, and the docstrings are corrected in 05cb808. The ledger -0.000 is left as you suggested.