Skip to content

test: pin the NIS re-acceptance bands and make the open-gap xfail strict (T1-11) - #125

Merged
Telschow merged 1 commit into
mainfrom
test/t1-11-nis-xfails
Oct 19, 2026
Merged

Telschow merged 1 commit into
mainfrom
test/t1-11-nis-xfails

Conversation

@Telschow

Copy link
Copy Markdown
Owner

Follows the XPASS finding in PR #123. Test-only; nothing under src/ changed.

Before and after

TestPermanenceIsStillUndetected in tests/test_nis_monitor.py had two xfail(strict=False) tests.

Test Before After
test_a_permanently_adopted_offset_is_locked_out xfail(strict=False) xfail(strict=True), same reason string. Closing the ADR-0007 gap now turns the run red (XPASS strict) and forces the marker out. The class docstring says strict=True is deliberate.
test_the_admitted_offset_exceeds_what_the_drift_budget_can_explain xfail(strict=False), but it XPASSED, so the marker hid a pass removed; replaced by test_honest_return_and_large_spoof_ratios_stay_in_their_measured_bands, a plain passing test

Measured values

Measured by running the test's own scenario (_denial_then_spoof) with a spy on FdirManager._reaccept_with_inflation. The ratio is the innovation norm over the square root of the inflation variance, on the grant. Two repeated runs gave identical values to full precision. The scenario has fixed seeds, so there is no seed spread to report.

Post-outage offset Grants Ratio
0 m (honest return) 1 1.0942792661487726
10 m (modest spoof) 1 1.094659199498256
22 m (large spoof) 1 (after 18 non-grant calls) 1.9893961981661243

(5 m produced no grant at all.) The honest return and the 10 m spoof coincide, which is the overlap the old docstring described: a threshold on this ratio cannot separate an honest re-acquisition from a modest spoof.

The new test asserts each of the three values within a relative margin of 0.05, and that the 10 m value sits in the honest band. It also asserts exactly one grant per offset, so the test cannot read a stale ratio. Margin reason: the values repeat exactly, so the margin is not for noise. It covers platform and library float differences. It is far below the effect of a factor of two on the inflation variance, which moves a ratio by about 40%.

The old assertion (spoofed_ratio < 2.2) only bounded the 22 m value from above and said nothing about the honest return.

Mutation checks

  • (b) variance = 2.0 * self.nis.inflation_variance(...) in FdirManager._reaccept_with_inflation (temporary):
    E       assert np.float64(0.7737722896056359) == 1.0942792661487726 ± 0.054714
    1 failed, 64 deselected, 1 xfailed
    
    Restored with git checkout; src/ is untouched in the diff.
  • (a) assertion temporarily replaced by assert True (the lockout "fires"):
    [XPASS(strict)] one grant, then permanent adoption; no second grant to count
    1 failed, 1 passed, 64 deselected
    
    Restored.

Other edits

Two sentences in docs/adr/0008-frozen-anchor-cross-check.md and one assertion message in test_nis_monitor.py said "the ADR-0007 xfails". There is one xfail now, so they say "xfail". No numbers added. ADR-0007 text (xfail for the permanence case) is still true and unchanged. Test count is unchanged (one test replaced by one), so metrics.py --write made no change to docs/data/metrics.json.

Suite summary

1459 passed, 24 skipped, 2 xfailed in 584.52s (0:09:44)

No XPASS. The task expected 1 xfailed, but 2 is right: the other is the strict test_known_gaps.py xfail (legacy form), which was in the count before too (previously 2 xfailed, 1 xpassed). Skips are 24 here against 25 in the T1-5 run. I did not chase the difference; I ran scripts/run_benchmark.py in this worktree for the table check, which may account for it.

Checks

ruff check, ruff format --check, mypy src, metrics.py --check and check_doc_tables.py (against a fresh benchmark: Scenario, Configuration, named-row tables) pass. The coverage ratchet is left to CI.

Left

The permanence gap itself (ADR-0007 open question) stays open, as designed.

🤖 Generated with Claude Code

…ict (T1-11)

The permanence lockout test is now xfail(strict=True), so closing the
ADR-0007 gap turns the run red and forces the marker out. The drift-budget
ratio test was an XPASS hidden by a non-strict xfail. It is now a plain
passing test with two bands taken from a measurement of its own scenario.
Two ADR-0008 sentences and one assertion message now say "xfail", singular.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Telschow
Telschow merged commit 69502a9 into main Oct 19, 2026
20 checks passed
@Telschow
Telschow deleted the test/t1-11-nis-xfails branch October 19, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant