Repository navigation
test: pin the NIS re-acceptance bands and make the open-gap xfail strict (T1-11) - #125
Merged
Merged
Conversation
…ict (T1-11) The permanence lockout test is now xfail(strict=True), so closing the ADR-0007 gap turns the run red and forces the marker out. The drift-budget ratio test was an XPASS hidden by a non-strict xfail. It is now a plain passing test with two bands taken from a measurement of its own scenario. Two ADR-0008 sentences and one assertion message now say "xfail", singular. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows the XPASS finding in PR #123. Test-only; nothing under
src/changed.Before and after
TestPermanenceIsStillUndetectedintests/test_nis_monitor.pyhad twoxfail(strict=False)tests.test_a_permanently_adopted_offset_is_locked_outxfail(strict=False)xfail(strict=True), same reason string. Closing the ADR-0007 gap now turns the run red (XPASS strict) and forces the marker out. The class docstring saysstrict=Trueis deliberate.test_the_admitted_offset_exceeds_what_the_drift_budget_can_explainxfail(strict=False), but it XPASSED, so the marker hid a passtest_honest_return_and_large_spoof_ratios_stay_in_their_measured_bands, a plain passing testMeasured values
Measured by running the test's own scenario (
_denial_then_spoof) with a spy onFdirManager._reaccept_with_inflation. The ratio is the innovation norm over the square root of the inflation variance, on the grant. Two repeated runs gave identical values to full precision. The scenario has fixed seeds, so there is no seed spread to report.(5 m produced no grant at all.) The honest return and the 10 m spoof coincide, which is the overlap the old docstring described: a threshold on this ratio cannot separate an honest re-acquisition from a modest spoof.
The new test asserts each of the three values within a relative margin of 0.05, and that the 10 m value sits in the honest band. It also asserts exactly one grant per offset, so the test cannot read a stale ratio. Margin reason: the values repeat exactly, so the margin is not for noise. It covers platform and library float differences. It is far below the effect of a factor of two on the inflation variance, which moves a ratio by about 40%.
The old assertion (
spoofed_ratio < 2.2) only bounded the 22 m value from above and said nothing about the honest return.Mutation checks
variance = 2.0 * self.nis.inflation_variance(...)inFdirManager._reaccept_with_inflation(temporary):git checkout;src/is untouched in the diff.assert True(the lockout "fires"):Other edits
Two sentences in
docs/adr/0008-frozen-anchor-cross-check.mdand one assertion message intest_nis_monitor.pysaid "the ADR-0007 xfails". There is one xfail now, so they say "xfail". No numbers added. ADR-0007 text (xfailfor the permanence case) is still true and unchanged. Test count is unchanged (one test replaced by one), sometrics.py --writemade no change todocs/data/metrics.json.Suite summary
No XPASS. The task expected 1 xfailed, but 2 is right: the other is the strict
test_known_gaps.pyxfail (legacy form), which was in the count before too (previously 2 xfailed, 1 xpassed). Skips are 24 here against 25 in the T1-5 run. I did not chase the difference; I ranscripts/run_benchmark.pyin this worktree for the table check, which may account for it.Checks
ruff check,ruff format --check,mypy src,metrics.py --checkandcheck_doc_tables.py(against a fresh benchmark: Scenario, Configuration, named-row tables) pass. The coverage ratchet is left to CI.Left
The permanence gap itself (ADR-0007 open question) stays open, as designed.
🤖 Generated with Claude Code