Skip to content

Say that tests check a heuristic's routes, not its thresholds, unless a route fails past one - #915

Merged
perlinm merged 1 commit into
mainfrom
docs/heuristic-threshold-tests
Nov 16, 2026
Merged

perlinm merged 1 commit into
mainfrom
docs/heuristic-threshold-tests

Conversation

@perlinm

@perlinm perlinm commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The testing standard says not to pin implementation choices, but the review rules count "claimed behavior that no test verifies" as a significant finding. The two collide for a heuristic whose routes each give a correct result, such as method="auto" in #913, which picks among exact distance-search methods: its guide documents its thresholds, so reviews asked for a test of each. About 260 of the 1,144 test lines that a draft of #913 added checked the heuristic's arithmetic at its exact boundaries, such as probing at a work budget minus one but not at the budget, although an off-by-one there changes only which correct method runs.

Changes

  • standards/testing.rst: tests of such a heuristic check that each route gives correct results, and that cases far from its thresholds take the intended route. Its thresholds are implementation choices, even where documented. The exception is a threshold past which a route would fail or give a wrong result even with unlimited time and memory, whose nearest cases on each side are still tested. This keeps tests such as the one of the fault search's cap of 2**31 pairs, past which its 32-bit slot numbers overflow, where a case far from the cap would miss a slip such as 2**32.
  • reviewing.rst: "claimed behavior that no test verifies" now excepts an implementation choice that the testing standard says not to pin, so that reviewers of every angle read the two rules alike.

The change adds about 90 words to the roughly 9,600 words of the rules, and replaces no rule.

… a route fails past one

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@perlinm
perlinm merged commit 5e7dc2e into main Nov 16, 2026
8 checks passed
@perlinm
perlinm deleted the docs/heuristic-threshold-tests branch November 16, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant