Skip to content

test(editor): time the routine scan against itself, not a fixed limit - #3999

Merged
datlechin merged 1 commit into
mainfrom
test/editor-routine-scan-linear
Nov 2, 2026
Merged

datlechin merged 1 commit into
mainfrom
test/editor-routine-scan-linear

Conversation

@datlechin

Copy link
Copy Markdown
Member

Summary

  • StatementRunPerformanceGuardTests.routineBodiesStayLinear failed on build(release): open the DMG to a laid-out Finder window #3990, which changes no app code, at 0.551 s against a fixed 0.5 s limit. Green runs on main took 0.25 to 0.38 s for the same scan. One sample swings about 2x between runs, and the limit left less than 2x of room.
  • The scanner is linear, so the code stays as it is. A scan costs the same per routine at 1,000 and at 10,000 routines: about 1.6 µs in a release build and 12 to 19 µs in a debug build, depending on the Mac's load. Nothing is rescanned and nothing grows per block. In a debug build most of the time goes to OptionSet generics behind grammar.contains, which every statement pays, not to block tracking.
  • The test now times the scanner against itself. One scan of 8,000 routines must take less than twice as long as four scans of 2,000. Linear scanning makes the two equal and quadratic scanning makes the long one four times slower, so the bound sits a factor of two from each. Both sides do the same work per sample and take the fastest of three interleaved samples, so a stall on the runner is as likely to hit either side, and the fastest sample leaves it out.
  • This changes the test on purpose (AGENTS.md, Principles). It now asserts what its name says, and a slow runner no longer fails it.

Test plan

  • verify.sh test StatementRunPerformanceGuardTests SQLFoldPerformanceGuardTests: PASS, 9 executed, 9 passed. The new case takes 0.6 s on this Mac in a debug build.
  • routineBodiesStayLinear with -test-iterations 20 -run-tests-until-failure: 20 of 20 passed.
  • The same comparison in a standalone debug build: ratios 0.95 to 1.29 over eight runs at 2,500 and 10,000 routines with the Mac at load average 35 to 38, and 1.00 to 1.14 over six runs at 2,000 and 8,000 at load average 14.
  • With a quadratic step planted in the scanner, a walk back over 1/16 of the text at every ; inside a body, the ratio was 2.5 to 4.3, so the test fails on it.

routineBodiesStayLinear failed on a PR with no app code at 0.551 s
against a fixed 0.5 s limit, while main measured the same scan at
0.25 to 0.38 s. The scanner is linear; one sample on a shared runner
swings about 2x.

The test now compares one scan of 8,000 routines with four scans of
2,000, the fastest of three interleaved samples each, and fails when
the long scan takes twice as long. Linear scanning gives 1x and
quadratic scanning 4x.
@datlechin
datlechin enabled auto-merge (squash) November 2, 2026 20:02
@datlechin
datlechin disabled auto-merge November 2, 2026 20:51
@datlechin
datlechin merged commit a6520ad into main Nov 2, 2026
7 of 8 checks passed
@datlechin
datlechin deleted the test/editor-routine-scan-linear branch November 2, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant