Skip to content

feat: test: lighter suites: one pass per route, targeted local runs, machine-wide lock, native VRT - #547

Merged
mvilacad merged 12 commits into
mainfrom
feat/546-test-lighter-suites-one-pass-per
Oct 25, 2026
Merged

mvilacad merged 12 commits into
mainfrom
feat/546-test-lighter-suites-one-pass-per

Conversation

@mvilacad

@mvilacad mvilacad commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Closes #546

Not ready to land: test:full does not meet its 240 s budget (see the table). Everything else below is pushed and validate passes on every push.

What changed

  • Budgets (scripts/test-run.ts, BUDGETS): changed 60 s, validate 90 s, land 180 s, full 240 s. Over budget fails the run and prints the 10 slowest tests. On an oversubscribed machine (load above the core count) the verdict is printed as NOT ENFORCED, because it would only measure other processes.
  • Machine-wide lock in the same script: a second run waits and prints who holds it; a dead holder is taken over; nested runs reuse their parent's lock.
  • Affected-only default: bun run test = scripts/test-changed.ts. Component tests are picked by a static import graph (scripts/test-graph.ts), because Vitest's --changed selected 198 of 198 files for a one-line Avatar edit in browser mode. Routes by scripts/route-impact.ts, which fails safe (shared path or unknown file selects every route and says why). Shared files select every component test.
  • One pass per route: routes.spec.ts (smoke + layout audit), routes.axe.spec.ts (own axe project, contrast on in dark and light, forced colors on every route, <pre> excluded), dist-html.spec.ts (no browser). makeAxe runs axe.run in the page: 4 to 7 times faster than @axe-core/playwright's builder with identical findings (checked on 8 page/scheme pairs with injected defects, frames included).
  • Native VRT: platform in every baseline name, darwin baselines generated natively (405 component, 20 site, each checked twice), vrt-docker.sh removed, site shots are viewport-only for landing/templates/theme. Docker is gone. CI is untouched and stays disabled.
  • Component tests: settled() jumps running animations to their end state, the pointer park only runs after a pointer event, Carousel's 4 real waits run on a fake clock, Shiki palette contrast is a component test.
  • release.ts runs test:full before versioning; issue-land.sh runs test-changed --land (validate runs on the push).
  • postinstall now installs WebKit (the iphone project needs it and test:full includes it).

Found and fixed on the way

  • My first site baselines were screenshots of another project. Port 4321 was taken by a different project's dev server and reuseExistingServer was on, so the suite compared that site with itself. Now: one port per checkout, no reuse, and a global setup that fails the run if the served <title> is not CodeAtlas. Both proven with a deliberate collision. Baselines regenerated and checked twice against the real site.
  • The lock test inherited the harness's lock variables under validate and failed only there; two manifest tests had no timeout under load.
  • 14 docs pages emit duplicate heading ids (docs: 14 pages emit the same heading id more than once (e.g. id="props" ×4 on /docs/display/table/) #548); allow-listed so a new one fails.

Measured (wall time; machine shared, load noted; never a quiet window for the full runs)

Budget Measured
test (one-component edit, fresh dist) 60 s 28 s (unit 0.4, components 4.6, screenshots 6.7, routes 16.6; load 15)
land gate (same edit, with builds) 180 s 104 s (builds 17 + 58; load 15)
validate 90 s 51 s at load 11; 96 to 237 s at load 20 to 90
components, all 109 s (load 25); was 178 to 227 s
component screenshots, all 105 to 137 s
route sweep (layout + axe + dist-html) 168 s at load 29 (was killed at 467 s, 1998 tests)
unit 30 to 40 s
test:full 240 s one run: 1092 s at load 50 (peak 99), 27 failures that are now fixed (20 contaminated baselines, 7 missing WebKit)

Still open

  1. test:full is over budget by construction: components 109 + screenshots ~105 + routes 168 + unit 40 is ~7 minutes. Needs a decision on the 240 s budget or on what moves out of it.
  2. The remaining real sleeps (VirtualList, MotionStage, ...) and the settled() effect on the full component run have no clean measurement yet.
  3. isolate: false was measured: 775 of 2095 tests fail, 642 s. Not an option.
  4. Docs are updated (AGENTS.md table with budgets and measured times, CONTRIBUTING.md). Memory notes about linux-only baselines are stale and are the maintainer's to update.

@mvilacad
mvilacad force-pushed the feat/546-test-lighter-suites-one-pass-per branch 2 times, most recently from 99d3ec7 to 6c66fb1 Compare October 25, 2026 04:37
@mvilacad
mvilacad marked this pull request as ready for review October 25, 2026 04:37
@mvilacad
mvilacad force-pushed the feat/546-test-lighter-suites-one-pass-per branch from 6c66fb1 to 13b5015 Compare October 25, 2026 06:55
@mvilacad
mvilacad merged commit b8e217e into main Oct 25, 2026
1 check passed
@mvilacad
mvilacad deleted the feat/546-test-lighter-suites-one-pass-per branch October 25, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: lighter suites: one pass per route, targeted local runs, machine-wide lock, native VRT

1 participant