Skip to content

feat(run): add --cpu-throttle <rate> to reproduce CI-only timing flakes - #37

Merged
kevinccbsg merged 2 commits into
BRIKEV:mainfrom
nanotower:feat/cpu-throttle
Oct 1, 2026
Merged

kevinccbsg merged 2 commits into
BRIKEV:mainfrom
nanotower:feat/cpu-throttle

Conversation

@nanotower

Copy link
Copy Markdown
Contributor
npx twd-cli run --cpu-throttle 6 --test "checkout flow"

Slows the browser's CPU with Chrome's own CPU throttling (page.emulateCPUThrottling), so a race that a fast machine always wins gets a chance to lose. "cpuThrottle": 4 in twd.config.json does the same for a CI job that should always run slowed. The flag overrides it.

Why

A Vite + React 19 + React Router app tested with TWD had two tests that passed locally and failed in CI for days. twd-cli had no way to slow the browser, so neither failure could be reproduced until I patched page.emulateCPUThrottling(rate) into a local copy:

Test Throttle Failed
A, react-router 8.4.0 10x 5 of 6 runs
A, react-router 8.3.1, same React 10x 0 of 6 runs
B 6x to 10x 4 of 4 runs
A and B after the fixes same 0 of 5 runs each

Both were real app bugs, not test bugs:

  • Test A: the throttled runs made a bisect possible, and it found a regression. A component's effect keyed on fetcher.data never ran.
  • Test B was a focus race. A Radix dropdown returned focus to its trigger after a drawer had opened, the drawer's focus trap re-selected its input, and the text being typed was replaced.

That is the case for the flag: slow CI runners expose races that fast laptops hide. Without a way to slow the browser, the only reproduction loop is "push and wait for CI". There is precedent for the knob itself: Lighthouse's mobile preset applies a 4x CPU slowdown. Neither Playwright nor Cypress has a built-in option for it; both can only send the CDP command by hand.

What it does

  • --cpu-throttle <rate> on run, in both --flag value and --flag=value forms, fractional rates allowed. It is null when absent, so the config still applies.
  • cpuThrottle in twd.config.json, default 1. --cpu-throttle 1 runs a throttled config at full speed.
  • page.emulateCPUThrottling(rate) runs right after browser.newPage(), before the viewport and the navigation, so the app's own boot is throttled too. At a rate of 1 nothing is called.
  • A throttled run says so twice: once in a line before it navigates, and again as CPU throttle: 6x under the duration in the run-complete block. At full speed the block is byte-identical to before.
CPU throttling: 6x (the browser runs 6 times slower; the dev server does not).
Navigating to http://127.0.0.1:5174 ...
Running 71 test(s)...

--- Run complete ---
  Passed: 71 | Failed: 0 | Skipped: 0
  Duration: 0.9s
  CPU throttle: 6x

Refusing a bad rate

Puppeteer asserts rate >= 1 itself (Throttling rate should be greater or equal to 1), but only once the browser is up. src/cpuThrottle.js checks first, and both the flag and the config go through it, so they refuse the same values in the same words:

$ npx twd-cli run --cpu-throttle 0.5
Invalid --cpu-throttle: expected a rate of 1 or more, got "0.5". 1 is full speed; 4 makes the browser's CPU four times slower.

It throws, like --shard, rather than dropping the value the way --record-speed and --record-pace do. A dropped rate means a full-speed run the caller believes was throttled, and "it passes under throttling" is the one conclusion this flag must not get wrong. The config value is checked even when the flag overrides it, so a bad value fails on a laptop instead of first in CI, where nobody passes the flag. CLAUDE.md records the third guard next to the existing pace/speed note, so nobody harmonises the three.

Verified

test-example-app under Vite, through the real bin, 71 tests, three runs per rate:

Invocation Run duration Wall clock Throttle lines
no flag 0.5s ~0.95s none
--cpu-throttle 4 0.7s ~1.18s 4x
--cpu-throttle 6 0.9s ~1.34s 6x
--cpu-throttle 10 1.2s ~1.70s 10x

All 71 passed at every rate, and each rate gave the same duration on all three runs. The duration grows much less than the rate, which is what you'd expect from a suite that spends most of its time waiting rather than computing. At 10x the app still booted well inside the 10s sidebar wait.

The precedence and error paths, also through the bin:

Setup Result
"cpuThrottle": 6, no flag throttled 6x, both lines printed
"cpuThrottle": 6 with --cpu-throttle 1 full speed, no throttle lines
"cpuThrottle": 0 exit 1 before launch, error names twd.config.json
--cpu-throttle=0.5 exit 1 before launch, error on stderr

npm run test:ci: 34 files and 757 tests, up from 715. src/cpuThrottle.js is at 100%. The repo has no lint script.

The tests were written first:

  • parseArgs: both forms, null when absent, fractional rates, and throws for 0, 0.5, -1, a non-number or a missing value. --cpu-throtle suggests the flag.
  • config: the default.
  • runTests: throttling is called before goto and not called at 1 or when unset. The flag wins over the config, including 1 over a throttled config. A bad config value is refused before puppeteer.launch. The header and run-complete lines print, including in the partial block of an interrupted run.
  • testSummary: where the line goes, and a byte-identical block at 1.
  • cli.test.js: one real-process case.

Caveats, documented in the README

  • It raises the odds of hitting a race; it does not reproduce one every time. The README shows a loop.
  • Runs are slower, so it is not a default. The page load is throttled too, so a heavy app may need a higher timeout or protocolTimeout at high rates.
  • Retries can hide a flake. The README says to use "retryCount": 1 while hunting, because retryCount counts attempts. 0 does not mean "no retries". twd-js's attempt loop never runs and it calls onFail(test, null); twd-cli's in-page onFail then reads err.message off that null. The run dies on the first chunk with TypeError: Cannot read properties of null (reading 'message'), which I reproduced on test-example-app. That seems worth its own fix, outside this PR.
  • Only the browser is slowed, not the dev server.

Not in this PR

  • --repeat <n>: run the selected tests n times and report pass/fail counts. Flakiness is statistical, and for now the README's shell loop stands in for it.
  • Network throttling. I haven't checked whether Chrome's network emulation applies to responses a service worker synthesizes, which is how TWD serves its mocks. Per-mock delay already covers latency.
  • The throttle in run.json. It is in the console output only. Adding it to the report's run header would let summary.md and index.html show it, which matters for a CI job that always runs throttled. It is additive and needs no schema bump.
  • No CHANGELOG.md entry, since entries are written at release.

Slows the browser's CPU with page.emulateCPUThrottling, applied to the new
page before navigation so the app boots throttled too. The cpuThrottle key in
twd.config.json does the same for a CI job that always runs slowed; the flag
overrides it, and 1 (the default) is full speed.

A rate below 1 is refused before the browser launches, from either source,
instead of being dropped: a dropped rate is a full-speed run the caller
believes was throttled. A throttled run says so before it navigates and again
in the run-complete block.

@kevinccbsg kevinccbsg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I added a comment but I will handle it after merging this change

Comment thread README.md
@kevinccbsg
kevinccbsg merged commit 49f226c into BRIKEV:main Oct 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants