feat(run): add --cpu-throttle <rate> to reproduce CI-only timing flakes - #37
Merged
Merged
Conversation
Slows the browser's CPU with page.emulateCPUThrottling, applied to the new page before navigation so the app boots throttled too. The cpuThrottle key in twd.config.json does the same for a CI job that always runs slowed; the flag overrides it, and 1 (the default) is full speed. A rate below 1 is refused before the browser launches, from either source, instead of being dropped: a dropped rate is a full-speed run the caller believes was throttled. A throttled run says so before it navigates and again in the run-complete block.
kevinccbsg
approved these changes
Sep 30, 2026
kevinccbsg
left a comment
Member
There was a problem hiding this comment.
LGTM I added a comment but I will handle it after merging this change
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
npx twd-cli run --cpu-throttle 6 --test "checkout flow"Slows the browser's CPU with Chrome's own CPU throttling (
page.emulateCPUThrottling), so a race that a fast machine always wins gets a chance to lose."cpuThrottle": 4intwd.config.jsondoes the same for a CI job that should always run slowed. The flag overrides it.Why
A Vite + React 19 + React Router app tested with TWD had two tests that passed locally and failed in CI for days. twd-cli had no way to slow the browser, so neither failure could be reproduced until I patched
page.emulateCPUThrottling(rate)into a local copy:Both were real app bugs, not test bugs:
fetcher.datanever ran.That is the case for the flag: slow CI runners expose races that fast laptops hide. Without a way to slow the browser, the only reproduction loop is "push and wait for CI". There is precedent for the knob itself: Lighthouse's mobile preset applies a 4x CPU slowdown. Neither Playwright nor Cypress has a built-in option for it; both can only send the CDP command by hand.
What it does
--cpu-throttle <rate>onrun, in both--flag valueand--flag=valueforms, fractional rates allowed. It isnullwhen absent, so the config still applies.cpuThrottleintwd.config.json, default1.--cpu-throttle 1runs a throttled config at full speed.page.emulateCPUThrottling(rate)runs right afterbrowser.newPage(), before the viewport and the navigation, so the app's own boot is throttled too. At a rate of 1 nothing is called.CPU throttle: 6xunder the duration in the run-complete block. At full speed the block is byte-identical to before.Refusing a bad rate
Puppeteer asserts
rate >= 1itself (Throttling rate should be greater or equal to 1), but only once the browser is up.src/cpuThrottle.jschecks first, and both the flag and the config go through it, so they refuse the same values in the same words:It throws, like
--shard, rather than dropping the value the way--record-speedand--record-pacedo. A dropped rate means a full-speed run the caller believes was throttled, and "it passes under throttling" is the one conclusion this flag must not get wrong. The config value is checked even when the flag overrides it, so a bad value fails on a laptop instead of first in CI, where nobody passes the flag.CLAUDE.mdrecords the third guard next to the existing pace/speed note, so nobody harmonises the three.Verified
test-example-appunder Vite, through the real bin, 71 tests, three runs per rate:--cpu-throttle 44x--cpu-throttle 66x--cpu-throttle 1010xAll 71 passed at every rate, and each rate gave the same duration on all three runs. The duration grows much less than the rate, which is what you'd expect from a suite that spends most of its time waiting rather than computing. At 10x the app still booted well inside the 10s sidebar wait.
The precedence and error paths, also through the bin:
"cpuThrottle": 6, no flag"cpuThrottle": 6with--cpu-throttle 1"cpuThrottle": 0twd.config.json--cpu-throttle=0.5npm run test:ci: 34 files and 757 tests, up from 715.src/cpuThrottle.jsis at 100%. The repo has no lint script.The tests were written first:
parseArgs: both forms,nullwhen absent, fractional rates, and throws for0,0.5,-1, a non-number or a missing value.--cpu-throtlesuggests the flag.config: the default.runTests: throttling is called beforegotoand not called at1or when unset. The flag wins over the config, including1over a throttled config. A bad config value is refused beforepuppeteer.launch. The header and run-complete lines print, including in the partial block of an interrupted run.testSummary: where the line goes, and a byte-identical block at1.cli.test.js: one real-process case.Caveats, documented in the README
timeoutorprotocolTimeoutat high rates."retryCount": 1while hunting, becauseretryCountcounts attempts.0does not mean "no retries". twd-js's attempt loop never runs and it callsonFail(test, null); twd-cli's in-pageonFailthen readserr.messageoff thatnull. The run dies on the first chunk withTypeError: Cannot read properties of null (reading 'message'), which I reproduced ontest-example-app. That seems worth its own fix, outside this PR.Not in this PR
--repeat <n>: run the selected tests n times and report pass/fail counts. Flakiness is statistical, and for now the README's shell loop stands in for it.delayalready covers latency.run.json. It is in the console output only. Adding it to the report'srunheader would letsummary.mdandindex.htmlshow it, which matters for a CI job that always runs throttled. It is additive and needs no schema bump.CHANGELOG.mdentry, since entries are written at release.