diff --git a/CLAUDE.md b/CLAUDE.md index 60ba02d..4122301 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,7 +44,7 @@ The numeric guards on the two recording flags differ **on purpose**, so do not h **`src/cpuThrottle.js`**: `parseCpuThrottle(value, source)` validates a CPU throttling rate for both `--cpu-throttle` and `cpuThrottle`, so the two refuse the same values in the same words. Puppeteer asserts `>= 1` itself, but only after the launch. `cpuThrottle` (default `1`, full speed) is checked even when the flag overrides it, so a bad value fails on a laptop and not first in CI, where nobody passes the flag. -`retryCount` counts **attempts**, not retries: twd-js loops `for (attempt = 1; attempt <= retryCount; …)`. `1` means no retry. `0` does not mean "no retries": the loop body never runs, twd-js calls `onFail(test, null)`, and the in-page `onFail` here reads `err.message` off that `null`, so the first chunk throws and the run dies with `TypeError: Cannot read properties of null (reading 'message')`. Say `1` wherever docs tell someone to turn retries off, as the throttling notes in the README do. +`retryCount` counts **attempts**, not retries: twd-js loops `for (attempt = 1; attempt <= retryCount; …)`. `1` means no retry. `0` does not mean "no retries": the loop body never runs, twd-js calls `onFail(test, null)`, and the in-page `onFail` here reads `err.message` off that `null`, so the first chunk throws and the run dies with `TypeError: Cannot read properties of null (reading 'message')`. Say `1` wherever docs tell someone to turn retries off, as `docs/cpu-throttling.md` does. `record` (`DEFAULT_RECORD`) is the only **nested** config key, so the merge goes two levels deep: `record` merges over `DEFAULT_RECORD`, and `record.viewport` merges over the default viewport. A flat spread would wipe sibling defaults. Recording is off by default and never runs unless explicitly requested. diff --git a/README.md b/README.md index 9c52e5b..4ead446 100644 --- a/README.md +++ b/README.md @@ -124,51 +124,16 @@ Notes: ### Reproducing CI-only timing flakes -A test that passes on your machine and fails on a CI runner is often a race -that a fast CPU always wins. `--cpu-throttle ` slows the browser's CPU -down that many times, using Chrome's own CPU throttling, so the slow side of -the race gets its chance to show up locally: +A test that passes locally and fails on a slow CI runner is often a race a fast +CPU always wins. `--cpu-throttle ` slows the browser's CPU down that many +times so the race can lose on your machine too: ```bash npx twd-cli run --cpu-throttle 6 --test "checkout flow" ``` -For a CI job that always runs slowed, set `"cpuThrottle": 4` in -`twd.config.json`. The flag overrides it, and `--cpu-throttle 1` runs that -config at full speed. A throttled run says so before it navigates and again in -the run-complete block, so a slow or red run is not read as an ordinary one: - -``` -CPU throttling: 6x (the browser runs 6 times slower; the dev server does not). -... ---- Run complete --- - Passed: 71 | Failed: 0 | Skipped: 0 - Duration: 0.9s - CPU throttle: 6x -``` - -Notes: - -- **It raises the odds of hitting a race; it does not reproduce one every - time.** Run it in a loop before concluding anything, in either direction: - ```bash - for i in $(seq 10); do - npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL - done - ``` -- **Turn retries off while you hunt.** The default `retryCount` of `2` gives a - failed test a second attempt, and a flake that passes on the second attempt - is exactly what you are looking for. `retryCount` counts attempts, so - `"retryCount": 1` means no retry. A test that only passed on a retry is - listed under `Retried` in the run-complete block. -- **Only the browser is slowed.** The dev server, and anything else outside - the page, runs at full speed. -- **It is slower, so it is not a default.** How much slower depends on how - much of the suite is CPU work rather than waiting. The page load is - throttled too, so a heavy app may need a higher `timeout` (the wait for the - TWD sidebar) or `protocolTimeout` at high rates. -- A rate below `1` is refused before the browser launches, whether it comes - from the flag or from `twd.config.json`. +See [docs/cpu-throttling.md](docs/cpu-throttling.md) for the config key, what a +throttled run prints, and how to hunt a flake reliably. ### Configuration @@ -205,7 +170,7 @@ Create a `twd.config.json` file in your project root: | `protocolTimeout` | number | `300000` | Puppeteer CDP `protocolTimeout` in ms (5 min). Tests run in chunks via `runByIds`, so this bounds a **single chunk's browser call** (not the entire run). Raise it (e.g. `600000`) for slow CI or if individual chunks hang; `0` means no timeout. Defaults above Puppeteer's implicit 180000ms ceiling | | `maxFailures` | number | `10` | Stop the run once this many tests have failed in total; the CLI prints the results gathered so far and exits non-zero. Set `0` to disable and always run every test | | `chunkSize` | number | `10` | How many tests run per browser call. Smaller values make the failure limit and timeouts more granular (less work lost if one chunk hangs); larger values reduce overhead. `0` runs everything in one call | -| `cpuThrottle` | number | `1` | How many times slower the browser's CPU runs, e.g. `4`, to reproduce timing flakes a slow CI runner shows. `1` is full speed and below `1` is refused. `--cpu-throttle` overrides it. See [Reproducing CI-only timing flakes](#reproducing-ci-only-timing-flakes) | +| `cpuThrottle` | number | `1` | How many times slower the browser's CPU runs, e.g. `4`, to reproduce timing flakes a slow CI runner shows. `1` is full speed and below `1` is refused. `--cpu-throttle` overrides it. See [CPU throttling](docs/cpu-throttling.md) | | `contracts` | array | none | OpenAPI contract validation specs (see [Contract Validation](#contract-validation)) | | `contractReportPath` | string | none | Path to write a markdown report for CI/PR integration | | `viewport` | object | `{ "width": 1280, "height": 800 }` | Browser viewport for every run. Layout snapshots are only reproducible when this is fixed and explicit. While recording, `record.viewport` wins | diff --git a/docs/cpu-throttling.md b/docs/cpu-throttling.md new file mode 100644 index 0000000..718d78b --- /dev/null +++ b/docs/cpu-throttling.md @@ -0,0 +1,47 @@ +# Reproducing CI-only timing flakes + +A test that passes on your machine and fails on a CI runner is often a race +that a fast CPU always wins. `--cpu-throttle ` slows the browser's CPU +down that many times, using Chrome's own CPU throttling, so the slow side of +the race gets its chance to show up locally: + +```bash +npx twd-cli run --cpu-throttle 6 --test "checkout flow" +``` + +For a CI job that always runs slowed, set `"cpuThrottle": 4` in +`twd.config.json`. The flag overrides it, and `--cpu-throttle 1` runs that +config at full speed. A throttled run says so before it navigates and again in +the run-complete block, so a slow or red run is not read as an ordinary one: + +``` +CPU throttling: 6x (the browser runs 6 times slower; the dev server does not). +... +--- Run complete --- + Passed: 71 | Failed: 0 | Skipped: 0 + Duration: 0.9s + CPU throttle: 6x +``` + +Notes: + +- **It raises the odds of hitting a race; it does not reproduce one every + time.** Run it in a loop before concluding anything, in either direction: + ```bash + for i in $(seq 10); do + npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL + done + ``` +- **Turn retries off while you hunt.** The default `retryCount` of `2` gives a + failed test a second attempt, and a flake that passes on the second attempt + is exactly what you are looking for. `retryCount` counts attempts, so + `"retryCount": 1` means no retry. A test that only passed on a retry is + listed under `Retried` in the run-complete block. +- **Only the browser is slowed.** The dev server, and anything else outside + the page, runs at full speed. +- **It is slower, so it is not a default.** How much slower depends on how + much of the suite is CPU work rather than waiting. The page load is + throttled too, so a heavy app may need a higher `timeout` (the wait for the + TWD sidebar) or `protocolTimeout` at high rates. +- A rate below `1` is refused before the browser launches, whether it comes + from the flag or from `twd.config.json`.