Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The numeric guards on the two recording flags differ **on purpose**, so do not h

**`src/cpuThrottle.js`**: `parseCpuThrottle(value, source)` validates a CPU throttling rate for both `--cpu-throttle` and `cpuThrottle`, so the two refuse the same values in the same words. Puppeteer asserts `>= 1` itself, but only after the launch. `cpuThrottle` (default `1`, full speed) is checked even when the flag overrides it, so a bad value fails on a laptop and not first in CI, where nobody passes the flag.

`retryCount` counts **attempts**, not retries: twd-js loops `for (attempt = 1; attempt <= retryCount; …)`. `1` means no retry. `0` does not mean "no retries": the loop body never runs, twd-js calls `onFail(test, null)`, and the in-page `onFail` here reads `err.message` off that `null`, so the first chunk throws and the run dies with `TypeError: Cannot read properties of null (reading 'message')`. Say `1` wherever docs tell someone to turn retries off, as the throttling notes in the README do.
`retryCount` counts **attempts**, not retries: twd-js loops `for (attempt = 1; attempt <= retryCount; …)`. `1` means no retry. `0` does not mean "no retries": the loop body never runs, twd-js calls `onFail(test, null)`, and the in-page `onFail` here reads `err.message` off that `null`, so the first chunk throws and the run dies with `TypeError: Cannot read properties of null (reading 'message')`. Say `1` wherever docs tell someone to turn retries off, as `docs/cpu-throttling.md` does.

`record` (`DEFAULT_RECORD`) is the only **nested** config key, so the merge goes two levels deep: `record` merges over `DEFAULT_RECORD`, and `record.viewport` merges over the default viewport. A flat spread would wipe sibling defaults. Recording is off by default and never runs unless explicitly requested.

Expand Down
47 changes: 6 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,51 +124,16 @@ Notes:

### Reproducing CI-only timing flakes

A test that passes on your machine and fails on a CI runner is often a race
that a fast CPU always wins. `--cpu-throttle <rate>` slows the browser's CPU
down that many times, using Chrome's own CPU throttling, so the slow side of
the race gets its chance to show up locally:
A test that passes locally and fails on a slow CI runner is often a race a fast
CPU always wins. `--cpu-throttle <rate>` slows the browser's CPU down that many
times so the race can lose on your machine too:

```bash
npx twd-cli run --cpu-throttle 6 --test "checkout flow"
```

For a CI job that always runs slowed, set `"cpuThrottle": 4` in
`twd.config.json`. The flag overrides it, and `--cpu-throttle 1` runs that
config at full speed. A throttled run says so before it navigates and again in
the run-complete block, so a slow or red run is not read as an ordinary one:

```
CPU throttling: 6x (the browser runs 6 times slower; the dev server does not).
...
--- Run complete ---
Passed: 71 | Failed: 0 | Skipped: 0
Duration: 0.9s
CPU throttle: 6x
```

Notes:

- **It raises the odds of hitting a race; it does not reproduce one every
time.** Run it in a loop before concluding anything, in either direction:
```bash
for i in $(seq 10); do
npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL
done
```
- **Turn retries off while you hunt.** The default `retryCount` of `2` gives a
failed test a second attempt, and a flake that passes on the second attempt
is exactly what you are looking for. `retryCount` counts attempts, so
`"retryCount": 1` means no retry. A test that only passed on a retry is
listed under `Retried` in the run-complete block.
- **Only the browser is slowed.** The dev server, and anything else outside
the page, runs at full speed.
- **It is slower, so it is not a default.** How much slower depends on how
much of the suite is CPU work rather than waiting. The page load is
throttled too, so a heavy app may need a higher `timeout` (the wait for the
TWD sidebar) or `protocolTimeout` at high rates.
- A rate below `1` is refused before the browser launches, whether it comes
from the flag or from `twd.config.json`.
See [docs/cpu-throttling.md](docs/cpu-throttling.md) for the config key, what a
throttled run prints, and how to hunt a flake reliably.

### Configuration

Expand Down Expand Up @@ -205,7 +170,7 @@ Create a `twd.config.json` file in your project root:
| `protocolTimeout` | number | `300000` | Puppeteer CDP `protocolTimeout` in ms (5 min). Tests run in chunks via `runByIds`, so this bounds a **single chunk's browser call** (not the entire run). Raise it (e.g. `600000`) for slow CI or if individual chunks hang; `0` means no timeout. Defaults above Puppeteer's implicit 180000ms ceiling |
| `maxFailures` | number | `10` | Stop the run once this many tests have failed in total; the CLI prints the results gathered so far and exits non-zero. Set `0` to disable and always run every test |
| `chunkSize` | number | `10` | How many tests run per browser call. Smaller values make the failure limit and timeouts more granular (less work lost if one chunk hangs); larger values reduce overhead. `0` runs everything in one call |
| `cpuThrottle` | number | `1` | How many times slower the browser's CPU runs, e.g. `4`, to reproduce timing flakes a slow CI runner shows. `1` is full speed and below `1` is refused. `--cpu-throttle` overrides it. See [Reproducing CI-only timing flakes](#reproducing-ci-only-timing-flakes) |
| `cpuThrottle` | number | `1` | How many times slower the browser's CPU runs, e.g. `4`, to reproduce timing flakes a slow CI runner shows. `1` is full speed and below `1` is refused. `--cpu-throttle` overrides it. See [CPU throttling](docs/cpu-throttling.md) |
| `contracts` | array | none | OpenAPI contract validation specs (see [Contract Validation](#contract-validation)) |
| `contractReportPath` | string | none | Path to write a markdown report for CI/PR integration |
| `viewport` | object | `{ "width": 1280, "height": 800 }` | Browser viewport for every run. Layout snapshots are only reproducible when this is fixed and explicit. While recording, `record.viewport` wins |
Expand Down
47 changes: 47 additions & 0 deletions docs/cpu-throttling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Reproducing CI-only timing flakes

A test that passes on your machine and fails on a CI runner is often a race
that a fast CPU always wins. `--cpu-throttle <rate>` slows the browser's CPU
down that many times, using Chrome's own CPU throttling, so the slow side of
the race gets its chance to show up locally:

```bash
npx twd-cli run --cpu-throttle 6 --test "checkout flow"
```

For a CI job that always runs slowed, set `"cpuThrottle": 4` in
`twd.config.json`. The flag overrides it, and `--cpu-throttle 1` runs that
config at full speed. A throttled run says so before it navigates and again in
the run-complete block, so a slow or red run is not read as an ordinary one:

```
CPU throttling: 6x (the browser runs 6 times slower; the dev server does not).
...
--- Run complete ---
Passed: 71 | Failed: 0 | Skipped: 0
Duration: 0.9s
CPU throttle: 6x
```

Notes:

- **It raises the odds of hitting a race; it does not reproduce one every
time.** Run it in a loop before concluding anything, in either direction:
```bash
for i in $(seq 10); do
npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL
done
```
- **Turn retries off while you hunt.** The default `retryCount` of `2` gives a
failed test a second attempt, and a flake that passes on the second attempt
is exactly what you are looking for. `retryCount` counts attempts, so
`"retryCount": 1` means no retry. A test that only passed on a retry is
listed under `Retried` in the run-complete block.
- **Only the browser is slowed.** The dev server, and anything else outside
the page, runs at full speed.
- **It is slower, so it is not a default.** How much slower depends on how
much of the suite is CPU work rather than waiting. The page load is
throttled too, so a heavy app may need a higher `timeout` (the wait for the
TWD sidebar) or `protocolTimeout` at high rates.
- A rate below `1` is refused before the browser launches, whether it comes
from the flag or from `twd.config.json`.
Loading