Skip to content

feat: unique test results per run (--run-name) - #3

Open
stanleykim-ux wants to merge 1 commit into
feat/parallel-local-devicesfrom
feat/named-test-runs
Open

feat: unique test results per run (--run-name)#3
stanleykim-ux wants to merge 1 commit into
feat/parallel-local-devicesfrom
feat/named-test-runs

Conversation

@stanleykim-ux

Copy link
Copy Markdown

Stacked on #2 (feat/parallel-local-devices). Review only the last commit.

Why

Two appwright runs on one machine (for example yarn test:ios and yarn test:android side by side, which #2 makes practical) clobber each other's output: Playwright wipes test-results/ at the start of every run, the html reporter deletes playwright-report/ before rebuilding it, and appwright's own video / worker-info store was hardcoded to playwright-report/data/videos-store (there was a TODO in workerInfo.ts asking for exactly this fix).

What

Every run writes into its own folders:

test-results/<run>/               Playwright output (per-test artifacts, .last-run.json)
test-results/<run>/videos-store/  appwright worker videos and worker-info files
playwright-report/<run>/          HTML report  ->  npx playwright show-report playwright-report/<run>
  • appwright test --run-name <name> names the run. The flag is stripped by the appwright CLI (Playwright rejects unknown options) and passed down as APPWRIGHT_RUN_NAME, which can also be set directly (CI).
  • Default name: <project>-<YYYYMMDD>-<HHmmss>-<4 random chars>, e.g. android-20260910-153915-k3i4. Sortable and tells you which platform at a glance.
  • Names are sanitized to [A-Za-z0-9._+-] so they are always a safe path segment.
  • A custom outputDir / html outputFolder in the consumer config is kept as the base; the run name is nested under it.
  • Global setup logs one line with the run's folders so interleaved output from concurrent runs can be told apart.
  • Consumer (tulip-player-mobile/e2e) needs no change: yarn test:android --run-name smoke passes straight through.

Notes for reviewers

  • Playwright evaluates the config file again inside every worker process, so defineConfig cannot generate a fresh name; resolveRunName() reads APPWRIGHT_RUN_NAME and only generates (and stores) one when it is unset. That also makes the direct npx playwright test -c appwright.config.ts path work.
  • The video store moves from playwright-report/data/videos-store to test-results/<run>/videos-store. The html reporter removes its whole output folder in onEnd before it reads attachment files, so anything stored under playwright-report/ was lost at report time. Playwright clears outputDir at run start, so the new location is fresh per run.
  • --last-failed reads .last-run.json from the run's folder, so it needs the same --run-name again. Documented.
  • Run folders are never deleted automatically. Retention is deliberately out of scope; documented as rm -rf test-results playwright-report.

Verification

  • npm run build, npm test (75 tests, 38 new covering name generation/sanitizing, flag parsing, reporter mapping, CLI invocation).
  • Real runs against a booted Android emulator with the example Wikipedia APK via a scratch project pointing at the built dist/:
    • appwright test --project android --run-name smoketest-results/smoke/{.last-run.json,videos-store/worker-info-0.json}, playwright-report/smoke/index.html.
    • appwright test --project android (no flag) → android-20260910-153915-k3i4.
    • npx playwright test --config appwright.config.ts --project android (bypassing the appwright CLI) → android-20260910-153922-r2em.
    • All three folder sets survived side by side.
  • Not run: two runs literally concurrent on two platforms (only one emulator handy). Per-run isolation follows from the folder layout above, so the risk is low.

🤖 Generated with Claude Code

Every run now writes into test-results/<run> (Playwright output plus appwright's
video store) and playwright-report/<run>, so concurrent runs on one machine no
longer clobber each other. The name comes from the appwright CLI's new --run-name
flag or APPWRIGHT_RUN_NAME, and defaults to <project>-<YYYYMMDD>-<HHmmss>-<rand4>.
The config re-reads the name from the environment because Playwright evaluates it
again in every worker. The video store moves out of playwright-report/, which the
html reporter deletes before it copies attachments.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@stanleykim-ux
stanleykim-ux added this pull request to stack #4 September 10, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant