Skip to content

fix(replay): fresh-account routines replay on a new account - #200

Merged
merge-loop[bot] merged 2 commits into
mainfrom
fix/fresh-account-routine-replay
Nov 16, 2026
Merged

merge-loop[bot] merged 2 commits into
mainfrom
fix/fresh-account-routine-replay

Conversation

@joaquim-verges

@joaquim-verges joaquim-verges commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #197.

What broke

Every new-user retest on 11-16 came back unclear, because the onboarding routines it depends on could not replay on a fresh account. There were four causes. I reproduced each one on dev with step-level replay logging.

  1. One-time values replayed literally. The routines typed the usernames the explorer picked when recording them (bugpatrolnew77, bug-patrol-77, and others). Dev's /user/username/check still reports these as taken after their accounts are deleted. So Save Profile stays disabled and the replay never leaves the Profile step.
  2. The connect hand-off. screen-onboarding-agents-run starts with open /. The replay ran it straight after connect, while /test/e2e-login?token=... was still seeding the session (it redirects itself about 1s after load). The app came up on "Welcome to Nebula". In a Playwright probe, 2 of 2 immediate opens landed signed out; an open after the redirect stayed signed in. networkidle fires before the redirect, so it doesn't help.
  3. Point fallback before the element renders. The username field renders about 1s after Last name is filled. resolveTarget found no named match yet and fell straight to the stored point (step 2 type … degraded=true). The value went nowhere, and every later step typed into the Profile page through the bare input selector or clicked stored points.
  4. The recorder cut the real invite. compactSteps removes a "detour" when the app returns to an earlier place. That place was the screen key plus element names, with no field values. So "type not-an-email, Send invite (error), type {{TEAMMATE_EMAIL}}" looked like a return to the same unchanged form, and the step that typed the real address was dropped. 19 mobile routines replayed only the invalid address and stalled on Team.

Change

  • {{RUN_ID}} (vars.ts, prompts.ts, the explorer's type tool): every run gets a public RUN_ID of 8 lowercase letters and digits, and a setup capture of the same name overrides it. The explorer is told to put it in any value the app accepts only once. The trail stores the unresolved text, so routines keep the template.
  • Hand-off wait (web.ts): when the connect URL has a query, connect() waits up to 15s for the page to leave it.
  • Named-target wait (dom.ts, web.ts): a replayed target with a test id, or a role and a name, gets up to 5s to appear before the text, selector or point stands in. Explorer actions use refs and are unchanged.
  • Value-aware screen key (explorer.ts locationKey): each element's value is part of the place, so typing that changes a form is never taken for a detour.
  • Docs: configuration.md documents {{RUN_ID}} and the hand-off wait.

Verification

  • New tests, each seen failing without its fix:
    • web-origin.test.ts (hand-off): with awaitHandOff commented out, it received /test/login?token=abc.
    • dom.test.ts (named-target wait): fails with the wait loop disabled.
    • agents-flow.test.ts (form retyped after an error): with the old key, the routine lost friend@example.com and its tap.
    • vars.test.ts (RUN_ID): unknown variable without the change.
  • Local: pnpm typecheck, pnpm check and pnpm knip are clean. pnpm test: everything passes except the 2 launchElectron tests, which need a full Chromium build this machine doesn't have. CI has it.
  • End to end on dev with a local build of this branch, against the Nebula Mobile patrol worktree:
    • Before: screen-onboarding-agents-live failed at step 3 (Save Profile).
    • After causes 1-3 were fixed, steps 0-9 replayed with no fallback, through Profile, Workspace and Team to /onboarding/team. The replay then stalled on Team, which exposed cause 4.
    • Retest verdicts: iss_a850d7b4195b came back fixed and iss_9c35fb29d760 came back fixed.

Rollout notes

  • Existing routines on the patrol box were migrated once, with a backup in ~/.bughunters/routines-backup-20261116-bp197/:
    • literal bugpatrol… usernames were changed to bugpatrolnew{{RUN_ID}};
    • the 19 Team steps that lost their invite got {{TEAMMATE_EMAIL}} back.
  • New recordings get both automatically.
  • agent-labs-dev/nebula-mobile#1874 and agent-labs-dev/nebula-desktop#5139 switch the onboarding guides to bugpatrolnew{{RUN_ID}}. Merge them only after a release that includes this PR.

🤖 Generated with Claude Code

Two things broke every new-user retest (bugpatrol#197):

- One-time values. The explorer recorded literal usernames
  (`bugpatrolnew77`, `bug-patrol-77`) into the onboarding routines. Dev keeps
  a username taken after its account is deleted, so on the next fresh account
  Save Profile stays disabled, the replay never leaves the Profile step, and
  every later step falls back to stored points. Vars now carries a public
  `{{RUN_ID}}`, new on each run, and the explorer is told to put it in any
  value the app accepts only once. Its trail already keeps placeholders, so a
  routine records `bugpatrolnew{{RUN_ID}}` and each replay types a new name.

- The connect hand-off. A connect URL like `/test/e2e-login?token=...` signs
  in ~1s after load and then redirects itself. A routine that starts with
  `open /` reloaded the page before that, and the app came up signed out
  (2 of 2 immediate opens; 0 of 1 after the redirect). connect() now waits,
  up to 15s, for a connect URL with a query to move on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nebula AI Agent <277912164+nebula-ai-agent[bot]@users.noreply.github.com>
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
bugpatrol Skipped Skipped Nov 16, 2026 4:31pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Bugpatrol review: dashboard

No problem introduced · tested a87d516 against main a228555

Tested

  • Overview page
  • Issue detail: Save button overlaps account text (fix, diff, attempts render)

Not tested

  • The rest of the change (replay hand-off wait, named-target wait, RUN_ID, locationKey): outside the dashboard

Updated on each push. Tests only what the diff can affect. Does not block the merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Bugpatrol review: fixture-app

No problem introduced · tested a87d516 against main a228555

Tested

  • Home page (index.html at /)
  • Settings page (/settings) via the Home Settings link

Not tested

  • The rest of the change (packages/agents, packages/drivers, docs/configuration.md): outside the fixture app

Updated on each push. Tests only what the diff can affect. Does not block the merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Bugpatrol review: cli

No problem introduced · tested a87d516 against main a228555

Next run: The explorer used its 20 minutes. Raise agents.explorer.timeoutMs.

Tested

The explorer stopped before it finished (timeout), so it tested a part of the change only.

Updated on each push. Tests only what the diff can affect. Does not block the merge.

compactSteps cuts a detour when the app comes back to an earlier place,
and the place was the screen key plus the element names. Field values were
not part of it, so on the onboarding Team step "type not-an-email, Send
invite (error), type {{TEAMMATE_EMAIL}}" looked like a return to the same,
unchanged form, and the real invite was cut: 19 Nebula Mobile routines
replayed only the invalid address and stalled on Team (#197).

The location key now includes each element's value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nebula AI Agent <277912164+nebula-ai-agent[bot]@users.noreply.github.com>
@joaquim-verges joaquim-verges added the mergepatrol Queue this PR: Mergepatrol rebases, tests and merges it label Nov 16, 2026 — with Nebula AI Agent
@merge-loop

merge-loop Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Merged as a216d5b.

@merge-loop merge-loop Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MergeLoop's review approves a87d516: nothing it found holds this PR.

@merge-loop
merge-loop Bot merged commit a216d5b into main Nov 16, 2026
12 checks passed
@merge-loop merge-loop Bot removed the mergepatrol Queue this PR: Mergepatrol rebases, tests and merges it label Nov 16, 2026

This branch was previously deployed

1 inactive deployment
Preview — a87d516f Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fresh-account routines replay one-time values, so new-user retests can't rebuild state

1 participant