Repository navigation
fix(replay-vision): say when replay is off or the change needs a deploy - #448
Conversation
The enable-replay step now checks how many sessions the project records and hands off three facts: the replay setting, whether the run changed code, and the recording count. The report leads with replay being off as an error, and tells the user to deploy when the run changed code or the project has no recordings yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
A project with no recordings and no code change from the run may just have no traffic yet. The report now asks for a deploy only when the run changed code. With no change and no recordings, it says scanners start with the first recording, and that the live app may not include the SDK yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
| Settings → Session replay, and finish. | ||
| - **Permission rejection**: record a follow-up to enable it from a | ||
| project-admin account, and finish — scanners created later sit idle until | ||
| - **Tool missing on this deploy**: don't fail the task — your handoff says |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
A 403 or missing tool means "unknown", not "off". Otherwise a non-admin on a project that already records gets a false error.
| - **Session replay is off**: lead with it as an error, not a follow-up — | ||
| session replay is off in this project, and no scanner runs until a project | ||
| admin turns on "Record user sessions" under Settings → Session replay. | ||
| - **This run changed code**: lead with "Deploy this change: session replay |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
If recordings already exist, scanners bill right away, so "cost nothing until then" is wrong. Only say this when the count is 0 or unknown?
| sessions here. Say so in your handoff and finish — scanner tasks read your | ||
| handoff and skip what doesn't apply. | ||
|
|
||
| Then read what actually records: call `vision-scanners-estimate` once with no |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
This only counts sessions long enough to scan, so a site with short visits reads 0 and gets "SDK may not be deployed".
| - **Tool missing on this deploy**: don't fail the task — your handoff says | ||
| session replay is **off** and the user must turn on "Record user sessions" | ||
| under Settings → Session replay. Then finish. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
A missing tool doesn't mean replay is off, so this can lead the report with a false error. Hand off "unknown" instead? (A 403 does mean off, so that case is fine.)
| sessions here. Say so in your handoff and finish — scanner tasks read your | ||
| handoff and skip what doesn't apply. | ||
|
|
||
| Then read what actually records: call `vision-scanners-estimate` once with no |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
The "Then finish" exits above come before this step, so those paths never hand off "code changed". Move this step above them?
| - **Replay setting**: on, or off and why. | ||
| - **Code changed**: yes or no — yes when this run installed or initialized | ||
| the SDK (upstream handoffs) or you edited the init. Name the files. | ||
| - **Recordings so far**: the count, or "unknown" if the estimate call failed. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
This count covers only the last window_days (up to 7) and only sessions that pass the quality filter. So "no recordings yet" can be wrong for a project with short sessions. #449 also needs window_days to turn the count into a monthly number. Hand that off too?
| When code changed, the handoff says the user must deploy (merge and release) | ||
| this change before any recording, and so any scanner observation, arrives. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
If this run wrote the init, the key is usually in a gitignored env file. A deploy records nothing until those env vars are set in prod, so the report should name them.
| Write the report to `./posthog-replay-vision-report.md` (the wizard shows this file at the end of the run), then summarize it for the user. Cover, briefly and concretely: | ||
| Write the report to `./posthog-replay-vision-report.md` (the wizard shows this file at the end of the run), then summarize it for the user. Open with what still blocks the first observation, before anything else: | ||
|
|
||
| - **Session replay is off**: lead with it as an error, not a follow-up - no scanner runs until a project admin turns on "Record user sessions" under Settings → Session replay. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
The linear flow has the report rule but not the enable-time rule, which lives only in enable-replay.md. So the agent never learns when replay counts as off. It also has no rule for when replay is off and code changed, or when the count is unknown.
…v vars A missing products-enable tool now hands off the replay setting as unknown instead of off. Every path reaches the handoff, which now carries every recording in 30 days, the scannable-session estimate with window_days, and the env vars a deploy needs in production. The report says scanners cost nothing only when there are no recordings, and the linear skill gets the same rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
edwinyjlim
left a comment
edwinyjlim
left a comment
There was a problem hiding this comment.
looks reasonable! if you tested the wizard runs with the new prompt, let's ship
|
/wizard-ci replay-vision |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
When products-enable is missing or rejected, the agent reads session_recording_opt_in with project-get and hands off on, off, or unknown from what it reads. A token that cannot turn replay on no longer produces a false "replay is off" error. The deploy list leaves out env files and anything gitignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
|
/wizard-ci replay-vision |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
|
/trunk merge |
Problem
The Replay Vision wizard turns on session replay and creates scanners, but it never tells the user that a code change made on their machine records nothing until it is deployed. Many wizard-created scanners sit enabled with no observations because their projects never record a session. The report also says "what is now recording" even when nothing records, and treats "replay is off" as a follow-up note.
Changes
enable-replayreads the project's recent recording count (vision-scanners-estimatewith no query) and hands off three facts: the replay setting, whether the run changed code (and which files), and the count.replay-visionskill gets the same rules.Why
Users read "configured" and wait, while their change is still only local. They need to know that the deploy is the step that starts their scanners.
How did you test this code?
npm test(220 passed) andnpm run buildpass. Prose-only change to the agent steps; I did not run a full wizard run against a test app.🤖 Generated with Claude Code