Repository navigation
Conversation
Every scanner task now estimates its exact query before creating it, and never creates a query that matches nothing in a project that records sessions. The breakage monitor scopes pages with a `visited_page` regex grounded in real recorded paths instead of a `$current_url` event filter. The frustration monitor falls back to active sessions at a low rate, excluding the breakage pages, when the project has no rage clicks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a wizard branch:
Add Results will be posted here when complete. |
…tcome classifier The replay-vision command now picks a tier from the project's monthly sessions. Low-volume projects get scanners with no filters, sampled to fit the free credits. Projects with no recordings yet get unfiltered scanners with free_only on. Busy projects keep the targeted, grounded queries. The command now creates a problems monitor (breakage and getting stuck in one scanner), a key flow outcome classifier, and session summaries. The breakage and frustration briefs stay for self-driving step 6c. Page scope rules move to the shared core skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
ksvat
changed the title
The free credits cover about 500 scans a month on the default model, so scanners with no filters only make sense for projects under about 250 sessions a month. Busier projects use the targeted queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
| 1.0, and add `"free_only": true`: the scanner then never adds to the bill, | ||
| and pauses if the organization's free credits run out. Say in your handoff | ||
| that the volume was unknown. | ||
| - **Low volume** (under 250 sessions a month, about what the free credits |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
The low tier has no cap: three unfiltered scanners, no credit_limit, no free_only. It also sizes from the org-wide free credits, ignoring what other scanners already spend. Set "free_only": true and size from free_only_monthly_credits_left?
| - **No recordings yet, or low volume**: no filters — `{"kind": | ||
| "RecordingsQuery"}`. `not-attempted` then tells the user how many sessions | ||
| never reach the flow. | ||
| - **High volume**: a `visited_page` scope on the flow's first page, at |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
If this scope matches nothing, core says "follow your brief's fallback", but there isn't one here. The agent lands on core's busiest paths and classifies a flow that isn't there. Say "skip the classifier and explain why"?
…an unmatched classifier Low-volume scanners have no filters, so their spend grows with traffic. They are now free_only, and their sampling rate is sized from free_only_monthly_credits_left, so it accounts for credits the organization already used. The flow outcome classifier has no page fallback: when its flow scope matches no session, it is skipped. The busiest-paths fallback moves from the core skill into the problems and breakage briefs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
| that the organization's own scanners need. Set `sampling_rate` to | ||
| `min(1, share × free_left / (sessions_per_month × | ||
| credits_per_observation))`, rounded down to two decimals. `share` is in |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
Rounding down with no floor turns any rate below 0.01 into 0, which pauses the scanner. So 1 credit left is worse than 0 left, which gets 0.1. Floor at 0.01?
| your brief. `free_left` is `free_only_monthly_credits_left` and | ||
| `credits_per_observation` comes from the same estimate of this scanner's | ||
| query: the free credits still left after what the organization already | ||
| used and its other free-only scanners. The three scanners' shares add up | ||
| to 1, so together they fit what is left. When `free_left` is 0, create the | ||
| scanner at `sampling_rate` 0.1 and say in your handoff that it waits until | ||
| the free credits reset. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
free_left already subtracts the scanners this setup made on a previous run, so rates shrink on every rerun. It's also this month's leftover, but the rate never changes after the reset. Size from the monthly ceiling minus the other free-only scanners instead?
| - **High volume** (250 sessions a month or more): your brief's targeted | ||
| query and sampling rate, checked against real sessions as below. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
Nothing sets free_only: false here. A project that grows past 250 a month keeps free-only scanners, and they pause every month.
| = `matched_sessions_in_window` / `window_days` × 30. Reuse the enable-replay | ||
| handoff's count when it gives one. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
This needs #448 to land first, and the handoff to include window_days. Also, L41 ("don't change sampling rates") contradicts the tiers.
| ```sql | ||
| SELECT path(url) AS page, uniq(session_id) AS sessions | ||
| FROM session_replay_events | ||
| ARRAY JOIN all_urls AS url | ||
| WHERE start_time >= now() - INTERVAL 7 DAY | ||
| GROUP BY page | ||
| HAVING sessions >= 5 | ||
| ORDER BY sessions DESC | ||
| LIMIT 100 | ||
| ``` |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
On small projects HAVING sessions >= 5 can return no rows, and nothing covers that case. Also path(url) drops the # part, so on hash-routed apps every page becomes /.
| **Earlier runs of this command** created a separate breakage monitor and a | ||
| frustration monitor (`emits_signals: false`). A breakage monitor from an | ||
| earlier run (match phrase `moments where the product visibly broke for the | ||
| user`) is this brief's predecessor: update it in place into this brief, per | ||
| the core re-run rules. Leave a frustration monitor from an earlier run | ||
| (match phrase `clear signs the user got stuck or frustrated`) untouched, and | ||
| say in your handoff that it now overlaps this monitor, so the user can turn | ||
| it off. |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
When the old breakage monitor is updated in place, what happens to its credit_limit and to a rate the user set themselves?
…tings on re-runs Low-volume sampling now comes from the monthly free credits minus other free-only scanners, leaving out this setup's own earlier scanners, with a 0.01 floor. High volume sets free_only off, and a re-run moves a scanner's query, rate and free_only only when its tier changed. credit_limit is never changed. Tier counts come from the enable-replay handoff or are read directly. The page query keeps hash routes and covers an empty result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
|
/wizard-ci replay-vision |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
|
/wizard-ci replay-vision |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
| `min(1, share × free_pool / (sessions_per_month × | ||
| credits_per_observation))`, rounded down to two decimals, and never below | ||
| 0.01. `share` is in your brief, and `credits_per_observation` comes from | ||
| the estimate of this scanner's query. `free_pool` is the monthly free |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
Free-only budgets draw on the org's total spend, paid scanners included (quota.py:750), and vision-scanners-list only sees this project. Could this size from the estimate's free_only_monthly_credits_left, adding back this setup's own earlier scanners? That keeps re-runs stable too.
| such projects stay small. Use your brief's low-volume query (no filters) | ||
| at `sampling_rate` 1.0, with `"free_only": true`. Say in your handoff that | ||
| the volume was unknown. | ||
| - **Low volume** (under 250 sessions a month, about what the free credits |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
Recordings > 0 with 0 scannable sessions divides by zero here, and the "never create a query that matches nothing" rule then blocks all three. Treat it like the no-recordings tier?
| - **`credit_limit`**: never change it. Setup never sets one, so it is always | ||
| the user's. | ||
| - **Same tier as before**: keep its `sampling_rate` and `free_only`. Read the | ||
| earlier tier from the scanner itself: `free_only` with no filters is a low |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
The high-tier summarizer has no filters and free_only: false, so it matches neither rule. Read the tier from free_only alone?
| - **Same tier as before**: keep its `sampling_rate` and `free_only`. Read the | ||
| earlier tier from the scanner itself: `free_only` with no filters is a low | ||
| tier; any filter is the high tier. | ||
| - **The tier changed** (the project grew past 250 sessions a month, or |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
The handoff tells users to turn off Free credits only, and the next re-run turns it back on. Never flip free_only from false to true on an update, and mention the mismatch in the handoff instead.
|
|
||
| Then pick the tier: | ||
|
|
||
| - **No recordings yet** (recordings is 0): the volume is unknown, and most |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
At rate 1.0 with no shares, the three can drain the free pool in days once traffic lands, and nothing re-tunes them. Apply the 0.4/0.4/0.2 shares here too, or tell the user to re-run setup once recordings arrive?
| ``` | ||
|
|
||
| A hash-routed app shows its pages after `#` (`/#/booking`), so anchor those | ||
| regexes on the fragment: `^https?://[^/]+/#/booking(/|\?|$)`. If the query |
There was a problem hiding this comment.
🤖 Agent-drafted, reviewed by Tue.
nit: this misses hash apps under a path (/app/#/booking). Something like ^https?://[^/]+(/[^#]*)?#/booking(/|\?|$)?
sarahxsanders
left a comment
sarahxsanders
left a comment
There was a problem hiding this comment.
hey @ksvat!! checked Tue's latest comments. before we ship, can you fix:
- the free-credit budget calculation
- handle recordings with 0 scannable sessions
- make reruns preserve the summarizer's settings and the user's free-only choice?
we also need a run that creates the scanners, both supported-app runs deferred them because of missing perms
the node failure is expected so that isn't blocking
… on re-runs Free-only sampling now comes from free_only_monthly_credits_left, which covers the whole organization, adding back this setup's own earlier scanners. A project whose recordings are all too short to scan uses the no-scannable-sessions tier instead of dividing by zero, and that tier now applies the free-credit shares. Re-runs read the earlier tier from free_only alone and never change it from false to true. The hash-route regex covers apps under a path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 158f9145-6821-4694-9719-574ec121ec75
|
/wizard-ci replay-vision |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
Merge after #448: the volume tiers read the recordings and scannable-session counts that #448 adds to the enable-replay handoff. Without it, the scanner tasks read the counts themselves.
Problem
Many wizard-created Replay Vision scanners are enabled but never write an observation. Most wizard projects record few sessions, yet the wizard always narrows its scanners with filters guessed from the repo: a
$rageclickgate and a$current_urlpage filter. On a small project these filters often match nothing. The wizard also never creates a classifier, the scanner type people open most often when they build scanners in the app.Changes
replay-vision-scanners-core): the agent reads the project's sessions per month from one unfiltered estimate.free_only: true, sized for 250 sessions a month.free_only: true. Sampling comes fromfree_only_monthly_credits_left(org-wide), adding back this setup's own earlier scanners, with a 0.01 floor.visited_pagepaths grounded in real recordings and checked withvision-scanners-estimatebefore create.wizard replay-vision:free_only, keepcredit_limit, and keep the query and rate unless the tier changed. They never changefree_onlyfrom false to true. High volume setsfree_only: false.How did you test this code?
npm test(220 passed) andnpm run buildpass, and the build packages both new briefs. The query shapes (visited_pageregex,not_regex,active_seconds, unfiltered estimate) and the page-path SQL return results against a real project through the MCP. A combined pages-OR-rage-click query did not widen the match, so the briefs do not use it. No full wizard run against a test app.🤖 Generated with Claude Code