From 940afca1910b0431f1b6c82649c7224216586ed6 Mon Sep 17 00:00:00 2001 From: Yaacov Date: Tue, 17 Nov 2026 03:07:12 +0200 Subject: [PATCH 1/2] feat(events): stop sending analytics to PostHog; D1 is the only store Remove the desktop and website event export, the identity-link export and the PostHog-only operator scripts (dashboards, API client, HogQL insights, D1-vs-PostHog reconciliation). Nothing is sent to PostHog any more. Installation erasure still deletes copies PostHog already holds, using POSTHOG_PERSONAL_API_KEY and POSTHOG_PROJECT_ID under its own lease. That is the only remaining PostHog code and goes once the PostHog project is deleted. POSTHOG_PROJECT_TOKEN and DESKTOP_POSTHOG_EXPORT_ENABLED are no longer read, /health drops the forwarding fields, and the analytics report no longer treats posthog_state='pending' as a delivery backlog. Co-Authored-By: Claude Opus 5.5 --- README.md | 170 ++---- package.json | 2 - scripts/analytics-reconciliation.mjs | 63 --- scripts/analytics-reconciliation.test.mjs | 36 -- scripts/analytics-report.mjs | 25 +- scripts/analytics-report.test.mjs | 22 +- scripts/manage-posthog-dashboards.mjs | 171 ------ scripts/posthog-api.mjs | 62 --- scripts/posthog-api.test.mjs | 53 -- scripts/posthog-dashboard-manifest.mjs | 525 ------------------ scripts/posthog-dashboard-manifest.test.mjs | 342 ------------ scripts/product-insights-hogql.mjs | 91 --- scripts/reconcile-analytics-pipeline.mjs | 88 --- .../events/src/attributionPipeline.test.ts | 2 - .../src/{exportLease.ts => deletionLease.ts} | 9 +- workers/events/src/desktopPipeline.test.ts | 56 +- workers/events/src/index.test.ts | 145 +---- workers/events/src/index.ts | 386 +------------ workers/events/src/people.test.ts | 30 +- workers/events/src/readiness.test.ts | 222 +------- workers/events/src/transport.ts | 17 +- workers/events/worker-configuration.d.ts | 6 +- workers/events/wrangler.jsonc | 5 +- 23 files changed, 130 insertions(+), 2398 deletions(-) delete mode 100644 scripts/analytics-reconciliation.mjs delete mode 100644 scripts/analytics-reconciliation.test.mjs delete mode 100644 scripts/manage-posthog-dashboards.mjs delete mode 100644 scripts/posthog-api.mjs delete mode 100644 scripts/posthog-api.test.mjs delete mode 100644 scripts/posthog-dashboard-manifest.mjs delete mode 100644 scripts/posthog-dashboard-manifest.test.mjs delete mode 100644 scripts/product-insights-hogql.mjs delete mode 100644 scripts/reconcile-analytics-pipeline.mjs rename workers/events/src/{exportLease.ts => deletionLease.ts} (85%) diff --git a/README.md b/README.md index 11d8ed5..ad49bde 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,16 @@ Local and Cloudflare preview hosts do not write events, which keeps production c ## Analytics gateway -The Worker under `workers/events` is ScientFactory's first-party telemetry and identity gateway. Desktop clients submit bounded event batches to `https://events.scientfactory.com/v1/events`; the Worker stores them in D1 first and can then forward pseudonymous copies to the ScientFactory EU PostHog project. PostHog is an optional analysis layer rather than the primary event store. +The Worker under `workers/events` is ScientFactory's first-party telemetry and identity gateway. Desktop clients submit bounded event batches to `https://events.scientfactory.com/v1/events`; the Worker stores them in D1, the only analytics store. + +Until 17 November 2026 the Worker also forwarded pseudonymous copies of website +and desktop usage events, and website identity links, to the ScientFactory EU +PostHog project. It no longer sends anything there. The only remaining PostHog +code is erasure: installation deletion requests still delete that +installation's earlier copies from PostHog. Remove the deletion code, +`POSTHOG_PERSONAL_API_KEY` and `POSTHOG_PROJECT_ID` once that PostHog project is +deleted. The `posthog_*` columns of shipped migrations stay; new event and identity +link rows keep `posthog_state='pending'` forever and nothing reads it. Desktop ingestion works like T3's: the Worker accepts any event the app records, with no list of event or property names. It checks schema version 1, the event's shape and size (an event name, @@ -106,25 +115,23 @@ limit per opaque installation ID. It does not use or store an IP address as a rate-limit key. Keep the variable absent or false during preparation and use it as the immediate ingestion kill switch during a selected-user rollout. -Desktop forwarding to PostHog has its own gate, -`DESKTOP_POSTHOG_EXPORT_ENABLED`, also false by default. Turning off ingress -does not drain queued data; turning off export prevents queued desktop copies -from being forwarded. Neither gate affects existing website forwarding. - `POST /v1/installations/delete` authenticates an installation, deletes its D1 -events, consent, identity links, and identity record, and queues the matching -PostHog person and historical-event deletion by opaque distinct ID. A request from an installation that has never uploaded is +events, consent, identity links, and identity record, and, if that installation's +events were ever sent to PostHog, queues the matching PostHog person and +historical-event deletion by opaque distinct ID. A request from an installation that has never uploaded is acknowledged idempotently so the desktop can still clear local data and rotate its anonymous identity. A minimal opaque-ID/authentication-hash tombstone blocks late uploads from recreating deleted history. Migration 0006 preserves legacy erasure tombstones conservatively; missing legacy credentials cannot be guessed. No behavioral payload is retained in the tombstone. -Erasure completes without PostHog only if no export was ever attempted. Otherwise +Erasure completes without PostHog only if no export was ever attempted (every +installation first seen after the export stopped). Otherwise the scheduled Worker submits/polls PostHog's person/event deletion, with bounded failure retries and an operator-visible blocked state. Provider verification, -not submission, completes the gateway request. Export/deletion share a lease; -deleted identities are tombstoned and never deliberately reused. Completion +not submission, completes the gateway request. A database lease, renewed before +each PostHog request, keeps overlapping cron runs from submitting the same +deletion twice; deleted identities are tombstoned and never deliberately reused. Completion records PostHog's verified operation, not a synchronous transaction covering every ambiguous capture or a guaranteed physical-deletion deadline. Do not claim remote deletion is complete while its state is pending or blocked. @@ -132,18 +139,14 @@ Do not claim remote deletion is complete while its state is pending or blocked. The scheduled Worker prunes canonical raw events older than 180 days in bounded batches of 5,000; diagnostics have a 30-day limit. An unfinished backlog does not count as a successful retention pass. Desktop occurrence age is also -enforced during ingestion, export and pruning, so offline delivery does not -reset its retention clock. D1 remains the source of truth -for retention and delivery; dashboard filters are not physical retention controls. -Desktop Diagnostic-class events remain only in Scient's central D1 ledger and -are never exported to PostHog. `bun run analytics:report` exposes their bounded -30-day aggregate breakdown alongside maintenance health; no access to a user's -computer is required. Essential/Product-class events can be exported even when -the user's consent level is Diagnostic. PostHog retention for those copies is -provider-managed; its query-access window is not a physical-deletion deadline. -Do not advertise 13-month or 30-day PostHog deletion guarantees. - -Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a website visitor to an account after Scient's account service has authenticated that user. Desktop linking is rejected until its per-installation erasure model is qualified. Browser and desktop clients cannot claim account identity. Website linking preserves consent; the PostHog identity event is forwarded only for Product-or-higher consent. +enforced during ingestion and pruning, so offline delivery does not +reset its retention clock. `bun run analytics:report` exposes the bounded +30-day aggregate breakdown of desktop diagnostics alongside maintenance health; +no access to a user's computer is required. Copies already in PostHog follow +PostHog's own retention until that project is deleted; do not advertise a +PostHog deletion deadline. + +Website visitors, desktop installations, sessions, and future Scient accounts use separate opaque identifiers. The service-authenticated `POST /v1/identity/link` endpoint can connect a website visitor to an account after Scient's account service has authenticated that user. Desktop linking is rejected until its per-installation erasure model is qualified. Browser and desktop clients cannot claim account identity. Website linking preserves consent and is stored only in D1. Generate binding types and validate the Worker with: @@ -160,8 +163,8 @@ candidate, then run: SCIENT_ANALYTICS_DESKTOP_ROOT=/absolute/desktop bun run test workers/events/src/desktopPipeline.test.ts ``` -This test connects the built desktop worker to a loopback gateway and mocked -PostHog exporter, checks forbidden-data removal and runtime-source metadata, +This test connects the built desktop worker to a loopback gateway, checks +forbidden-data removal and the stored runtime-source metadata, then exercises consent, deletion, and late-replay rejection. It does not touch production and is intentionally skipped when no explicit desktop path is set. Record the desktop revision/build as well as the website revision; ordinary @@ -176,45 +179,35 @@ Deploy the Worker only from an approved production change: bun run events:deploy ``` -`POSTHOG_PROJECT_TOKEN`, `POSTHOG_PERSONAL_API_KEY`, and `IDENTITY_LINK_TOKEN` -are Cloudflare Worker secrets and must never be committed. The personal key is -used only for queued deletion and needs the reviewed person read/write scopes -for lookup, submission, and verification (qualify the exact provider permissions); -`POSTHOG_PROJECT_ID` selects the project and is committed as non-secret Worker -configuration. If the project token is absent, -ingestion continues and events remain queued in D1 for later delivery. If the +`POSTHOG_PERSONAL_API_KEY` and `IDENTITY_LINK_TOKEN` are Cloudflare Worker +secrets and must never be committed. The personal key is used only for queued +PostHog deletion and needs the reviewed person read/write scopes for lookup, +submission, and verification; `POSTHOG_PROJECT_ID` selects the project and is +committed as non-secret Worker configuration. The old `POSTHOG_PROJECT_TOKEN` +secret is no longer read and can be deleted. If the deletion key or project ID is absent, accepted erasures remain queued in D1. If the identity-link token is absent, account linking returns `503` while ordinary ingestion continues. Before activating an owner-approved rollout: -1. Apply the approved migrations and deploy the reviewed Worker with **both - desktop gates false**. Website Pages deployment is not Worker deployment. +1. Apply the approved migrations and deploy the reviewed Worker with the + desktop ingestion gate false. Website Pages deployment is not Worker deployment. 2. Verify `/health` against the exact deployed revision: required schema and a recent successful retention pass are checked, and the Worker version ID, tag, and creation time identify the deployed artifact. Configured secrets are not proof of valid permissions. Tag production uploads with the reviewed Git commit. -3. Confirm the approved first-party-only diagnostic routing and truthful - PostHog-managed retention wording. Verify asynchronous provider erasure with - synthetic identifiers; an arbitrary delay or repeat-delete loop is not proof. +3. Verify asynchronous PostHog erasure with synthetic identifiers; an arbitrary + delay or repeat-delete loop is not proof. 4. Exercise authorized synthetic end-to-end delivery, - rejection, erasure, retry, retention and aggregate reconciliation. Never use + rejection, erasure, retry and retention. Never use live researchers' records for a test or expose credentials in logs. 5. Complete human privacy-copy/consent/cohort review. Enable the approved - ingress/export gates only after qualification. A desktop release is still needed; + ingestion gate only after qualification. A desktop release is still needed; packaged apps are on by default, and `SCIENT_ANALYTICS_ENABLED=false` is the only off switch. -Export uses stable capture UUIDs and bounded retries, with a database lease -renewed before each outbound call. This prevents concurrent local exporters; -it is not a provider-side transactional fence. Persisted desktop properties are -revalidated so malformed/legacy rows cannot bypass today's privacy contract. -`posthog_state='sent'` means capture acknowledged, not erasure settled; -`'excluded'` marks rows written that way because export never sends them (desktop -diagnostics and identity observations). Event export runs from the five-minute -cron, up to ten batches of 100 per run, not after each upload, so an upload -costs one D1 round trip. Each upload logs its status, event count and stage +An upload costs one D1 round trip. Each upload logs its status, event count and stage durations (`read_ms`, `rate_limit_ms`, `d1_ms`, `total_ms`) without identifiers. A rejected upload's response and log carry a closed `reason` and `stage` (`headers`, `body`, `batch`, `event`, `privacy` or `size`), listed in @@ -243,7 +236,7 @@ SCIENT_IDENTITY_LINK_TOKEN=... bun run identity:link \ This command is an operational bridge, not a substitute for account authentication. The eventual account service should call the endpoint server-to-server after sign-in; no link token belongs in a client bundle. -## PostHog dashboards +## Product insights `bun run analytics:insights` reads an aggregate-only product report from D1: feature observations/repeat days, provider/model terminal usage, reported token @@ -261,84 +254,17 @@ state is the latest observation in the reporting window; an install operation count is a separate measure. Historical `other` cannot be attributed to Scient Agent. Ready observations are not a current sign-in inventory, installations are not people, and observed population is not feature eligibility. See the desktop -analytics document for producer meanings and omissions. Existing delivery and -erasure diagnostics remain in `analytics:report`. - -The companion PostHog query definitions remain prepared, not installed or -live-qualified. Qualify their execution and project timezone before publishing; -match UTC to the D1 report when reconciling periods. Deploy the generated -revision-3 validator before releasing new desktop producers. No database -migration, collection-gate change or dashboard mutation is part of this extension. - -The managed dashboard manifest is `scripts/posthog-dashboard-manifest.mjs`. -It records all planned product dashboards, their source-backed queries, and the -exact events each one needs. -The manager is read-only by default and retrieves the personal API key from the -`scient-posthog-personal-api-key` macOS Keychain item (or the -`POSTHOG_PERSONAL_API_KEY` environment variable): - -```sh -bun run analytics:dashboards -``` - -Validate every prepared query against PostHog without creating or changing a -dashboard: - -```sh -bun run analytics:dashboards --validate-queries -``` - -Only dashboards marked current and backed by observed events are eligible for -creation. Planned dashboards are not created as empty or misleading shells. -After reviewing the readiness output, an authorized operator can idempotently -create or update ready dashboards: - -```sh -bun run analytics:dashboards --apply-ready -``` - -Operator API requests are project-origin restricted, time/body bounded, and do -not blindly retry ambiguous creates. Pagination is bounded. The script never -deletes dashboards or insights. D1 delivery state remains the -operational source of truth and should be reconciled with PostHog using -`bun run analytics:report` before relying on a dashboard. That aggregate-only -report includes pending/blocked deletion, exhausted or quarantined delivery, -and missing/stale maintenance; it is not a claim of end-to-end healthy delivery. -Maintenance rows have a status and timestamp, not an event count. The exact per-event -D1-sent and PostHog counts can be checked without exposing the personal API key: - -```sh -bun run analytics:reconcile -``` +analytics document for producer meanings and omissions. -Reconciliation defaults to desktop events in the last seven days, excluding -the newest hour, and compares the same occurrence window and deduplicated event -IDs in both stores, excluding first-party-only diagnostics. Override it with `ANALYTICS_RECONCILE_SOURCE`, -`ANALYTICS_RECONCILE_FROM`, and `ANALYTICS_RECONCILE_TO` (maximum 30 days). -The one-hour delay is a reporting convention, not an erasure guarantee. -Mismatch, pending events, outstanding deletions, and no data exit non-zero; -an empty dashboard is not a verified pipeline. - -Prepared metrics count consenting installation profiles, not all people. Product -success rates use a consistent Product/Diagnostic population and exclude terminal -stops. Activation uses the gateway's first-observed Product cohort anchor, -excludes unknown legacy anchors, requires ordered steps, and reports immature -cohorts separately. Retention needs complete follow-up windows. Billing allowances -are not hardcoded as current facts. Exact HogQL execution and installed-dashboard -behavior still require authorized provider-side qualification. - -Scientific `source-import` outcomes have item-attempt grain, not batch grain. -Saved source-store results count as completions even when later batch cleanup -fails; duplicate/possible-match skips are reported separately as -`scient.operation.skipped` and do not qualify for meaningful-use metrics. -Retries are new attempts, not a second completion of an already saved source. -These events do not claim batch-conversion or human-review coverage. +`bun run analytics:report` lists pending or blocked erasures and missing or +stale maintenance; it is not a claim of end-to-end healthy delivery. +Maintenance rows have a status and timestamp, not an event count. ### Account analytics activation Revision 11 accepts provider account/profile observations. `IDENTITY_ANALYTICS_ENABLED` is committed as `true` in both the Pages and event Worker configurations. Existing -usage ingestion/export stays independent. With this flag off, new identity events +usage ingestion stays independent. With this flag off, new identity events are acknowledged and discarded rather than queued for later activation. Migration 0007 must be applied before deploying the compatible gateway. Website @@ -348,7 +274,7 @@ The owner authorized production migration and activation on 10 November 2026, with the public privacy-wording review to follow preparation and activation. With activation enabled, dedicated account, profile, device-hash and request-network -observations remain in Cloudflare and are excluded from PostHog. Token-authenticated +observations remain in Cloudflare. Token-authenticated installation deletion removes its observations too. Usage follows two-year inactivity retention; diagnostic events expire after 60 days. The fallback 180/30-day retention path applies only when identity analytics is disabled. Local desktop outbox retention diff --git a/package.json b/package.json index bf8b0b4..551e9f9 100644 --- a/package.json +++ b/package.json @@ -18,8 +18,6 @@ "db:migrate": "wrangler d1 migrations apply scientfactory-downloads --remote", "analytics:report": "node scripts/analytics-report.mjs", "analytics:insights": "node scripts/product-insights.mjs", - "analytics:dashboards": "node scripts/manage-posthog-dashboards.mjs", - "analytics:reconcile": "node scripts/reconcile-analytics-pipeline.mjs", "identity:link": "node scripts/link-analytics-identity.mjs", "events:types": "wrangler types workers/events/worker-configuration.d.ts --config workers/events/wrangler.jsonc --env-interface AnalyticsWorkerBindings --include-runtime false", "events:types:check": "wrangler types workers/events/worker-configuration.d.ts --config workers/events/wrangler.jsonc --env-interface AnalyticsWorkerBindings --include-runtime false --check", diff --git a/scripts/analytics-reconciliation.mjs b/scripts/analytics-reconciliation.mjs deleted file mode 100644 index 340adbb..0000000 --- a/scripts/analytics-reconciliation.mjs +++ /dev/null @@ -1,63 +0,0 @@ -/** Same source, occurrence window and event-ID grain on both sides of the gateway. */ -export function reconciliationQueries({ source = "desktop", from, to }) { - if (!["desktop", "website"].includes(source)) throw new Error("Invalid source"); - const start = new Date(from); - const end = new Date(to); - if ( - !Number.isFinite(+start) || - !Number.isFinite(+end) || - +end <= +start || - +end - +start > 30 * 86400000 - ) { - throw new Error("Use an increasing window no longer than 30 days"); - } - const since = start.toISOString(); - const until = end.toISOString(); - return { - source, - from: since, - to: until, - d1: `SELECT event_name, posthog_state, COUNT(*) AS event_count - FROM analytics_events WHERE source = '${source}' - AND (source <> 'desktop' OR privacy_level <> 'diagnostic') - AND posthog_state <> 'excluded' - AND julianday(occurred_at) >= julianday('${since}') AND julianday(occurred_at) < julianday('${until}') - GROUP BY event_name, posthog_state ORDER BY event_name, posthog_state`, - posthog: `SELECT event, uniqExact(properties.event_id) FROM events - WHERE properties.source = '${source}' AND properties.event_id IS NOT NULL - AND (properties.source != 'desktop' OR coalesce(properties.privacy_level, '') != 'diagnostic') - AND timestamp >= parseDateTimeBestEffort('${since}') AND timestamp < parseDateTimeBestEffort('${until}') - GROUP BY event ORDER BY event`, - backlog: `SELECT posthog_state, count(*) AS request_count FROM analytics_deletion_requests - WHERE posthog_state <> 'completed' GROUP BY posthog_state`, - }; -} - -export function comparePipeline(d1Rows, posthogRows, deletionBacklog = 0) { - const counts = new Map(); - const rowFor = (name) => { - if (!counts.has(name)) counts.set(name, { name, sent: 0, pending: 0, posthog: 0 }); - return counts.get(name); - }; - const count = (value) => { - const result = Number(value); - if (!Number.isSafeInteger(result) || result < 0) - throw new Error("Invalid reconciliation count"); - return result; - }; - for (const row of d1Rows) { - if (!["sent", "pending"].includes(row.posthog_state)) throw new Error("Unknown delivery state"); - rowFor(String(row.event_name))[row.posthog_state] += count(row.event_count); - } - for (const [name, value] of posthogRows) rowFor(String(name)).posthog += count(value); - const rows = [...counts.values()].sort((a, b) => a.name.localeCompare(b.name)); - const status = - count(deletionBacklog) > 0 || rows.some((row) => row.pending > 0) - ? "unsettled" - : rows.length === 0 - ? "no-data" - : rows.some((row) => row.sent !== row.posthog) - ? "mismatch" - : "matched"; - return { status, rows, deletionBacklog }; -} diff --git a/scripts/analytics-reconciliation.test.mjs b/scripts/analytics-reconciliation.test.mjs deleted file mode 100644 index 33d4275..0000000 --- a/scripts/analytics-reconciliation.test.mjs +++ /dev/null @@ -1,36 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { comparePipeline, reconciliationQueries } from "./analytics-reconciliation.mjs"; - -describe("analytics reconciliation", () => { - it("uses a shared bounded source/time window and deduplicated capture IDs", () => { - const query = reconciliationQueries({ from: "2026-08-20", to: "2026-08-27" }); - for (const sql of [query.d1, query.posthog]) { - expect(sql).toContain("desktop"); - expect(sql).toContain("2026-08-20T00:00:00.000Z"); - expect(sql).toContain("2026-08-27T00:00:00.000Z"); - } - expect(query.posthog).toContain("uniqExact(properties.event_id)"); - expect(query.d1).toContain("privacy_level <> 'diagnostic'"); - // Never-exported rows would otherwise fail as an unknown delivery state. - expect(query.d1).toContain("posthog_state <> 'excluded'"); - expect(query.posthog).toContain("coalesce(properties.privacy_level, '') != 'diagnostic'"); - expect(() => - reconciliationQueries({ source: "desktop' OR 1=1", from: "2026-08-20", to: "2026-08-27" }), - ).toThrow(); - expect(() => reconciliationQueries({ from: "2026-01-01", to: "2026-08-27" })).toThrow(); - }); - it("does not call empty, pending or deletion-affected populations healthy", () => { - const rows = [{ event_name: "app.health", posthog_state: "sent", event_count: 2 }]; - expect(comparePipeline([], []).status).toBe("no-data"); - expect(comparePipeline([], [], 1).status).toBe("unsettled"); - expect(comparePipeline(rows, [["app.health", 2]]).status).toBe("matched"); - expect(comparePipeline(rows, [["app.health", 1]]).status).toBe("mismatch"); - expect(comparePipeline(rows, [["app.health", 2]], 1).status).toBe("unsettled"); - expect( - comparePipeline( - [...rows, { event_name: "app.health", posthog_state: "pending", event_count: 1 }], - [["app.health", 2]], - ).status, - ).toBe("unsettled"); - }); -}); diff --git a/scripts/analytics-report.mjs b/scripts/analytics-report.mjs index 4031e44..e847e49 100644 --- a/scripts/analytics-report.mjs +++ b/scripts/analytics-report.mjs @@ -4,7 +4,7 @@ import { fileURLToPath } from "node:url"; export const analyticsReportQuery = ` WITH expected_maintenance(name) AS ( - VALUES ('retention'), ('deletion'), ('identity-export'), ('event-export') + VALUES ('retention'), ('deletion') ) SELECT '30_day_identity' AS report_section, @@ -138,17 +138,6 @@ export const analyticsReportQuery = ` UNION ALL - SELECT - 'posthog_delivery' AS report_section, - CASE WHEN posthog_state = 'excluded' OR (source = 'desktop' AND privacy_level = 'diagnostic') - THEN 'first-party-only' ELSE posthog_state END AS item, - COUNT(*) AS event_count, - MAX(received_at) AS latest_event - FROM analytics_events - GROUP BY item - - UNION ALL - SELECT '30_day_desktop_diagnostics', event_name || ':' || COALESCE(json_extract(properties_json, '$.deliveryClass'), 'unknown'), COUNT(*), MAX(occurred_at) @@ -159,18 +148,6 @@ export const analyticsReportQuery = ` UNION ALL - SELECT - 'delivery_attention' AS report_section, - source || ':' || CASE WHEN posthog_last_error = 'contract-rejected' THEN 'contract-rejected' - ELSE 'retry-exhausted' END AS item, - COUNT(*) AS event_count, - MAX(received_at) AS latest_event - FROM analytics_events - WHERE posthog_state = 'pending' AND posthog_attempts >= 20 - GROUP BY item - - UNION ALL - SELECT 'deletion_state', posthog_state, COUNT(*), MAX(requested_at) FROM analytics_deletion_requests GROUP BY posthog_state diff --git a/scripts/analytics-report.test.mjs b/scripts/analytics-report.test.mjs index 26ca6ea..f562693 100644 --- a/scripts/analytics-report.test.mjs +++ b/scripts/analytics-report.test.mjs @@ -18,17 +18,13 @@ it("keeps remote report chunks within the D1 compound SELECT limit", () => { } }); -it("reports centrally retained diagnostics separately from PostHog delivery", () => { +it("reports centrally retained diagnostics", () => { const store = testDatabase(); try { store.sqlite.exec(`INSERT INTO analytics_events (event_id, event_name, source, privacy_level, occurred_at, distinct_id, properties_json) VALUES ('diagnostic', 'app.diagnostics', 'desktop', 'diagnostic', '2026-08-30', 'private-id', '{"deliveryClass":"retrying"}');`); const rows = store.sqlite.prepare(fixedQuery).all(); - expect(rows.find((row) => row.report_section === "posthog_delivery")).toMatchObject({ - item: "first-party-only", - event_count: 1, - }); expect(rows.find((row) => row.report_section === "30_day_desktop_diagnostics")).toMatchObject({ event_count: 1, }); @@ -61,20 +57,14 @@ it("keeps the aggregate report's 30-day window correct across SQLite and ISO tim } }); -it("shows blocked erasure, quarantine and missing or stale maintenance without raw errors", () => { +it("shows blocked erasure and missing or stale maintenance without identifiers", () => { const store = testDatabase(); try { - store.sqlite.exec(`INSERT INTO analytics_events - (event_id, event_name, source, privacy_level, occurred_at, distinct_id, properties_json, posthog_attempts, posthog_last_error) - VALUES ('retry', 'app.health', 'desktop', 'essential', '2026-08-30', 'private-id', '{}', 20, 'private-error-text'); - INSERT INTO analytics_deletion_requests (request_id, installation_id, requested_at, posthog_state) + store.sqlite + .exec(`INSERT INTO analytics_deletion_requests (request_id, installation_id, requested_at, posthog_state) VALUES ('erase', 'private-id', '2026-08-30', 'blocked'); INSERT INTO analytics_maintenance_status VALUES ('retention', '2026-08-31 11:00:00', 'ok');`); const rows = store.sqlite.prepare(fixedQuery).all(); - expect(rows.find((row) => row.report_section === "delivery_attention")).toMatchObject({ - item: "desktop:retry-exhausted", - event_count: 1, - }); expect(rows.find((row) => row.report_section === "deletion_state")).toMatchObject({ item: "blocked", event_count: 1, @@ -82,11 +72,9 @@ it("shows blocked erasure, quarantine and missing or stale maintenance without r const maintenance = rows.filter((row) => row.report_section === "maintenance"); expect(maintenance.map((row) => row.item).sort()).toEqual([ "deletion:never-run", - "event-export:never-run", - "identity-export:never-run", "retention:stale-ok", ]); - expect(JSON.stringify(rows)).not.toMatch(/private-id|private-error-text/); + expect(JSON.stringify(rows)).not.toContain("private-id"); } finally { store.close(); } diff --git a/scripts/manage-posthog-dashboards.mjs b/scripts/manage-posthog-dashboards.mjs deleted file mode 100644 index 25b5d30..0000000 --- a/scripts/manage-posthog-dashboards.mjs +++ /dev/null @@ -1,171 +0,0 @@ -#!/usr/bin/env node - -import { execFileSync } from "node:child_process"; - -import { dashboards } from "./posthog-dashboard-manifest.mjs"; -import { createPosthogApi } from "./posthog-api.mjs"; - -const PROJECT_ID = "228610"; -const KEYCHAIN_SERVICE = "scient-posthog-personal-api-key"; -const apply = process.argv.includes("--apply-ready"); -const validateQueries = process.argv.includes("--validate-queries"); - -function personalApiKey() { - if (process.env.POSTHOG_PERSONAL_API_KEY) return process.env.POSTHOG_PERSONAL_API_KEY; - if (process.platform !== "darwin") return undefined; - try { - return execFileSync("security", ["find-generic-password", "-w", "-s", KEYCHAIN_SERVICE], { - encoding: "utf8", - stdio: ["ignore", "pipe", "ignore"], - }).trim(); - } catch { - return undefined; - } -} - -const apiKey = personalApiKey(); -if (!apiKey) { - console.error( - `PostHog personal API key unavailable. Set POSTHOG_PERSONAL_API_KEY or add macOS Keychain service '${KEYCHAIN_SERVICE}'.`, - ); - process.exit(1); -} - -const api = createPosthogApi({ apiKey, projectId: PROJECT_ID }); - -async function observedEvents() { - const response = await api("query/", { - method: "POST", - body: JSON.stringify({ - query: { - kind: "HogQLQuery", - query: - "SELECT event, count() FROM events WHERE timestamp >= now() - INTERVAL 30 DAY GROUP BY event ORDER BY event", - }, - }), - }); - return new Map(response.results.map(([name, count]) => [String(name), Number(count)])); -} - -async function allPages(path) { - const results = []; - let next = `${path}${path.includes("?") ? "&" : "?"}limit=100`; - const seen = new Set(); - while (next) { - if (seen.has(next) || seen.size >= 100) throw new Error("PostHog pagination limit exceeded"); - seen.add(next); - const page = await api(next); - if (!Array.isArray(page.results)) throw new Error("Invalid PostHog pagination response"); - results.push(...page.results); - next = page.next ?? ""; - } - return results; -} - -async function ensureDashboard(definition, existingDashboards, existingInsights) { - const acceptedNames = new Set([definition.name, ...(definition.aliases ?? [])]); - let dashboard = existingDashboards.find((candidate) => acceptedNames.has(candidate.name)); - if (!dashboard) { - dashboard = await api("dashboards/", { - method: "POST", - body: JSON.stringify({ - name: definition.name, - description: definition.description, - tags: ["scient-managed", "analytics-contract-v2"], - }), - }); - console.log(`created dashboard: ${definition.name}`); - } else { - dashboard = await api(`dashboards/${dashboard.id}/`, { - method: "PATCH", - body: JSON.stringify({ - name: definition.name, - description: definition.description, - tags: ["scient-managed", "analytics-contract-v2"], - }), - }); - console.log(`updated dashboard: ${definition.name}`); - } - - for (const insightDefinition of definition.insights ?? []) { - const missing = (insightDefinition.requiredEvents ?? []).filter((name) => !observed.has(name)); - if (missing.length > 0) { - console.log(`skipped insight: ${insightDefinition.name} (unobserved required events)`); - continue; - } - const acceptedInsightNames = new Set([ - insightDefinition.name, - ...(insightDefinition.aliases ?? []), - ]); - const existing = existingInsights.find( - (candidate) => - acceptedInsightNames.has(candidate.name) && candidate.tags?.includes("scient-managed"), - ); - const currentDashboards = existing?.dashboards ?? []; - const payload = { - name: insightDefinition.name, - description: insightDefinition.description, - query: insightDefinition.query, - dashboards: [...new Set([...currentDashboards, dashboard.id])], - tags: ["scient-managed", `scient-dashboard-${definition.key}`], - }; - if (existing) { - const updated = await api(`insights/${existing.id}/`, { - method: "PATCH", - body: JSON.stringify(payload), - }); - Object.assign(existing, updated); - console.log(`updated insight: ${insightDefinition.name}`); - } else { - const created = await api("insights/", { - method: "POST", - body: JSON.stringify(payload), - }); - existingInsights.push(created); - console.log(`created insight: ${insightDefinition.name}`); - } - } -} - -const observed = await observedEvents(); -const readiness = dashboards.map((dashboard) => { - const missing = dashboard.requiredEvents.filter((name) => !observed.has(name)); - return { - dashboard, - missing, - ready: dashboard.phase === "current" && missing.length === 0, - }; -}); - -for (const { dashboard, missing, ready } of readiness) { - const state = ready ? "READY" : dashboard.phase === "planned" ? "PLANNED" : "BLOCKED"; - console.log(`${state.padEnd(7)} ${dashboard.name}`); - if (missing.length > 0) console.log(` missing events: ${missing.join(", ")}`); -} - -if (validateQueries) { - for (const { dashboard } of readiness) { - for (const insight of dashboard.insights ?? []) { - await api("query/", { - method: "POST", - body: JSON.stringify({ query: insight.query }), - }); - console.log(`valid ${dashboard.key}: ${insight.name}`); - } - } -} - -if (!apply) { - console.log( - "\nRead-only check complete. Use --validate-queries to execute definitions read-only or --apply-ready to update only READY dashboards.", - ); - process.exit(0); -} - -const [existingDashboards, existingInsights] = await Promise.all([ - allPages("dashboards/"), - allPages("insights/"), -]); -for (const item of readiness.filter((candidate) => candidate.ready)) { - await ensureDashboard(item.dashboard, existingDashboards, existingInsights); -} diff --git a/scripts/posthog-api.mjs b/scripts/posthog-api.mjs deleted file mode 100644 index 964f692..0000000 --- a/scripts/posthog-api.mjs +++ /dev/null @@ -1,62 +0,0 @@ -const ORIGIN = "https://eu.posthog.com"; -const MAX_RESPONSE_BYTES = 2 * 1024 * 1024; - -/** Operator-only API transport. Never runs in the desktop app or on page visits. */ -export function createPosthogApi({ apiKey, projectId, fetchImpl = fetch }) { - if (!/^\d+$/.test(projectId)) throw new Error("Invalid PostHog project"); - const root = `${ORIGIN}/api/projects/${projectId}/`; - return async (path, init = {}) => { - const url = new URL(path, root); - if ( - url.origin !== ORIGIN || - !url.pathname.startsWith(new URL(root).pathname) || - url.username || - url.password - ) { - throw new Error("PostHog API URL is outside the configured project"); - } - // No blind retries: a timed-out create may already have succeeded. - const response = await fetchImpl(url.href, { - ...init, - redirect: "error", - signal: AbortSignal.timeout(30_000), - headers: { - ...(init.body ? { "Content-Type": "application/json" } : {}), - Authorization: `Bearer ${apiKey}`, - }, - }); - if (!response.ok) { - await response.body?.cancel(); - throw new Error(`PostHog API request failed (${response.status})`); - } - const reader = response.body?.getReader(); - if (!reader) throw new Error("PostHog API returned no response"); - let size = 0; - const chunks = []; - try { - while (true) { - const result = await reader.read(); - if (result.done) break; - size += result.value.byteLength; - if (size > MAX_RESPONSE_BYTES) throw new Error("PostHog API response exceeded limit"); - chunks.push(result.value); - } - } catch (error) { - await reader.cancel().catch(() => undefined); - throw error; - } finally { - reader.releaseLock(); - } - const bytes = new Uint8Array(size); - let offset = 0; - for (const chunk of chunks) { - bytes.set(chunk, offset); - offset += chunk.byteLength; - } - try { - return JSON.parse(new TextDecoder().decode(bytes)); - } catch { - throw new Error("PostHog API returned invalid JSON"); - } - }; -} diff --git a/scripts/posthog-api.test.mjs b/scripts/posthog-api.test.mjs deleted file mode 100644 index 9601737..0000000 --- a/scripts/posthog-api.test.mjs +++ /dev/null @@ -1,53 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; -import { createPosthogApi } from "./posthog-api.mjs"; - -describe("operator PostHog API", () => { - it("restricts pagination and credentials to the configured project", async () => { - const fetchImpl = vi.fn().mockResolvedValue(Response.json({ results: [] })); - const api = createPosthogApi({ apiKey: "synthetic", projectId: "123", fetchImpl }); - for (const path of [ - "https://elsewhere.invalid/api/projects/123/", - "https://eu.posthog.com/api/projects/456/", - "//elsewhere.invalid/", - "../456/", - ]) { - await expect(api(path)).rejects.toThrow("outside the configured project"); - } - expect(fetchImpl).not.toHaveBeenCalled(); - expect(await api("insights/?limit=100")).toEqual({ results: [] }); - expect(fetchImpl).toHaveBeenCalledWith( - "https://eu.posthog.com/api/projects/123/insights/?limit=100", - expect.objectContaining({ - redirect: "error", - signal: expect.any(AbortSignal), - headers: { Authorization: "Bearer synthetic" }, - }), - ); - }); - it("does not leak error bodies or retry ambiguous creates", async () => { - const fetchImpl = vi - .fn() - .mockResolvedValue(new Response("private upstream detail", { status: 503 })); - const api = createPosthogApi({ apiKey: "synthetic", projectId: "123", fetchImpl }); - await expect(api("dashboards/", { method: "POST", body: "{}" })).rejects.toThrow( - "PostHog API request failed (503)", - ); - expect(fetchImpl).toHaveBeenCalledOnce(); - }); - it("bounds streamed responses and cancels oversized bodies", async () => { - const cancel = vi.fn(); - const body = new ReadableStream({ - start(controller) { - controller.enqueue(new Uint8Array(2 * 1024 * 1024 + 1)); - }, - cancel, - }); - const api = createPosthogApi({ - apiKey: "synthetic", - projectId: "123", - fetchImpl: async () => new Response(body), - }); - await expect(api("query/", { method: "POST" })).rejects.toThrow("exceeded limit"); - expect(cancel).toHaveBeenCalledOnce(); - }); -}); diff --git a/scripts/posthog-dashboard-manifest.mjs b/scripts/posthog-dashboard-manifest.mjs deleted file mode 100644 index c3c573b..0000000 --- a/scripts/posthog-dashboard-manifest.mjs +++ /dev/null @@ -1,525 +0,0 @@ -import { providerUsageInsights, featureUsageInsights } from "./product-insights-hogql.mjs"; - -const event = (name, customName = name, math = "total") => ({ - kind: "EventsNode", - event: name, - name, - custom_name: customName, - math, -}); - -const trends = ({ series, display = "ActionsLineGraph", dateFrom = "-30d", interval = "day" }) => ({ - kind: "InsightVizNode", - source: { - kind: "TrendsQuery", - series, - version: 4, - interval, - dateRange: { date_from: dateFrom, explicitDate: false }, - properties: [], - trendsFilter: { - display, - showLegend: series.length > 1, - yAxisScaleType: "linear", - showValuesOnSeries: false, - smoothingIntervals: 1, - showPercentStackView: false, - aggregationAxisFormat: "numeric", - showAlertThresholdLines: false, - }, - breakdownFilter: { breakdown_type: "event" }, - filterTestAccounts: false, - }, -}); - -const hogql = (query) => ({ kind: "HogQLQuery", query }); - -const preparedInsight = (name, description, query, aliases = []) => ({ - name, - description, - query: hogql(query), - aliases, -}); - -// Product denominators never mix Product successes with Essential-only failures. -// These are participating installations/profiles, not accounts or all Scient users. -export const PRODUCT_POPULATION = - "properties.source = 'desktop' AND properties.consent_level IN ('product', 'diagnostic')"; -export const SCIENTIFIC_OUTCOME = - "event = 'scient.operation.completed' AND properties.operationKind IN ('pdf-export', 'source-import', 'compute-run', 'compute-artifact', 'latex-build', 'document-export')"; -const JOURNEYS = `cohort_events AS ( - SELECT distinct_id, event, timestamp, properties.productFirstSeenAt AS cohort_start - FROM events - WHERE ${PRODUCT_POPULATION} AND properties.productFirstSeenAt IS NOT NULL - AND timestamp >= now() - INTERVAL 180 DAY -), projects AS ( - SELECT distinct_id, - min(parseDateTimeBestEffort(cohort_start)) AS first_seen, - minIf(timestamp, event = 'project.opened') AS project_opened - FROM cohort_events - GROUP BY distinct_id - HAVING first_seen >= now() - INTERVAL 180 DAY -), providers AS ( - SELECT projects.distinct_id, projects.first_seen, projects.project_opened, - minIf(cohort_events.timestamp, cohort_events.event = 'provider.session.started' - AND cohort_events.timestamp >= projects.project_opened) AS provider_started - FROM projects INNER JOIN cohort_events ON projects.distinct_id = cohort_events.distinct_id - GROUP BY projects.distinct_id, projects.first_seen, projects.project_opened -), journeys AS ( - SELECT providers.distinct_id, providers.first_seen, providers.project_opened, providers.provider_started, - minIf(cohort_events.timestamp, cohort_events.event = 'provider.turn.completed' - AND cohort_events.timestamp >= providers.provider_started) AS turn_completed - FROM providers INNER JOIN cohort_events ON providers.distinct_id = cohort_events.distinct_id - GROUP BY providers.distinct_id, providers.first_seen, providers.project_opened, providers.provider_started -)`; -const ACTIVATED = - "project_opened >= first_seen AND provider_started >= project_opened AND turn_completed >= provider_started AND turn_completed <= first_seen + INTERVAL 7 DAY"; - -export const dashboards = [ - { - key: "pipeline", - name: "90 — Scient analytics pipeline and data quality", - aliases: ["00 — Scient analytics: current coverage"], - phase: "current", - requiredEvents: ["server.boot.heartbeat"], - description: - "Current first-party event coverage. D1 remains the delivery source of truth; use bun run analytics:report for queue reconciliation.", - insights: [ - { - name: "Current event volume", - description: "Daily volume for the event families that currently exist in this project.", - query: trends({ - series: [ - event("page_viewed", "Website page views"), - event("server.boot.heartbeat", "Desktop server heartbeats"), - event("outbound_link_clicked", "Website outbound links"), - event("provider.sessions.stopped_all", "Desktop shutdowns"), - event("download_clicked", "Website downloads"), - ], - }), - }, - { - name: "Active desktop installations", - description: - "Unique pseudonymous installations emitting server heartbeats. This is not an account or named-user count.", - query: trends({ - series: [event("server.boot.heartbeat", "Active installations", "dau")], - interval: "day", - }), - }, - { - name: "Observed website identities", - aliases: ["Active website visitors"], - description: - "Consent-dependent identity count, not a visitor count: without persistent consent each event may have its own identity.", - query: trends({ - series: [event("page_viewed", "Observed website identities", "dau")], - interval: "day", - }), - }, - { - name: "Desktop heartbeat volume (30 days)", - description: "Total accepted desktop heartbeat events over the last 30 days.", - query: trends({ - series: [event("server.boot.heartbeat", "Heartbeats")], - display: "BoldNumber", - }), - }, - preparedInsight( - "Monthly event volume", - "Rolling 30-day event volume. Billing limits are configured separately; this is not a cost estimate or calendar-month invoice count.", - "SELECT count() AS events_last_30_days FROM events WHERE timestamp >= now() - INTERVAL 30 DAY", - ["Monthly event volume and free-tier budget"], - ), - { - ...preparedInsight( - "Observed provider lifecycle outcomes", - "Observed starts and terminal outcomes, not clicks or a current-state fleet. Product/Diagnostic population only; failures at Essential consent appear in the failure view.", - `SELECT properties.appVersion AS app_version, properties.provider AS provider, - properties.action AS action, properties.runtimeSource AS runtime_source, event, - uniqExact(properties.event_id) AS observations -FROM events WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND event IN ('provider.lifecycle.started', 'provider.lifecycle.completed', 'provider.lifecycle.failed', 'provider.lifecycle.cancelled') -GROUP BY app_version, provider, action, runtime_source, event ORDER BY observations DESC`, - ), - requiredEvents: ["provider.lifecycle.started"], - }, - { - ...preparedInsight( - "Observed provider readiness transitions", - "Reported changes only. Missing or offline installations are not assumed ready, and no observations is not a healthy zero.", - `SELECT properties.provider AS provider, properties.from AS previous_state, - properties.to AS next_state, uniqExact(properties.event_id) AS transitions -FROM events WHERE ${PRODUCT_POPULATION} AND event = 'provider.readiness.changed' - AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY provider, previous_state, next_state ORDER BY transitions DESC`, - ), - requiredEvents: ["provider.readiness.changed"], - }, - { - ...preparedInsight( - "Observed app health outcomes", - "Server startup and renderer termination observations by release and consent. Desktop-update and migration coverage is not implied; counts are not a success rate.", - `SELECT properties.appVersion AS app_version, properties.component AS component, - properties.operation AS operation, properties.outcome AS outcome, properties.consent_level AS consent, - uniqExact(properties.event_id) AS observations -FROM events WHERE properties.source = 'desktop' AND event = 'app.health' - AND properties.outcome IN ('completed', 'failed', 'abnormal') AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY app_version, component, operation, outcome, consent ORDER BY observations DESC`, - ["Observed server health outcomes"], - ), - requiredEvents: ["app.health"], - }, - ], - }, - { - key: "executive", - name: "00 — Scient executive product health", - phase: "planned", - requiredEvents: ["app.session.started", "project.opened", "provider.turn.completed"], - description: - "Activation, meaningful weekly use, retention, and reliability guardrails in one decision view.", - plannedInsights: [ - "Weekly Meaningful Active Installations", - "Seven-day activation rate", - "Week-one and week-four retained activation", - "Successful assistant-turn rate", - ], - insights: [ - preparedInsight( - "Weekly Meaningful Active Installations", - "Twelve complete calendar weeks. An installation qualifies after three completed turns across two sessions, or one completed scientific operation; the current partial week is excluded.", - `SELECT week, countIf(turns >= 3 AND sessions >= 2 OR scientific_operations >= 1) AS meaningful_installations -FROM ( - SELECT toStartOfWeek(timestamp) AS week, distinct_id, - uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns, - uniqIf(properties.$session_id, event = 'provider.turn.completed') AS sessions, - uniqExactIf(properties.event_id, ${SCIENTIFIC_OUTCOME}) AS scientific_operations - FROM events - WHERE ${PRODUCT_POPULATION} AND timestamp >= toStartOfWeek(now()) - INTERVAL 12 WEEK - AND timestamp < toStartOfWeek(now()) - GROUP BY week, distinct_id -) -GROUP BY week ORDER BY week`, - ), - preparedInsight( - "Successful assistant-turn rate", - "Product-consenting completed turns divided by completed plus failed turns. Stops/cancellations are reported separately; Essential-only failures are excluded from this denominator.", - `SELECT toStartOfDay(timestamp) AS day, - uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS completed, - uniqExactIf(properties.event_id, event IN ('provider.turn.completed', 'provider.turn.failed')) AS terminal, - round(100 * completed / nullIf(terminal, 0), 1) AS success_percent -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND (event <> 'provider.turn.failed' OR properties.failureClass NOT IN ('cancelled', 'interrupted')) -GROUP BY day ORDER BY day`, - ), - preparedInsight( - "Activated installations", - "Ordered project → provider session → completed turn within seven days of first observed Product participation. Only complete seven-day windows and known cohort origins count; not install-to-activation conversion.", - `WITH ${JOURNEYS} -SELECT countIf(first_seen <= now() - INTERVAL 7 DAY) AS eligible_installations, - countIf(first_seen > now() - INTERVAL 7 DAY) AS immature_installations, - countIf(first_seen <= now() - INTERVAL 7 DAY AND ${ACTIVATED}) AS activated_installations -FROM journeys`, - ), - preparedInsight( - "Week-one and week-four retained activation", - "Activated Product cohorts qualifying for meaningful later-week use. Separate mature denominators exclude incomplete week-one/week-four windows; absent cohort history is unknown, not new.", - `WITH ${JOURNEYS}, activated AS ( - SELECT distinct_id, toStartOfWeek(greatest(project_opened, greatest(provider_started, turn_completed))) AS activation_week - FROM journeys - WHERE ${ACTIVATED} -), meaningful AS ( - SELECT distinct_id, toStartOfWeek(timestamp) AS week, - uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns, - uniqIf(properties.$session_id, event = 'provider.turn.completed') AS sessions, - uniqExactIf(properties.event_id, ${SCIENTIFIC_OUTCOME}) AS scientific_operations - FROM events - WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 180 DAY - GROUP BY distinct_id, week - HAVING turns >= 3 AND sessions >= 2 OR scientific_operations >= 1 -) -SELECT activation_week, - uniqExactIf(activated.distinct_id, activation_week + INTERVAL 2 WEEK <= toStartOfWeek(now())) AS eligible_week_one, - uniqExactIf(activated.distinct_id, activation_week + INTERVAL 5 WEEK <= toStartOfWeek(now())) AS eligible_week_four, - uniqExactIf(activated.distinct_id, activation_week + INTERVAL 2 WEEK <= toStartOfWeek(now()) AND meaningful.week = activation_week + INTERVAL 1 WEEK) AS retained_week_one, - uniqExactIf(activated.distinct_id, activation_week + INTERVAL 5 WEEK <= toStartOfWeek(now()) AND meaningful.week = activation_week + INTERVAL 4 WEEK) AS retained_week_four -FROM activated -LEFT JOIN meaningful ON activated.distinct_id = meaningful.distinct_id -GROUP BY activation_week ORDER BY activation_week`, - ), - ], - }, - { - key: "activation", - name: "01 — Scient acquisition and activation", - phase: "planned", - requiredEvents: [ - "app.session.started", - "project.added", - "project.opened", - "provider.session.started", - "provider.turn.completed", - ], - description: - "First-observed Product participation and seven-day activation through an ordered project/provider/answer journey. Not install conversion.", - plannedInsights: [ - "App start → project → provider session → successful turn funnel", - "Median time-to-activation bucket", - "Activation by build channel", - ], - insights: [ - preparedInsight( - "Activation stage reach", - "Product-participating installations observed at each stage. These independent reach counts are not an ordered funnel or conversion rate.", - `SELECT event, uniqExact(distinct_id) AS installations -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND event IN ('app.session.started', 'project.opened', 'provider.session.started', 'provider.turn.completed') -GROUP BY event ORDER BY installations DESC`, - ), - preparedInsight( - "Answer-producing installations by build channel", - "Product-participating installations with a completed turn, grouped by its build channel. This is not first-use activation.", - `SELECT properties.buildChannel AS build_channel, uniqExact(distinct_id) AS installations -FROM events -WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.completed' AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY build_channel ORDER BY installations DESC`, - ["First-answer activation by build channel"], - ), - ], - }, - { - key: "engagement", - name: "02 — Scient engagement and retention", - phase: "planned", - requiredEvents: ["app.session.started", "provider.turn.completed", "surface.opened"], - description: - "Weekly meaningful active installations, depth of core use, and week-one/week-four return behavior.", - plannedInsights: [ - "Weekly Meaningful Active Installations", - "Meaningful sessions per installation", - "Week-one and week-four retention cohorts", - "Core surface use by returning installations", - ], - insights: [ - preparedInsight( - "Completed turns per active installation", - "Distribution of completed provider turns per pseudonymous installation over the last 30 days.", - `SELECT turns, count() AS installations FROM ( - SELECT distinct_id, uniqExactIf(properties.event_id, event = 'provider.turn.completed') AS turns - FROM events WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY GROUP BY distinct_id -) GROUP BY turns ORDER BY turns`, - ), - preparedInsight( - "Returning active installations by week", - "Product-participating installations with completed turns in consecutive complete calendar weeks. Not the stricter meaningful-use retention KPI.", - `WITH weekly AS ( - SELECT distinct_id, toStartOfWeek(timestamp) AS week - FROM events WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.completed' - AND timestamp >= toStartOfWeek(now()) - INTERVAL 13 WEEK AND timestamp < toStartOfWeek(now()) - GROUP BY distinct_id, week -) -SELECT current.week, uniqExact(current.distinct_id) AS returning_installations -FROM weekly AS current -INNER JOIN weekly AS previous ON current.distinct_id = previous.distinct_id AND previous.week = current.week - INTERVAL 1 WEEK -GROUP BY current.week ORDER BY current.week`, - ), - ], - }, - { - key: "providers", - name: "03 — Scient providers and agent runtime", - phase: "planned", - requiredEvents: [ - "provider.turn.usage", - "provider.session.started", - "provider.session.recovered", - "provider.turn.completed", - "provider.turn.failed", - ], - description: "Provider connection, recovery, success, failure class, and latency guardrails.", - plannedInsights: [ - "Provider session starts and recoveries", - "Successful and failed turns by provider", - "Turn duration buckets by provider", - "Runtime-mode distribution", - ], - insights: [ - ...providerUsageInsights, - preparedInsight( - "Provider terminal outcomes", - "Completed, failed and stopped turns for the same Product-consenting population, by provider. Stopped turns are not failures.", - `SELECT properties.provider AS provider, event, uniqExact(properties.event_id) AS turns -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND event IN ('provider.turn.completed', 'provider.turn.failed', 'provider.turn.stopped') -GROUP BY provider, event ORDER BY provider, event`, - ), - preparedInsight( - "Model selection", - "Attempted turns by maintained public model key; private custom model names collapse to other.", - `SELECT properties.modelKey AS model_key, uniqExact(properties.event_id) AS turns -FROM events -WHERE ${PRODUCT_POPULATION} AND event = 'provider.turn.sent' AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY model_key ORDER BY turns DESC`, - ), - preparedInsight( - "Provider failure classes", - "Bounded provider failure classes without raw messages or stack traces.", - `SELECT properties.provider AS provider, properties.failureClass AS failure_class, uniqExact(properties.event_id) AS failures -FROM events -WHERE properties.source = 'desktop' AND event = 'provider.turn.failed' AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY provider, failure_class ORDER BY failures DESC`, - ), - ], - }, - { - key: "features", - name: "04 — Scient feature adoption", - phase: "planned", - requiredEvents: [ - "surface.opened", - "panel.viewed", - "settings.viewed", - "usage.viewed", - "project.initialization.completed", - "thread.fork.completed", - "voice.transcription.completed", - ], - description: - "Repeated adoption of projects, surfaces, forking, voice, settings, and later Scient-owned capabilities.", - plannedInsights: [ - "Feature adoption by active installation", - "Repeated feature use", - "Project initialization outcomes", - "Fork and voice completion", - ], - insights: [ - ...featureUsageInsights, - preparedInsight( - "Feature completion by installation", - "Unique installations completing bounded Scient feature outcomes.", - `SELECT event, uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS completions -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND event IN ('project.initialization.completed', 'thread.fork.completed', 'thread.revert.completed', 'voice.transcription.completed') -GROUP BY event ORDER BY installations DESC`, - ), - preparedInsight( - "Selected surfaces opened", - "Once-per-session style surface signals; this is deliberately not clickstream tracking.", - `SELECT properties.surface AS surface, uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS opens -FROM events -WHERE ${PRODUCT_POPULATION} AND event = 'surface.opened' AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY surface ORDER BY installations DESC`, - ), - preparedInsight( - "Measured settings choices", - "Bounded direction, theme, and notification choices only.", - `SELECT properties.setting AS setting, properties.value AS value, uniqExact(properties.event_id) AS changes -FROM events -WHERE ${PRODUCT_POPULATION} AND event = 'setting.changed' AND timestamp >= now() - INTERVAL 30 DAY -GROUP BY setting, value ORDER BY setting, changes DESC`, - ), - ], - }, - { - key: "reliability", - name: "05 — Scient reliability and release health", - phase: "planned", - requiredEvents: [ - "provider.turn.completed", - "provider.turn.failed", - "project.add.failed", - "project.initialization.failed", - "thread.fork.failed", - "voice.transcription.failed", - ], - description: "Release, provider, project, recovery, and voice reliability guardrails.", - plannedInsights: [ - "Successful assistant-turn rate by version", - "Bounded failure classes by version", - "Project, fork, revert, and voice success rates", - "Release regression comparison", - ], - insights: [ - preparedInsight( - "Failures by class and release", - "Bounded failures grouped by event, class, and application version.", - `SELECT properties.appVersion AS app_version, event, properties.failureClass AS failure_class, uniqExact(properties.event_id) AS failures -FROM events -WHERE properties.source = 'desktop' AND timestamp >= now() - INTERVAL 30 DAY - AND (event IN ('provider.turn.failed', 'project.add.failed', 'project.initialization.failed', 'thread.fork.failed', 'thread.revert.failed', 'voice.transcription.failed', 'provider.lifecycle.failed', 'scient.operation.failed') - OR (event = 'app.health' AND properties.outcome IN ('failed', 'abnormal'))) -GROUP BY app_version, event, failure_class ORDER BY failures DESC`, - ), - preparedInsight( - "Duration bucket distribution", - "Coarse terminal-outcome latency histograms, not exact percentiles. Starts are excluded; missing duration remains unknown.", - `SELECT event, properties.component AS component, properties.operation AS operation, - properties.operationKind AS operation_kind, properties.outcome AS health_outcome, - properties.durationBucket AS duration_bucket, uniqExact(properties.event_id) AS outcomes -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND (event IN ('provider.turn.completed', 'provider.turn.failed', 'provider.turn.stopped', - 'provider.lifecycle.completed', 'provider.lifecycle.failed', 'provider.lifecycle.cancelled', - 'scient.operation.completed', 'scient.operation.failed', 'scient.operation.cancelled', 'scient.operation.skipped', 'voice.transcription.completed') - OR (event = 'app.health' AND properties.outcome IN ('completed', 'failed', 'abnormal'))) -GROUP BY event, component, operation, operation_kind, health_outcome, duration_bucket ORDER BY event, outcomes DESC`, - ), - ], - }, - { - key: "scientific-workflows", - name: "06 — Scient scientific workflows", - phase: "planned", - requiredEvents: [ - "scient.operation.started", - "scient.operation.completed", - "scient.operation.failed", - ], - description: - "Prepared for actual compute-run, latex-build, agent pdf-export and per-item source-import outcomes. Import skips are separate from saved sources; technical completion does not prove scientific review.", - plannedInsights: [ - "Completed scientific operations", - "Reviewed outcome rate", - "Failure class by registered operation", - "Repeat workflow use", - ], - insights: [ - preparedInsight( - "Scientific operation outcomes", - "Technical completions, failures, cancellations and no-op skips for qualified producers in one consent population. Source imports count individual attempts, not batches; no reviewed-outcome rate is inferred.", - `SELECT properties.operationKind AS operation_kind, event, uniqExact(properties.event_id) AS outcomes -FROM events -WHERE ${PRODUCT_POPULATION} AND timestamp >= now() - INTERVAL 30 DAY - AND event IN ('scient.operation.completed', 'scient.operation.failed', 'scient.operation.cancelled', 'scient.operation.skipped') -GROUP BY operation_kind, event ORDER BY outcomes DESC`, - ), - ], - }, - { - key: "cloud-mobile", - name: "07 — Scient cloud and mobile", - phase: "planned", - requiredEvents: ["cloud.session.started", "mobile.session.started"], - description: "Future selected-user cloud and mobile activation, health, and meaningful use.", - plannedInsights: [ - "Selected-user cloud activation and reliability", - "Mobile meaningful use and retention", - "Cross-surface continuation after authenticated linking", - ], - insights: [ - preparedInsight( - "Cloud and mobile session health", - "Session starts by future governed client surface after those event contracts ship.", - `SELECT event, uniqExact(distinct_id) AS active_identities, count() AS sessions -FROM events -WHERE event IN ('cloud.session.started', 'mobile.session.started') -GROUP BY event ORDER BY sessions DESC`, - ), - ], - }, -]; diff --git a/scripts/posthog-dashboard-manifest.test.mjs b/scripts/posthog-dashboard-manifest.test.mjs deleted file mode 100644 index bd059aa..0000000 --- a/scripts/posthog-dashboard-manifest.test.mjs +++ /dev/null @@ -1,342 +0,0 @@ -import { DatabaseSync } from "node:sqlite"; -import { expect, it } from "vitest"; -import { - dashboards, - PRODUCT_POPULATION, - SCIENTIFIC_OUTCOME, -} from "./posthog-dashboard-manifest.mjs"; - -const insights = dashboards.flatMap((dashboard) => dashboard.insights ?? []); -const query = (name) => insights.find((insight) => insight.name === name).query.query; - -it("keeps reliability ratios within one consent population and stops separate", () => { - expect(query("Successful assistant-turn rate")).toContain(PRODUCT_POPULATION); - expect(query("Successful assistant-turn rate")).toContain("nullIf(terminal, 0)"); - expect(query("Successful assistant-turn rate")).not.toContain("provider.turn.stopped"); - expect(query("Provider terminal outcomes")).toContain("provider.turn.stopped"); - expect(query("Provider terminal outcomes")).toContain(PRODUCT_POPULATION); -}); - -it("uses durable known Product cohort origins and complete later-week denominators", () => { - const activation = query("Activated installations"); - expect(activation).toContain("properties.productFirstSeenAt IS NOT NULL"); - expect(activation).not.toContain("minIf(timestamp, event = 'app.session.started')"); - expect(activation).toContain("eligible_installations"); - expect(activation).toContain("immature_installations"); - expect(activation).toContain("provider_started >= project_opened"); - const retention = query("Week-one and week-four retained activation"); - expect(retention).toContain("eligible_week_one"); - expect(retention).toContain("eligible_week_four"); - expect(retention).toContain("INTERVAL 2 WEEK <= toStartOfWeek(now())"); - expect(retention).toContain("INTERVAL 5 WEEK <= toStartOfWeek(now())"); -}); - -it("does not count opening a browser/file as a scientific outcome", () => { - expect(query("Weekly Meaningful Active Installations")).toContain(SCIENTIFIC_OUTCOME); - expect(SCIENTIFIC_OUTCOME).not.toContain("file-preview"); - expect(SCIENTIFIC_OUTCOME).not.toContain("browser"); - expect( - insights.some((insight) => insight.name === "First-answer activation by build channel"), - ).toBe(false); -}); - -// Execute the actual query text against synthetic records. This small adapter -// covers only the date/property/aggregate functions used below; it does not claim -// to qualify PostHog's parser, timezone configuration, permissions or live data. -const DAY = 86_400; -const WEEK = 7 * DAY; -const NOW = Date.parse("2026-08-31T12:00:00Z") / 1000; -const startOfWeek = (timestamp) => { - const date = new Date(timestamp * 1000); - return ( - Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate() - date.getUTCDay()) / 1000 - ); -}; -const runQuery = (name, rows) => { - const database = new DatabaseSync(":memory:"); - try { - database.exec( - "CREATE TABLE events (distinct_id TEXT, event TEXT, timestamp INTEGER, properties TEXT)", - ); - const insert = database.prepare("INSERT INTO events VALUES (?, ?, ?, ?)"); - rows.forEach((row, index) => - insert.run( - row.id ?? "desktop-one", - row.event, - row.timestamp ?? NOW - DAY, - JSON.stringify({ - source: "desktop", - consent_level: "product", - contractRevision: "7", - buildChannel: "stable", - event_id: `event-${index}`, - ...row.properties, - }), - ), - ); - database.function("now", () => NOW); - database.function("toStartOfDay", (timestamp) => Math.floor(timestamp / DAY) * DAY); - database.function("toDate", (timestamp) => - new Date(timestamp * 1000).toISOString().slice(0, 10), - ); - database.function("parseDateTimeBestEffort", (value) => Date.parse(value) / 1000); - database.function("toStartOfWeek", startOfWeek); - database.function("greatest", (left, right) => Math.max(left, right)); - database.aggregate("countIf", { - start: 0, - step: (count, condition) => count + (condition ? 1 : 0), - }); - database.aggregate("minIf", { - start: () => null, - step: (minimum, value, condition) => - condition ? (minimum === null ? value : Math.min(minimum, value)) : minimum, - result: (minimum) => minimum ?? 0, - }); - database.aggregate("uniqExact", { - start: () => new Set(), - step: (values, value) => { - if (value !== null) values.add(value); - return values; - }, - result: (values) => values.size, - }); - for (const aggregate of ["uniqExactIf", "uniqIf"]) { - database.aggregate(aggregate, { - start: () => new Set(), - step: (values, value, condition) => { - if (condition && value !== null) values.add(value); - return values; - }, - result: (values) => values.size, - }); - } - const sql = query(name) - .replace( - /properties\.([A-Za-z_$][\w$]*)/g, - (_, key) => `json_extract(properties, '$."${key}"')`, - ) - .replace(/INTERVAL (\d+) (DAY|WEEK)/g, (_, amount, unit) => - String(Number(amount) * (unit === "DAY" ? DAY : WEEK)), - ); - return database - .prepare(sql) - .all() - .map((row) => ({ ...row })); - } finally { - database.close(); - } -}; - -it("deduplicates named-agent prompts and installs while excluding dev traffic and terminal outcomes", () => { - const records = [ - { event: "provider.turn.sent", properties: { provider: "scient", event_id: "same" } }, - { event: "provider.turn.sent", properties: { provider: "scient", event_id: "same" } }, - { - event: "provider.turn.sent", - id: "two", - properties: { provider: "scient", buildChannel: "beta" }, - }, - { - event: "provider.turn.sent", - properties: { provider: "scient", buildChannel: "development" }, - }, - { event: "provider.turn.sent", properties: { provider: "scient", contractRevision: "2" } }, - { event: "provider.turn.sent", properties: { provider: "scient" }, timestamp: NOW }, - { event: "provider.turn.completed", properties: { provider: "scient" } }, - { event: "provider.turn.usage", properties: { provider: "scient" } }, - { - event: "provider.lifecycle.completed", - properties: { provider: "scient", action: "install", event_id: "install" }, - }, - { - event: "provider.lifecycle.completed", - properties: { provider: "scient", action: "install", event_id: "install" }, - }, - { event: "provider.lifecycle.completed", properties: { provider: "scient", action: "repair" } }, - ]; - expect(runQuery("Daily submitted prompts by provider", records)).toEqual([ - { day: "2026-08-30", provider: "scient", installations: 2, submitted_prompts: 2 }, - ]); - expect(runQuery("Successful provider install operations", records)).toEqual([ - { provider: "scient", installations: 1, install_completions: 1 }, - ]); -}); - -it("finds a valid ordered activation even after earlier out-of-order attempts", () => { - const firstSeen = NOW - 14 * DAY; - const properties = { productFirstSeenAt: new Date(firstSeen * 1000).toISOString() }; - const records = [ - ["provider.session.started", 0], - ["provider.turn.completed", 60], - ["project.opened", 100], - ["provider.session.started", 120], - ["provider.turn.completed", 150], - ].map(([event, offset]) => ({ event, timestamp: firstSeen + offset, properties })); - const unordered = records.slice(0, 3).map((row) => ({ ...row, id: "not-activated" })); - expect(runQuery("Activated installations", [...records, ...unordered])).toEqual([ - { eligible_installations: 2, immature_installations: 0, activated_installations: 1 }, - ]); -}); - -it("does not count website or Essential-only identities in desktop engagement", () => { - const website = Array.from({ length: 100 }, (_, index) => ({ - id: `website-${index}`, - event: "page_viewed", - properties: { source: "website" }, - })); - expect( - runQuery("Completed turns per active installation", [ - ...website, - { - id: "essential-only", - event: "provider.turn.failed", - properties: { consent_level: "essential" }, - }, - { event: "provider.turn.completed" }, - ]), - ).toEqual([{ turns: 1, installations: 1 }]); -}); - -it("includes the entire oldest calendar week and excludes the current partial week", () => { - const oldest = startOfWeek(NOW) - 12 * WEEK; - const turns = [0, DAY, 2 * DAY].map((offset, index) => ({ - event: "provider.turn.completed", - timestamp: oldest + offset, - properties: { $session_id: `session-${index % 2}`, event_id: `turn-${index}` }, - })); - expect( - runQuery("Weekly Meaningful Active Installations", [ - ...turns, - turns[0], // Duplicate delivery must not add a completed turn. - { - event: "scient.operation.completed", - timestamp: startOfWeek(NOW) + 1, - properties: { operationKind: "pdf-export" }, - }, - { - event: "scient.operation.completed", - timestamp: oldest - 1, - properties: { operationKind: "compute-run" }, - }, - ]), - ).toEqual([{ week: oldest, meaningful_installations: 1 }]); -}); - -it("counts terminal duration observations without counting their start event", () => { - const properties = { component: "server", operation: "startup", durationBucket: "lt_1s" }; - const result = runQuery("Duration bucket distribution", [ - { event: "app.health", properties: { ...properties, outcome: "started" } }, - { event: "app.health", properties: { ...properties, outcome: "completed" } }, - { - event: "scient.operation.started", - properties: { operationKind: "latex-build", durationBucket: "unknown" }, - }, - ]); - expect(result).toHaveLength(1); - expect(result[0]).toMatchObject({ - event: "app.health", - health_outcome: "completed", - outcomes: 1, - }); -}); - -it("reports import skips separately without counting them as meaningful work", () => { - const week = startOfWeek(NOW) - WEEK; - const records = [ - { - id: "skipped-only", - event: "scient.operation.skipped", - timestamp: week + DAY, - properties: { operationKind: "source-import", durationBucket: "under-1s" }, - }, - { - id: "imported", - event: "scient.operation.completed", - timestamp: week + DAY, - properties: { operationKind: "source-import", durationBucket: "under-1s" }, - }, - { - id: "essential-only", - event: "scient.operation.failed", - timestamp: week + DAY, - properties: { operationKind: "source-import", consent_level: "essential" }, - }, - ]; - expect(runQuery("Weekly Meaningful Active Installations", records)).toEqual([ - { week, meaningful_installations: 1 }, - ]); - expect(runQuery("Scientific operation outcomes", records)).toEqual( - expect.arrayContaining([ - { operation_kind: "source-import", event: "scient.operation.skipped", outcomes: 1 }, - { operation_kind: "source-import", event: "scient.operation.completed", outcomes: 1 }, - ]), - ); - expect(runQuery("Scientific operation outcomes", records)).toHaveLength(2); - expect( - runQuery("Duration bucket distribution", records) - .map((row) => row.event) - .toSorted(), - ).toEqual(["scient.operation.completed", "scient.operation.skipped"]); -}); - -it("uses separate mature week-one and week-four retention denominators", () => { - const matureWeek = startOfWeek(NOW) - 8 * WEEK; - const recentWeek = startOfWeek(NOW) - 2 * WEEK; - const activation = (id, week) => - ["project.opened", "provider.session.started", "provider.turn.completed"].map( - (event, index) => ({ - id, - event, - timestamp: week + DAY + index, - properties: { - productFirstSeenAt: new Date((week + DAY) * 1000).toISOString(), - $session_id: "activation", - }, - }), - ); - const returns = [0, DAY, 2 * DAY].map((offset, index) => ({ - id: "mature", - event: "provider.turn.completed", - timestamp: matureWeek + WEEK + offset, - properties: { - productFirstSeenAt: new Date((matureWeek + DAY) * 1000).toISOString(), - $session_id: `return-${index % 2}`, - }, - })); - expect( - runQuery("Week-one and week-four retained activation", [ - ...activation("mature", matureWeek), - ...activation("recent", recentWeek), - ...returns, - { - id: "mature", - event: "scient.operation.completed", - timestamp: matureWeek + 4 * WEEK + DAY, - properties: { operationKind: "latex-build" }, - }, - ]), - ).toEqual([ - { - activation_week: matureWeek, - eligible_week_one: 1, - eligible_week_four: 1, - retained_week_one: 1, - retained_week_four: 1, - }, - { - activation_week: recentWeek, - eligible_week_one: 1, - eligible_week_four: 0, - retained_week_one: 0, - retained_week_four: 0, - }, - ]); -}); - -it("keeps managed insight aliases unique so corrected names update rather than duplicate", () => { - const names = insights.flatMap((insight) => [insight.name, ...(insight.aliases ?? [])]); - expect(new Set(names).size).toBe(names.length); - expect( - insights.find((insight) => insight.name === "Observed website identities").aliases, - ).toContain("Active website visitors"); -}); diff --git a/scripts/product-insights-hogql.mjs b/scripts/product-insights-hogql.mjs deleted file mode 100644 index 5b13beb..0000000 --- a/scripts/product-insights-hogql.mjs +++ /dev/null @@ -1,91 +0,0 @@ -// Same Product population and complete-day window as product-insights.mjs. -const population = `properties.source = 'desktop' AND properties.consent_level IN ('product', 'diagnostic') -AND properties.contractRevision IN ('3', '4', '5', '6', '7', '8', '9', '10') -AND properties.buildChannel IN ('stable', 'beta', 'nightly') -AND timestamp >= toStartOfDay(now()) - INTERVAL 30 DAY AND timestamp < toStartOfDay(now())`; -const insight = (name, description, query) => ({ - name, - description, - query: { kind: "HogQLQuery", query }, - aliases: [], -}); - -export const providerUsageInsights = [ - insight( - "Daily submitted prompts by provider", - "Accepted run admissions and distinct accepted steering messages per day and provider, including queued work. Progress, queued-to-steer promotions, retries, restart continuations and legacy queue hydration do not add submissions. Older releases with missing submissions cannot be reconstructed. Installations are profiles, not people.", - `SELECT toDate(timestamp) AS day, properties.provider AS provider, -uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS submitted_prompts -FROM events WHERE ${population} AND event = 'provider.turn.sent' -GROUP BY day, provider ORDER BY day, provider`, - ), - insight( - "Successful provider install operations", - "Observed successful install operations and participating installation profiles by provider. Repairs and updates are separate; this is not the latest installed inventory or a count of people.", - `SELECT properties.provider AS provider, -uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS install_completions -FROM events WHERE ${population} AND event = 'provider.lifecycle.completed' AND properties.action = 'install' -GROUP BY provider ORDER BY install_completions DESC`, - ), - insight( - "Provider and model usage", - "Deduplicated live terminal usage reports, not model-picker clicks. Missing models are unknown; private and mixed-model labels use other. Counts include failed and stopped turns.", - `SELECT properties.provider AS provider, properties.modelKey AS model, -uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS reported_turns -FROM events WHERE ${population} AND event = 'provider.turn.usage' -GROUP BY provider, model ORDER BY reported_turns DESC`, - ), - insight( - "Reported tokens and coverage", - "Main-agent reported counts only. Cache and reasoning subsets are not added again. Missing counts are not zero; complete/partial/unavailable turns show coverage. Not all-provider billing or subagent totals.", - `SELECT provider, -count() AS reported_turns, countIf(status = 'complete') AS complete_turns, -countIf(status = 'partial') AS partial_turns, countIf(status = 'unavailable') AS unavailable_turns, -count(input_tokens) AS input_reporting_turns, count(output_tokens) AS output_reporting_turns, -if(count(input_tokens) = 0, NULL, sum(input_tokens)) AS reported_input_tokens, -if(count(output_tokens) = 0, NULL, sum(output_tokens)) AS reported_output_tokens -FROM (SELECT properties.event_id AS id, any(properties.provider) AS provider, -any(properties.usageStatus) AS status, any(toFloat(properties.inputTokens)) AS input_tokens, -any(toFloat(properties.outputTokens)) AS output_tokens FROM events WHERE ${population} -AND event = 'provider.turn.usage' GROUP BY id) -GROUP BY provider ORDER BY reported_turns DESC`, - ), - insight( - "Latest observed provider installations", - "Latest explicit installed state per pseudonymous installation and provider. Bundled-but-missing providers and historical readiness are not counted as installed.", - `SELECT provider, countIf(installed = true) AS installations -FROM ( - SELECT distinct_id, properties.provider AS provider, - argMax(if(event = 'provider.installation.observed', properties.installed, properties.toInstalled), - tuple(timestamp, properties.event_id)) AS installed - FROM events WHERE ${population} AND properties.contractRevision IN ('4', '5', '6', '7', '8', '9', '10') - AND event IN ('provider.installation.observed', 'provider.installation.changed') - GROUP BY distinct_id, provider -) -GROUP BY provider HAVING installations > 0 ORDER BY installations DESC`, - ), -]; - -export const featureUsageInsights = [ - insight( - "Panel adoption and repeat days", - "Observed visible category entries, including a visible restored panel. Background tabs are excluded. Repeat means at least two distinct days, not runtime sessions; no eligibility-adjusted adoption rate is claimed.", - `SELECT category, count() AS installations, sum(observations) AS observations, countIf(days >= 2) AS repeat_installations -FROM (SELECT distinct_id, properties.category AS category, uniqExact(properties.event_id) AS observations, -uniqExact(toDate(timestamp)) AS days FROM events WHERE ${population} AND event = 'panel.viewed' -GROUP BY distinct_id, category) GROUP BY category ORDER BY installations DESC`, - ), - insight( - "Settings sections visited", - "Visible section entries, not preference changes or evidence of preference. Routes and search text are never collected.", - `SELECT properties.section AS section, uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS observations -FROM events WHERE ${population} AND event = 'settings.viewed' GROUP BY section ORDER BY installations DESC`, - ), - insight( - "Usage tools viewed", - "Visible metric/range/breakdown combinations. Counts reflect view changes, not imported transcript usage or monetary spend.", - `SELECT properties.metric AS metric, properties.window AS range_days, properties.breakdown AS breakdown, -uniqExact(distinct_id) AS installations, uniqExact(properties.event_id) AS observations -FROM events WHERE ${population} AND event = 'usage.viewed' GROUP BY metric, range_days, breakdown ORDER BY observations DESC`, - ), -]; diff --git a/scripts/reconcile-analytics-pipeline.mjs b/scripts/reconcile-analytics-pipeline.mjs deleted file mode 100644 index bf52812..0000000 --- a/scripts/reconcile-analytics-pipeline.mjs +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env node - -import { execFileSync, spawnSync } from "node:child_process"; -import { reconciliationQueries, comparePipeline } from "./analytics-reconciliation.mjs"; -import { createPosthogApi } from "./posthog-api.mjs"; - -const PROJECT_ID = "228610"; -const KEYCHAIN_SERVICE = "scient-posthog-personal-api-key"; - -function fail(message) { - console.error(message); - process.exit(1); -} - -function personalApiKey() { - if (process.env.POSTHOG_PERSONAL_API_KEY) return process.env.POSTHOG_PERSONAL_API_KEY; - if (process.platform !== "darwin") return undefined; - try { - return execFileSync("security", ["find-generic-password", "-w", "-s", KEYCHAIN_SERVICE], { - encoding: "utf8", - stdio: ["ignore", "pipe", "ignore"], - }).trim(); - } catch { - return undefined; - } -} - -// Exclude the newest hour, where accepted captures may still be processing. -// A settled window is a reporting convention, not proof of completed deletion. -const until = process.env.ANALYTICS_RECONCILE_TO ?? new Date(Date.now() - 3600000).toISOString(); -const queries = reconciliationQueries({ - source: process.env.ANALYTICS_RECONCILE_SOURCE ?? "desktop", - from: - process.env.ANALYTICS_RECONCILE_FROM ?? - new Date(Date.parse(until) - 7 * 86400000).toISOString(), - to: until, -}); -function queryD1(query) { - const d1 = spawnSync( - "wrangler", - ["d1", "execute", "scientfactory-downloads", "--remote", "--json", "--command", query], - { encoding: "utf8", timeout: 60_000, maxBuffer: 2 * 1024 * 1024 }, - ); - if (d1.error) fail("Unable to run the bounded D1 query"); - if (d1.status !== 0) fail("D1 query failed; verify operator access and database availability"); - - let d1Body; - try { - d1Body = JSON.parse(d1.stdout); - } catch { - fail("D1 returned an unreadable reconciliation response"); - } - const d1Rows = d1Body?.[0]?.results; - if (!Array.isArray(d1Rows)) fail("D1 reconciliation response has no result rows"); - return d1Rows; -} -const d1Rows = queryD1(queries.d1); -const deletionBacklog = queryD1(queries.backlog).reduce( - (sum, row) => sum + Number(row.request_count), - 0, -); - -const apiKey = personalApiKey(); -if (!apiKey) { - fail( - `PostHog personal API key unavailable. Set POSTHOG_PERSONAL_API_KEY or add macOS Keychain service '${KEYCHAIN_SERVICE}'.`, - ); -} -const api = createPosthogApi({ apiKey, projectId: PROJECT_ID }); -const posthogBody = await api("query/", { - method: "POST", - body: JSON.stringify({ - query: { - kind: "HogQLQuery", - query: queries.posthog, - }, - }), -}); -if (!Array.isArray(posthogBody.results)) fail("PostHog returned no reconciliation rows"); - -const report = comparePipeline(d1Rows, posthogBody.results, deletionBacklog); -console.log(`${queries.source}: [${queries.from}, ${queries.to}) — event-ID counts`); -console.table(report.rows); -console.log(`Result: ${report.status}. Outstanding deletions: ${report.deletionBacklog}.`); -console.log( - "Identity-link events are intentionally excluded. No data does not mean the pipeline is verified.", -); -if (report.status !== "matched") process.exitCode = 2; diff --git a/workers/events/src/attributionPipeline.test.ts b/workers/events/src/attributionPipeline.test.ts index 49eacf5..cd4562c 100644 --- a/workers/events/src/attributionPipeline.test.ts +++ b/workers/events/src/attributionPipeline.test.ts @@ -68,7 +68,6 @@ it.skipIf(!desktopRoot)( DESKTOP_INGESTION_ENABLED: "true", ANALYTICS_INGESTION_RATE_LIMITER: { limit: async () => ({ success: true }) }, }; - const context = { waitUntil: () => undefined } as unknown as ExecutionContext; const uploads: string[] = []; const server = createServer((request, response) => { void (async () => { @@ -88,7 +87,6 @@ it.skipIf(!desktopRoot)( body, }) as Request, env, - context, ); response.writeHead(result.status, { "Content-Type": "application/json" }); response.end(await result.text()); diff --git a/workers/events/src/exportLease.ts b/workers/events/src/deletionLease.ts similarity index 85% rename from workers/events/src/exportLease.ts rename to workers/events/src/deletionLease.ts index 6e1a722..cb5d451 100644 --- a/workers/events/src/exportLease.ts +++ b/workers/events/src/deletionLease.ts @@ -1,5 +1,8 @@ -/** Serialize PostHog exports and erasures across Worker invocations, not just one isolate. */ -export async function withExportLease( +/** + * Serialize PostHog erasures across Worker invocations, not just one isolate, so + * overlapping cron runs never submit the same deletion twice. + */ +export async function withDeletionLease( database: D1Database, operation: (beforeRequest: () => Promise) => Promise, ): Promise { @@ -28,7 +31,7 @@ export async function withExportLease( .bind(owner) .first<{ expires_at: number }>(); if (!renewed || renewed.expires_at - Date.now() < 10_000) { - throw new Error("export-lease-lost"); + throw new Error("deletion-lease-lost"); } }); } finally { diff --git a/workers/events/src/desktopPipeline.test.ts b/workers/events/src/desktopPipeline.test.ts index 238928c..859b00f 100644 --- a/workers/events/src/desktopPipeline.test.ts +++ b/workers/events/src/desktopPipeline.test.ts @@ -5,7 +5,7 @@ import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { pathToFileURL } from "node:url"; import { expect, it, vi } from "vitest"; -import worker, { flushPendingEvents } from "./index"; +import worker from "./index"; import { testDatabase } from "./sqlite.testSupport"; // Opt-in cross-repository qualification, not an implicit sibling-checkout dependency. @@ -20,7 +20,6 @@ it.skipIf(!desktopRoot)( ); const fixture = mkdtempSync(join(tmpdir(), "scient-pipeline-proof-")); const store = testDatabase(); - const tasks: Promise[] = []; const uploads: { token: string; body: string }[] = []; const env = { ANALYTICS_DB: store.database, @@ -28,9 +27,6 @@ it.skipIf(!desktopRoot)( IDENTITY_ANALYTICS_ENABLED: "true", ANALYTICS_INGESTION_RATE_LIMITER: { limit: async () => ({ success: true }) }, }; - const context = { - waitUntil: (task: Promise) => tasks.push(task), - } as unknown as ExecutionContext; const server = createServer((request, response) => { void (async () => { const chunks: Buffer[] = []; @@ -45,7 +41,6 @@ it.skipIf(!desktopRoot)( body, }) as Request, env, - context, ); response.writeHead(result.status, { "Content-Type": "application/json" }); response.end(await result.text()); @@ -191,18 +186,6 @@ it.skipIf(!desktopRoot)( ), ), ).toMatchObject({ provider: "codex", serviceTier: "fast", inputTokens: 900 }); - const exportRequest = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", exportRequest); - expect( - await flushPendingEvents({ - ...env, - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - POSTHOG_PROJECT_TOKEN: "synthetic", - }), - ).toBe(25); - expect(exportRequest).toHaveBeenCalledOnce(); - expect(String(exportRequest.mock.calls[0]![1].body)).not.toContain("PRIVATE"); - expect(String(exportRequest.mock.calls[0]![1].body)).not.toContain("synthetic@scient.test"); expect( store.sqlite.prepare("SELECT email FROM analytics_installation_accounts").get()!.email, ).toBe("synthetic@scient.test"); @@ -212,17 +195,30 @@ it.skipIf(!desktopRoot)( expect( store.sqlite.prepare("SELECT count(*) AS n FROM analytics_installation_devices").get()!.n, ).toBe(1); - const exported = JSON.parse(String(exportRequest.mock.calls[0]![1].body)); + // Parsed JSON is untyped, as the exported PostHog batch was before. + const stored: { batch: any[] } = { + batch: store.sqlite + .prepare( + `SELECT event_name AS event, properties_json FROM analytics_events + WHERE event_name NOT IN ('install.account_observed', 'install.profile_observed') + AND privacy_level <> 'diagnostic'`, + ) + .all() + .map((row) => ({ + event: String(row.event), + properties: JSON.parse(String(row.properties_json)), + })), + }; expect( - exported.batch.find((event: { event: string }) => event.event === "install.owner_observed") + stored.batch.find((event: { event: string }) => event.event === "install.owner_observed") .properties, ).toMatchObject({ ownerSource: "openai", ownerId, observedBy: "server" }); expect( - exported.batch.find((event: { event: string }) => event.event === "provider.turn.usage") + stored.batch.find((event: { event: string }) => event.event === "provider.turn.usage") .properties, ).toMatchObject({ createdBy: "user", creationSource: "web", scheduled: false }); for (const provider of ["scient", "omp", "muse", "acpRegistry"]) { - const events = exported.batch.filter( + const events = stored.batch.filter( (event: { properties: { provider?: string } }) => event.properties.provider === provider, ); expect(events.map((event: { event: string }) => event.event).sort()).toEqual([ @@ -232,10 +228,10 @@ it.skipIf(!desktopRoot)( expect(events[0].properties.contractRevision).toBe("11"); } expect( - exported.batch.find((event: { event: string }) => event.event === "feature.viewed") + stored.batch.find((event: { event: string }) => event.event === "feature.viewed") .properties, ).toMatchObject({ feature: "latex-visual", contractRevision: "11" }); - const visualOutcomes = exported.batch.filter( + const visualOutcomes = stored.batch.filter( (event: { properties: { operationKind?: string } }) => event.properties.operationKind === "latex-visual-edit", ); @@ -248,7 +244,7 @@ it.skipIf(!desktopRoot)( contractRevision: "11", }); expect( - exported.batch.find((event: { event: string }) => event.event === "provider.turn.usage") + stored.batch.find((event: { event: string }) => event.event === "provider.turn.usage") .properties, ).toMatchObject({ provider: "codex", @@ -259,14 +255,13 @@ it.skipIf(!desktopRoot)( cachedInputTokens: 200, }); expect( - exported.batch.find( + stored.batch.find( (event: { event: string }) => event.event === "provider.lifecycle.completed", ).properties, ).toMatchObject({ source: "desktop", runtimeSource: "scient_managed" }); expect( - exported.batch.find( - (event: { event: string }) => event.event === "scient.operation.skipped", - ).properties, + stored.batch.find((event: { event: string }) => event.event === "scient.operation.skipped") + .properties, ).toMatchObject({ operationKind: "source-import", trigger: "user" }); await runtime.setConsent("off"); expect(runtime.record("project.opened")).toBe(false); @@ -276,7 +271,7 @@ it.skipIf(!desktopRoot)( expect( store.sqlite.prepare("SELECT posthog_state FROM analytics_deletion_requests").get()! .posthog_state, - ).toBe("pending"); + ).toBe("completed"); // An acknowledged deletion must resist delayed uploads with the old credential. const replay = await originalFetch(endpoint, { method: "POST", @@ -292,7 +287,6 @@ it.skipIf(!desktopRoot)( await runtime.close(); server.closeAllConnections(); await new Promise((done) => server.close(() => done())); - await Promise.allSettled(tasks); store.close(); rmSync(fixture, { recursive: true, force: true }); vi.unstubAllGlobals(); diff --git a/workers/events/src/index.test.ts b/workers/events/src/index.test.ts index 1277491..111bb2b 100644 --- a/workers/events/src/index.test.ts +++ b/workers/events/src/index.test.ts @@ -2,8 +2,6 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import worker, { flushPendingDeletions, - flushPendingEvents, - flushPendingIdentityLinks, validateIdentityLinkPayload, validateIngestionPayload, } from "./index"; @@ -259,7 +257,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); const response = await submit(events); expect(await response.json()).toEqual({ accepted: 3 }); @@ -319,7 +316,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database, { enabled: false }), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(503); @@ -344,7 +340,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(400); @@ -396,7 +391,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(500); @@ -430,7 +424,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(202); @@ -459,7 +452,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database, { rateLimitSuccess: false }), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(429); @@ -469,7 +461,6 @@ describe("event gateway routes", () => { it("stores a valid desktop event before accepting it", async () => { const database = createDatabase(); - const waitUntil = vi.fn(); const response = await worker.fetch!( incoming( new Request("https://events.scientfactory.com/v1/events", { @@ -482,7 +473,6 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil } as unknown as ExecutionContext, ); expect(response.status).toBe(202); @@ -522,12 +512,10 @@ describe("event gateway routes", () => { "installation:16ace444-e7c3-4b26-893f-98713188ae52", "session:8e0ee7d5-2c4b-48b6-8209-08f1e536f665", "product", - "pending", "installation:16ace444-e7c3-4b26-893f-98713188ae52", expect.any(String), ); // Export runs from the cron, not after every ingest. - expect(waitUntil).not.toHaveBeenCalled(); }); it("rejects browser submissions from unknown origins", async () => { @@ -544,19 +532,17 @@ describe("event gateway routes", () => { }), ), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(403); expect(database.batch).not.toHaveBeenCalled(); }); - it("reports whether optional PostHog forwarding is configured", async () => { + it("reports storage readiness and which optional services are configured", async () => { const database = realDatabase(); const response = await worker.fetch!( incoming(new Request("https://events.scientfactory.com/health")), gatewayEnv(database.database), - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(200); @@ -569,7 +555,7 @@ describe("event gateway routes", () => { storage: "ready", retention: "pending_verification", activation_prerequisites_configured: false, - posthog_forwarding: "pending_configuration", + posthog_deletion: "pending_configuration", identity_linking: "pending_configuration", }); }); @@ -592,7 +578,6 @@ describe("event gateway routes", () => { ANALYTICS_DB: database.database, IDENTITY_LINK_TOKEN: "secret", } as unknown as Cloudflare.Env, - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(401); @@ -601,7 +586,6 @@ describe("event gateway routes", () => { it("links consented visitor history to an authenticated account id", async () => { const database = createDatabase(); - const waitUntil = vi.fn(); const accountId = "account:16ace444-e7c3-4b26-893f-98713188ae52"; const installationId = "visitor:8e0ee7d5-2c4b-48b6-8209-08f1e536f665"; const response = await worker.fetch!( @@ -623,7 +607,6 @@ describe("event gateway routes", () => { ANALYTICS_DB: database.database, IDENTITY_LINK_TOKEN: "secret", } as unknown as Cloudflare.Env, - { waitUntil } as unknown as ExecutionContext, ); expect(response.status).toBe(200); @@ -633,7 +616,6 @@ describe("event gateway routes", () => { }); expect(database.batch).toHaveBeenCalledOnce(); expect(database.bind).toHaveBeenCalledWith(accountId, installationId); - expect(waitUntil).toHaveBeenCalledOnce(); }); it("refuses to reassign an identity to a different account", async () => { @@ -657,7 +639,6 @@ describe("event gateway routes", () => { ANALYTICS_DB: database.database, IDENTITY_LINK_TOKEN: "secret", } as unknown as Cloudflare.Env, - { waitUntil: vi.fn() } as unknown as ExecutionContext, ); expect(response.status).toBe(400); @@ -683,65 +664,7 @@ describe("identity link validation", () => { }); }); -describe("PostHog forwarding", () => { - it("forwards stored events with only the deletable pseudonymous person profile", async () => { - const row = { - event_id: "8e0ee7d5-2c4b-48b6-8209-08f1e536f665", - event_name: "provider.turn.sent", - source: "desktop", - privacy_level: "product", - occurred_at: new Date().toISOString(), - distinct_id: "installation:16ace444-e7c3-4b26-893f-98713188ae52", - canonical_id: "installation:16ace444-e7c3-4b26-893f-98713188ae52", - identity_type: "desktop_installation", - session_id: "session:8e0ee7d5-2c4b-48b6-8209-08f1e536f665", - consent_level: "product", - properties_json: JSON.stringify(validPayload().events[0]!.properties), - }; - const { database, sqlite } = realDatabase(); - sqlite - .prepare( - `INSERT INTO analytics_events - (event_id, event_name, source, privacy_level, occurred_at, distinct_id, canonical_id, identity_type, session_id, consent_level, properties_json) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, - ) - .run(...Object.values(row)); - const fetchMock = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetchMock); - - const forwarded = await flushPendingEvents({ - ANALYTICS_DB: database, - POSTHOG_PROJECT_TOKEN: "phc_scientfactory_test", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }); - - expect(forwarded).toBe(1); - expect(fetchMock).toHaveBeenCalledOnce(); - const [url, request] = fetchMock.mock.calls[0] ?? []; - expect(url).toBe("https://eu.i.posthog.com/batch"); - const payload = JSON.parse(String(request?.body)) as { - api_key: string; - batch: ReadonlyArray<{ properties: Record }>; - }; - expect(payload.api_key).toBe("phc_scientfactory_test"); - expect(payload.batch[0]?.properties).toMatchObject({ - provider: "codex", - appVersion: "0.0.32", - buildChannel: "development", - event_id: row.event_id, - $insert_id: row.event_id, - source: "desktop", - privacy_level: "product", - consent_level: "product", - identity_type: "desktop_installation", - $session_id: row.session_id, - $process_person_profile: true, - }); - expect(sqlite.prepare("SELECT posthog_state FROM analytics_events").get()!.posthog_state).toBe( - "sent", - ); - }); - +describe("PostHog erasure", () => { it("resolves the person before submitting an erasure and keeps it pending verification", async () => { const row = { request_id: "delete-1", @@ -808,66 +731,4 @@ describe("PostHog forwarding", () => { posthog_submitted_at: expect.any(String), }); }); - - it("forwards anonymous-to-account identity events from the first-party link queue", async () => { - const row = { - link_id: "link-1", - source_identity_id: "visitor:16ace444-e7c3-4b26-893f-98713188ae52", - canonical_id: "account:8e0ee7d5-2c4b-48b6-8209-08f1e536f665", - linked_at: new Date().toISOString(), - }; - const { database, sqlite } = realDatabase(); - const addIdentity = sqlite.prepare(`INSERT INTO analytics_identities - (identity_id, identity_type, canonical_id, consent_level, first_seen_at, last_seen_at) - VALUES (?, ?, ?, 'essential', ?, ?)`); - addIdentity.run(row.canonical_id, "account", row.canonical_id, row.linked_at, row.linked_at); - addIdentity.run( - row.source_identity_id, - "web_visitor", - row.canonical_id, - row.linked_at, - row.linked_at, - ); - sqlite - .prepare( - `INSERT INTO analytics_identity_links (link_id, source_identity_id, canonical_id, linked_at) - VALUES (?, ?, ?, ?)`, - ) - .run(...Object.values(row)); - const fetchMock = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetchMock); - - const env = { - ANALYTICS_DB: database, - POSTHOG_PROJECT_TOKEN: "phc_scientfactory_test", - }; - expect(await flushPendingIdentityLinks(env)).toBe(0); - expect(fetchMock).not.toHaveBeenCalled(); - sqlite - .prepare("UPDATE analytics_identities SET consent_level = 'product' WHERE identity_id = ?") - .run(row.source_identity_id); - const forwarded = await flushPendingIdentityLinks(env); - - expect(forwarded).toBe(1); - expect( - sqlite.prepare("SELECT posthog_state FROM analytics_identity_links").get()!.posthog_state, - ).toBe("sent"); - const [, request] = fetchMock.mock.calls[0] ?? []; - const payload = JSON.parse(String(request?.body)) as { - batch: ReadonlyArray<{ - event: string; - distinct_id: string; - properties: Record; - }>; - }; - expect(payload.batch[0]).toMatchObject({ - event: "$identify", - distinct_id: row.canonical_id, - properties: { - $anon_distinct_id: row.source_identity_id, - $insert_id: row.link_id, - $process_person_profile: true, - }, - }); - }); }); diff --git a/workers/events/src/index.ts b/workers/events/src/index.ts index 2916b5d..1f1a707 100644 --- a/workers/events/src/index.ts +++ b/workers/events/src/index.ts @@ -1,5 +1,4 @@ import { - DIAGNOSTIC_SQL, diagnosticObservation, usageRetentionCutoff, installationObservationStatements, @@ -14,8 +13,8 @@ import { PRIVACY_LEVELS, type PrivacyLevel, } from "./eventContract"; -import { posthogEventUuid, posthogRequest, readBoundedJson, TransportFailure } from "./transport"; -import { withExportLease } from "./exportLease"; +import { posthogRequest, readBoundedJson, TransportFailure } from "./transport"; +import { withDeletionLease } from "./deletionLease"; const ALLOWED_WEB_ORIGINS = new Set(["https://scientfactory.com", "https://www.scientfactory.com"]); const IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$/; @@ -26,10 +25,9 @@ const SESSION_ID_PATTERN = /^session:[0-9a-f-]{36}$/i; const MAX_REQUEST_BYTES = 128 * 1024; const MAX_BATCH_SIZE = 50; const MAX_PROPERTIES_BYTES = 16 * 1024; -const POSTHOG_BATCH_SIZE = 100; -// The five-minute cron is the only event export; this bounds one run's work. -const POSTHOG_MAX_BATCHES_PER_EXPORT = 10; -const POSTHOG_HOST = "https://eu.i.posthog.com"; +// D1 is the only analytics store. Nothing is sent to PostHog any more. Its only +// remaining use is erasing the copies sent before 17 November 2026; remove this +// and the deletion code below once that PostHog project is deleted. const POSTHOG_API_HOST = "https://eu.posthog.com"; const POSTHOG_DELETION_BATCH_SIZE = 1; const POSTHOG_DELETION_MAX_ATTEMPTS = 10; @@ -45,19 +43,15 @@ type AnalyticsEnv = Omit< | "CF_VERSION_METADATA" | "IDENTITY_ANALYTICS_ENABLED" | "DESKTOP_INGESTION_ENABLED" - | "DESKTOP_POSTHOG_EXPORT_ENABLED" | "POSTHOG_PERSONAL_API_KEY" | "POSTHOG_PROJECT_ID" - | "POSTHOG_PROJECT_TOKEN" > & { readonly CF_VERSION_METADATA?: WorkerVersionMetadata; - readonly POSTHOG_PROJECT_TOKEN?: string; readonly POSTHOG_PERSONAL_API_KEY?: string; readonly POSTHOG_PROJECT_ID?: string; readonly IDENTITY_LINK_TOKEN?: string; readonly IDENTITY_ANALYTICS_ENABLED?: string; readonly DESKTOP_INGESTION_ENABLED?: string; - readonly DESKTOP_POSTHOG_EXPORT_ENABLED?: string; readonly ANALYTICS_INGESTION_RATE_LIMITER?: RateLimit; }; @@ -73,28 +67,6 @@ interface AcceptedEvent { readonly properties: Record; } -interface PendingEventRow { - readonly event_id: string; - readonly event_name: string; - readonly source: string; - readonly privacy_level: string; - readonly occurred_at: string; - readonly distinct_id: string; - readonly canonical_id: string; - readonly identity_type: string; - readonly session_id: string | null; - readonly consent_level: string; - readonly properties_json: string; - readonly product_first_seen_at?: string | null; -} - -interface PendingIdentityLinkRow { - readonly link_id: string; - readonly source_identity_id: string; - readonly canonical_id: string; - readonly linked_at: string; -} - interface PendingDeletionRow { readonly request_id: string; readonly posthog_distinct_id: string; @@ -511,12 +483,11 @@ async function persistEvents( identity_type, canonical_id, session_id, - consent_level, - posthog_state + consent_level ) SELECT ?, ?, 'desktop', ?, ?, ?, ?, 'desktop_installation', COALESCE((SELECT canonical_id FROM analytics_identities WHERE identity_id = ?), ?), - ?, ?, ? + ?, ? WHERE EXISTS ( SELECT 1 FROM analytics_identities WHERE identity_id = ? AND deletion_token_hash = ? @@ -551,7 +522,6 @@ async function persistEvents( event.distinctId, event.sessionId, event.consentLevel, - neverExported(event) ? "excluded" : "pending", event.distinctId, deletionTokenHash, ), @@ -577,303 +547,6 @@ async function persistEvents( } } -/** The PostHog export never sends these, so they never enter its queue. */ -function neverExported(event: AcceptedEvent): boolean { - return event.privacyLevel === "diagnostic" || isIdentityObservation(event); -} - -async function posthogEvent(row: PendingEventRow): Promise> { - let properties: Record = {}; - try { - const parsed = JSON.parse(row.properties_json) as unknown; - if (isRecord(parsed)) properties = parsed; - } catch { - // The gateway writes valid JSON; retaining an empty object makes a malformed legacy row retryable. - } - return { - uuid: await posthogEventUuid(row.event_id), - event: row.event_name, - distinct_id: row.canonical_id, - timestamp: row.occurred_at, - properties: { - ...properties, - event_id: row.event_id, - $insert_id: row.event_id, - source: row.source, - privacy_level: row.privacy_level, - consent_level: row.consent_level, - identity_type: row.identity_type, - ...(row.product_first_seen_at ? { productFirstSeenAt: row.product_first_seen_at } : {}), - ...(row.session_id ? { $session_id: row.session_id } : {}), - // A minimal pseudonymous person record is necessary for PostHog's - // supported distinct-id event deletion API. No person properties are set. - $process_person_profile: true, - }, - }; -} - -async function markPosthogFailure( - database: D1Database, - rows: ReadonlyArray, - error: string, -): Promise { - if (rows.length === 0) return; - await database.batch( - rows.map((row) => - database - .prepare( - `UPDATE analytics_events - SET posthog_attempts = posthog_attempts + 1, - posthog_last_error = ?, - posthog_next_attempt_at = datetime('now', '+' || min(1800, 30 * (1 << min(posthog_attempts, 6))) || ' seconds') - WHERE event_id = ? AND posthog_state = 'pending'`, - ) - .bind(error.slice(0, 500), row.event_id), - ), - ); -} - -export async function flushPendingEvents(env: AnalyticsEnv): Promise { - if (!env.POSTHOG_PROJECT_TOKEN) return 0; - return withExportLease(env.ANALYTICS_DB, async (beforeRequest) => { - let exported = 0; - for (let batch = 0; batch < POSTHOG_MAX_BATCHES_PER_EXPORT; batch += 1) { - const sent = await exportPendingEvents(env, beforeRequest); - exported += sent; - if (sent < POSTHOG_BATCH_SIZE) break; - } - return exported; - }); -} - -async function exportPendingEvents( - env: AnalyticsEnv, - beforeRequest: () => Promise, -): Promise { - const result = await env.ANALYTICS_DB.prepare( - `SELECT - event_id, - event_name, - source, - privacy_level, - occurred_at, - distinct_id, - canonical_id, - identity_type, - session_id, - consent_level, - properties_json, - (SELECT product_first_seen_at FROM analytics_identities WHERE identity_id = analytics_events.distinct_id) AS product_first_seen_at - FROM analytics_events - WHERE posthog_state = 'pending' - AND event_name NOT IN ('install.account_observed', 'install.profile_observed') - AND NOT (event_name = 'install.owner_observed' AND CASE WHEN json_valid(properties_json) THEN COALESCE(json_extract(properties_json, '$.ownerSource'), '') ELSE '' END = 'device') - AND (source <> 'desktop' OR privacy_level <> 'diagnostic') - AND (source <> 'desktop' OR ? = 1) - AND (source <> 'desktop' OR ( - (? = 1 OR julianday(occurred_at) >= julianday('now', CASE WHEN privacy_level = 'diagnostic' THEN '-${DIAGNOSTIC_EVENT_RETENTION_DAYS} days' ELSE '-${RAW_EVENT_RETENTION_DAYS} days' END)) - AND (? = 1 OR julianday(received_at) >= julianday('now', CASE WHEN privacy_level = 'diagnostic' THEN '-${DIAGNOSTIC_EVENT_RETENTION_DAYS} days' ELSE '-${RAW_EVENT_RETENTION_DAYS} days' END)) - AND julianday(occurred_at) <= julianday('now', '+1 day') - )) - AND (? = 0 OR NOT ${DIAGNOSTIC_SQL} OR (julianday(occurred_at) >= julianday('now', '-60 days') AND julianday(received_at) >= julianday('now', '-60 days'))) - AND posthog_attempts < 20 - AND (posthog_next_attempt_at IS NULL OR julianday(posthog_next_attempt_at) <= julianday('now')) - AND NOT EXISTS (SELECT 1 FROM analytics_deleted_installations WHERE installation_id = analytics_events.distinct_id) - ORDER BY received_at, event_id - LIMIT ?`, - ) - .bind( - env.DESKTOP_POSTHOG_EXPORT_ENABLED === "true" ? 1 : 0, - env.IDENTITY_ANALYTICS_ENABLED === "true" ? 1 : 0, - env.IDENTITY_ANALYTICS_ENABLED === "true" ? 1 : 0, - env.IDENTITY_ANALYTICS_ENABLED === "true" ? 1 : 0, - POSTHOG_BATCH_SIZE, - ) - .all(); - let rows = result.results; - if (rows.length === 0) return 0; - - // Revalidate persisted desktop rows too: a legacy/corrupt row must not bypass - // today's privacy contract, nor poison every later event in its batch. - const rejected: string[] = []; - rows = rows.filter((row) => { - if (row.source !== "desktop") return true; - try { - const properties: unknown = JSON.parse(row.properties_json); - if (isRecord(properties) && isIdentityObservation({ name: row.event_name, properties })) - return false; - if ( - isRecord(properties) && - PRIVACY_LEVELS.includes(row.privacy_level as PrivacyLevel) && - PRIVACY_LEVELS.includes(row.consent_level as PrivacyLevel) && - eventContractViolation({ - name: row.event_name, - privacyLevel: row.privacy_level as PrivacyLevel, - consentLevel: row.consent_level as PrivacyLevel, - properties, - }) === null - ) - return true; - } catch { - /* Quarantine by a fixed class, never by raw properties/error text. */ - } - rejected.push(row.event_id); - return false; - }); - if (rejected.length > 0) - await env.ANALYTICS_DB.batch( - rejected.map((id) => - env.ANALYTICS_DB.prepare( - "UPDATE analytics_events SET posthog_attempts = 20, posthog_last_error = 'contract-rejected' WHERE event_id = ? AND posthog_state = 'pending'", - ).bind(id), - ), - ); - if (rows.length === 0) return 0; - - // Persist before sending: a concurrent erasure must know about an uncertain export. - await env.ANALYTICS_DB.batch( - rows.map((row) => - env.ANALYTICS_DB.prepare( - "UPDATE analytics_identities SET posthog_attempted = 1 WHERE identity_id = ?", - ).bind(row.distinct_id), - ), - ); - const surviving = await env.ANALYTICS_DB.prepare( - `SELECT event_id FROM analytics_events WHERE event_id IN (${rows.map(() => "?").join(",")})`, - ) - .bind(...rows.map((row) => row.event_id)) - .all<{ event_id: string }>(); - const survivingIds = new Set(surviving.results.map((row) => row.event_id)); - rows = rows.filter((row) => survivingIds.has(row.event_id)); - if (rows.length === 0) return 0; - - let response: Response; - try { - const batch = await Promise.all(rows.map(posthogEvent)); - await beforeRequest(); - response = await posthogRequest(`${POSTHOG_HOST}/batch`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - api_key: env.POSTHOG_PROJECT_TOKEN, - batch, - }), - }); - } catch (error) { - const message = error instanceof TransportFailure ? error.kind : "internal"; - await markPosthogFailure(env.ANALYTICS_DB, rows, message); - throw error; - } - - await response.body?.cancel(); - - await env.ANALYTICS_DB.batch( - rows.map((row) => - env.ANALYTICS_DB.prepare( - `UPDATE analytics_events - SET posthog_state = 'sent', - posthog_attempts = posthog_attempts + 1, - posthog_last_error = NULL, - posthog_sent_at = CURRENT_TIMESTAMP - WHERE event_id = ? AND posthog_state = 'pending'`, - ).bind(row.event_id), - ), - ); - return rows.length; -} - -async function identityIdentifyEvent( - row: PendingIdentityLinkRow, -): Promise> { - return { - uuid: await posthogEventUuid(row.link_id), - event: "$identify", - distinct_id: row.canonical_id, - timestamp: row.linked_at, - properties: { - $anon_distinct_id: row.source_identity_id, - $insert_id: row.link_id, - link_id: row.link_id, - source: "identity_gateway", - $process_person_profile: true, - }, - }; -} - -export async function flushPendingIdentityLinks(env: AnalyticsEnv): Promise { - if (!env.POSTHOG_PROJECT_TOKEN) return 0; - return withExportLease(env.ANALYTICS_DB, (beforeRequest) => - exportPendingIdentityLinks(env, beforeRequest), - ); -} - -async function exportPendingIdentityLinks( - env: AnalyticsEnv, - beforeRequest: () => Promise, -): Promise { - const result = await env.ANALYTICS_DB.prepare( - `SELECT links.link_id, links.source_identity_id, links.canonical_id, links.linked_at - FROM analytics_identity_links AS links - JOIN analytics_identities AS identities - ON identities.identity_id = links.source_identity_id - WHERE links.posthog_state = 'pending' - AND links.posthog_attempts < 20 - AND (links.posthog_next_attempt_at IS NULL OR julianday(links.posthog_next_attempt_at) <= julianday('now')) - AND identities.identity_type = 'web_visitor' - AND identities.consent_level IN ('product', 'diagnostic', 'contribution') - ORDER BY links.linked_at, links.link_id - LIMIT ?`, - ) - .bind(POSTHOG_BATCH_SIZE) - .all(); - const rows = result.results; - if (rows.length === 0) return 0; - - let response: Response; - try { - const batch = await Promise.all(rows.map(identityIdentifyEvent)); - await beforeRequest(); - response = await posthogRequest(`${POSTHOG_HOST}/batch`, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - api_key: env.POSTHOG_PROJECT_TOKEN, - batch, - }), - }); - } catch (error) { - const message = error instanceof TransportFailure ? error.kind : "internal"; - await env.ANALYTICS_DB.batch( - rows.map((row) => - env.ANALYTICS_DB.prepare( - `UPDATE analytics_identity_links - SET posthog_attempts = posthog_attempts + 1, - posthog_last_error = ?, - posthog_next_attempt_at = datetime('now', '+' || min(1800, 30 * (1 << min(posthog_attempts, 6))) || ' seconds') - WHERE link_id = ? AND posthog_state = 'pending'`, - ).bind(message.slice(0, 500), row.link_id), - ), - ); - throw error; - } - - await response.body?.cancel(); - - await env.ANALYTICS_DB.batch( - rows.map((row) => - env.ANALYTICS_DB.prepare( - `UPDATE analytics_identity_links - SET posthog_state = 'sent', - posthog_attempts = posthog_attempts + 1, - posthog_last_error = NULL, - posthog_sent_at = CURRENT_TIMESTAMP - WHERE link_id = ? AND posthog_state = 'pending'`, - ).bind(row.link_id), - ), - ); - return rows.length; -} - async function markPosthogDeletionFailure( database: D1Database, row: PendingDeletionRow, @@ -895,10 +568,13 @@ async function markPosthogDeletionFailure( .run(); } -/** A capture acknowledgement is not an erasure acknowledgement. Poll verified status. */ +/** + * Erases an installation's copies from PostHog, the only PostHog request left. + * A capture acknowledgement is not an erasure acknowledgement. Poll verified status. + */ export async function flushPendingDeletions(env: AnalyticsEnv): Promise { if (!env.POSTHOG_PERSONAL_API_KEY || !env.POSTHOG_PROJECT_ID) return 0; - return withExportLease(env.ANALYTICS_DB, (beforeRequest) => + return withDeletionLease(env.ANALYTICS_DB, (beforeRequest) => processPendingDeletion(env, beforeRequest), ); } @@ -974,8 +650,7 @@ async function processPendingDeletion( ) .run(); // Completion is PostHog's verified asynchronous erasure, not its - // submission acknowledgement. The tombstone rejects future uploads; - // the shared export lease prevents a later exporter reusing this ID. + // submission acknowledgement. The tombstone rejects future uploads. if (verified) completed += 1; continue; } @@ -1133,12 +808,7 @@ async function persistIdentityLinks( ) VALUES (?, ?, ?, ?) ON CONFLICT(source_identity_id) DO UPDATE SET canonical_id = excluded.canonical_id, - linked_at = excluded.linked_at, - posthog_state = CASE - WHEN analytics_identity_links.canonical_id = excluded.canonical_id - THEN analytics_identity_links.posthog_state - ELSE 'pending' - END`, + linked_at = excluded.linked_at`, ) .bind(crypto.randomUUID(), identityId, accountId, linkedAt), database @@ -1153,11 +823,7 @@ async function persistIdentityLinks( await database.batch(statements); } -async function handleIdentityLink( - request: Request, - env: AnalyticsEnv, - context: Pick, -): Promise { +async function handleIdentityLink(request: Request, env: AnalyticsEnv): Promise { if (!env.IDENTITY_LINK_TOKEN) { return jsonResponse({ error: "Identity linking is not configured" }, 503); } @@ -1175,16 +841,6 @@ async function handleIdentityLink( return jsonResponse({ error: "Desktop account linking is not available" }, 409); } await persistIdentityLinks(env.ANALYTICS_DB, link.accountId, link.identityIds); - context.waitUntil( - flushPendingIdentityLinks(env).catch((error: unknown) => { - console.error( - JSON.stringify({ - message: "PostHog identity linking failed", - error_class: error instanceof TransportFailure ? error.kind : "internal", - }), - ); - }), - ); return jsonResponse({ linked: link.identityIds.length, canonical_id: link.accountId }, 200); } catch (error) { if (error instanceof RequestValidationError) { @@ -1466,14 +1122,14 @@ async function handleIngestion(request: Request, env: AnalyticsEnv): Promise) { + async fetch(request, env) { const url = new URL(request.url); if (request.method === "GET" && url.pathname === "/health") { let storageReady = false; let retentionReady = false; try { // Compile against the required schema without scanning or exposing user rows. - await env.ANALYTICS_DB.prepare(`SELECT events.posthog_next_attempt_at, identities.posthog_attempted, + await env.ANALYTICS_DB.prepare(`SELECT events.consent_level, identities.posthog_attempted, deletions.posthog_person_uuid, tomb.deletion_token_hash, leases.expires_at, maintenance.outcome FROM analytics_events AS events, analytics_identities AS identities, analytics_deletion_requests AS deletions, analytics_deleted_installations AS tomb, @@ -1502,7 +1158,6 @@ const worker = { storageReady && retentionReady && env.ANALYTICS_INGESTION_RATE_LIMITER && - env.POSTHOG_PROJECT_TOKEN && env.POSTHOG_PERSONAL_API_KEY && env.POSTHOG_PROJECT_ID, ), @@ -1510,9 +1165,6 @@ const worker = { rate_limiting: env.ANALYTICS_INGESTION_RATE_LIMITER ? "configured" : "pending_configuration", - posthog_forwarding: env.POSTHOG_PROJECT_TOKEN ? "configured" : "pending_configuration", - desktop_posthog_export: - env.DESKTOP_POSTHOG_EXPORT_ENABLED === "true" ? "enabled" : "disabled", posthog_deletion: env.POSTHOG_PERSONAL_API_KEY && env.POSTHOG_PROJECT_ID ? "configured" @@ -1542,7 +1194,7 @@ const worker = { return handleIngestion(request, env); } if (request.method === "POST" && url.pathname === "/v1/identity/link") { - return handleIdentityLink(request, env, context); + return handleIdentityLink(request, env); } if (request.method === "POST" && url.pathname === "/v1/installations/delete") { return handleInstallationDeletion(request, env); @@ -1562,8 +1214,6 @@ const worker = { : pruneExpiredAnalyticsEvents(env.ANALYTICS_DB), ], ["deletion", () => flushPendingDeletions(env)], - ["identity-export", () => flushPendingIdentityLinks(env)], - ["event-export", () => flushPendingEvents(env)], ] as const) { let outcome = "ok"; try { diff --git a/workers/events/src/people.test.ts b/workers/events/src/people.test.ts index c8238dd..86337b8 100644 --- a/workers/events/src/people.test.ts +++ b/workers/events/src/people.test.ts @@ -1,5 +1,5 @@ import { afterEach, expect, it, vi } from "vitest"; -import worker, { flushPendingEvents } from "./index"; +import worker from "./index"; import { testDatabase } from "./sqlite.testSupport"; import { prunePeopleAnalytics } from "../../../functions/_lib/analyticsPeople"; import { insertSiteEvent } from "../../../functions/_lib/events"; @@ -15,12 +15,10 @@ afterEach(() => { function fixture(enabled = true) { const store = testDatabase(); stores.push(store); - const tasks: Promise[] = []; const env = { ANALYTICS_DB: store.database, DESKTOP_INGESTION_ENABLED: "true", IDENTITY_ANALYTICS_ENABLED: String(enabled), - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", ANALYTICS_INGESTION_RATE_LIMITER: { limit: async () => ({ success: true }) }, }; const send = async ( @@ -64,11 +62,7 @@ function fixture(enabled = true) { asOrganization: "Synthetic network", }, }); - return worker.fetch!(request as Request, env, { - waitUntil: (task: Promise) => { - tasks.push(task); - }, - }); + return worker.fetch!(request as Request, env); }; const erase = () => worker.fetch!( @@ -78,9 +72,8 @@ function fixture(enabled = true) { body: JSON.stringify({ schema_version: 1, installation_id: installation }), }) as Request, env, - { waitUntil: () => {} }, ); - return { ...store, env, send, erase, tasks }; + return { ...store, env, send, erase }; } it("keeps collection disabled until activation, while acknowledging newer clients", async () => { @@ -101,7 +94,7 @@ it("keeps collection disabled until activation, while acknowledging newer client expect(f.sqlite.prepare(`SELECT count(*) AS n FROM ${table}`).get()!.n).toBe(0); }); -it("stores identity separately and never exports its events or request network", async () => { +it("stores identity observations and request network in their own tables", async () => { const f = fixture(); expect( ( @@ -125,14 +118,6 @@ it("stores identity separately and never exports its events or request network", expect( f.sqlite.prepare("SELECT ip_address, country FROM analytics_installation_sightings").get(), ).toMatchObject({ ip_address: "192.0.2.4", country: "IL" }); - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - expect(await flushPendingEvents({ ...f.env, POSTHOG_PROJECT_TOKEN: "synthetic" })).toBe(1); - const body = String(fetcher.mock.calls[0]![1].body); - expect(body).not.toContain("synthetic@scient.test"); - expect(body).not.toContain("192.0.2.4"); - expect(body).not.toContain('"ownerSource":"device"'); - expect(body).toContain("provider.turn.sent"); }); it("rejects wrong tokens and ignores event replays without inflating totals or changing identity", async () => { @@ -278,7 +263,7 @@ it("ignores malformed device hashes without failing an otherwise valid batch", a ).toBe(0); }); -it("expires failure observations after 60 days without dropping old active usage exports", async () => { +it("expires failure observations after 60 days without dropping old active usage", async () => { const f = fixture(); await f.send("provider.turn.failed", {}); f.sqlite.exec("UPDATE analytics_events SET occurred_at = datetime('now', '-61 days')"); @@ -288,9 +273,8 @@ it("expires failure observations after 60 days without dropping old active usage f.sqlite.exec( "UPDATE analytics_events SET occurred_at = datetime('now', '-400 days'), received_at = datetime('now', '-400 days')", ); - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - expect(await flushPendingEvents({ ...f.env, POSTHOG_PROJECT_TOKEN: "synthetic" })).toBe(1); + await prunePeopleAnalytics(f.database); + expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(1); }); it("retains a canonical account while any linked installation or visitor remains active", async () => { diff --git a/workers/events/src/readiness.test.ts b/workers/events/src/readiness.test.ts index 7f1280f..51d85c8 100644 --- a/workers/events/src/readiness.test.ts +++ b/workers/events/src/readiness.test.ts @@ -1,12 +1,8 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import worker, { - flushPendingDeletions, - flushPendingEvents, - pruneExpiredAnalyticsEvents, -} from "./index"; +import worker, { flushPendingDeletions, pruneExpiredAnalyticsEvents } from "./index"; import { testDatabase } from "./sqlite.testSupport"; -import { posthogEventUuid, posthogRequest, readBoundedJson } from "./transport"; -import { withExportLease } from "./exportLease"; +import { posthogRequest, readBoundedJson } from "./transport"; +import { withDeletionLease } from "./deletionLease"; const installation = "installation:10000000-0000-4000-8000-000000000001"; const token = "a".repeat(64); @@ -33,7 +29,6 @@ function fixture(...hooks: Parameters) { body: JSON.stringify(data), }) as Request, env, - context, ); const event = (id = crypto.randomUUID()) => ({ schema_version: 1, @@ -78,70 +73,12 @@ it("does not follow PostHog redirects or turn them into network failures", async vi.stubGlobal("fetch", fetcher); await expect( - posthogRequest("https://eu.i.posthog.com/batch", { method: "POST" }), + posthogRequest("https://eu.posthog.com/api/projects/1/persons/", { method: "GET" }), ).rejects.toThrow("http"); expect(fetcher.mock.calls[0]![1].redirect).toBe("manual"); }); describe("inactive gateway readiness with real SQL", () => { - it("keeps diagnostic events queryable in D1 without exporting them to PostHog", async () => { - const f = fixture(); - await f.request("/v1/events", f.event()); - f.sqlite.exec(`UPDATE analytics_events SET event_name = 'app.diagnostics', - privacy_level = 'diagnostic', consent_level = 'diagnostic', properties_json = - '{"queuedCountBucket":"0","droppedCountBucket":"0","retryCountBucket":"0","deliveryClass":"idle"}'`); - const fetcher = vi.fn(); - vi.stubGlobal("fetch", fetcher); - expect( - await flushPendingEvents({ - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }), - ).toBe(0); - expect(fetcher).not.toHaveBeenCalled(); - expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(1); - }); - it("keeps desktop export off independently of existing website forwarding", async () => { - const f = fixture(); - await f.request("/v1/events", f.event()); - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - const env = { ...f.env, POSTHOG_PROJECT_TOKEN: "synthetic" }; - expect(await flushPendingEvents(env)).toBe(0); - expect(fetcher).not.toHaveBeenCalled(); - f.sqlite.exec("UPDATE analytics_events SET source = 'website', event_name = 'page_viewed'"); - expect(await flushPendingEvents(env)).toBe(1); - expect(fetcher).toHaveBeenCalledOnce(); - }); - - it("quarantines invalid persisted desktop properties without blocking valid rows", async () => { - const f = fixture(); - const invalid = f.event(); - await f.request("/v1/events", invalid); - await f.request("/v1/events", f.event()); - f.sqlite - .prepare("UPDATE analytics_events SET properties_json = ? WHERE event_id = ?") - .run(JSON.stringify({ prompt: "private research" }), invalid.events[0]!.id); - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - expect( - await flushPendingEvents({ - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }), - ).toBe(1); - expect(String(fetcher.mock.calls[0]?.[1]?.body)).not.toContain("private research"); - expect( - f.sqlite - .prepare( - "SELECT posthog_attempts, posthog_last_error FROM analytics_events WHERE event_id = ?", - ) - .get(invalid.events[0]!.id), - ).toMatchObject({ posthog_attempts: 20, posthog_last_error: "contract-rejected" }); - }); - it("keeps retention bounded and reports an outstanding backlog as unqualified", async () => { const f = fixture(); f.sqlite @@ -153,61 +90,6 @@ describe("inactive gateway readiness with real SQL", () => { expect(await pruneExpiredAnalyticsEvents(f.database)).toBe(1); }); - it("writes never-exported rows as excluded and leaves export to the cron", async () => { - const f = fixture(); - const session = f.event().events[0]!; - const diagnostic = { - ...session, - id: crypto.randomUUID(), - name: "app.diagnostics", - privacy_level: "diagnostic", - consent_level: "diagnostic", - properties: { - appVersion: "0.6.8", - buildChannel: "stable", - queuedCountBucket: "0", - droppedCountBucket: "0", - retryCountBucket: "0", - deliveryClass: "delivered", - }, - }; - const response = await f.request("/v1/events", { - ...f.event(), - events: [session, { ...diagnostic, consent_level: "diagnostic" }], - }); - expect(response.status).toBe(202); - expect(f.tasks).toHaveLength(0); - expect( - f.sqlite - .prepare("SELECT event_name, posthog_state FROM analytics_events ORDER BY event_name") - .all(), - ).toEqual([ - { event_name: "app.diagnostics", posthog_state: "excluded" }, - { event_name: "app.session.started", posthog_state: "pending" }, - ]); - }); - - it("drains several PostHog batches in one cron export", async () => { - const f = fixture(); - for (let batch = 0; batch < 5; batch += 1) { - const events = Array.from({ length: 50 }, () => ({ - ...f.event().events[0]!, - id: crypto.randomUUID(), - })); - expect((await f.request("/v1/events", { ...f.event(), events })).status).toBe(202); - } - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - expect( - await flushPendingEvents({ - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }), - ).toBe(250); - expect(fetcher).toHaveBeenCalledTimes(3); - }); - it("backfills excluded rows without recounting totals and keeps the table's triggers", async () => { const insert = `INSERT INTO analytics_events (event_id, event_name, source, privacy_level, occurred_at, distinct_id, properties_json, posthog_state) VALUES (?, ?, 'desktop', ?, ?, ?, ?, ?)`; @@ -260,7 +142,7 @@ describe("inactive gateway readiness with real SQL", () => { ).toThrow("deleted-installation"); }); - it("does not reset diagnostic retention when old offline events arrive or await export", async () => { + it("does not reset diagnostic retention when old offline events arrive", async () => { const f = fixture(); const old = new Date(Date.now() - 31 * 86400000).toISOString(); const diagnostic = { @@ -296,16 +178,6 @@ describe("inactive gateway readiness with real SQL", () => { f.sqlite .prepare("UPDATE analytics_events SET privacy_level = 'diagnostic', occurred_at = ?") .run(old); - const fetcher = vi.fn().mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetcher); - expect( - await flushPendingEvents({ - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }), - ).toBe(0); - expect(fetcher).not.toHaveBeenCalled(); expect(await pruneExpiredAnalyticsEvents(f.database)).toBe(1); }); @@ -345,7 +217,7 @@ describe("inactive gateway readiness with real SQL", () => { ).toEqual({ outcome: "ok" }); expect( f.sqlite.prepare("SELECT count(*) AS n FROM analytics_maintenance_status").get()!.n, - ).toBe(4); + ).toBe(2); expect(console.error).not.toHaveBeenCalledWith( expect.stringContaining("private remote detail"), ); @@ -403,11 +275,11 @@ describe("inactive gateway readiness with real SQL", () => { const f = fixture(); let requests = 0; await expect( - withExportLease(f.database, async (beforeRequest) => { + withDeletionLease(f.database, async (beforeRequest) => { await beforeRequest(); f.sqlite.exec("UPDATE analytics_maintenance_leases SET expires_at = 0"); expect( - await withExportLease(f.database, async (newOwnerRequest) => { + await withDeletionLease(f.database, async (newOwnerRequest) => { await newOwnerRequest(); requests += 1; return 1; @@ -417,7 +289,7 @@ describe("inactive gateway readiness with real SQL", () => { requests += 1; return 1; }), - ).rejects.toThrow("export-lease-lost"); + ).rejects.toThrow("deletion-lease-lost"); expect(requests).toBe(1); }); @@ -434,7 +306,6 @@ describe("inactive gateway readiness with real SQL", () => { }), }) as Request, { ...f.env, IDENTITY_LINK_TOKEN: "synthetic" }, - f.context, ); expect(response.status).toBe(409); expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_identity_links").get()!.n).toBe(0); @@ -475,75 +346,6 @@ describe("inactive gateway readiness with real SQL", () => { expect((await f.request("/v1/events", f.event())).status).toBe(403); }); - it("keeps an in-flight export and deletion truthful and serializes exporters", async () => { - const f = fixture(); - await f.request("/v1/events", f.event()); - let finish!: (response: Response) => void; - let started!: () => void; - const signal = new Promise((resolve) => { - started = resolve; - }); - vi.stubGlobal( - "fetch", - vi.fn(() => { - started(); - return new Promise((resolve) => { - finish = resolve; - }); - }), - ); - const env = { - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - POSTHOG_PROJECT_ID: "1", - POSTHOG_PERSONAL_API_KEY: "synthetic", - }; - const sending = flushPendingEvents(env); - await signal; - expect(await flushPendingEvents(env)).toBe(0); - const receipt = await (await f.erase()).json(); - expect(receipt).toMatchObject({ posthog_state: "pending" }); - expect(await (await f.erase()).json()).toEqual(receipt); - expect(await flushPendingDeletions(env)).toBe(0); - finish(new Response("{}", { status: 200 })); - await sending; - expect(f.sqlite.prepare("SELECT count(*) AS n FROM analytics_events").get()!.n).toBe(0); - expect( - f.sqlite.prepare("SELECT posthog_state FROM analytics_deletion_requests").get()! - .posthog_state, - ).toBe("pending"); - }); - - it("uses a stable UUID and bounded transport for retries, without raw error persistence", async () => { - const f = fixture(); - await f.request("/v1/events", f.event()); - const fetcher = vi - .fn() - .mockRejectedValueOnce(new Error("private credential/path")) - .mockResolvedValue(new Response("{}")); - vi.stubGlobal("fetch", fetcher); - const env = { - ...f.env, - POSTHOG_PROJECT_TOKEN: "synthetic", - DESKTOP_POSTHOG_EXPORT_ENABLED: "true", - }; - await expect(flushPendingEvents(env)).rejects.toThrow("network"); - expect( - f.sqlite.prepare("SELECT posthog_last_error FROM analytics_events").get()!.posthog_last_error, - ).toBe("network"); - f.sqlite.exec("UPDATE analytics_events SET posthog_next_attempt_at = NULL"); - expect(await flushPendingEvents(env)).toBe(1); - const first = JSON.parse(fetcher.mock.calls[0]![1].body).batch[0]; - const second = JSON.parse(fetcher.mock.calls[1]![1].body).batch[0]; - expect(first).toEqual(second); - expect(first.uuid).toMatch( - /^[a-f0-9]{8}-[a-f0-9]{4}-8[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/, - ); - expect(fetcher.mock.calls[0]![1].redirect).toBe("manual"); - expect(fetcher.mock.calls[0]![1].signal).toBeInstanceOf(AbortSignal); - }); - it("completes only after provider-verified erasure and continues rejecting the old identity", async () => { const f = fixture(); await f.request("/v1/events", f.event()); @@ -630,7 +432,7 @@ describe("inactive gateway readiness with real SQL", () => { }); }); -it("bounds unknown-length bodies while streaming and preserves stable legacy IDs", async () => { +it("bounds unknown-length bodies while streaming", async () => { let cancelled = false; const body = new ReadableStream({ pull(controller) { @@ -642,8 +444,4 @@ it("bounds unknown-length bodies while streaming and preserves stable legacy IDs }); await expect(readBoundedJson(new Response(body), 100)).rejects.toThrow("body-too-large"); expect(cancelled).toBe(true); - expect(await posthogEventUuid("legacy:event-1")).toBe(await posthogEventUuid("legacy:event-1")); - expect(await posthogEventUuid("legacy:event-1")).not.toBe( - await posthogEventUuid("legacy:event-2"), - ); }); diff --git a/workers/events/src/transport.ts b/workers/events/src/transport.ts index 21fe433..eec735c 100644 --- a/workers/events/src/transport.ts +++ b/workers/events/src/transport.ts @@ -1,4 +1,4 @@ -/** Bounded transport shared by ingestion and the optional analysis exporter. */ +/** Bounded transport shared by ingestion and PostHog erasure. */ export class TransportFailure extends Error { constructor(readonly kind: "body-too-large" | "invalid-json" | "network" | "timeout" | "http") { super(kind); @@ -40,7 +40,7 @@ export async function readBoundedJson( } } -/** Never follow redirects with an analysis project token or deletion credential. */ +/** Never follow redirects with the PostHog deletion credential. */ export async function posthogRequest(url: string, init: RequestInit): Promise { try { const response = await fetch(url, { @@ -48,7 +48,7 @@ export async function posthogRequest(url: string, init: RequestInit): Promise { - const hash = new Uint8Array( - await crypto.subtle.digest("SHA-256", new TextEncoder().encode(`scient:analytics:event:${id}`)), - ); - hash[6] = (hash[6]! & 0x0f) | 0x80; - hash[8] = (hash[8]! & 0x3f) | 0x80; - const hex = [...hash.subarray(0, 16)].map((byte) => byte.toString(16).padStart(2, "0")).join(""); - return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`; -} diff --git a/workers/events/worker-configuration.d.ts b/workers/events/worker-configuration.d.ts index f36aab8..c1838a7 100644 --- a/workers/events/worker-configuration.d.ts +++ b/workers/events/worker-configuration.d.ts @@ -1,14 +1,12 @@ /* eslint-disable */ -// Generated by Wrangler by running `wrangler types --config=workers/events/wrangler.jsonc --include-runtime=false --env-interface=AnalyticsWorkerBindings workers/events/worker-configuration.d.ts` (hash: 0843d6ed5ec01c49d2b2e375e45a033b) +// Generated by Wrangler by running `wrangler types --config=workers/events/wrangler.jsonc --include-runtime=false --env-interface=AnalyticsWorkerBindings workers/events/worker-configuration.d.ts` (hash: aa7b4d52796921e8d790a51df2b252b1) interface __BaseEnv_AnalyticsWorkerBindings { ANALYTICS_DB: D1Database; ANALYTICS_INGESTION_RATE_LIMITER: RateLimit; CF_VERSION_METADATA: WorkerVersionMetadata; IDENTITY_ANALYTICS_ENABLED: "true"; DESKTOP_INGESTION_ENABLED: "true"; - DESKTOP_POSTHOG_EXPORT_ENABLED: "true"; POSTHOG_PROJECT_ID: "228610"; - POSTHOG_PROJECT_TOKEN: string; POSTHOG_PERSONAL_API_KEY: string; } declare namespace Cloudflare { @@ -27,9 +25,7 @@ declare namespace NodeJS { Cloudflare.Env, | "IDENTITY_ANALYTICS_ENABLED" | "DESKTOP_INGESTION_ENABLED" - | "DESKTOP_POSTHOG_EXPORT_ENABLED" | "POSTHOG_PROJECT_ID" - | "POSTHOG_PROJECT_TOKEN" | "POSTHOG_PERSONAL_API_KEY" > > {} diff --git a/workers/events/wrangler.jsonc b/workers/events/wrangler.jsonc index af6a928..8bb7b5f 100644 --- a/workers/events/wrangler.jsonc +++ b/workers/events/wrangler.jsonc @@ -7,11 +7,12 @@ "vars": { "IDENTITY_ANALYTICS_ENABLED": "true", "DESKTOP_INGESTION_ENABLED": "true", - "DESKTOP_POSTHOG_EXPORT_ENABLED": "true", "POSTHOG_PROJECT_ID": "228610", }, + // PostHog is used only to erase copies sent before 17 November 2026. Remove + // POSTHOG_PROJECT_ID and this secret once that PostHog project is deleted. "secrets": { - "required": ["POSTHOG_PROJECT_TOKEN", "POSTHOG_PERSONAL_API_KEY"], + "required": ["POSTHOG_PERSONAL_API_KEY"], }, "version_metadata": { "binding": "CF_VERSION_METADATA", From a7b5d72caa7e742be466dba2fea83e96ca920bcc Mon Sep 17 00:00:00 2001 From: Yaacov Date: Tue, 17 Nov 2026 03:07:12 +0200 Subject: [PATCH 2/2] docs(privacy): usage is no longer copied to PostHog; name built-in browser sites Co-Authored-By: Claude Opus 5.5 --- src/pages/privacy.astro | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/pages/privacy.astro b/src/pages/privacy.astro index 6977a8f..9896ae9 100644 --- a/src/pages/privacy.astro +++ b/src/pages/privacy.astro @@ -9,7 +9,7 @@ import Layout from "../layouts/Layout.astro"; description="How ScientFactory handles website, Desktop analytics, phone app and Scient Connect information." > @@ -34,7 +34,7 @@ import Layout from "../layouts/Layout.astro";

Scient Desktop

- Scient collects usage, reliability, account, device and network information to understand how people use the app and improve it. This includes which features, agents and models are used and whether they work as expected. + Scient collects usage, reliability, account, device and network information to understand how people use the app and improve it. This includes which features, agents and models are used, whether they work as expected, and the names of the websites opened in Scient's built-in browser.

Where available, account information includes provider sign-in emails and reported plans. Device and network information helps us associate installations and understand where the app is used. You may also optionally tell us about your work, field and role, or skip these questions. @@ -88,7 +88,7 @@ import Layout from "../layouts/Layout.astro";

Storage and retention

- Website and Desktop analytics are stored in ScientFactory's Cloudflare database. Ordinary usage information may also be processed in our EU-hosted PostHog project. Account details, optional work-profile answers, device identifiers and network observations stay in Cloudflare and are excluded from those exports. + Website and Desktop analytics are stored in ScientFactory's Cloudflare database. Until 17 November 2026, ordinary usage information was also copied to our EU-hosted PostHog project; we no longer send it there.

Detailed installation usage is retained while the installation remains active and for two years after its last activity. Diagnostic and failure records expire after 60 days. Aggregate totals without personal identifiers may be retained indefinitely.