diff --git a/docs/docs/reference/telemetry.md b/docs/docs/reference/telemetry.md index d09077da66..c5e1639caf 100644 --- a/docs/docs/reference/telemetry.md +++ b/docs/docs/reference/telemetry.md @@ -85,8 +85,10 @@ We collect the following categories of events: | `rules_learning_changed` | The person turned learning from sessions on or off for a project, or back to the workspace's setting, with the resulting state and who decides it (`organization`, `you`, `workspace` or `default`). | | `rules_command` | `/rules` opened in a client (`view`), or a `learn` subcommand run (its name only, never its arguments). | | `lineage_report` | What a person did with the Claude Code plugin's lineage report, sent by the plugin through `altimate-code tool report-event`: the end-of-task summary was shown, **Open report** was pressed, `/lineage-report` was run, or the report was built, failed to build or found no browser to open in. Its payload is the action and the Claude Code session id, beside the fields every event carries (below): no model, column, file or project names. The report page itself makes no network requests. | +| `plugin_session_start` | A Claude Code session started with the Altimate Code plugin, sent by the plugin through `altimate-code tool report-event`. Its payload is the plugin version, this altimate-code's version (`altimate_code_version`), the OS as Claude Code names it (`host_os`: `macos`, `linux`, `windows`, `wsl` or `unknown`; named apart from the `os` other events carry as `process.platform`) and the plugin's anonymous install id (a random UUID the plugin makes on first run and keeps in its Claude Code plugin store). It carries no session id, machine id, project id or account (`user_id`, `tenant`), nor the per-launch `launch_id`, so it cannot be joined to the CLI's other events: beside its own fields, only `cli_version`, `client` and `source`, keyed to the install id (`ai.user.id`). It is sent in a request of its own, never in one batch with the CLI's other events. | +| `plugin_feature_used` | One of the plugin's panels or commands was opened (`lineage`, `dbt_show`, `profile_ctes`, `scan_cost`, `comments`, `sql_graph`, `health`, `runs`, `to_model`), or one of its gates held or allowed a command (`cost_gate`, `health_push_gate`). Its payload is the feature, the outcome (`opened`; `failed` when the panel could not open; `held` or `allowed` for a gate), the plugin version and the install id: no model, column, file or project names, SQL or session text. Like `plugin_session_start`, it carries no session id, machine id, project id, launch id or account: beside its own fields, only `cli_version`, `client` and `source`, keyed to the install id. | -Each event includes a timestamp, session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and a machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and not derived from your hardware or identity). When the machine is signed in to Altimate, each event also carries `user_id` and `tenant`, described under [User Identification](#new-user-identification). +Each event (except the plugin's `plugin_session_start` and `plugin_feature_used`, described above) includes a timestamp, session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and a machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and not derived from your hardware or identity). The final `telemetry_opted_out`, in the same table, is the other exception: it carries the timestamp, CLI version and machine ID, but no session ID, `launch_id` or account. When the machine is signed in to Altimate, each event also carries `user_id` and `tenant`, described under [User Identification](#new-user-identification). ### Workspace events @@ -228,6 +230,8 @@ What identifies the sender of an event depends on whether the machine is signed - **Not signed in:** A random UUID is generated using `crypto.randomUUID()` on first run and stored at `~/.altimate/machine-id`. This ID is not tied to your hardware, operating system, or identity — it is purely random and serves only to distinguish one machine from another in aggregate analytics. It is also the Application Insights user ID (`ai.user.id`). - **Signed in to Altimate:** Events also carry `user_id`, your Altimate user ID (a UUID), and `tenant`, the name of your Altimate instance; `ai.user.id` is your user ID instead of the machine ID. These identify your account, so signed-in events are not anonymous. To find them, Altimate Code makes one `GET /users/me` request to the Altimate API with the credential you signed in with, and caches the answer at `~/.altimate/telemetry-identity.json` (readable only by you). The cache holds the user ID, the tenant and a fingerprint of the credential — no API key or token. A new credential is looked up again; a sign-out, or a credential the API rejects, removes the cache. No email address, hashed or not, is sent. +The Claude Code plugin's `plugin_session_start` and `plugin_feature_used` are the exception: signed in or not, they carry neither the machine ID nor your account, and their `ai.user.id` is the plugin's own random install id. + These identifiers are sent only while telemetry is enabled. The final `telemetry_opted_out` event carries the machine ID and nothing that identifies your account, and no `/users/me` lookup is made while telemetry is disabled. Disable telemetry entirely with `ALTIMATE_TELEMETRY_DISABLED=true` or the config option above. ### CLI Authentication Flow diff --git a/docs/docs/usage/tool.md b/docs/docs/usage/tool.md index d630c6ebd0..de1146e04f 100644 --- a/docs/docs/usage/tool.md +++ b/docs/docs/usage/tool.md @@ -84,3 +84,5 @@ For clients that cannot use the `altimate-code` MCP server, the Claude Code plug ## Plugin events `altimate-code tool report-event --session claude:` is called by the Claude Code plugin to record what a person did with its lineage report. `` is one of `summary_shown`, `open_pressed`, `command_run`, `built`, `build_failed` or `no_browser`. It sends one `lineage_report` usage event (see [Telemetry](../reference/telemetry.md)) and nothing when telemetry is off. It is not meant to be run by hand. + +The plugin also calls `altimate-code tool report-event plugin_session_start --plugin-version --install-id --os ` at each session start, and `altimate-code tool report-event plugin_feature_used --plugin-version --install-id --feature --outcome ` when a panel or command opens (or its panel fails to open) or a gate holds or allows a command. They take no `--session`. Each value is checked against a fixed list or pattern, and any other flag is refused, so the events carry only those fields (see [Telemetry](../reference/telemetry.md)). diff --git a/packages/opencode/src/altimate/telemetry/index.ts b/packages/opencode/src/altimate/telemetry/index.ts index 6ecb4cfead..5df4d68dc3 100644 --- a/packages/opencode/src/altimate/telemetry/index.ts +++ b/packages/opencode/src/altimate/telemetry/index.ts @@ -5,6 +5,8 @@ import { Log } from "@/altimate/util/log" import { TraceContext } from "@/altimate/observability/trace-context" // altimate_change — shared machine-id helper (race-safe, UUID-validated, size-capped) import { getOrCreateMachineId } from "@/altimate/util/machine-id" +// altimate_change — the Claude Code plugin's actions, whose events are sent anonymous (see ANONYMOUS_EVENTS) +import { PLUGIN_ACTIONS } from "@/altimate/tool-cli/report-event" import { TelemetryIdentity } from "./identity" import { OptOutMarker } from "./opt-out" import { TelemetryConfig } from "./config-state" @@ -928,6 +930,39 @@ export namespace Telemetry { action: "summary_shown" | "open_pressed" | "command_run" | "built" | "build_failed" | "no_browser" } // altimate_change end + // altimate_change start — the Claude Code plugin's own usage, sent through `tool report-event` + | { + /** A Claude Code session started with the plugin. Versions, OS and the plugin's random install id only: no session id. */ + type: "plugin_session_start" + timestamp: number + plugin_version: string + altimate_code_version: string + /** Claude Code's name for the OS; not `os`, which other events carry as `process.platform`. */ + host_os: "macos" | "linux" | "windows" | "wsl" | "unknown" + /** A random UUID the plugin made on first run and keeps in its Claude Code plugin store. */ + install_id: string + } + | { + /** A plugin panel or command was opened, or one of its gates held or allowed a command. */ + type: "plugin_feature_used" + timestamp: number + plugin_version: string + install_id: string + feature: + | "lineage" + | "dbt_show" + | "profile_ctes" + | "scan_cost" + | "comments" + | "sql_graph" + | "health" + | "runs" + | "to_model" + | "cost_gate" + | "health_push_gate" + outcome: "opened" | "failed" | "held" | "allowed" + } + // altimate_change end | { // Sent before connecting, so an attempt that hangs or ends the process is still counted. type: "warehouse_connect_started" @@ -2383,6 +2418,12 @@ export namespace Telemetry { // altimate_change — `ids` lets a caller build envelopes for a machine and identity other than // the module's current state (the opt-out marker runs while telemetry is disabled and must not // touch the state an enabled init may be using concurrently). + // altimate_change start — the Claude Code plugin's usage events are anonymous and unlinkable: they carry + // none of the identifiers the CLI's other events share (account, machine id, project, session, launch id), + // so they cannot be joined to a signed-in row. They are keyed to the plugin's own random install id only. + const ANONYMOUS_EVENTS: ReadonlySet = new Set(PLUGIN_ACTIONS) + // altimate_change end + function toAppInsightsEnvelopes( events: Event[], cfg: AppInsightsConfig, @@ -2400,7 +2441,11 @@ export namespace Telemetry { const clientSource = Flag.ALTIMATE_CLI_CLIENT return events.map((event) => { const { type, timestamp, ...fields } = event as any - const sid: string = fields.session_id ?? sessionId + // altimate_change start — the plugin's anonymous events: no account, machine id, project, session or launch id. + const anonymous = ANONYMOUS_EVENTS.has(type) + const account = anonymous ? undefined : who + const sid: string = anonymous ? "" : (fields.session_id ?? sessionId) + // altimate_change end const properties: Record = { cli_version: InstallationVersion, @@ -2408,17 +2453,19 @@ export namespace Telemetry { // still says how this CLI was launched (cli, tui, serve, datamates). client: "altimate-code", source: clientSource, - project_id: fields.project_id ?? projectId, - ...(mid && { machine_id: mid }), + ...(!anonymous && { project_id: fields.project_id ?? projectId }), + ...(mid && !anonymous && { machine_id: mid }), // altimate_change — the backend user uuid and tenant when signed in to Altimate, so a CLI // row joins the PostHog person and the backend record for the same account. - ...(who && { user_id: who.user_id, tenant: who.tenant }), + ...(account && { user_id: account.user_id, tenant: account.tenant }), // altimate_change — groups every event from one process launch. The onboarding funnel // spans the TUI main thread and the server worker, and most of it runs before any chat // session exists, so `session_id` is empty for the first half and real for the second — // leaving no key to join a single run on. This is not persisted, not derived from the // machine or the user, and not reused across launches; it only says "same run". - launch_id: launchId(), + // altimate_change — not on the plugin's anonymous events: shared with the same process's other rows + // (a first_launch carries the machine id), it would join them. + ...(!anonymous && { launch_id: launchId() }), } const measurements: Record = {} @@ -2438,15 +2485,15 @@ export namespace Telemetry { time: new Date(timestamp).toISOString(), iKey: cfg.iKey, tags: { - "ai.session.id": sid || "startup", + "ai.session.id": sid || (anonymous ? "" : "startup"), // altimate_change start — the backend user uuid when signed in, else the machine id. // Anonymous machines each keep a distinct identity (the random machine UUID, already a // custom property) instead of sharing ai.user.id="". No email, hashed or not, is sent. - "ai.user.id": who?.user_id || mid || "", + "ai.user.id": anonymous ? String(fields.install_id ?? "") : account?.user_id || mid || "", // altimate_change end "ai.cloud.role": "altimate", "ai.application.ver": InstallationVersion, - ...(fields._operation_id && { "ai.operation.id": fields._operation_id }), + ...(fields._operation_id && !anonymous && { "ai.operation.id": fields._operation_id }), }, data: { baseType: "EventData", @@ -3009,15 +3056,27 @@ export namespace Telemetry { const controller = new AbortController() activeFlushAbort = controller const timeout = setTimeout(() => controller.abort(), timeoutMs) + // altimate_change start — the plugin's anonymous events go in a request of their own, never in one + // body beside rows that carry the machine id or the account. Only an unsent batch is retried. + const batches = [events.filter((e) => !ANONYMOUS_EVENTS.has(e.type)), events.filter((e) => ANONYMOUS_EVENTS.has(e.type))].filter( + (batch) => batch.length > 0, + ) + const sent = new Set() + const attempted = new Set() + // altimate_change end try { - const response = await fetch(appInsights.endpoint, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(toAppInsightsEnvelopes(events, appInsights, { machineId: mid, identity: who })), - signal: controller.signal, - }) - if (!response.ok) { - log.debug("telemetry flush failed", { status: response.status }) + for (const batch of batches) { + for (const e of batch) attempted.add(e) + const response = await fetch(appInsights.endpoint, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(toAppInsightsEnvelopes(batch, appInsights, { machineId: mid, identity: who })), + signal: controller.signal, + }) + for (const e of batch) sent.add(e) + if (!response.ok) { + log.debug("telemetry flush failed", { status: response.status }) + } } } catch { // altimate_change — no write-back during shutdown. The buffer is cleared a few lines later @@ -3025,9 +3084,10 @@ export namespace Telemetry { // shipped by whatever lifecycle comes next, under a different launch id. if (shuttingDown) return // Re-add events that haven't been retried yet to avoid data loss - const retriable = events.filter((e) => !(e as any)._retried) + const retriable = events.filter((e) => !sent.has(e) && !(e as any)._retried) + // altimate_change — a batch never attempted (an earlier one failed) keeps its retry. for (const e of retriable) { - ;(e as any)._retried = true + if (attempted.has(e)) (e as any)._retried = true } const space = Math.max(0, MAX_BUFFER_SIZE - buffer.length) buffer.unshift(...retriable.slice(0, space)) diff --git a/packages/opencode/src/altimate/tool-cli/report-event.ts b/packages/opencode/src/altimate/tool-cli/report-event.ts index 2e31df7461..0cbb1f97e8 100644 --- a/packages/opencode/src/altimate/tool-cli/report-event.ts +++ b/packages/opencode/src/altimate/tool-cli/report-event.ts @@ -1,21 +1,125 @@ // altimate_change - new file // -// `tool report-event`: the Claude Code plugin records what a person did with its lineage report, as one -// usage-telemetry event. Only the action and the host session are sent: no model, column, file or project -// names. The plugin's page itself stays offline; this runs in the CLI, under its telemetry settings, so the -// usual opt-out turns it off. +// `tool report-event`: the Claude Code plugin records usage as one usage-telemetry event each: what a +// person did with its lineage report, a session start (plugin and altimate-code versions, OS, the +// plugin's anonymous install id), or a feature used (which panel, command or gate, and its outcome). +// Each event is built from an allow-list of fields, each checked against a fixed pattern or list, so no +// model, column, file or project name, SQL or session text can ride along. This runs in the CLI, under +// its telemetry settings, so the usual opt-out turns it off. + +import { InstallationVersion } from "@opencode-ai/core/installation/version" export const REPORT_ACTIONS = ["summary_shown", "open_pressed", "command_run", "built", "build_failed", "no_browser"] as const export type ReportAction = (typeof REPORT_ACTIONS)[number] +/** The plugin's own usage actions. */ +export const PLUGIN_ACTIONS = ["plugin_session_start", "plugin_feature_used"] as const +export type PluginAction = (typeof PLUGIN_ACTIONS)[number] + +/** The plugin's panels and commands, and its two gates. */ +export const PLUGIN_FEATURES = [ + "lineage", + "dbt_show", + "profile_ctes", + "scan_cost", + "comments", + "sql_graph", + "health", + "runs", + "to_model", + "cost_gate", + "health_push_gate", +] as const +export type PluginFeature = (typeof PLUGIN_FEATURES)[number] + +/** A panel opened or could not open; a gate held a command or let it through. */ +export const PLUGIN_OUTCOMES = ["opened", "failed", "held", "allowed"] as const +export type PluginOutcome = (typeof PLUGIN_OUTCOMES)[number] + +/** The OS as Claude Code names it. */ +export const PLUGIN_OS = ["macos", "linux", "windows", "wsl", "unknown"] as const +export type PluginOs = (typeof PLUGIN_OS)[number] + /** A Claude Code session id as the plugin passes it: `claude:` and the host's own id. */ const HOST_SESSION = /^claude:[A-Za-z0-9_-]{1,128}$/ +/** A SemVer 2.0 version (prerelease and build metadata included), up to 64 characters, or `unknown` when the plugin could not read its own. */ +const NUMBER = "(?:0|[1-9]\\d*)" +const PRERELEASE_PART = "(?:0|[1-9]\\d*|\\d*[A-Za-z-][0-9A-Za-z-]*)" +const PLUGIN_VERSION = new RegExp( + `^(?:${NUMBER}\\.${NUMBER}\\.${NUMBER}(?:-${PRERELEASE_PART}(?:\\.${PRERELEASE_PART})*)?(?:\\+[0-9A-Za-z-]+(?:\\.[0-9A-Za-z-]+)*)?|unknown)$`, +) +const MAX_VERSION = 64 +/** The plugin's install id: a random (version 4) UUID it made on first run, lowercase. */ +const INSTALL_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/ export type ReportEvent = { type: "lineage_report"; timestamp: number; session_id: string; action: ReportAction } +export type PluginSessionStartEvent = { + type: "plugin_session_start" + timestamp: number + plugin_version: string + altimate_code_version: string + /** Claude Code's name for the OS; not `os`, which other events carry as `process.platform`. */ + host_os: PluginOs + install_id: string +} +export type PluginFeatureUsedEvent = { + type: "plugin_feature_used" + timestamp: number + plugin_version: string + install_id: string + feature: PluginFeature + outcome: PluginOutcome +} +export type UsageEvent = ReportEvent | PluginSessionStartEvent | PluginFeatureUsedEvent + +/** The fields the plugin's actions take, as the command line gives them; a field the action does not take is refused. */ +export type PluginFields = { pluginVersion?: unknown; installId?: unknown; os?: unknown; feature?: unknown; outcome?: unknown } + +const ACTIONS: readonly string[] = [...REPORT_ACTIONS, ...PLUGIN_ACTIONS] +/** The fields each kind of action takes; any other one given is refused. */ +const TAKES: Record<"lineage" | PluginAction, readonly (keyof PluginFields)[]> = { + lineage: [], + plugin_session_start: ["pluginVersion", "installId", "os"], + plugin_feature_used: ["pluginVersion", "installId", "feature", "outcome"], +} +const FLAG: Record = { + pluginVersion: "--plugin-version", + installId: "--install-id", + os: "--os", + feature: "--feature", + outcome: "--outcome", +} +const oneOf = (list: readonly T[], value: unknown): value is T => list.includes(value as T) + +/** + * The event for an action, or why there is none. A lineage report step takes the host session; the plugin's + * own actions take their fields and no session, so they cannot be joined to anything else the CLI sends. + */ +export function reportEvent( + action: unknown, + session: unknown, + now = Date.now(), + fields: PluginFields = {}, + cliVersion: string = InstallationVersion, +): UsageEvent | { error: string } { + if (!oneOf(ACTIONS, action)) return { error: `the action must be one of ${ACTIONS.join(", ")}` } + const plugin = oneOf(PLUGIN_ACTIONS, action) + if (plugin && session !== undefined) return { error: `${action} does not take --session` } + if (!plugin && (typeof session !== "string" || !HOST_SESSION.test(session))) return { error: "--session takes a Claude Code session id: claude:" } + const takes = TAKES[plugin ? action : "lineage"] + const extra = (Object.keys(FLAG) as (keyof PluginFields)[]).filter((key) => fields[key] !== undefined && !takes.includes(key)) + if (extra.length > 0) return { error: `${action} does not take ${extra.map((key) => FLAG[key]).join(", ")}` } + if (oneOf(REPORT_ACTIONS, action)) return { type: "lineage_report", timestamp: now, session_id: session as string, action } -/** The event for an action and a host session, or why there is none. */ -export function reportEvent(action: unknown, session: unknown, now = Date.now()): ReportEvent | { error: string } { - if (!REPORT_ACTIONS.includes(action as ReportAction)) return { error: `the action must be one of ${REPORT_ACTIONS.join(", ")}` } - if (typeof session !== "string" || !HOST_SESSION.test(session)) return { error: "--session takes a Claude Code session id: claude:" } - return { type: "lineage_report", timestamp: now, session_id: session, action: action as ReportAction } + if (typeof fields.pluginVersion !== "string" || fields.pluginVersion.length > MAX_VERSION || !PLUGIN_VERSION.test(fields.pluginVersion)) + return { error: "--plugin-version takes the plugin's version: a semver, or unknown" } + if (typeof fields.installId !== "string" || !INSTALL_ID.test(fields.installId)) return { error: "--install-id takes a lowercase random (version 4) UUID" } + const common = { timestamp: now, plugin_version: fields.pluginVersion, install_id: fields.installId } + if (action === "plugin_session_start") { + if (!oneOf(PLUGIN_OS, fields.os)) return { error: `--os must be one of ${PLUGIN_OS.join(", ")}` } + return { type: "plugin_session_start", ...common, altimate_code_version: cliVersion, host_os: fields.os } + } + if (!oneOf(PLUGIN_FEATURES, fields.feature)) return { error: `--feature must be one of ${PLUGIN_FEATURES.join(", ")}` } + if (!oneOf(PLUGIN_OUTCOMES, fields.outcome)) return { error: `--outcome must be one of ${PLUGIN_OUTCOMES.join(", ")}` } + return { type: "plugin_feature_used", ...common, feature: fields.feature, outcome: fields.outcome } } diff --git a/packages/opencode/src/cli/cmd/tool.ts b/packages/opencode/src/cli/cmd/tool.ts index 5ba4c2faf6..43389f4e3d 100644 --- a/packages/opencode/src/cli/cmd/tool.ts +++ b/packages/opencode/src/cli/cmd/tool.ts @@ -332,14 +332,34 @@ function writeAll(text: string) { const ToolReportEventCommand = cmd({ command: "report-event ", - describe: "record what a person did with the Claude Code plugin's lineage report (usage telemetry; nothing is sent when telemetry is off)", + describe: "record the Claude Code plugin's usage: its lineage report, a session start, a feature used (usage telemetry; nothing is sent when telemetry is off)", builder: (yargs: Argv) => yargs - .positional("action", { type: "string", describe: "what happened to the report", demandOption: true }) - .option("session", { type: "string", describe: "the Claude Code session: claude:", demandOption: true }), + .positional("action", { + type: "string", + describe: + "a lineage report step (summary_shown, open_pressed, command_run, built, build_failed, no_browser), plugin_session_start or plugin_feature_used", + demandOption: true, + }) + .option("session", { type: "string", describe: "the Claude Code session: claude: (lineage report steps only)" }) + .option("plugin-version", { type: "string", describe: "the plugin's version (plugin_* actions)" }) + .option("install-id", { type: "string", describe: "the plugin's random install id, a UUID (plugin_* actions)" }) + .option("os", { type: "string", describe: "macos, linux, windows, wsl or unknown (plugin_session_start)" }) + .option("feature", { + type: "string", + describe: + "lineage, dbt_show, profile_ctes, scan_cost, comments, sql_graph, health, runs, to_model, cost_gate or health_push_gate (plugin_feature_used)", + }) + .option("outcome", { type: "string", describe: "opened, failed, held or allowed (plugin_feature_used)" }), handler: async (args) => { const { reportEvent } = await import("../../altimate/tool-cli/report-event") - const event = reportEvent(args.action, args.session) + const event = reportEvent(args.action, args.session, Date.now(), { + pluginVersion: args.pluginVersion, + installId: args.installId, + os: args.os, + feature: args.feature, + outcome: args.outcome, + }) if ("error" in event) { console.error(event.error) process.exitCode = 1 diff --git a/packages/opencode/test/altimate/telemetry/identity.test.ts b/packages/opencode/test/altimate/telemetry/identity.test.ts index 25735b8a72..df63ec5995 100644 --- a/packages/opencode/test/altimate/telemetry/identity.test.ts +++ b/packages/opencode/test/altimate/telemetry/identity.test.ts @@ -47,6 +47,8 @@ describe("telemetry identity", () => { let gates: Record | undefined> = {} /** Resolved when a /users/me request for that key has reached its gate. */ let entered: Record void> = {} + /** The next this-many /v2/track posts fail with a network error. */ + let failTrackPosts = 0 function writeCreds(apiKey = "key-1") { mkdirSync(path.join(home, ".altimate"), { recursive: true }) @@ -77,11 +79,16 @@ describe("telemetry identity", () => { meBody = undefined gates = {} entered = {} + failTrackPosts = 0 // The auth store lives in the real data directory; never let a test read it. authSpy = spyOn(Auth, "get").mockResolvedValue(undefined) fetchSpy = spyOn(globalThis, "fetch").mockImplementation((async (input: any, init?: RequestInit) => { const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url calls.push({ url, method: init?.method ?? "GET", headers: headersOf(init), body: String(init?.body ?? "") }) + if (url.includes("/v2/track") && failTrackPosts > 0) { + failTrackPosts-- + throw new Error("network down") + } if (url === `${API}/users/me` || url === `${DEFAULT_API}/users/me`) { const key = headersOf(init)["authorization"]?.replace("Bearer ", "") ?? "" const gate = gates[key] @@ -213,6 +220,72 @@ describe("telemetry identity", () => { expect(last.headers["authorization"]).toBe("Bearer key-F") }) + test("signed in, the Claude Code plugin's usage events share no identifier with the account's events: no user_id, tenant, machine id, project or session", async () => { + process.env.APPLICATIONINSIGHTS_CONNECTION_STRING = + "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingest.example.invalid/" + writeCreds("key-A") + await Telemetry.shutdown() + await Telemetry.init() + await Telemetry.refreshIdentity() + expect(Telemetry.identityForTest()).toEqual({ user_id: UUID, tenant: "acme" }) + const common = { timestamp: Date.now(), plugin_version: "1.8.2", install_id: UUID_B } + Telemetry.track({ type: "plugin_session_start", ...common, altimate_code_version: "0.13.0", host_os: "macos" }) + Telemetry.track({ type: "plugin_feature_used", ...common, feature: "lineage", outcome: "opened" }) + Telemetry.track({ type: "startup_ready", session_id: "claude:abc", command: "a", surface: "a", duration_ms: 1, fresh_install: false, timestamp: Date.now() }) + await Telemetry.flush() + // Tracked together, sent apart: no request body holds a plugin row beside the account's. + const bodies = trackCalls().map((c) => JSON.parse(c.body) as any[]) + const isPlugin = (e: any) => String(e.data?.baseData?.name).startsWith("plugin_") + expect(bodies.filter((body) => body.some(isPlugin))).toHaveLength(1) + for (const body of bodies) expect(body.every(isPlugin) || !body.some(isPlugin)).toBe(true) + const sent = bodies.flat() + const named = (name: string) => sent.find((e) => e.data?.baseData?.name === name) + const account = named("startup_ready") + // The account's own row, as before: user, tenant and machine id. + expect(account.data.baseData.properties.user_id).toBe(UUID) + expect(account.tags["ai.user.id"]).toBe(UUID) + const machine = account.data.baseData.properties.machine_id + expect(machine).toMatch(/^[0-9a-f-]{36}$/) + for (const name of ["plugin_session_start", "plugin_feature_used"]) { + const event = named(name) + expect(event).toBeDefined() + for (const key of ["user_id", "tenant", "machine_id", "project_id", "session_id", "launch_id"]) expect(event.data.baseData.properties).not.toHaveProperty(key) + expect(event.tags["ai.user.id"]).toBe(UUID_B) + expect(event.tags["ai.session.id"]).toBe("") + const text = JSON.stringify(event) + // Nothing the account's row carries to join on, the process's launch id included. + const launch = account.data.baseData.properties.launch_id + expect(launch).toBeTruthy() + for (const shared of [UUID, "acme", machine, "claude:abc", launch]) expect(text).not.toContain(shared) + } + }) + + test("a plugin batch never attempted because the batch before it failed keeps its retry", async () => { + process.env.APPLICATIONINSIGHTS_CONNECTION_STRING = + "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingest.example.invalid/" + await Telemetry.shutdown() + await Telemetry.init() + Telemetry.track({ type: "plugin_feature_used", timestamp: Date.now(), plugin_version: "1.8.2", install_id: UUID_B, feature: "runs", outcome: "opened" }) + // Not an anchor event: nothing is sent until the flushes below. + Telemetry.track({ type: "rules_command", timestamp: Date.now(), session_id: "claude:abc", command: "view" }) + // Twice the account's batch fails first, so the plugin's is not tried; the third flush sends it. + failTrackPosts = 1 + await Telemetry.flush() + failTrackPosts = 1 + await Telemetry.flush() + await Telemetry.flush() + expect(trackCalls().map((c) => (JSON.parse(c.body) as any[]).map((e) => e.data?.baseData?.name))).toEqual([ + ["rules_command"], + ["rules_command"], + ["plugin_feature_used"], + ]) + const delivered = trackCalls() + .slice(-1) + .flatMap((c) => JSON.parse(c.body) as any[]) + .map((e) => e.data?.baseData?.name) + expect(delivered).toEqual(["plugin_feature_used"]) + }) + test("a credential change on disk is noticed at the next flush, and a buffered batch keeps the identity it was tracked under", async () => { process.env.APPLICATIONINSIGHTS_CONNECTION_STRING = "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingest.example.invalid/" diff --git a/packages/opencode/test/altimate/tool-cli/report-event.test.ts b/packages/opencode/test/altimate/tool-cli/report-event.test.ts index 556d77cc46..454b4effc6 100644 --- a/packages/opencode/test/altimate/tool-cli/report-event.test.ts +++ b/packages/opencode/test/altimate/tool-cli/report-event.test.ts @@ -1,5 +1,17 @@ -import { describe, expect, test } from "bun:test" -import { REPORT_ACTIONS, reportEvent } from "../../../src/altimate/tool-cli/report-event" +import { afterAll, beforeAll, describe, expect, test } from "bun:test" +import fs from "node:fs/promises" +import os from "node:os" +import path from "node:path" +import { + PLUGIN_FEATURES, + PLUGIN_OS, + PLUGIN_OUTCOMES, + REPORT_ACTIONS, + reportEvent, +} from "../../../src/altimate/tool-cli/report-event" + +const SESSION = "claude:3f6c2a10-7b2e-4c1d-9a55-0e1f2a3b4c5d" +const INSTALL = "0f3c2a1e-6b7d-4c8e-9f10-1a2b3c4d5e6f" describe("tool report-event", () => { test("an action and a Claude Code session make one event with nothing else in it", () => { @@ -15,11 +27,242 @@ describe("tool report-event", () => { }) test("an action outside the list is refused", () => { - for (const action of ["clicked_column", "", undefined, "built; rm -rf /"]) expect(reportEvent(action, "claude:abc")).toHaveProperty("error") + for (const action of ["clicked_column", "", undefined, "built; rm -rf /", "plugin_installed"]) + expect(reportEvent(action, "claude:abc")).toHaveProperty("error") }) test("a session that is not claude: is refused, so no path or name can ride along", () => { for (const session of ["abc", "codex:abc", "claude:", "claude:/home/someone/project", "claude:a b", `claude:${"x".repeat(129)}`, undefined]) expect(reportEvent("built", session)).toHaveProperty("error") }) + + test("a lineage report step takes only its action and session: a plugin field with it is refused", () => { + expect(reportEvent("built", SESSION, 7, {})).toEqual({ type: "lineage_report", timestamp: 7, session_id: SESSION, action: "built" }) + expect(reportEvent("built", SESSION, 7, { pluginVersion: "1.8.2", installId: INSTALL })).toEqual({ error: "built does not take --plugin-version, --install-id" }) + }) +}) + +describe("tool report-event: the plugin's usage", () => { + test("a session start carries the plugin and altimate-code versions, the OS and the install id, and nothing else", () => { + for (const os of PLUGIN_OS) { + expect(reportEvent("plugin_session_start", undefined, 1000, { pluginVersion: "1.8.2", installId: INSTALL, os }, "0.13.4")).toEqual({ + type: "plugin_session_start", + timestamp: 1000, + plugin_version: "1.8.2", + altimate_code_version: "0.13.4", + host_os: os, + install_id: INSTALL, + }) + } + }) + + test("a feature used carries the feature and its outcome, and nothing else", () => { + expect(PLUGIN_FEATURES).toEqual([ + "lineage", + "dbt_show", + "profile_ctes", + "scan_cost", + "comments", + "sql_graph", + "health", + "runs", + "to_model", + "cost_gate", + "health_push_gate", + ]) + expect(PLUGIN_OUTCOMES).toEqual(["opened", "failed", "held", "allowed"]) + for (const feature of PLUGIN_FEATURES) + for (const outcome of PLUGIN_OUTCOMES) + expect(reportEvent("plugin_feature_used", undefined, 1000, { pluginVersion: "1.8.2-beta.1", installId: INSTALL, feature, outcome })).toEqual({ + type: "plugin_feature_used", + timestamp: 1000, + plugin_version: "1.8.2-beta.1", + install_id: INSTALL, + feature, + outcome, + }) + }) + + test("a plugin version that is not a semver is refused, so no name or path rides in it", () => { + for (const pluginVersion of [ + "", + "1.8", + "1.8.2 /Users/me/project", + "/opt/plugin", + "v1.8.2", + "01.8.2", + "1.8.2-", + "1.8.2-..", + "1.8.2-foo..bar", + "1.8.2-01", + "1.8.2+", + "1.8.2+a..b", + "1.8.2-" + "x".repeat(60), + undefined, + 182, + ]) + expect({ pluginVersion, event: reportEvent("plugin_session_start", undefined, 1, { pluginVersion, installId: INSTALL, os: "macos" }) }).toMatchObject({ + pluginVersion, + event: { error: expect.any(String) }, + }) + for (const pluginVersion of ["unknown", "0.0.0", "1.8.2", "20240101.0.0", "1.0.1000000", "1.8.2-beta.1", "1.8.2-0.3.7", "1.8.2-x-y.7z", "1.8.2+build.7", "1.8.2-rc.1+sha.5114f85"]) + expect(reportEvent("plugin_session_start", undefined, 1, { pluginVersion, installId: INSTALL, os: "macos" })).not.toHaveProperty("error") + }) + + test("a plugin action takes no session: its events cannot be joined to the CLI's other events about that session", () => { + for (const action of ["plugin_session_start", "plugin_feature_used"]) + expect(reportEvent(action, SESSION, 1, { pluginVersion: "1.8.2", installId: INSTALL, os: "macos", feature: "health", outcome: "opened" })).toHaveProperty("error") + expect(reportEvent("plugin_session_start", SESSION, 1, { pluginVersion: "1.8.2", installId: INSTALL, os: "macos" })).toEqual({ error: "plugin_session_start does not take --session" }) + }) + + test("each plugin action refuses a field it does not take", () => { + expect(reportEvent("plugin_session_start", undefined, 1, { pluginVersion: "1.8.2", installId: INSTALL, os: "macos", feature: "lineage" })).toEqual({ + error: "plugin_session_start does not take --feature", + }) + expect(reportEvent("plugin_feature_used", undefined, 1, { pluginVersion: "1.8.2", installId: INSTALL, os: "macos", feature: "lineage", outcome: "opened" })).toEqual({ + error: "plugin_feature_used does not take --os", + }) + }) + + test("an install id that is not a lowercase UUID is refused", () => { + // Version 0 and a non-RFC variant are not what crypto.randomUUID makes. + for (const installId of ["", "machine", "orders_model", INSTALL.toUpperCase(), `${INSTALL}x`, "../../etc/passwd", "0f3c2a1e-6b7d-0c8e-9f10-1a2b3c4d5e6f", "0f3c2a1e-6b7d-4c8e-7f10-1a2b3c4d5e6f", undefined]) + expect(reportEvent("plugin_feature_used", undefined, 1, { pluginVersion: "1.8.2", installId, feature: "health", outcome: "opened" })).toHaveProperty("error") + }) + + test("an OS, feature or outcome outside its list is refused", () => { + for (const os of ["darwin", "Linux", "", undefined, "macos; select 1"]) + expect(reportEvent("plugin_session_start", undefined, 1, { pluginVersion: "1.8.2", installId: INSTALL, os })).toHaveProperty("error") + for (const feature of ["fct_orders", "select * from t", "lineage-report", "", undefined]) + expect(reportEvent("plugin_feature_used", undefined, 1, { pluginVersion: "1.8.2", installId: INSTALL, feature, outcome: "opened" })).toHaveProperty("error") + for (const outcome of ["clicked", "", undefined, "opened /repo/models"]) + expect(reportEvent("plugin_feature_used", undefined, 1, { pluginVersion: "1.8.2", installId: INSTALL, feature: "runs", outcome })).toHaveProperty("error") + }) +}) + +// The real CLI, its telemetry pointed at a local sink: what reaches Application Insights, and that the opt-out sends nothing. +describe("tool report-event through the CLI's telemetry", () => { + const cliEntry = path.resolve(import.meta.dir, "../../../src/index.ts") + let root: string + let server: ReturnType + let received: any[] = [] + /** Each request's body, as the sink got it. */ + let requests: any[][] = [] + + beforeAll(async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), "report-event-")) + server = Bun.serve({ + port: 0, + async fetch(req) { + if (new URL(req.url).pathname === "/v2/track") { + const body = await req.json().catch(() => []) + const rows = Array.isArray(body) ? body : [body] + requests.push(rows) + received.push(...rows) + } + return Response.json({ itemsReceived: 1, itemsAccepted: 1, errors: [] }) + }, + }) + }) + + afterAll(async () => { + server.stop(true) + await fs.rm(root, { recursive: true, force: true }) + }) + + async function run(args: string[], extra: Record = {}, installed = false) { + const home = await fs.mkdtemp(path.join(root, "home-")) + // Just installed: the postinstall marker makes this process send first_launch, with the machine id. + if (installed) { + await fs.mkdir(path.join(home, ".local/share/altimate-code"), { recursive: true }) + await fs.writeFile(path.join(home, ".local/share/altimate-code/.installed-version"), "0.13.0") + } + const env: Record = { + ...(process.env as Record), + OPENCODE_TEST_HOME: home, + HOME: home, + XDG_CONFIG_HOME: path.join(home, ".config"), + XDG_DATA_HOME: path.join(home, ".local/share"), + XDG_STATE_HOME: path.join(home, ".local/state"), + XDG_CACHE_HOME: path.join(home, ".cache"), + OPENCODE_DISABLE_AUTOUPDATE: "1", + OPENCODE_DISABLE_MODELS_FETCH: "1", + APPLICATIONINSIGHTS_CONNECTION_STRING: `InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=http://127.0.0.1:${server.port}/`, + ...extra, + } + delete env.ALTIMATE_TELEMETRY_DISABLED + delete env.OPENCODE_DISABLE_TELEMETRY + Object.assign(env, extra) + const proc = Bun.spawn(["bun", "run", "--conditions=browser", cliEntry, "tool", "report-event", ...args], { cwd: home, env, stdout: "pipe", stderr: "pipe" }) + const [exitCode, stderr] = await Promise.all([proc.exited, new Response(proc.stderr).text()]) + return { exitCode, stderr } + } + + const plugin = () => received.filter((one) => String(one?.data?.baseData?.name ?? "").startsWith("plugin_")) + + test( + "a session start and a feature used each send one event, with the allowed fields only", + async () => { + received = [] + requests = [] + const start = await run(["plugin_session_start", "--plugin-version", "1.8.2", "--install-id", INSTALL, "--os", "macos"], {}, true) + expect(start.exitCode).toBe(0) + const used = await run(["plugin_feature_used", "--plugin-version", "1.8.2", "--install-id", INSTALL, "--feature", "cost_gate", "--outcome", "held"]) + expect(used.exitCode).toBe(0) + const events = plugin() + expect(events.map((one) => one.data.baseData.name)).toEqual(["plugin_session_start", "plugin_feature_used"]) + // The envelope's own keys, and the event's allow-listed fields: nothing else. No machine id, project, session or launch id. + const ENVELOPE = ["cli_version", "client", "source"] + const [startEvent, usedEvent] = events.map((one) => one.data.baseData.properties as Record) + expect(Object.keys(startEvent!).sort()).toEqual([...ENVELOPE, "plugin_version", "altimate_code_version", "host_os", "install_id"].sort()) + expect(Object.keys(usedEvent!).sort()).toEqual([...ENVELOPE, "plugin_version", "install_id", "feature", "outcome"].sort()) + expect(startEvent).toMatchObject({ plugin_version: "1.8.2", host_os: "macos", install_id: INSTALL }) + expect(startEvent!.altimate_code_version).toBe(startEvent!.cli_version) + expect(usedEvent).toMatchObject({ feature: "cost_gate", outcome: "held", install_id: INSTALL }) + // A fresh home: the same processes sent first_launch rows with the machine id. Nothing joins the plugin's rows to them. + const others = received.filter((one) => !String(one?.data?.baseData?.name ?? "").startsWith("plugin_")) + expect(others.some((one) => one.data.baseData.name === "first_launch")).toBe(true) + // Nor does any request carry a plugin row beside another. + const isPlugin = (one: any) => String(one?.data?.baseData?.name ?? "").startsWith("plugin_") + for (const rows of requests) expect(rows.every(isPlugin) || !rows.some(isPlugin)).toBe(true) + const joinable = others.flatMap((one) => [one.data.baseData.properties.machine_id, one.data.baseData.properties.launch_id, one.tags["ai.user.id"], one.tags["ai.session.id"]]) + for (const one of events) { + const text = JSON.stringify(one) + for (const value of joinable) if (value && value !== "startup") expect(text).not.toContain(value) + expect(one.tags["ai.session.id"]).toBe("") + expect(one.tags["ai.user.id"]).toBe(INSTALL) + expect(one.tags).not.toHaveProperty("ai.operation.id") + expect(JSON.stringify(one)).not.toMatch(/report-event-/) + } + }, + 60_000, + ) + + test( + "with ALTIMATE_TELEMETRY_DISABLED, or telemetry off in the config, nothing is sent", + async () => { + received = [] + const args = ["plugin_feature_used", "--plugin-version", "1.8.2", "--install-id", INSTALL, "--feature", "lineage", "--outcome", "opened"] + expect((await run(args, { ALTIMATE_TELEMETRY_DISABLED: "true" })).exitCode).toBe(0) + expect((await run(args, { OPENCODE_CONFIG_CONTENT: JSON.stringify({ telemetry: { disabled: true } }) })).exitCode).toBe(0) + expect(plugin()).toEqual([]) + }, + 60_000, + ) + + test( + "a value outside the allow-list, or a field the event does not take, sends nothing and says why", + async () => { + received = [] + const ok = ["plugin_feature_used", "--plugin-version", "1.8.2", "--install-id", INSTALL] + const bad = await run([...ok, "--feature", "fct_orders", "--outcome", "opened"]) + expect(bad.exitCode).toBe(1) + expect(bad.stderr).toContain("--feature must be one of") + const extra = await run([...ok, "--feature", "lineage", "--outcome", "opened", "--sql", "select secret from t"]) + expect(extra.exitCode).toBe(1) + expect(plugin()).toEqual([]) + expect(JSON.stringify(received)).not.toContain("select secret") + }, + 60_000, + ) }) diff --git a/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap b/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap index 5a59f74b6b..020fee6fb1 100644 --- a/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap +++ b/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap @@ -536,9 +536,9 @@ Commands: altimate-code tool run run one data tool without a model and print its result altimate-code tool daemon [action] run, inspect or stop the long-running tool process for a project - altimate-code tool report-event record what a person did with the Claude Code plugin's - lineage report (usage telemetry; nothing is sent when - telemetry is off) + altimate-code tool report-event record the Claude Code plugin's usage: its lineage + report, a session start, a feature used (usage + telemetry; nothing is sent when telemetry is off) Options: -h, --help show help [boolean]