feat(onboarding): first-run funnel telemetry

docs/docs/reference/telemetry.md

@@ -47,8 +47,32 @@ We collect the following categories of events:
 | `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
 | `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
 | `validator_retries_exhausted` | A session terminated with unresolved validator failures after exhausting the synthetic-retry budget — names of the failing validators (no failure body content). |
-
-Each event includes a timestamp, anonymous session ID, CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
+| `onboarding_started` | The first-run setup gate opened (fresh launch with no usable model). |
+| `model_picker_shown` | The provider picker was displayed. `trigger` distinguishes the first run from `/connect`, from declining Big Pickle, and from the prompt gate. |
+| `provider_selected` | A provider row was chosen (`altimate_gateway`, `anthropic`, `openai`, `google`, `big_pickle`, `search_all`). Recorded at the moment of choice, so a sign-in that is then cancelled still counts. |
+| `big_pickle_confirm_shown` / `big_pickle_choice` | The Big Pickle interstitial was shown, and what the user decided (`accept`/`cancel`). |
+| `gateway_device_code_issued` | The Altimate Gateway authorize URL was built and the browser open attempted. **Name note:** the flow is a browser loopback OAuth — there is no device code. The name follows the original event spec. |
+| `gateway_auth_completed` / `gateway_auth_failed` | Gateway sign-in outcome. `reason` is `timeout`, `denied`, or `error` — never the underlying message, which can contain the instance name. An unrecognised callback state does not reject the pending attempt, so a CSRF mismatch surfaces as `timeout`. |
+| `instance_connected` | Credentials received and saved. `time_to_connect_ms` runs from the start of the authorize call, so it includes the browser launch. No instance or tenant name is sent. |
+| `onboarding_completed` | A model is ready and chat is live. |
+| `scan_gate_shown` / `scan_gate_choice` | The "scan your environment?" gate appeared, and whether the user chose `scan` or `skip`. |
+| `environment_scan_completed` | A `project_scan` finished — `has_dbt`, `has_warehouse`, `is_repo`, `connections_found`, and a bounded list of short `degraded` detection keys. No paths, hostnames, or connection details. Fires for every scan, including `/discover`, not just onboarding. |
+| `sample_setup_completed` | The sample dbt project was materialised. `success`, `models`, `tables`, and `reused` — the tool is deliberately re-callable, so this is per invocation. The target path is never sent. |
+| `activation_menu_shown` | The activation menu was (very likely) rendered. `variant` is `warehouse` or `no_data`. **Derived** — see the note below. |
+| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. **Derived** — see the note below. |
+| `first_prompt_sent` | The user's first typed message in an onboarding session. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. |
+| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `big_pickle_confirm`, `gateway_auth`, or `connected`. Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. |
+
+### A note on the derived activation events
+
+`activation_menu_shown`, `activation_job_selected`, and `first_job_completed` are **inferred, not observed**. The activation menu is not a UI element: it is text the model writes from a prompt template, and the user picks a job by replying in free text. Nothing in the CLI can see either moment directly.
+
+They are therefore inferred from the closest deterministic signals — the menu from the command dispatch or the completed environment scan, the job from the first matching tool or skill invocation that follows. Treat the counts as **lower bounds**, and note two specific gaps:
+
+- The "something else" branch has no tool signature at all and is never counted.
+- `first_job_completed` only fires for jobs with a real completion signal. Skill-driven jobs (downstream impact, SQL review, cost) load an instruction bundle and then do their work through other tools, so their completion is not observable and they are absent from this event rather than wrongly counted in it.
+
+Each event includes a timestamp, anonymous session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
 
 ## Delivery & Reliability
 

packages/opencode/src/altimate/plugin/altimate.ts

@@ -3,6 +3,28 @@ import { createServer } from "http"
 import { randomBytes } from "crypto"
 import open from "open"
 import { AltimateApi } from "../api/client"
+// altimate_change — onboarding telemetry for the gateway sign-in funnel
+import * as OnboardingTelemetry from "../telemetry/onboarding"
+
+/**
+ * Why a failure reason is attached at the rejection site rather than inferred from the message:
+ * `gateway_auth_failed.reason` is a closed enum, and the callback's catch sees only an Error.
+ * Matching on message text would silently drift the moment any of these strings is reworded, and
+ * the `error` query param is attacker-influenced text we must not parse or forward. Tagging the
+ * error where the cause is known keeps the classification deterministic — and note that an
+ * unknown/invalid `state` never rejects a pending flow at all (the handler 400s without touching
+ * the map), so a CSRF mismatch legitimately surfaces later as `timeout`, not `denied`.
+ */
+type GatewayFailureReason = "timeout" | "denied" | "error"
+
+function markReason(err: Error, reason: GatewayFailureReason): Error {
+  return Object.assign(err, { altimateGatewayReason: reason })
+}
+
+function reasonOf(err: unknown): GatewayFailureReason {
+  const tagged = (err as { altimateGatewayReason?: GatewayFailureReason } | undefined)?.altimateGatewayReason
+  return tagged ?? "error"
+}
 
 // Loopback port the CLI listens on for the browser to deliver the gateway
 // credential after sign-in. Must match the redirect the web authorize page posts
@@ -121,7 +143,8 @@ async function startCallbackServer(): Promise<void> {
 
     const error = url.searchParams.get("error")
     if (error) {
-      entry.reject(new Error(error))
+      // altimate_change — the browser reported an explicit failure: the only true `denied` signal
+      entry.reject(markReason(new Error(error), "denied"))
       html(200, HTML_ERROR(error))
       return
     }
@@ -180,7 +203,8 @@ function stopCallbackServer() {
 function registerPending(state: string, timeoutMs = 5 * 60 * 1000): Promise<CallbackResult> {
   return new Promise<CallbackResult>((resolve, reject) => {
     const timeout = setTimeout(() => {
-      if (pending.delete(state)) reject(new Error("Timed out waiting for browser sign-in"))
+      // altimate_change — tag as `timeout` for gateway_auth_failed classification
+      if (pending.delete(state)) reject(markReason(new Error("Timed out waiting for browser sign-in"), "timeout"))
       // If the dialog was dismissed and callback() never ran its finally, the
       // loopback server would otherwise stay bound past the timeout. Free the
       // port once nothing is waiting on it.
@@ -209,7 +233,18 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
           label: "Altimate LLM Gateway",
           async authorize() {
             const state = randomBytes(16).toString("hex")
-            await startCallbackServer()
+            // altimate_change start — the attempt starts here, before the callback server and the
+            // browser open. startCallbackServer() throws when port 7317 is taken, which is a real
+            // and reasonably common gateway-auth failure that happens before any callback object
+            // exists — without this catch it would never appear in the funnel.
+            const startedAt = Date.now()
+            try {
+              await startCallbackServer()
+            } catch (err) {
+              void OnboardingTelemetry.emit({ type: "gateway_auth_failed", reason: "error" })
+              throw err
+            }
+            // altimate_change end
             // Register the pending flow BEFORE opening the browser so an instant
             // redirect can be matched by state rather than dropped as CSRF.
             const result = registerPending(state)
@@ -232,6 +267,20 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
 
             await open(authorizeUrl).catch(() => undefined)
 
+            // altimate_change start — onboarding funnel: gateway sign-in started.
+            // Spec name is `gateway_device_code_issued`; this flow is a browser loopback OAuth
+            // with no device code, so the event means "authorize URL built, browser open
+            // attempted". open() failures are swallowed above (the URL is also printed for the
+            // user to paste), so this fires even when no browser actually launched.
+            // The URL is never sent — it carries the CSRF `state`.
+            void OnboardingTelemetry.emit({ type: "gateway_device_code_issued" })
+
+            // One outcome per attempt. callback() closes over `result` and re-runs its whole body
+            // on every invocation, so a repeated call would otherwise re-emit completion/failure
+            // (and re-report a connect time measured from the original attempt).
+            let outcomeReported = false
+            // altimate_change end
+
             return {
               url: authorizeUrl,
               instructions: "Complete sign-in in your browser to connect Altimate LLM Gateway.",
@@ -255,11 +304,33 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
                     altimateInstanceName: creds.instance,
                     altimateApiKey: authToken,
                   })
+                  // altimate_change start — onboarding funnel: auth succeeded and the instance
+                  // is live. The instance name is the customer's tenant identifier and is never
+                  // sent. time_to_connect_ms runs from the start of authorize() — before the
+                  // callback server and browser open, both of which are part of the wait the user
+                  // actually experiences — and lives in this attempt's closure, so a concurrent
+                  // attempt cannot overwrite it.
+                  if (!outcomeReported) {
+                    outcomeReported = true
+                    void OnboardingTelemetry.emit({ type: "gateway_auth_completed" })
+                    void OnboardingTelemetry.emit({
+                      type: "instance_connected",
+                      time_to_connect_ms: Date.now() - startedAt,
+                    })
+                  }
+                  // altimate_change end
                   return { type: "success", key: authToken, provider: "altimate-backend" }
                 } catch (err) {
                   // Log the reason (CSRF / timeout / invalid instance / …). Runs in the
                   // server process, so this goes to the log, not the TUI display.
                   console.error("[altimate] gateway sign-in failed:", err instanceof Error ? err.message : err)
+                  // altimate_change — onboarding funnel: only the classified enum is sent. The
+                  // message can embed the instance name (see the invalid-instance throw above),
+                  // so it never reaches telemetry.
+                  if (!outcomeReported) {
+                    outcomeReported = true
+                    void OnboardingTelemetry.emit({ type: "gateway_auth_failed", reason: reasonOf(err) })
+                  }
                   return { type: "failed" }
                 } finally {
                   // Keep the shared server up while another flow is still waiting.

packages/opencode/src/altimate/plugin/onboarding-telemetry.ts

@@ -0,0 +1,138 @@
+// altimate_change start — activation-funnel telemetry, as a plugin.
+//
+// The three activation events are DERIVED. The activation menu is not UI: it is text the model
+// writes from src/command/template/onboard-connect.txt, and the user picks a job by replying in
+// free text. Nothing in the codebase observes either moment. What is deterministic is which
+// command started the session and which tool ran next, so that is what this infers from — and
+// the events are documented as lower bounds rather than exact counts.
+//
+// Implemented as a plugin rather than as edits inside session/prompt.ts because the hooks
+// already exist (`command.execute.before`, `tool.execute.after`) and carry everything needed.
+// This keeps the inference logic in one fork-owned file instead of scattering it across the
+// session loop.
+import type { Hooks, PluginInput } from "@opencode-ai/plugin"
+import * as OnboardingTelemetry from "../telemetry/onboarding"
+
+const ONBOARD_CONNECT = "onboard-connect"
+
+/** Tool/skill → the spec's activation job enum. */
+type ActivationJob = "sample_duck_db" | "breaks_downstream" | "sql_review" | "cost" | "something_else"
+
+const JOB_BY_TOOL: Record<string, ActivationJob> = {
+  sample_setup: "sample_duck_db",
+}
+
+const JOB_BY_SKILL: Record<string, ActivationJob> = {
+  "dbt-analyze": "breaks_downstream",
+  "sql-review": "sql_review",
+  "cost-report": "cost",
+}
+
+/**
+ * Which activation job, if any, a tool call represents. Skills arrive as the generic `skill`
+ * tool with the skill name in `args.name`, so both shapes have to be checked.
+ *
+ * Returns undefined for the many tools that are not jobs (reads, writes, project_scan itself).
+ * Note that `something_else` — the "just let me chat" branch — has no tool signature at all and
+ * therefore can never be detected here; that branch is systematically missing from these counts.
+ */
+function jobForTool(tool: string, args: unknown): ActivationJob | undefined {
+  if (tool === "skill") {
+    const name = (args as { name?: unknown } | undefined)?.name
+    if (typeof name === "string") return JOB_BY_SKILL[name]
+    return undefined
+  }
+  return JOB_BY_TOOL[tool]
+}
+
+/**
+ * Total warehouse connections a project_scan found, from its `metadata.connections` breakdown
+ * (`{existing, new_dbt, new_docker, new_env}` — there is no pre-summed total). All four count:
+ * a user whose only warehouse was discovered from a dbt profile, a docker compose file, or env
+ * vars still has a warehouse, and reporting `no_data` for them would send them down the
+ * sample-project branch of the activation menu.
+ */
+function countScanConnections(metadata: unknown): number {
+  const connections = (metadata as { connections?: Record<string, unknown> } | undefined)?.connections
+  if (!connections) return 0
+  return ["existing", "new_dbt", "new_docker", "new_env"].reduce((total, key) => {
+    const value = connections[key]
+    return total + (typeof value === "number" ? value : 0)
+  }, 0)
+}
+
+/**
+ * Whether a completed tool call is evidence the JOB finished, not merely that a tool returned.
+ *
+ * This distinction is the whole reason `first_job_completed` is narrower than
+ * `activation_job_selected`. `sample_setup` genuinely does the work and reports `metadata.success`.
+ * The `skill` tool does not: it loads an instruction bundle and returns, after which the agent
+ * does the actual analysis with other tools. Treating a successful skill load as job completion
+ * would report "downstream impact analysis complete" the moment the instructions were read.
+ *
+ * There is no reliable signal for when a skill-driven job finishes, so those jobs are absent from
+ * `first_job_completed` rather than wrong in it.
+ */
+function isJobCompletion(tool: string, output: { metadata?: unknown }): boolean {
+  if (tool !== "sample_setup") return false
+  return (output.metadata as { success?: unknown } | undefined)?.success !== false
+}
+
+export async function OnboardingTelemetryPlugin(_input: PluginInput): Promise<Hooks> {
+  return {
+    "command.execute.before": async (input) => {
+      // Mark the session BEFORE flagging the submission: noteCommandSubmission only touches
+      // sessions already tracked (so ordinary slash commands cannot churn the capped map), and
+      // /onboard-connect is the command that creates the record in the first place.
+      if (input.command === ONBOARD_CONNECT) OnboardingTelemetry.markOnboardingSession(input.sessionID)
+
+      // Any slash command means the next user message was not typed by the user — needed so
+      // `first_prompt_sent` measures a real first prompt rather than the scan gate's hidden
+      // `/onboard-connect` submission.
+      OnboardingTelemetry.noteCommandSubmission(input.sessionID)
+
+      if (input.command !== ONBOARD_CONNECT) return
+
+      // `skip` renders the menu immediately, with no scan to wait for, so the variant is known
+      // now. The `scan` branch cannot be resolved here — the menu follows the scan, and the
+      // variant depends on what the scan finds — so it is emitted from the scan result below.
+      if (input.arguments?.trim() === "skip" && OnboardingTelemetry.claimActivationMenu(input.sessionID)) {
+        void OnboardingTelemetry.emit({ type: "activation_menu_shown", variant: "no_data" }, input.sessionID)
+      }
+    },
+
+    "tool.execute.after": async (input, output) => {
+      if (!OnboardingTelemetry.isOnboardingSession(input.sessionID)) return
+
+      // The scan branch: the menu is rendered right after project_scan returns, and the variant
+      // is whatever the scan found. This is the closest deterministic proxy for the menu
+      // actually being shown — closer than the command dispatch, which happens before the agent
+      // has done anything and would over-count sessions that error out first.
+      if (input.tool === "project_scan" && OnboardingTelemetry.claimActivationMenu(input.sessionID)) {
+        void OnboardingTelemetry.emit(
+          {
+            type: "activation_menu_shown",
+            // `no_data` tracks the template's own menu variant, which keys on whether a warehouse
+            // connection exists. A dbt project with no warehouse gets the no-data menu too.
+            variant: countScanConnections(output.metadata) > 0 ? "warehouse" : "no_data",
+          },
+          input.sessionID,
+        )
+        return
+      }
+
+      const job = jobForTool(input.tool, input.args)
+      if (!job) return
+
+      // Selection lands on the first job-shaped tool call. A job that starts and then fails still
+      // counts as selected — the user did choose it.
+      if (OnboardingTelemetry.claimActivationJobSelected(input.sessionID)) {
+        void OnboardingTelemetry.emit({ type: "activation_job_selected", job }, input.sessionID)
+      }
+      if (isJobCompletion(input.tool, output) && OnboardingTelemetry.claimFirstJobCompleted(input.sessionID)) {
+        void OnboardingTelemetry.emit({ type: "first_job_completed", job }, input.sessionID)
+      }
+    },
+  }
+}
+// altimate_change end