feat(onboarding): first-run funnel telemetry - #1049
Draft
sahrizvi wants to merge 14 commits into
Draft
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
full receipts (4 sessions)
orchestrator ·
|
| subagent | cost |
|---|---|
| You are exploring the altimate-code monorepo to inform how a **preloaded sample… | 3,624,157 tokens |
| Task: survey the altimate-code repo for conventions I need to follow while fixi… | 2,039,125 tokens |
| You are exploring the altimate-code repo to inform the design of two new featur… | 2,460,388 tokens |
orchestrator · 3559fc02
- - - - - - - - - - - - - - - - - - - - - - - - -
AIRECEIPTS
“Investigate telemetry hooks implementation pl…”
Claude Code · Jul 29 2026 05:50 UTC · 4h 15m
claude-opus-5 100%
cache served 99% of input tokens
pre-edit: 5% of tokens (45/317 turns)
(share before the first named edit tool)
Bash...................53,016,584 tok (200 calls)
Edit....................27,693,201 tok (68 calls)
(thinking/reply).........9,889,096 tok (31 turns)
Read.....................6,149,396 tok (28 calls)
Write....................5,357,175 tok (14 calls)
ToolSearch..................567,411 tok (5 calls)
mcp__atlassian__addCommen…...511,535 tok (1 call)
TaskUpdate..................489,339 tok (3 calls)
TaskCreate..................288,417 tok (2 calls)
AskUserQuestion..............150,188 tok (1 call)
mcp__atlassian__getJiraIss…...52,719 tok (1 call)
--------------------------------------------------
TOTAL..............................104,165,061 tok
no price table matched
- - - - - - - - - - - - - - - - - - - - - - - - -
npx aireceipts-cli
github.com/anandgupta42/receipts
- - - - - - - - - - - - - - - - - - - - - - - - -
codex · 9f29ce8c
- - - - - - - - - - - - - - - - - - - - - - - - -
AIRECEIPTS
“commit b4252c5f699b70365b9b709d2ec468abca9456…”
Codex · Jul 29 2026 09:37:58 UTC · 3m 56s
gpt-5.5 100%
cache served 93% of input tokens
pre-edit: no named edit tool observed
(share before the first named edit tool)
exec_command.............1,368,318 tok (39 calls)
--------------------------------------------------
TOTAL................................1,368,318 tok
no price table matched
- - - - - - - - - - - - - - - - - - - - - - - - -
npx aireceipts-cli
github.com/anandgupta42/receipts
- - - - - - - - - - - - - - - - - - - - - - - - -
codex · 55530872
- - - - - - - - - - - - - - - - - - - - - - - - -
AIRECEIPTS
“eba46bfba0 fix(onboarding): TUI funnel review…”
Codex · Jul 29 2026 09:56:26 UTC · 2m 44s
gpt-5.5 100%
cache served 78% of input tokens
pre-edit: no named edit tool observed
(share before the first named edit tool)
exec_command...............352,832 tok (19 calls)
--------------------------------------------------
TOTAL..................................352,832 tok
no price table matched
- - - - - - - - - - - - - - - - - - - - - - - - -
npx aireceipts-cli
github.com/anandgupta42/receipts
- - - - - - - - - - - - - - - - - - - - - - - - -
Generated by aireceipts
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
8 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
sahrizvi
pushed a commit
that referenced
this pull request
Jul 30, 2026
…ness The Part 2 gate was triggered by useReady(), which is `connected() || setupComplete()`. `connected()` flips as soon as a provider lands in sync data — and the BYOK confirm handlers do exactly that inside `await sync.bootstrap()`, before going on to open the model picker (dialog-provider ApiMethod and CodeMethod). So for every BYOK provider the gate mounted mid-handler, the handler's own `dialog.replace(<DialogModel/>)` destroyed it a moment later, and the one-shot latch meant it never came back. `/onboard-connect` was never submitted, which made activation_menu_shown, activation_job_selected, first_job_completed, sample_setup_completed and first_prompt_sent unreachable on the majority onboarding path — while onboarding_completed and scan_gate_shown were still reported for a gate the user never saw. The gateway path was unaffected because its success branch does not replace the dialog, which is why this survived manual testing and the Big Pickle E2E. Now driven by setup completion alone, which is only set once a model is genuinely chosen — the model picker, the Big Pickle accept path, and the gateway auto-select. That is also what the spec means by "a model is ready". Reported in consensus review of #1049 (CRITICAL), flagged independently by two reviewers, with the ordering verified against sync.tsx: the store write is inside `batch()` within an awaited promise chain, two `.then()` hops before bootstrap() resolves, so the effect flushes before the dialog.replace under either effect-scheduling model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
sahrizvi
force-pushed
the
feat/onboarding-telemetry
branch
from
July 30, 2026 14:23
ecd94c8 to
447f8b6
Compare
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
sahrizvi
pushed a commit
that referenced
this pull request
Jul 30, 2026
Addresses the findings from the review that fall inside this PR. MAJOR — markSetupComplete() ran before the model-availability check (dialog-provider). The branch below it deliberately refuses to claim success when the gateway connects but offers nothing usable, and said so in a comment — but completion had already been marked, so telemetry reported a finished onboarding while the user was told to go pick a model. Now marked where the model is actually set. This matters more since the scan gate moved onto the same signal. MAJOR — the flushTimer leak. doInit() assigned a new interval without clearing the previous handle, and shutdown() only ever clears the current one, so a second doInit() stranded a timer for the life of the process. doShutdown also nulled initPromise unconditionally, discarding a doInit() that init() had chained onto the in-flight shutdown. Both fixed; see the note in doShutdown on why only the first is covered by a test. MAJOR — instance_connected and onboarding_abandoned could both be reported for one launch. The gateway success events are emitted on the worker thread while abandonment state is main-thread-owned, so a user who finished in the browser and quit before the TUI observed the new provider was reported as abandoning at gateway_auth. The exit path now checks whether credentials landed. MAJOR — environment_scan_completed fired on every project_scan, including /discover and any model-initiated call, so a funnel query could exceed 100% conversion. Now guarded on isOnboardingSession. MINOR — command.execute.before created a tracking record for every slash command in every session, churning the capped map and evicting genuine onboarding sessions, after which their remaining activation events were silently dropped. noteCommandSubmission now only touches sessions already tracked, and /onboard-connect marks the session before flagging its own submission. MINOR — the cross-package event parity test that two comments claimed but which did not exist. Now a compile-time assertion pinning the packages/tui event union to the Telemetry variants; verified by renaming a property and watching the build fail. Required exporting the context subpath from packages/tui. MINOR — a raw HOME path could reach LLM-visible tool output on a sample-setup failure. Paths are masked in `output`; the full message stays in metadata, which the model never sees. MINOR — documented why sample_setup_completed uses a success boolean instead of a _failed sibling event. NIT — launchId() no longer writes process.env. The worker receives the id explicitly through WorkerOptions.env, so the write only leaked it into every subprocess the CLI spawns. Also removes the stray `// scratch` line at the end of cli/cmd/tui.ts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
1 similar comment
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
Adds the onboarding funnel event taxonomy and wires the first six events, all emitted from the TUI worker thread. - `Telemetry.Event`: 19 onboarding variants under the spec's own event and property names. Doc comments record the two things that would otherwise mislead whoever writes the queries — `gateway_device_code_issued` describes a browser loopback OAuth (this flow has no device code), and the activation events are inferred from proxies rather than observed, so their counts are lower bounds. - `altimate/telemetry/onboarding.ts`: typed emitter that awaits the idempotent `Telemetry.init()` before tracking. Onboarding events can fire before any prompt runs, and in the TUI worker `init()` only happens via `session/prompt.ts` — a user who quits during gateway auth never reaches a prompt, so those events would sit in the pre-init buffer and never ship. Also owns the monotonic funnel-stage state behind `onboarding_abandoned`. - `Telemetry.shutdown()`: serialize concurrent calls. Three paths can now overlap, and each would enter `flush()`, which splices a shared buffer — one caller could post a half-empty batch while the other dropped events. - Close two flush holes. `cmd/tui.ts` ends its handler with `process.exit(0)`, skipping the outer `finally` in `index.ts` that normally flushes; and the TUI worker's `rpc.shutdown()` drained traces but never its own telemetry buffer. The two threads hold separate `Telemetry` module instances, so both need one. Bounded at 2s each: `flush()` can block for `REQUEST_TIMEOUT_MS` (10s), which exceeds the 5s the shutdown RPC is given and would be a visible hang on exit. Events wired: the four gateway auth events, `environment_scan_completed`, and `sample_setup_completed`. `gateway_auth_failed.reason` is classified by tagging the error at each rejection site rather than by matching message text. The `error` query param is attacker-influenced text we must neither parse nor forward, and message matching would drift the moment a string is reworded. Note that an unknown `state` never rejects a pending flow — the handler returns 400 without touching the map — so a CSRF mismatch correctly surfaces as `timeout`, not `denied`. `has_warehouse` and `connections_found` derive from `totalConnections`, not `connections.alreadyConfigured`: warehouses discovered from dbt profiles, docker, or env vars are counted separately, and a user whose only warehouse was auto-discovered would otherwise be recorded as having none. `sample_setup_completed` counts models and seeds by walking the shipped sample tree. A constant would drift the first time someone adds a model, and dbt's `target/manifest.json` is ~17k lines to extract two integers on a path the user is waiting on. No instance or tenant names, filesystem paths, raw error text, or authorize URLs (which carry the CSRF `state`) reach telemetry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Follow-up to the first telemetry batch, from a review pass over it.
Shutdown correctness:
- `init()` now waits out an in-flight `shutdown()`. The previous commit only
serialized shutdown callers, which was half the problem: `session/prompt.ts`
init()s at the start of every session loop and shuts down at the end, so a new
session could begin while the previous shutdown was still awaiting `flush()`.
`init()` returned the stale `initPromise`, the new session's events landed in
`buffer`, and the in-flight `doShutdown()` then cleared that buffer — every
event tracked in the gap was lost.
- The 2s bounds on the exit paths did not bound anything. `withTimeout()` and
`Promise.race()` do not cancel the promise they lose to, so a timed-out flush
kept running and reset module state after the caller had already resumed, and
the worker's raw race additionally left a live timer behind. The deadline is
now applied inside — `shutdown({timeoutMs})` → `flush(timeoutMs)` → the
existing `AbortController` — so the request is genuinely aborted and no
external timer exists.
Gateway funnel gaps:
- `gateway_auth_failed` was never emitted when `startCallbackServer()` throws,
which is what happens when port 7317 is already in use. That is a real and
reasonably common failure, and it occurs before any callback object exists, so
it was invisible to the funnel.
- Repeated `callback()` invocations re-ran the whole body and could emit
completion/failure more than once per attempt, re-reporting a connect time
measured from the original attempt. One outcome per attempt now.
- `time_to_connect_ms` started after the browser open, excluding the callback
server startup and the browser launch — both part of the wait the user
actually experiences. It now runs from the top of `authorize()`.
Coverage:
- `sample_setup_completed(success: false)` now also fires when the shipped
sample assets cannot be resolved (a CLI installed without its wrapper package
assets). Previously only materialization failures were counted, so failure
numbers silently excluded the broken-install case.
Also brackets the whole moved `shutdown()` body in `altimate_change` markers;
the previous commit left most of it outside, which would conflict awkwardly on a
future upstream merge.
Known and unchanged: `environment_scan_completed` and `sample_setup_completed`
fire on every invocation of their tools, including `/discover` and
model-initiated calls, so they are not onboarding-scoped. Scoping them needs the
session-level onboarding state that lands with the activation events.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Wires the four session-scoped funnel events, completing the worker-side half of the taxonomy. The three activation events are DERIVED, and the code says so. The activation menu is not UI — it is text the model writes from the `/onboard-connect` template — and the user picks a job by replying in free text, so neither moment is observable. What is deterministic is which command started the session and which tool ran next, so that is what they infer from: - `activation_menu_shown` — for the `skip` branch, at command dispatch, where the variant is known because there is no scan to wait for. For the `scan` branch, when `project_scan` returns, since the menu follows the scan and the variant depends on what the scan found. Emitting both at dispatch would have over-counted sessions that error out before reaching the menu. - `activation_job_selected` / `first_job_completed` — from the first job-shaped tool call in an onboarding session: `sample_setup`, or the `skill` tool carrying `dbt-analyze`, `sql-review`, or `cost-report`. A job that starts and then fails still counts as selected. `something_else` — the "just let me chat" branch — has no tool signature and therefore cannot be detected at all. Those counts are lower bounds, noted in both the event type and the plugin. `first_prompt_sent` excludes slash commands, which matters because the scan gate submits a hidden `/onboard-connect` as an ordinary user message: without the exclusion it would be recorded as the user's first prompt in every single fresh onboarding. Implemented as a plugin (`command.execute.before`, `tool.execute.after`) rather than as edits scattered through `session/prompt.ts` — the hooks already carry everything needed, and this keeps the inference logic in one fork-owned file. Only `first_prompt_sent` lives in the session loop, next to the existing `session_start`/`task_classified` telemetry it belongs with. Session-scoped state is capped and evicts in insertion order: a `serve` process is long-lived and sees unboundedly many sessions. The scan variant sums all four connection buckets from `project_scan` metadata (`existing`, `new_dbt`, `new_docker`, `new_env`) — there is no pre-summed total. A user whose only warehouse was discovered from a dbt profile, docker compose, or env vars still has a warehouse, and reporting `no_data` would send them down the sample-project branch of the menu. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
…n, claims Follow-up to the activation batch, from a review pass over it. `first_prompt_sent` was wrong in three ways at once: - It fired on every user turn, not once per session. The enclosing `step === 1` block reads like "first message of the session" but is not: `step` is declared inside `loop()`, and `loop()` runs once per user turn, so the block executes every turn. Now claimed once per session explicitly. - It was not scoped to onboarding at all, so an onboarding-taxonomy event fired for every session in the product — TUI, `run`, GitHub, API callers. - Combined with the above, a resumed session could emit it for a historical message. `first_job_completed` no longer fires on skill load. The `skill` tool loads an instruction bundle and returns; the agent then does the actual work with other tools. Treating that as completion reported "downstream impact analysis complete" the moment the instructions were read. Only `sample_setup`, which really does the work and reports `metadata.success`, is evidence of completion. Skill-driven jobs are now absent from this event rather than wrong in it — `activation_job_selected` still covers them, since selection IS observable. Session attribution: `emit()` now takes the caller's session id instead of always reading `Telemetry.getContext()`. That context is process-global and set by the session loop, so a plugin hook firing for session A while the context pointed at session B misattributed the event — and stamped an empty session for the gateway events, which run before any session exists. Session state is now one record per session instead of five parallel sets. Separate sets could evict a session from some and not others, leaving the worst possible state: still considered "onboarding" but with its once-per-session claims forgotten, so it re-emitted `first` events. One map evicts a session's whole state atomically. Cap raised 64 → 256, since a long-lived `serve` process can hold many concurrent sessions. Note for whoever reads the existing telemetry: the `step === 1` misreading above is not limited to this change. `session_start` and `task_classified` are emitted from the same block and therefore also fire once per user turn rather than once per session, which inflates session counts by the number of turns. That is pre-existing and untouched here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Wires the eight TUI-side funnel events, completing the taxonomy. `packages/tui` cannot import `packages/opencode`, where the Telemetry module lives, so the host injects a callback through `TuiInput` and a context carries it to the components — the same seam `context/exit.tsx` already uses for the host's exit function. No HTTP route, no SDK surface, no worker round-trip: the TUI renders on the main thread, so this reaches the process's own Telemetry instance, already initialized by the CLI middleware. The provider is mounted above `DialogProvider`, and that placement is load-bearing: `ui/dialog.tsx` renders dialog contents as a *sibling* of its children, so a provider placed around `<App>` is invisible to every dialog. `dialog-scan-gate.tsx` already documents the same trap for the prompt ref. For the scan gate specifically the event is emitted from the `onChoose` closure in `app.tsx`, reusing the prop bridge that exists for exactly this reason. Events and their anchors: - `onboarding_started` — only on the branch that actually opens the first-run gate, so returning users never enter the funnel. - `model_picker_shown` — carries `trigger`. The picker also opens from `/connect`, from declining Big Pickle, and from the prompt gate; without the discriminator every impression would read as a fresh first run. - `provider_selected` — through a single activation choke point so keyboard and mouse cannot diverge, plus the `/` shortcut, which is the same intent as the "Search all providers…" row. Fires on selection, before auth resolves: a cancelled sign-in still counts as a provider having been chosen. - `big_pickle_confirm_shown` / `big_pickle_choice`, `scan_gate_shown` / `scan_gate_choice` — both choice handlers are now guarded against double-submit. Keyboard and mouse both call them directly and nothing stopped two firing before unmount, which would double-count the choice and, for the scan gate, submit `/onboard-connect` twice. - `onboarding_completed` — the readiness false→true transition, which is exactly the spec's "a model is ready and chat is live". Selecting the gateway provider also marks the auth funnel stage on this thread, because the browser flow runs in the worker and cannot reach the main thread's abandonment state. The `name` → `type` remap in `cli/cmd/tui.ts` is the one untyped point in the chain: the package boundary forces `packages/tui` to declare its own mirror of the event union. Both sides are independently type-checked; a test will pin the two lists together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
… safety Follow-up to the TUI batch, from a review pass over it. Abandonment was going to be mostly noise. Nearly every piece of UI that emits a funnel event is also reachable outside onboarding: `/connect` opens the same welcome picker, `/model` opens the same Big Pickle interstitial. A returning user doing either advanced the funnel stage to `model_picker`, never emitted `onboarding_completed`, and was therefore reported as ABANDONED on exit. The headline drop-off metric would have been dominated by established users opening `/connect`. Stage tracking is now gated on the funnel actually having started — `onboarding_started`, which is emitted only from the branch that opens the first-run gate. Everything downstream is a no-op outside a real first run, which also fixes the same class of problem for the host marking `gateway_auth` on any gateway selection. Other fixes: - `prompt_gate` was never emitted. The prompt gate opened the picker without passing a trigger, so every "typed a prompt before connecting" impression was misattributed to `/connect`. - `provider_selected` was not fire-once. Keyboard return and mouse-up both reach the activation path with nothing stopping two firings before unmount — that double-counted the choice *and* started the provider flow twice. The `/` shortcut now routes through the same guarded path instead of emitting its own event. - The telemetry context no longer throws when absent. It was built on `createSimpleContext`, which throws for consumers outside its provider; these dialogs are rendered directly by tests and can be reused by other hosts, and analytics is not a reason to crash a UI. Missing provider now means no tracking. The host callback is also isolated, so a throwing tracker cannot propagate out of a mount handler or a keypress and take the dialog transition with it. Known and unchanged: under `OPENCODE_FAST_BOOT`, sync reports ready before credentials are populated, so a returning user can transiently look un-onboarded and see the first-run gate. That is a pre-existing product behaviour — the gate itself already had it — and these events faithfully report what the UI did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
A whole-feature review found the events individually correct but the funnel not reconstructable. These are the fixes. Correlation (see #1048): - Every event now carries `launch_id`, a random value generated once per process and shared across threads through the environment. The funnel spans the TUI main thread and the server worker, and most of it runs before any chat session exists — so the first half of the funnel stamped an empty session and the second half a real one, leaving no key to join a single run on. It is not persisted, not derived from the machine or the user, and not reused across launches. Placed in the event envelope alongside `machine_id`, so it applies to all telemetry rather than only the onboarding events. - `~/.altimate/machine-id` is now created exclusively. Both threads initialise their own copy of the telemetry module, and on a genuinely new install both can find the file missing at the same moment; with a plain write the loser overwrote the winner while each kept its own value in memory, so one first run reported two machine ids. The fallback identity broke on exactly the run it mattered most for. `wx` makes one writer fail and re-read. Abandonment: - `last_stage` no longer advertises values it can never report. `scan_gate` and `activation` both occur after `onboarding_completed`, which sets `completed` and suppresses abandonment entirely — abandonment is by definition quitting before connecting. They are removed rather than documented as live. - Added `provider_setup`, advanced by any provider choice. A user who picked Anthropic and quit during key entry was previously reported as abandoning at `model_picker`, as though they had never chosen anything. Shutdown: - `shutdown({timeoutMs})` no longer ignores the budget when a shutdown is already in flight. It returned the existing promise, which may have been running on the default 10s, so the exit path waited past its 2s budget and the worker was terminated mid-flush anyway. The in-flight flush cannot be shortened, but the caller no longer waits on it beyond its own deadline. Known and unchanged: choosing "Search all providers…" and then picking from the full catalogue records only `provider=search_all`. Fixing that means emitting from the full model picker, which is used outside first-run too, so it needs a product call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
…eanup block `test/cli/tui/command.test.ts` guards the ordering of the TUI worker cleanup by slicing `cli/cmd/tui.ts` between the "clean up TUI worker after failed --session validation" start marker and the next `altimate_change end`. The onboarding telemetry seam added a nested start/end pair inside that region, so the slice ended early and `await stop()` fell outside it — the guard failed with the cleanup index at -1. Uses a single-line marker instead, and says why in the comment so the next person adding something there does not reintroduce it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Answers "does this user action emit that event, with those properties" at each emission point, rather than trying to reconstruct a whole cross-thread stream. Expected values are written out from the product spec rather than computed the way the implementation computes them, so a test cannot agree with the code while both are wrong. Every assertion here was checked by breaking the implementation and confirming the test fails: - removing the funnel-start gate → the returning-user test fails - treating a skill load as job completion → the skill test fails - dropping `launch_id` from the envelope → the correlation test fails - removing the scan-gate guard → both double-submit tests fail Coverage of note: - A returning user opening the picker is not enrolled in the funnel and is never reported as abandoned. - Quitting after choosing a provider reports `provider_setup`, not `model_picker`. - A warehouse discovered from a dbt profile counts as having a warehouse — the adversarial fixture for the connection-total bug, where only `existing` was being read. - Loading a skill selects an activation job but does not complete one. - Tools run outside an onboarding session emit nothing. - `launch_id` is asserted at the envelope level, because it is attached during App Insights conversion and is invisible to a `Telemetry.track` spy — the transport-level test is the only place the correlation fix is observable. Not covered yet: mounting the provider picker and the Big Pickle interstitial needs the full sync/SDK stack (`LocalProvider` depends on both), so those belong with the app-lifecycle harness. The scan gate is covered because its choice is handed out through a prop and needs no telemetry context. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
The picker was the last uncovered emission point. I had written it off as needing the app-lifecycle harness because `LocalProvider` depends on sync and the SDK — that was wrong. `test/fixture/tui-sdk.ts` already provides a fake fetch and event source, so the real provider stack mounts fine. Covered: the impression carries the trigger that opened the picker; an unset trigger is attributed to `/connect` rather than to a first run; choosing the first row records the gateway; a rapid second Enter records one selection, not two; and the `/` shortcut records the same choice as the search row. Verified by mutation, as with the other funnel tests — removing the activation guard fails the double-Enter test, and changing the default trigger fails the attribution test. The trigger parameter is typed by extraction from the event union, so renaming a trigger breaks this test at compile time rather than silently drifting. The fake SDK serves an empty provider list. That is fine here: the curated rows are hardcoded and the funnel event is recorded at the moment of choice, before the provider lookup that would act on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
…threads Found by driving a real CLI process through a PTY against a local telemetry sink: the eight TUI-thread events arrived with one launch_id and the worker-thread events with a different one. The correlation id could not join the two halves of the funnel — which is the only thing it exists to do. Two approaches that look correct are not, both confirmed end to end: - Mutating `process.env` at runtime. A Bun Worker does not observe it, so the worker minted its own id. This is what the original implementation did, and it was lazy as well, so the value did not exist until the first flush — long after the worker had started. - Deriving it from the process (pid + start time). Both threads really are one process and share a pid, but `process.uptime()` is per-THREAD in Bun, so each thread computed a different start time. What works is handing the id to the worker explicitly at construction, which Bun's Worker supports through its `env` option. Verified: all eleven events from a full first run — across both threads and two different session ids — now carry a single launch_id. Also loosens the launch_id unit test to select envelopes by event name instead of by position. The telemetry buffer is module-global, so a sibling test file can leave events in it that flush alongside these, which made the exact-length assertion fail only when the files ran together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
…ssion `environment_scan_completed` and `sample_setup_completed` were emitted without a session, so they fell back to the process-global telemetry context — which is set per prompt loop. With two concurrent sessions in `serve`, or in the TUI worker, one session overwrites the other's context and a scan or sample setup gets attributed to whichever session most recently started a turn. Both tools have their own `ctx.sessionID`. The activation plugin was already passing it explicitly; these two were not. Same root cause as the launch-id bug: state that looks process-wide is not describing what you assume it describes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
…ness The Part 2 gate was triggered by useReady(), which is `connected() || setupComplete()`. `connected()` flips as soon as a provider lands in sync data — and the BYOK confirm handlers do exactly that inside `await sync.bootstrap()`, before going on to open the model picker (dialog-provider ApiMethod and CodeMethod). So for every BYOK provider the gate mounted mid-handler, the handler's own `dialog.replace(<DialogModel/>)` destroyed it a moment later, and the one-shot latch meant it never came back. `/onboard-connect` was never submitted, which made activation_menu_shown, activation_job_selected, first_job_completed, sample_setup_completed and first_prompt_sent unreachable on the majority onboarding path — while onboarding_completed and scan_gate_shown were still reported for a gate the user never saw. The gateway path was unaffected because its success branch does not replace the dialog, which is why this survived manual testing and the Big Pickle E2E. Now driven by setup completion alone, which is only set once a model is genuinely chosen — the model picker, the Big Pickle accept path, and the gateway auto-select. That is also what the spec means by "a model is ready". Reported in consensus review of #1049 (CRITICAL), flagged independently by two reviewers, with the ordering verified against sync.tsx: the store write is inside `batch()` within an awaited promise chain, two `.then()` hops before bootstrap() resolves, so the effect flushes before the dialog.replace under either effect-scheduling model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Addresses the findings from the review that fall inside this PR. MAJOR — markSetupComplete() ran before the model-availability check (dialog-provider). The branch below it deliberately refuses to claim success when the gateway connects but offers nothing usable, and said so in a comment — but completion had already been marked, so telemetry reported a finished onboarding while the user was told to go pick a model. Now marked where the model is actually set. This matters more since the scan gate moved onto the same signal. MAJOR — the flushTimer leak. doInit() assigned a new interval without clearing the previous handle, and shutdown() only ever clears the current one, so a second doInit() stranded a timer for the life of the process. doShutdown also nulled initPromise unconditionally, discarding a doInit() that init() had chained onto the in-flight shutdown. Both fixed; see the note in doShutdown on why only the first is covered by a test. MAJOR — instance_connected and onboarding_abandoned could both be reported for one launch. The gateway success events are emitted on the worker thread while abandonment state is main-thread-owned, so a user who finished in the browser and quit before the TUI observed the new provider was reported as abandoning at gateway_auth. The exit path now checks whether credentials landed. MAJOR — environment_scan_completed fired on every project_scan, including /discover and any model-initiated call, so a funnel query could exceed 100% conversion. Now guarded on isOnboardingSession. MINOR — command.execute.before created a tracking record for every slash command in every session, churning the capped map and evicting genuine onboarding sessions, after which their remaining activation events were silently dropped. noteCommandSubmission now only touches sessions already tracked, and /onboard-connect marks the session before flagging its own submission. MINOR — the cross-package event parity test that two comments claimed but which did not exist. Now a compile-time assertion pinning the packages/tui event union to the Telemetry variants; verified by renaming a property and watching the build fail. Required exporting the context subpath from packages/tui. MINOR — a raw HOME path could reach LLM-visible tool output on a sample-setup failure. Paths are masked in `output`; the full message stays in metadata, which the model never sees. MINOR — documented why sample_setup_completed uses a success boolean instead of a _failed sibling event. NIT — launchId() no longer writes process.env. The worker receives the id explicitly through WorkerOptions.env, so the write only leaked it into every subprocess the CLI spawns. Also removes the stray `// scratch` line at the end of cli/cmd/tui.ts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
sahrizvi
force-pushed
the
feat/onboarding-telemetry
branch
from
July 30, 2026 22:41
8e369a8 to
6843e2d
Compare
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the onboarding funnel event taxonomy to the CLI: 19 events covering first run, provider choice, gateway auth, the scan gate, activation, and drop-off.
Approach
Events are emitted from three places, each chosen because it is where the thing being measured actually happens:
packages/tuicannot import the Telemetry module, so the host injects a callback throughTuiInputand a context carries it down — the same seamcontext/exit.tsxalready uses. No HTTP route and no new API surface: the TUI renders on the process main thread, so this reaches the existing Telemetry instance directly.Telemetry.track()calls.command.execute.beforeandtool.execute.afterhooks.Things worth knowing before reviewing
Three activation events are derived, not observed. The activation menu is not UI — it is text the model writes from a prompt template, and the user replies in free text.
activation_menu_shown,activation_job_selectedandfirst_job_completedare inferred from the closest deterministic signals. Their counts are lower bounds. Two gaps are documented rather than papered over: the "something else" branch has no tool signature and is never counted, and skill-driven jobs cannot be confirmed complete so they are absent fromfirst_job_completedrather than wrongly counted in it.gateway_device_code_issuedis a spec name, not a description. The flow is a browser loopback OAuth; there is no device code. Kept for taxonomy fidelity, with the mismatch noted in the type and the docs.launch_id. The funnel spans two threads and most of it runs before any chat session exists, so the first half stamped an empty session and the second half a real one — nothing to join a run on. Every event now carries a per-launch random id. Not persisted, not derived from the machine or user, not reused across launches. See #1048.Also fixed along the way
Telemetry.shutdown()was not reentrancy-safe, and neither exit path flushed — the TUI handler exits viaprocess.exit(0), skipping the outer flush, and the worker never flushed its own buffer at all. Both threads hold separate buffers.~/.altimate/machine-idcould be created twice on a new install, giving one first run two machine ids.Known gaps
provider=search_all. Fixing it means emitting from the full model picker, which is used well outside first run — wanted a product call before touching it.OPENCODE_FAST_BOOT, sync reports ready before credentials load, so a returning user can transiently look un-onboarded and see the first-run gate. Pre-existing behaviour; these events faithfully report what the UI did.Status
Draft because automated tests for the new code are not written yet — that is the remaining work.
Typecheck is clean on both packages. Full
packages/opencodesuite: 10,780 pass / 7 fail. All 7 failures are pre-existing and unrelated (6 inacp permissions, which reproduce with this branch's changes reverted to the base commit; 1 flakyrunsubprocess test that passes in isolation).Unrelated bug found while working on this and filed separately: #1047.
🤖 Generated with Claude Code
https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Summary by cubic
Adds end-to-end first‑run onboarding funnel telemetry across CLI, TUI, and worker, correlated by a per‑launch
launch_id. Tightens first‑run UX and telemetry lifecycle to ensure accurate funnel data and prevent false abandonment.New Features
packages/opencode/src/altimate/telemetrywithlaunch_idon all events; docs updated indocs/reference/telemetry.md.packages/tui):onboarding_started,model_picker_shown(withtrigger),provider_selected,big_pickle_*,scan_gate_*,onboarding_completed; prompt‑gate attribution; guarded double‑submits; context inpackages/tui/src/context/onboarding-telemetry.tsx.OnboardingTelemetryPlugin: infersactivation_menu_shown,activation_job_selected,first_job_completed(sample‑only), andfirst_prompt_sent(once per onboarding session; excludes slash commands); bounded per‑session state.environment_scan_completed(onboarding‑scoped) andsample_setup_completedwith explicitsessionID.launch_idto the worker viaWorker({ env }).Bug Fixes
init()/shutdown(), apply real abort timeouts, fix flush‑timer leak; flush both TUI and worker buffers on exit; exclusive~/.altimate/machine-idcreation.gateway_auth_failed(incl. port‑in‑use), one outcome per attempt, connect time starts atauthorize(); no parsing of attacker‑controlled text.provider_setupstage; preventonboarding_abandonedafter a successful connect;environment_scan_completedemits only for onboarding; commands no longer churn the capped session map.sessionID; TUI and worker events share onelaunch_id.packages/tuievent union andTelemetry.Event.Written for commit 6843e2d. Summary will update on new commits.