test(cli): supabox-backed live test suite + cli-e2e-ci dispatch#5699
Merged
Conversation
Introduce a `live` Vitest project (`*.live.test.ts`) for black-box tests that run the built CLI against a real platform — a supabox stack in CI (supabase/cli-e2e-ci) — exercising real Management API, functions, database, and storage flows. - `tests/helpers/live.ts`: `runSupabaseLive` (retargets via SUPABASE_PROFILE=supabase-local), `describeLive` gate, readiness helpers. - `tests/live-global-setup.ts`: fail fast if the platform is unreachable when the live env is configured; no-op otherwise. - vitest `live` project + nx `test:live` target (auto-derived from the project name), `pnpm test:live` script, knip entry, coverage exclude. - Gated by SUPABASE_ACCESS_TOKEN so the suite is inert in the normal unit/integration/e2e loop. - Seed the canonical example: `orgs list` authenticated read-only smoke. Refs CLI-1834 (parent CLI-1825). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RNpmVr6SxaFQHbJSBYLUCj
Two defects surfaced running the live suite against a real supabox stack: - The `test:live` package script (`nx run-many -t test:live`) shadowed the nx-plugin `test:live` target and recursed into itself forever. Remove it; the nx target (auto-derived from the `live` Vitest project, `dependsOn: build`) is the single source of truth, invoked via nx like test:unit/integration/e2e, which have no package scripts either. - The global-setup readiness probe required a 2xx from `/v1/health`, but supabox's mgmt-api has no public health route (`/v1/health` 404s; an unauthenticated request is rejected by the auth middleware with 401). Make it a pure reachability gate: any HTTP response from `/v1/organizations` proves the API is up and routing; functional/auth coverage stays in the live tests. Validated end-to-end: the `orgs list` live smoke passes against a real supabox control plane. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a sender workflow that, on PRs labeled `run-live-e2e-ci`, fires a repository_dispatch (type: cli-pr) to supabase/cli-e2e-ci carrying this PR's head SHA, so the supabox-backed `test:live` suite runs against the PR's cli. Opt-in by label to keep the expensive full-stack run off every PR; cli-e2e-ci reports a `cli-e2e-ci / live` commit status back onto the head SHA. Distinct from live-e2e.yml (the staging cli-e2e package suite). Part of CLI-1831 (epic CLI-1825). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CLI-1834 follow-up: - `projects list` live scenario (account-level, read-only) plus a JSON-output assertion. Validated green against a real supabox control plane. - `describeLiveProject` / `liveProjectRef` / `requireLiveProjectRef` harness gate for project-scoped suites: skip unless SUPABASE_LIVE_PROJECT_REF is set (a provisioned project), so a control-plane-only stack (e.g. local macOS) skips rather than fails them. - `functions list` and `branches list` project-scoped scenarios behind that gate — the entry points for the edge-functions and branching lifecycle coverage, which need a provisioned project on the full stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…zed) A bad token must round-trip to the real Management API, come back 401, and surface as a non-zero exit carrying the upstream "Unauthorized" message — exercising the cli's auth + error mapping against the live stack, not just the golden path. Validated green against a real supabox control plane. Refs CLI-1834. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
More live coverage validatable on a control-plane-only stack: - `orgs list --output-format json`: stdout is payload-only parseable JSON (machine-output parity with `projects list`). - `functions list --project-ref <unknown>`: a well-formed but nonexistent ref round-trips to the live Management API, returns 404, and surfaces as a non-zero exit (not a crash, not "Unauthorized") — exercises the `--project-ref` request path + error mapping without a provisioned project, so it runs under `describeLive` rather than `describeLiveProject`. Validated green against a real supabox control plane (6 passed, 2 skipped). Refs CLI-1834. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@2e84e5ec686fad7e7ee5b624120278af6c5a6b58Preview package for commit |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3ef0579de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- dispatch-cli-e2e-ci.yml: guard the dispatch job on head.repo == base repo so a labeled fork PR (which doesn't receive secrets) skips cleanly instead of red-checking on the App-token step. - runSupabaseLive: default exitTimeoutMs to 240s (LIVE_EXIT_TIMEOUT_MS) so a slow-but-valid supabox call isn't killed by runSupabase's 60s default before the live tests' own (60–120s) timeouts fire; callers can still override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b55d0018a7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Split the env-only live helpers (isLiveConfigured, liveApiBaseUrl, project-ref accessors, profile/timeout constants) into tests/helpers/live-env.ts, which imports no Vitest test APIs. globalSetup now imports from there instead of helpers/live.ts (which pulls in `describe`), avoiding evaluating Vitest test APIs in the globalSetup context. live.ts re-exports the env helpers so *.live.test.ts keep a single import site. Validated against a real supabox stack: 8 passed, 0 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coly010
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
livetest category that exercises the built CLI against a real Supabase platform — a fulls supabox stack stood up in CI by cli-e2e-ci — and the cli-side trigger that runs it for a PR.What changed
liveVitest project (*.live.test.ts) + harness inapps/cli/tests/helpers/live.ts:runSupabaseLive(drives the built binary viaSUPABASE_PROFILE=supabase-local),describeLive(gated onSUPABASE_ACCESS_TOKEN), anddescribeLiveProject/requireLiveProjectRef(gated onSUPABASE_LIVE_PROJECT_REFfor project-scoped suites).tests/live-global-setup.tsfail-fast reachability probe.test:livetarget (auto-derived from the project) +nx.jsondefault; removed the recursivetest:livepackage script (it shadowed the nx target and looped).orgs list(+JSON, +invalid-token negative),projects list(+JSON),functions list/branches list(project-scoped), and afunctions listunknown-project (404) negative..github/workflows/dispatch-cli-e2e-ci.yml: on PRs labeledrun-live-e2e-ci, fires arepository_dispatchto cli-e2e-ci with the PR head SHA; cli-e2e-ci builds that SHA, runs the suite, and reports acli-e2e-ci / livecommit status back. Distinct from the staginglive-e2e.yml.Why
There was no backend-hitting live coverage — existing
*.e2e.test.tsuse fake tokens. This validates real Management-API flows end-to-end against supabox. Refs CLI-1825 / CLI-1834 / CLI-1831.Reviewer notes
liveproject is inert by default — it only runs whenSUPABASE_ACCESS_TOKENis set (the cli-e2e-ci runner provides supabox's seeded PAT), so it does not touch the normal unit/integration/e2e loop.SUPABASE_LIVE_PROJECT_REF).run-live-e2e-cilabel on this repo, the App'scontents: writeon cli-e2e-ci (to dispatch) andstatuses: writehere (for the back-status). The build/test half is proven without them.