test(cli): expand live e2e coverage across read-only commands#5708
Open
avallete wants to merge 5 commits into
Open
test(cli): expand live e2e coverage across read-only commands#5708avallete wants to merge 5 commits into
avallete wants to merge 5 commits into
Conversation
The `--output-format json` live tests only asserted `JSON.parse` succeeded,
which proves "command exited and emitted JSON" but not platform correctness.
Assert real shape + state instead:
- orgs list: `organizations` is a non-empty array of `{ id, slug, name }`
(the live token always belongs to at least one org).
- projects list: `projects` is an array, and when the runner provisioned a
project (SUPABASE_LIVE_PROJECT_REF set) that ref appears in the listing —
proving the JSON reflects real platform state.
Validated against a real supabox stack: 8 passed (the project-ref containment
assertion exercised with a provisioned project).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
avallete
force-pushed
the
test/deepen-live-assertions
branch
from
8becf86 to
9c23cd7
Compare
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@3b2049aba03ea161cee8bb1d8b12e272f85f83d4Preview package for commit |
Extends the live suite beyond orgs/projects/functions/branches with the
secrets resource — a new Management API control-plane surface.
- describeLiveProject: lists secrets for the provisioned project (exit 0,
not Unauthorized) and asserts `{ secrets: [...] }` JSON envelope shape.
- describeLive: a valid token with an unknown --project-ref round-trips to
the live Management API, returns 404, and exits non-zero — exercising the
request path + error mapping even on a control-plane-only stack.
Mirrors the functions/list live pattern. Secrets are edge-function env vars
served by the control plane, so it runs against a freshly provisioned project
regardless of data-plane health (empty list is a valid payload).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Broadens the live suite across the rest of the read-only command surface so the supabox-backed run exercises each command's auth + request path + output shaping against the real platform. Control-plane, project-scoped (success + unknown-ref 404): - backups list, network-bans get, network-restrictions get, postgres-config get, ssl-enforcement get, vanity-subdomains get, sso list — assert the JSON envelope shape (array key or single object) plus a 404 negative. Control-plane, 404-negative only (no stable success on a bare project, or a resource id is required): - domains get (custom hostname unset), branches get (needs a branch), sso show (needs a provider id). Data-plane, gated by the new `describeLiveDb` / SUPABASE_LIVE_DB_URL (connect to the project Postgres via --db-url, skipped until the runner resolves the pooler URL): - db dump, db advisors, migration list. Adds liveDbUrl()/requireLiveDbUrl() + describeLiveDb to the live helpers. All 18 live files collect and skip cleanly with no live env configured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
avallete
changed the title
avallete
added
the
run-live-e2e-ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f09bdfcc56
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Addresses Codex review of the read-only live coverage — several assertions assumed an error shape the handlers do not produce: - sso list / sso show / network-restrictions get: the 404 paths map to errors that omit the HTTP status code (SAML-disabled, NotFound, "received: <body>"), so the negatives now assert clean non-zero + non-Unauthorized instead of a literal "404". - sso list / vanity-subdomains get: success is entitlement-gated (SAML 2.0; Pro+ plan) and 404s/400s on a fresh project, so both are demoted to unknown-project negative-only rather than an unconditional success. - branches get: pass a placeholder branch name so the non-TTY subprocess does not enter the interactive branch-id prompt before reaching the API. Commands whose mappers do include the status code (backups, network-bans, postgres-config, ssl-enforcement, domains, secrets) keep the stricter 404 assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A supabox-backed live run showed `network-bans get` exits non-zero on a freshly provisioned project: the retrieve endpoint returns a non-200 there (the request reaches the API — not Unauthorized — but does not 200), same class as the SAML-gated sso and plan-gated vanity-subdomains commands. Drop the project-scoped success/JSON tests; keep the unknown-project 404 negative (which passed live), matching the established pattern for endpoints without a stable success path on supabox. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the supabox-backed live suite (#5699) by deepening the existing assertions and extending live coverage across the rest of the read-only command surface.
What changed
Deepened existing assertions (orgs/projects list): the
--output-format jsontests now assert real shape + platform state (non-emptyorganizations, provisioned project ref present) instead of only thatJSON.parsesucceeds.New control-plane live tests (project-scoped success + unknown-ref 404):
secrets list,backups list,network-bans get,network-restrictions get,postgres-config get,ssl-enforcement get,vanity-subdomains get,sso list— each asserts the JSON envelope shape (array key or single object) and a 404 negative.404-negative only (no stable success on a freshly provisioned project, or a resource id is required):
domains get,branches get,sso show.Data-plane tests behind a new
describeLiveDbgate (SUPABASE_LIVE_DB_URL), connecting to the project Postgres via--db-url:db dump,db advisors,migration list. These skip until the cli-e2e-ci runner exports a pooler URL.Notes for reviewers / runner
SUPABASE_LIVE_PROJECT_REFis set (cli-e2e-ci provisions it); the data-plane tests additionally needSUPABASE_LIVE_DB_URL. All 18 live files collect and skip cleanly with no live env configured.sso list's billing posture) are assumed served by supabox; the manual cli-e2e-ci dispatch is the gate that confirms which succeed against the stack.