Skip to content

feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root - #2291

Draft
hunleyd wants to merge 13 commits into
developfrom
hunleyd/indata-996-gate-pgbackrest-async-archiving-on-customer-tier
Draft

feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root#2291
hunleyd wants to merge 13 commits into
developfrom
hunleyd/indata-996-gate-pgbackrest-async-archiving-on-customer-tier

Conversation

@hunleyd

@hunleyd hunleyd commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The salt subcommand's systemd service runs as the unprivileged supabase-admin-agent OS user (group supabase-admin-agent only), but /etc/pgbackrest/conf.d is pgbackrest:postgres 02770 — only root/adminapi (in the postgres group) can write there.
  • Adds one NOPASSWD sudoers entry so the salt cycle can elevate just pgbackrest reconcile via sudo, mirroring the existing salt-call entry in the same file — no broader group membership needed.
  • Changes /opt/supabase-admin-agent (directory, unpacked binary) from owner supabase-admin-agent to owner root:root, so the sudoers grant above cannot be used to elevate a binary the unprivileged agent user could replace first. Pins explicit mode: "0755" on both so this doesn't depend on umask or the release tarball's stored permission bits. (The symlink itself carries no separate owner/groupstate: link with the file module's default follow: yes would only re-apply those to the link target, already covered by the unpack task above; the symlink still gets mode: "0755" explicitly.)

Why this is safe

  • Scoped exactly to one command.
    • The sudoers rule matches the full literal path /opt/supabase-admin-agent/supabase-admin-agent pgbackrest reconcile — it can't be used to run anything else as root. Confirmed against the actual call site in supabase-admin-agent PR Generate optimized config for PG #112: reconcileBin is a hardcoded constant equal to this exact path, with no --config flag and no os.Executable() resolution that could drift from it.
  • Root ownership blocks binary tampering — with one known gap.
    • The ownership change above is what stops the agent user from swapping the binary it's allowed to sudo-run as root. However, this AMI-time ownership is currently undone at instance boot: the platform repo's provisioning scripts (init-scripts/project/00-init.sh and init-scripts/suspend-and-wake/00-init.sh) recursively chown the whole directory back to the unprivileged agent user after every install. That's tracked as a hard blocker in INDATA-1151 and must ship before this sudoers rule is actually safe on real instances.
    • That same platform provisioning flow is also what writes /opt/supabase-admin-agent/config.yaml (root templates it, then chown supabase-admin-agent:supabase-admin-agent + chmod 0640init-scripts/{project,suspend-and-wake}/00-init.sh), and a periodic salt state (salt/state/formula/supabase-admin-agent/init.sls) re-templates it at 0644 running as the agent user. reconcile reads this file (--config, default /opt/supabase-admin-agent/config.yaml) and derives EffectiveAdminAPIUser() from it — so even with the binary root-owned, root-run reconcile still trusts a config the agent user can write. INDATA-1151's fix needs to explicitly decide config.yaml's final ownership (root-own it too, vs. carve it out as agent-writable for the salt refresh path) — it isn't automatically solved by scoping down the recursive chown on the binary/dir alone.
    • audit-specs/baselines/prod-deployed/files-opt-deployed.yml and baseline.yml still record /opt/supabase-admin-agent/README.md and the binary as owner 993/group 987 (the agent uid/gid). Because the platform boot scripts above chown -R real instances back to that uid/gid on every project creation and wake, those baselines stay accurate on live instances even after this PR merges — they only need updating once INDATA-1151 actually removes that revert. Left untouched here; update them as part of INDATA-1151 instead.
  • A second, separate revert path — not covered by INDATA-1151.
    • The platform repo also ships its own copy of this sudoers file at init-scripts/suspend-and-wake/supabase-admin-agent.sudoers.conf, containing only the pre-existing salt-call/gpg entries. setup_supabase_admin_agent() in init-scripts/suspend-and-wake/00-init.sh unconditionally reinstalls that copy to /etc/sudoers.d/supabase-admin-agent on every project wake, silently stripping the pgbackrest reconcile line this PR adds. The agent only logs the resulting sudo denial at Warn, so this fails silently. INDATA-1151 does not cover this — the platform repo's wake-path sudoers copy needs its own update. Tracked in INDATA-1171.
  • Same pattern already in production.
    • supabase-admin-agent already elevates salt-call and gpg via this exact mechanism (pkg/salt/exec.go); this isn't a new privilege model, just one more entry.
  • Syntax validated.
    • visudo -c parses the file clean.

Alternatives considered

  • Group membership instead of sudo. /etc/pgbackrest/conf.d is pgbackrest:postgres 02770 — the writable group is postgres, not a pgbackrest-specific group. setup-postgres.yml runs initdb with --allow-group-access, so the postgres group has read access to the entire live PGDATA (every table file, every WAL segment) on every instance — adding supabase-admin-agent to that group to reach conf.d would grant a much larger blast radius than the single-argv-path sudo grant this PR actually adds. Rejected in favor of keeping the narrowly-scoped sudo rule.

Related

  • Part of INDATA-996 (gate pgBackRest async archiving on customer tier) — this PR is the sudoers/ownership plumbing pgbackrest reconcile needs, not the tier-gating logic itself.
  • Blocked by INDATA-1151 — see "Why this is safe" above.
  • Blocked by INDATA-1171: platform repo's init-scripts/suspend-and-wake/supabase-admin-agent.sudoers.conf also needs the pgbackrest reconcile entry added, or the wake path strips it every time — see "Why this is safe" above.

Depends on

  • https://github.com/supabase/supabase-admin-agent/pull/112 (the pgbackrest reconcile command this grants access to)
  • INDATA-1151 (platform-repo fix so the root-ownership change actually survives instance provisioning)
  • INDATA-1171: platform repo update to init-scripts/suspend-and-wake/supabase-admin-agent.sudoers.conf (adds the pgbackrest reconcile NOPASSWD entry so the wake path stops stripping it)

Test plan

  • visudo -c -f ansible/files/supabase_admin_agent_config/supabase-admin-agent.sudoers.conf — parsed OK
  • Manual: once all three dependencies land, confirm the salt cycle's sudo-invoked pgbackrest reconcile succeeds on a test instance without a password prompt, and that /opt/supabase-admin-agent stays root-owned after a fresh project creation and a pause/wake cycle

Smoke Test Results

PASS — test-postgres-pitr.sh (engine 17)

Command: ./scripts/test-postgres-pitr.sh 'https://github.com/supabase/postgres/pull/2291' --engine 17

The script itself reported a failure at the final health-check step because the invocation was initially missing an unrelated CLI credential (SUPADEV_SYSTEM_API_KEY) — backup creation and PITR restore both succeeded before that point. Re-running just the health check (supadev project-health <restored-project>) once the credential was set confirmed all components ACTIVE_HEALTHY (Storage, Realtime, Postgres (Admin), Postgres (User), Supavisor (TXN), PgBouncer, and others). Net result: PITR backup/restore works correctly with this PR's ansible ownership change.

[INFO] Filtered to single engine: 17
============================================================
============================================================
[INFO] Testing PostgresEngines: 17
[INFO] Using region: us-east-1
[INFO] Project prefix: supadev-pitr-1784744543
[INFO] GitHub PR URL: https://github.com/supabase/postgres/pull/2291
============================================================
============================================================
[INFO] Source project name: supadev-pitr-1784744543-pg17
[INFO] Restored project name: supadev-pitr-1784744543-restored-pg17
[INFO] Region: us-east-1
[INFO] Start time: 2026-07-22 14:22:23
[INFO] Using version from PR: https://github.com/supabase/postgres/pull/2291
[INFO] Resolved version: 17.6.1.151
[STEP] Checking organization tier...
[INFO] Organization is already on tier_pro
[STEP] Creating project supadev-pitr-1784744543-pg17 with PITR enabled...
[SUCCESS] Project supadev-pitr-1784744543-pg17 created with PITR enabled
[INFO] Time to ACTIVE_HEALTHY: 373 seconds
[STEP] Waiting for physical backup data (30s sleep, then polling)...
[INFO] Polling attempt 1/20...
[SUCCESS] Physical backup data found
[STEP] Extracting earliest PITR backup timestamp...
[INFO] Earliest PITR backup: 2026-07-22 14:28:42
[STEP] Restoring PITR backup to new project supadev-pitr-1784744543-restored-pg17...
[SUCCESS] PITR backup restored to supadev-pitr-1784744543-restored-pg17
[INFO] Time to restore: 343 seconds
[STEP] Checking health of restored project...
[ERROR] Restored project health check failed (missing SUPADEV_SYSTEM_API_KEY in this invocation — see note above)

--- follow-up: supadev project-health supadev-pitr-1784744543-restored-pg17 (credential fixed) ---
  Storage:            healthy, ACTIVE_HEALTHY
  Realtime:           healthy, ACTIVE_HEALTHY
  Postgres (Admin):   healthy, ACTIVE_HEALTHY
  Postgres (User):    healthy, ACTIVE_HEALTHY
  Supavisor (TXN):    healthy, ACTIVE_HEALTHY
  PgBouncer:          healthy, ACTIVE_HEALTHY
SKIPPED — test-postgres-pause-restore.sh (engine 17)

Command: ./scripts/test-postgres-pause-restore.sh 'https://github.com/supabase/postgres/pull/2291' --engine 17

Blocked by a pre-existing local dev-stack defect ("IPv6 address is not defined for this project", 500 from mgmt-api) unrelated to this PR's diff (ansible sudoers/ownership only) — OrbStack's ipv6 config was already correct, so the fix requires an engine restart that drops every container on the host. Skipped rather than disrupt other work; PITR is the test most directly relevant to this diff's pgbackrest/archiving change and passed cleanly.

[INFO] Filtered to single engine: 17
============================================================
============================================================
[INFO] Testing PostgresEngines: 17
[INFO] Using region: us-east-1
[INFO] Project prefix: supadev-pause-1784745764
[INFO] GitHub PR URL: https://github.com/supabase/postgres/pull/2291
============================================================
============================================================
[STEP] Resolving version from PR for engine 17...
[SUCCESS] Resolved version for 17: 17.6.1.151
[STEP] Inserting version 17.6.1.151 as ga...
[SUCCESS] Version 17.6.1.151 inserted as ga
============================================================
============================================================
[INFO] Project name: supadev-pause-1784745764-pg17
[INFO] Region: us-east-1
[STEP] Checking organization tier (pause/restore requires Free tier)...
[INFO] Already on free tier, no changes needed
[STEP] Creating project with PostgresEngine 17...
[ERROR] Failed to create project supadev-pause-1784745764-pg17 with PostgresEngine 17
  (root cause: "IPv6 address is not defined for this project" — local dev stack defect, not this PR)

…s root

The salt subcommand's systemd service runs as the unprivileged
supabase-admin-agent OS user, but /etc/pgbackrest/conf.d is
pgbackrest:postgres 02770 — only root/adminapi can write there. Adds a
NOPASSWD sudoers entry for `pgbackrest reconcile`, mirroring the existing
salt-call entry, so the salt cycle can elevate just this one command
(via sudo) to keep async-archiving config in sync with billing tier
(INDATA-996) without granting the supabase-admin-agent user any broader
group membership.
@hunleyd hunleyd self-assigned this Jul 21, 2026
@blacksmith-sh

This comment has been minimized.

…priv agent

The new sudoers rule in this PR grants the supabase-admin-agent user
NOPASSWD root exec of /opt/supabase-admin-agent/supabase-admin-agent, but
the config dir, extracted archive, and symlink were all owned by that same
low-priv user/group. Since sudo matches on path only (not content/hash),
that account could replace the binary and use the new sudo grant to run
arbitrary code as root. Own the dir/binary/symlink as root:root instead,
matching the existing pgdata-chown/pgdata-signal pattern in this file — the
agent process only needs to execute it, never write to it.
@blacksmith-sh

This comment has been minimized.

@hunleyd
hunleyd marked this pull request as ready for review July 22, 2026 18:04
@hunleyd
hunleyd requested review from a team as code owners July 22, 2026 18:04
@hunleyd
hunleyd enabled auto-merge July 22, 2026 18:05
@hunleyd
hunleyd marked this pull request as draft July 23, 2026 01:52
auto-merge was automatically disabled July 23, 2026 01:52

Pull request was converted to draft

spool-path was never set explicitly, so pgBackRest defaulted to
/var/spool/pgbackrest, which lands on the AMI's root volume (10GB,
shared with the OS/systemd/journal) rather than the /data EBS volume
PGDATA lives on. archive-get's async replica-catch-up queue can hold
several 16MB+ WAL segments at once; once archive-async is enabled
fleet-wide (INDATA-996) that risks filling the root volume and taking
the instance down. Points spool-path at /data/pgbackrest_spool instead,
and updates the directory-creation task to match. Global option, set
once at provisioning — no supabase-admin-agent changes needed.
hunleyd added 10 commits July 23, 2026 11:18
…rationale

Single-line comments only per updated style rules: collapses the 4-line
sudoers rationale into one line, and adds a why-comment on the new
root:root ownership for /opt/supabase-admin-agent so a future edit doesn't
revert it back to the unprivileged service account.
The comment described /etc/pgbackrest/conf.d as root/adminapi-only, but
setup-pgbackrest.yml creates it pgbackrest:postgres 02770 -- the agent
just isn't in either group. Contradicted the PR's own accurate
description of the same path.
…chiving-on-customer-tier

* origin/develop: (36 commits)
  Lots of bootstrap script clean ups (#2326)
  ci: Fix nix-build PUSH_TO_CACHE (#2345)
  ci/nix-install-ephemeral: Drop sticky disk config (#2346)
  chore: bump postgres_release to cut fresh AMIs (includes #2334) (#2349)
  fix(multigres): stop base config data_directory from overriding pooler data dir (#2344)
  fix(cron): remove unnecessary TRIGGER grant on cron.job_run_details from postgres (#2334)
  chore: enable extension version restriction (warn) and cut AMIs (#2315)
  feat(nix): add site-env packages for rolling instance updates (#2283)
  ci: resolve package install sha from a flake input override (#2327)
  fix(multigres): remove dangling wal-g include from postgresql.conf (#2338)
  fix: Allow test CI pipelines to use old Int CA access. For staging only (#2330)
  chore(nix): remove maintainers field from package definitions (#2280)
  chore: bump multigres to b713432 (#2323)
  ci: Use arm-native-runner for kvm builds on aarch64-linux (#2319)
  Whole lot of ansible clean up (#2272)
  fix(ansible): drop no-op zpool loop item from zswap task (#2322)
  ci: Fix dockerhub-release-matrix matrix generation (#2320)
  chore: bump pgctld (#2318)
  fix(ansible): skip zswap params the kernel does not expose (#2321)
  feat(docker): add Dockerfile-supabase base image and rewrite Dockerfile-multigres as layered image (#2160)
  ...
…d unpack tasks

Owner/group alone left permissions umask- and tarball-derived instead of
deterministic; a UMASK 027 hardening pass would yield 0750 and break the
salt service, postgres archive_command, and adminapi execution paths.
A malformed /etc/sudoers.d/supabase-admin-agent breaks sudo fleet-wide,
including the adminapi entries pgBackRest restore depends on. visudo -csf
rejects a bad file before it's ever installed.
The comment implied this change alone stops the agent from ever gaining
root; the salt-call NOPASSWD entry above already grants that group
effectively unrestricted root. Reworded to describe what this change
actually closes: binary tampering ahead of the new sudo grant.
Per convention, comments only cite a Linear issue when tagged as a TODO
for follow-up work, not as a general reference to the ticket a change is
part of.
state: link with the file module's default follow: yes applies owner/group
to the link target, not the symlink itself, and the kernel ignores symlink
ownership for access control regardless. The target is already root-owned
by the unpack task above; these two lines re-asserted nothing.
Previous wording's trailing clause read as describing the entry it was
attached to; reworded so 'the salt subcommand above' unambiguously refers
to the entry on the preceding line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant