refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive #1843

hunleyd · 2025-10-14T13:46:22Z

What kind of change does this PR introduce?

Moved all config files that are included in the postgresql.conf into the /etc/postgresql-custom dir then updated postgresql.conf to use include_dir instead of include. To ensure the same ordering of the various GUCs (to ensure proper overriding of values), rename files with digits prepended.
Finally, split auto_explain and pg_cron out into their own config files instead of them being tacked on the end of the main PG config.

this is take 2, which should fix the supautils issue

…dir' directive and move logging.conf into the postgresql-customer dir

… preserve ordering

…wn config files

* 'develop' of github.com:supabase/postgres: (29 commits) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1818) fix: update Dockerfiles for changes to postgis multiversion (#1817) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1813) feat: support multiple versions of the postgis extension (#1667) Update pgsql-http to handle semver (#1816) feat: add retry policy for auth service routes (#1782) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1811) fix: move tmpdir for SAA to one that always exists (#1799) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1810) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1809) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1807) build: automatically cancel old tests/build on new push (#1808) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1804) docs: small adjust (#1806) docs: getting started guide in README (#1805) fix: templating of SAA service file fix: adjust the qemu vars filename refactor(ansible): bring our ansible up to modern ansible-lint standards feat: support multiple versions of the http extension (#1664) feat: supply a slightly different supabase-admin-agent configuration for qemu artifacts ...

…lute path

…les to ensure ordering

* origin/develop: fix(nix): remove '%' character from Nix trusted-public-keys configuration (#1840) chore: systemd hardening (#1837) Revert "refactor(postgresq): switch to 'include_dir' and then rename conf files to ensure ordering" (#1835) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1821) fix(pgmq): replace drop_queue function if exists (#1828) refactor(postgresq): switch to 'include_dir' and then renames conf files to ensure ordering (#1820) feat: support multiple versions of the pg_jsonschema extension (#1757) feat: bump auth to v2.180.0 (#1829) feat: update supautils confs w/ new tables feat(migrations): predefined role grants (#1815) feat: run pg_regress tests after installing the last version of the extension (#1826) fix(pgmq): add missing helper function in migration script (#1825) feat: support multiple versions of the pgmq extension (#1668) feat: run pg_regress during extension tests (#1812) feat: support multiple versions of the pgrouting extension (#1687) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1819) feat: support multiple versions of the pg-graphql extension (#1761)

…e numeric prefixes as per Sam

…vars

…ions

* fix: refactor tests to work with include_dir approach and be more accurate with production conditions * tests: use distinct savepoint names in test

* origin: refactor(ansible): bring our ansible up to modern ansible-lint standards (#1852) chore: bump to release (#1855) docs: getting started nix install or config revised (#1853) docs: how to update new structure (#1851) feat: support multiple versions of the plpgsql_check extension (#1684) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1848) feat(wrappers): support more versions (#1831) refactor(ansible): bring our ansible up to modern ansible-lint standards (#1833) fix: fine tune protection rules to unblock wal-g functionality (#1846) fix: incorporate v3.0.0 supautils (#1844) fix: remove git revision from postgres package

samrose · 2025-10-23T16:01:07Z

@hunleyd this is still in draft mode of course. But requesting that once we are ready for review, we need @soedirgo @steve-chavez @za-arthur and myself to review before we merge and deploy just due to all of the changes.

jchancojr · 2025-11-10T19:15:27Z

/data/pgdata/postgresql.conf still exists. Should we symlink this to postgresql-custom or something? @hunleyd

hunleyd · 2025-11-10T21:31:01Z

/data/pgdata/postgresql.conf still exists. Should we symlink this to postgresql-custom or something? @hunleyd

imho, we should, yes. or remove them entirely. all the .conf files in this dir (except postgresql.auto.conf obviously).

interested in @samrose 's thoughts though

samrose · 2025-11-13T18:06:50Z

@hunleyd see if you agree with all of this

Recommending remove the symlink entirely and maintain your current approach of configs in /etc/postgresql/ with lots of testing to make sure nothing breaks:

  Why This is Best:

  1. PostgreSQL Official Documentation endorses separating configs from data
  2. Debian/Ubuntu Standard Practice (which your Ansible builds follow)
  3. Your current postgresql.conf.j2 already implements this correctly:
  data_directory = '/var/lib/postgresql/data'
  hba_file = '/etc/postgresql/pg_hba.conf'
  include_dir = '/etc/postgresql-custom/conf.d'
  4. The symlink serves no purpose - PostgreSQL is already configured to read from /etc/

PostgreSQL Official Docs: https://www.postgresql.org/docs/current/runtime-config-file-locations.html
include_dir Best Practices: https://www.postgresql.org/docs/current/config-setting.html

Just please test as much as you can in local infra, and we should run e2e tests against your branch. Please create a playbook to document running e2e tests, but also reach out for help if you get stuck trying to do it. Thanks for the careful work on this.

* 'develop' of github.com:supabase/postgres: fix: don't run a nix installer on qemu image build (#1916) fix: restore ami release to us east 1 (#1913) feat: upgrade admin-mgr to 0.32.3 (#1915) fix: Prepare job (lines 23-25): Moved checkout before nix-install-ephemeral (#1914) feat: support multiple versions of the wal2json extension (#1875)

This change adds a task to remove `pg_hba.conf`, `pg_ident.conf`, and `postgresql.conf` from the PostgreSQL data directory. This ensures there is no confusion about what configuration file is active.

* origin: Revert "feat: update supautils (#1879)" (#1912)

Symlinks PostgreSQL configuration files (`pg_hba.conf`, `pg_ident.conf`, `postgresql.conf`) from `/etc/postgresql/` into the PGDATA directory (`/var/lib/postgresql/data/`). This ensures that the configuration files within PGDATA are always managed centrally and point to the correct, actively used configuration. The `force: true` option is added to overwrite any existing files, guaranteeing the symlink is correctly established.

* origin: feat: support multiple versions of the pg_plan_filter extension (#1873)

- Explicitly set the data directory using `--pgdata` in `pg_ctl initdb` commands.

* 'develop' of github.com:supabase/postgres: fix: add alpn_protocols to envoy (#1503)

…t gets rid of the arg errors

…-152 * 'INDATA-152' of github.com:supabase/postgres:

- Standardize Ansible module usage for PostgreSQL configuration files and related tasks. - Update the default PostgreSQL data directory for `initdb` to `/data/pgdata`.

This commit updates the Ansible tasks for setting up PostgreSQL. The primary changes involve: - Shifting the `initdb` executable path from `/usr/lib/postgresql/bin/pg_ctl` to `/var/lib/postgresql/.nix-profile/bin/initdb`, reflecting a Nix-based installation. - Adjusting the arguments passed to `initdb` for clarity and correctness. - Updating the `creates` path in the Ansible `args` to match the new data directory structure.

* 'develop' of github.com:supabase/postgres: feat: support multiple versions of the pg-safeupdate extension (#1871) feat(setup-postgres.yml): enable checksums via initdb for postgres (#1891)

Consolidate template tasks for postgresql.conf, pg_hba.conf, and pg_ident.conf into a single loop. Ensure placeholder config files are correctly placed within `/etc/postgresql-custom/conf.d/`. Remove the redundant `-D /data/pgdata` argument from the `initdb` commands, as the data directory is now managed by a symlink to `/data/pgdata`. Add debug commands to verify the data directory symlink.

* 'develop' of github.com:supabase/postgres: feat: support multiple versions of the pg_stat_monitor extension (#1683) feat: support multiple versions of the pgjwt extension (#1751)

blacksmith-sh · 2025-11-17T16:33:43Z

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
`z_17_pg_stat_monitor`	View Logs

hunleyd and others added 25 commits October 6, 2025 11:23

refactor(postgresql.conf): configure postgresql.conf to use 'include_…

07ab014

…dir' directive and move logging.conf into the postgresql-customer dir

refactor(ansible): rename all conf files in /etc/postgresql-custom to…

e0426ee

… preserve ordering

refactor(postgresql): split auto_explain and pg_cron out into their o…

7876920

…wn config files

fix(nix): it helps to use the correct filename everywhere

0c932e9

refactor(postgresql.conf): use relative path to include_dir

6ce9f8b

fix(postgresql.conf): revert the location of include_dir back to abso…

b2a64fd

…lute path

refactor(postgresql): switch to 'include_dir' and then rename conf fi…

d4092d8

…les to ensure ordering

refactor(docker): don't sed, just echo

a01355d

fix: test harness needed to handle config files correctly

e95145a

refactor(PG-configuration): move includes to conf.d and rename wth th…

161cd90

…e numeric prefixes as per Sam

fix(nix): nix doesn't like it when you try to assign vars to undef'd …

f18cd9e

…vars

fix: update path to 05-supautils

100a6e1

fix: update path to 00-logging

c51b05e

refactor(postgresql): mv 00-logging.conf to just logging.conf

55c0ea7

refactor(postgres): 01-generated-optimizations -> generated-optimizat…

bf4c03e

…ions

refactor(postgresql): 02-custom-overrides -> custom-overrides

fcf4c28

refactor(postgres): 03-wal-g -> wal-g

fdb7898

refactor(postgres): 04-read-replica -> read-replica

b96ed46

refactor(postgres): 05-supautils -> supautils

bf0133c

style(postgresql): style fixups

da987d4

fix: more path updates

9bba13d

Indata 152 run server (#1850)

1f82f5b

* fix: refactor tests to work with include_dir approach and be more accurate with production conditions * tests: use distinct savepoint names in test

hunleyd requested review from samrose, soedirgo, steve-chavez and za-arthur October 24, 2025 13:53

hunleyd and others added 3 commits November 11, 2025 10:46

Merge branch 'develop' into INDATA-152

b811ef4

Merge branch 'develop' into INDATA-152

917e43c

test(nix): update epected test outputs

519f7b0

hunleyd added 3 commits November 13, 2025 13:21

fix: Remove extraneous postgresql config files

c492227

This change adds a task to remove `pg_hba.conf`, `pg_ident.conf`, and `postgresql.conf` from the PostgreSQL data directory. This ensures there is no confusion about what configuration file is active.

Merge remote-tracking branch 'origin' into INDATA-152

6124db1

* origin: Revert "feat: update supautils (#1879)" (#1912)

hunleyd changed the title ~~refactor(postgresql.conf): configure postgresql.conf to use 'include_…dir' directive~~ refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive Nov 13, 2025

hunleyd added 2 commits November 13, 2025 16:34

Merge remote-tracking branch 'origin' into INDATA-152

e094056

* origin: feat: support multiple versions of the pg_plan_filter extension (#1873)

samrose self-requested a review November 14, 2025 14:02

hunleyd added 13 commits November 14, 2025 09:16

fix: Fix PostgreSQL initdb and config symlinks

bec3bfc

- Explicitly set the data directory using `--pgdata` in `pg_ctl initdb` commands.

Merge branch 'develop' into INDATA-152

5afbf50

Merge branch 'develop' of github.com:supabase/postgres into INDATA-152

17a7f54

* 'develop' of github.com:supabase/postgres: fix: add alpn_protocols to envoy (#1503)

fix(setup-postgres): switch from cmd to argv format to see if tha…

c8f5b9f

…t gets rid of the arg errors

Merge branch 'INDATA-152' of github.com:supabase/postgres into INDATA…

876f1d2

…-152 * 'INDATA-152' of github.com:supabase/postgres:

fix(setup-postgres): use the short form for the data dir option

5d9830a

fix(setup-postgres): use the short form for the data dir option

4882331

refactor: Update Postgres Ansible tasks and data dir

9be0668

- Standardize Ansible module usage for PostgreSQL configuration files and related tasks. - Update the default PostgreSQL data directory for `initdb` to `/data/pgdata`.

Merge branch 'develop' of github.com:supabase/postgres into INDATA-152

e8b9f55

* 'develop' of github.com:supabase/postgres: feat: support multiple versions of the pg-safeupdate extension (#1871) feat(setup-postgres.yml): enable checksums via initdb for postgres (#1891)

fix(setup-docker): copy supautils.conf in nixpkg_mode

55dc27a

Merge branch 'develop' of github.com:supabase/postgres into INDATA-152

7679ee5

* 'develop' of github.com:supabase/postgres: feat: support multiple versions of the pg_stat_monitor extension (#1683) feat: support multiple versions of the pgjwt extension (#1751)

supabase deleted a comment from blacksmith-sh bot Nov 17, 2025

hunleyd marked this pull request as draft November 17, 2025 16:38

hunleyd mentioned this pull request Nov 17, 2025

feat(tests): add updated and improved regression tests #1923

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive #1843

refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive #1843

hunleyd commented Oct 14, 2025

Uh oh!

samrose commented Oct 23, 2025

Uh oh!

jchancojr commented Nov 10, 2025

Uh oh!

hunleyd commented Nov 10, 2025

Uh oh!

samrose commented Nov 13, 2025

Uh oh!

blacksmith-sh bot commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive #1843

Are you sure you want to change the base?

refactor(postgresql.conf): configure postgresql.conf to use 'include_dir' directive #1843

Conversation

hunleyd commented Oct 14, 2025

Uh oh!

samrose commented Oct 23, 2025

Uh oh!

jchancojr commented Nov 10, 2025

Uh oh!

hunleyd commented Nov 10, 2025

Uh oh!

samrose commented Nov 13, 2025

Uh oh!

blacksmith-sh bot commented Nov 17, 2025

Failure

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants