|
| 1 | +--- |
| 2 | +sidebar_label: 'Publication Management' |
| 3 | +description: 'Best practices for managing PostgreSQL publications with ClickPipes' |
| 4 | +slug: /integrations/clickpipes/postgres/source/publication-management |
| 5 | +title: 'PostgreSQL Publication Management for ClickPipes' |
| 6 | +--- |
| 7 | + |
| 8 | +# PostgreSQL Publication Management for ClickPipes |
| 9 | + |
| 10 | +This guide covers best practices and considerations for managing PostgreSQL publications when using ClickPipes for data replication. |
| 11 | + |
| 12 | +## Publication Creation Options {#publication-creation-options} |
| 13 | + |
| 14 | +You have two main approaches for managing publications with ClickPipes: |
| 15 | + |
| 16 | +### Option 1: Automatic Publication Creation (Recommended for Simplicity) {#automatic-publication-creation} |
| 17 | + |
| 18 | +If you **don't specify a publication** during ClickPipe creation, ClickPipes will automatically create a publication scoped to only the tables you select for replication. |
| 19 | + |
| 20 | +**Requirements:** |
| 21 | +- The ClickPipes user must have **table owner permissions** for all tables you want to replicate |
| 22 | +- This is the easiest approach for getting started |
| 23 | + |
| 24 | +**Advantages:** |
| 25 | +- No manual publication management required |
| 26 | +- Publication is automatically scoped to only selected tables |
| 27 | +- No risk of billing for unwanted table data |
| 28 | + |
| 29 | +**Disadvantages:** |
| 30 | +- Requires higher privileges (table owner permissions) |
| 31 | +- If you add tables to the pipe later, ClickPipes will need to modify the publication, requiring continued owner permissions |
| 32 | + |
| 33 | +### Option 2: Manual Publication Creation (Recommended for Production) {#manual-publication-creation} |
| 34 | + |
| 35 | +Create and manage the publication yourself before setting up the ClickPipe. |
| 36 | + |
| 37 | +#### Table-Specific Publication (Recommended) {#table-specific-publication} |
| 38 | + |
| 39 | +```sql |
| 40 | +-- Create a publication for specific tables you wish to replicate |
| 41 | +-- Replace 'schema.table1', 'schema.table2' with your actual schema and table names |
| 42 | +CREATE PUBLICATION clickpipes_publication FOR TABLE schema.table1, schema.table2; |
| 43 | +``` |
| 44 | + |
| 45 | +**Advantages:** |
| 46 | +- Fine-grained control over which tables are included |
| 47 | +- Only requires SELECT permissions on tables (not ownership) |
| 48 | +- Clear visibility into what data will be replicated |
| 49 | +- Optimal billing - only pay for data you actually need |
| 50 | + |
| 51 | +**Disadvantages:** |
| 52 | +- Manual management required |
| 53 | +- Must update publication when adding/removing tables from replication |
| 54 | + |
| 55 | +#### All-Tables Publication (Use with Caution) {#all-tables-publication} |
| 56 | + |
| 57 | +```sql |
| 58 | +-- Creates publication for all current and future tables |
| 59 | +CREATE PUBLICATION clickpipes_publication FOR ALL TABLES; |
| 60 | +``` |
| 61 | + |
| 62 | +**Important Considerations:** |
| 63 | +- Though ClickPipes will only replicate tables you select in the UI, PostgreSQL will send data changes from **all tables** over the network |
| 64 | +- **You will be billed for the extra bytes** from tables you don't actually need |
| 65 | +- This approach is generally not recommended for production use |
| 66 | + |
| 67 | +## Permission Requirements {#permission-requirements} |
| 68 | + |
| 69 | +### For Automatic Publication Creation {#automatic-permissions} |
| 70 | +```sql |
| 71 | +-- Grant table ownership (required for automatic publication creation) |
| 72 | +ALTER TABLE schema.table1 OWNER TO clickpipes_user; |
| 73 | +ALTER TABLE schema.table2 OWNER TO clickpipes_user; |
| 74 | +``` |
| 75 | + |
| 76 | +### For Manual Publication Creation {#manual-permissions} |
| 77 | +```sql |
| 78 | +-- Minimal permissions (sufficient for manual publication approach) |
| 79 | +GRANT USAGE ON SCHEMA "your_schema" TO clickpipes_user; |
| 80 | +GRANT SELECT ON TABLE schema.table1, schema.table2 TO clickpipes_user; |
| 81 | +ALTER USER clickpipes_user REPLICATION; |
| 82 | + |
| 83 | +-- If you need to grant permissions for future tables in the schema: |
| 84 | +ALTER DEFAULT PRIVILEGES IN SCHEMA "your_schema" GRANT SELECT ON TABLES TO clickpipes_user; |
| 85 | +``` |
| 86 | + |
| 87 | +## Managing Publications Over Time {#managing-publications-over-time} |
| 88 | + |
| 89 | +### Adding Tables to an Existing Publication {#adding-tables} |
| 90 | + |
| 91 | +If you need to add tables to your ClickPipe later: |
| 92 | + |
| 93 | +```sql |
| 94 | +-- Add a single table |
| 95 | +ALTER PUBLICATION clickpipes_publication ADD TABLE schema.new_table; |
| 96 | + |
| 97 | +-- Add multiple tables |
| 98 | +ALTER PUBLICATION clickpipes_publication ADD TABLE schema.table3, schema.table4; |
| 99 | +``` |
| 100 | + |
| 101 | +### Removing Tables from a Publication {#removing-tables} |
| 102 | + |
| 103 | +```sql |
| 104 | +-- Remove a single table |
| 105 | +ALTER PUBLICATION clickpipes_publication DROP TABLE schema.old_table; |
| 106 | + |
| 107 | +-- Remove multiple tables |
| 108 | +ALTER PUBLICATION clickpipes_publication DROP TABLE schema.table3, schema.table4; |
| 109 | +``` |
| 110 | + |
| 111 | +### Viewing Publication Contents {#viewing-publication-contents} |
| 112 | + |
| 113 | +```sql |
| 114 | +-- See which tables are in your publication |
| 115 | +SELECT schemaname, tablename |
| 116 | +FROM pg_publication_tables |
| 117 | +WHERE pubname = 'clickpipes_publication'; |
| 118 | + |
| 119 | +-- See all publications |
| 120 | +SELECT * FROM pg_publication; |
| 121 | +``` |
| 122 | + |
| 123 | +## Best Practices {#best-practices} |
| 124 | + |
| 125 | +1. **Start with table-specific publications** for production workloads to maintain cost control and clarity |
| 126 | + |
| 127 | +2. **Use automatic creation for development/testing** when you want to get started quickly and don't mind granting higher privileges |
| 128 | + |
| 129 | +3. **Plan your table selection carefully** - adding/removing tables later requires publication changes |
| 130 | + |
| 131 | +4. **Monitor your data usage** to ensure you're only paying for data you actually need |
| 132 | + |
| 133 | +5. **Document your publication strategy** so team members understand the setup |
| 134 | + |
| 135 | +## Billing Implications {#billing-implications} |
| 136 | + |
| 137 | +- **Table-specific publications**: You pay only for data changes from explicitly listed tables |
| 138 | +- **All-tables publications**: You pay for data changes from all tables in the database, even if ClickPipes only processes a subset |
| 139 | +- **Automatic creation**: You pay only for data from selected tables (most cost-effective) |
| 140 | + |
| 141 | +## Additional Resources {#additional-resources} |
| 142 | + |
| 143 | +For more advanced publication options and syntax, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-createpublication.html). |
| 144 | + |
| 145 | +## Troubleshooting {#troubleshooting} |
| 146 | + |
| 147 | +### Permission Errors {#permission-errors} |
| 148 | +If you encounter permission errors: |
| 149 | +1. Verify the ClickPipes user has the required permissions for your chosen approach |
| 150 | +2. Check that the publication exists and contains the expected tables |
| 151 | +3. Ensure the user has REPLICATION privileges |
| 152 | + |
| 153 | +### Unexpected Billing {#unexpected-billing} |
| 154 | +If you're seeing higher than expected data transfer costs: |
| 155 | +1. Check if you're using an all-tables publication unintentionally |
| 156 | +2. Verify your publication only contains the tables you need |
| 157 | +3. Monitor which tables are actively changing in your database |
0 commit comments