Filter CDC changes at transform time, not just apply time - #52
Merged
Conversation
For clone --follow with --filters, the transform step had no awareness of filters: it fully JSON-decoded and rendered PREPARE/EXECUTE SQL for every change, including for excluded tables, and filtering only happened later at apply time. On a heavy-write source a large fraction of transformed statements can be for excluded tables, wasting transform CPU/disk and apply read/parse work on changes that are always discarded. Move the filter check upstream into parseMessage() so filtered-out tables are never materialized into the .sql files. The apply-time check stays as defense-in-depth. - Share shouldFilterOutTable() via filtering.c/.h (moved out of ld_apply.c). - Wire specs->filters into the transform StreamContext in stream_init_context. - Load filters from the catalog in the prefetch path (stream_start_in_mode), mirroring catchup, so standalone stream prefetch filters too, not just the in-memory clone --follow path. - Dequote identifiers before comparing: wal2json stores names escaped (e.g. "cron") while filter lists hold bare names. Fully-filtered transactions still emit BEGIN/COMMIT so the replication origin advances and the migration does not stall. tests: extend cdc-filtering with exclude-table and a fully-filtered transaction, and assert the generated .sql files contain zero references to excluded schemas/tables. Also wire the cdc-filtering target into tests/Makefile: it was listed in CI but had no target, so it never ran.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
For
clone --followwith--filters, the transform step had no awareness of filters: it fully JSON-decoded and renderedPREPARE/EXECUTESQL for every change, including for excluded tables. Filtering only happened later at apply time, which skipped running the already-rendered statements.On a heavy-write source a large fraction of transformed statements can be for excluded tables — measured ~44% of statements in one 16 MB WAL segment on a large customer — wasting transform CPU/disk and apply read/parse work on changes that were always going to be discarded. This widened the apply-vs-source divergence for filtered workloads.
Change
Move the filter check upstream into
parseMessage()(ld_transform.c) so filtered-out tables are never materialized into the.sqlfiles. The apply-time check stays as defense-in-depth (protects mixed-version rollouts).shouldFilterOutTable()viafiltering.c/filtering.h(moved out ofld_apply.c, unchanged logic).specs->filtersinto the transformStreamContextinstream_init_context()(the field existed but was never populated).stream_start_in_mode), mirroring catchup — so the standalonestream prefetchpath filters too, not only the in-memoryclone --followpath."cron") while the filter lists hold bare names; apply-time worked only because it unquotes when parsing PREPARE text.Fully-filtered transactions still emit
BEGIN/COMMIT, so the replication origin advances and the migration does not stall.Tests
Extended
tests/cdc-filtering:[exclude-table]case (public.filtered_events) and a transaction touching only excluded tables (exercises the empty-transaction / LSN-advance path)..sqlfiles contain zero references to excluded schemas/tables — the core regression test.BEGINcount ==COMMITcount (transactions stay balanced when fully filtered).Also wired the
cdc-filteringtarget intotests/Makefile: it was listed in the CI matrix but had no Makefile target, so it silently did "Nothing to be done" and never actually ran.Verification (PostgreSQL 18, local)
make build/citus_indent --checkcdc-filtering(extended).sql, BEGIN=COMMIT, target state correctcdc-wal2json,cdc-test-decoding,cdc-low-levelfollow-wal2json(theclone --followpath)make testsbatteryCI runs the full suite across PG 16/17/18.