Skip to content

Filter CDC changes at transform time, not just apply time - #52

Merged
teknogeek0 merged 1 commit into
mainfrom
fix/cdc-filter-at-transform-time
Jul 25, 2026
Merged

Filter CDC changes at transform time, not just apply time#52
teknogeek0 merged 1 commit into
mainfrom
fix/cdc-filter-at-transform-time

Conversation

@teknogeek0

Copy link
Copy Markdown
Collaborator

Problem

For clone --follow with --filters, the transform step had no awareness of filters: it fully JSON-decoded and rendered PREPARE/EXECUTE SQL for every change, including for excluded tables. Filtering only happened later at apply time, which skipped running the already-rendered statements.

On a heavy-write source a large fraction of transformed statements can be for excluded tables — measured ~44% of statements in one 16 MB WAL segment on a large customer — wasting transform CPU/disk and apply read/parse work on changes that were always going to be discarded. This widened the apply-vs-source divergence for filtered workloads.

Change

Move the filter check upstream into parseMessage() (ld_transform.c) so filtered-out tables are never materialized into the .sql files. The apply-time check stays as defense-in-depth (protects mixed-version rollouts).

  • Share shouldFilterOutTable() via filtering.c/filtering.h (moved out of ld_apply.c, unchanged logic).
  • Wire specs->filters into the transform StreamContext in stream_init_context() (the field existed but was never populated).
  • Load filters from the catalog in the prefetch path (stream_start_in_mode), mirroring catchup — so the standalone stream prefetch path filters too, not only the in-memory clone --follow path.
  • Dequote identifiers before comparing: wal2json stores names escaped (e.g. "cron") while the filter lists hold bare names; apply-time worked only because it unquotes when parsing PREPARE text.

Fully-filtered transactions still emit BEGIN/COMMIT, so the replication origin advances and the migration does not stall.

Tests

Extended tests/cdc-filtering:

  • Added an [exclude-table] case (public.filtered_events) and a transaction touching only excluded tables (exercises the empty-transaction / LSN-advance path).
  • Assert the generated .sql files contain zero references to excluded schemas/tables — the core regression test.
  • Assert BEGIN count == COMMIT count (transactions stay balanced when fully filtered).

Also wired the cdc-filtering target into tests/Makefile: it was listed in the CI matrix but had no Makefile target, so it silently did "Nothing to be done" and never actually ran.

Verification (PostgreSQL 18, local)

Check Result
make build / citus_indent --check clean
cdc-filtering (extended) pass — 0 excluded refs in .sql, BEGIN=COMMIT, target state correct
cdc-wal2json, cdc-test-decoding, cdc-low-level pass (both output plugins)
follow-wal2json (the clone --follow path) pass
full make tests battery in progress at push time; green through the CDC/follow suites

CI runs the full suite across PG 16/17/18.

For clone --follow with --filters, the transform step had no awareness of
filters: it fully JSON-decoded and rendered PREPARE/EXECUTE SQL for every
change, including for excluded tables, and filtering only happened later at
apply time. On a heavy-write source a large fraction of transformed
statements can be for excluded tables, wasting transform CPU/disk and apply
read/parse work on changes that are always discarded.

Move the filter check upstream into parseMessage() so filtered-out tables
are never materialized into the .sql files. The apply-time check stays as
defense-in-depth.

- Share shouldFilterOutTable() via filtering.c/.h (moved out of ld_apply.c).
- Wire specs->filters into the transform StreamContext in stream_init_context.
- Load filters from the catalog in the prefetch path (stream_start_in_mode),
  mirroring catchup, so standalone stream prefetch filters too, not just the
  in-memory clone --follow path.
- Dequote identifiers before comparing: wal2json stores names escaped
  (e.g. "cron") while filter lists hold bare names.

Fully-filtered transactions still emit BEGIN/COMMIT so the replication origin
advances and the migration does not stall.

tests: extend cdc-filtering with exclude-table and a fully-filtered
transaction, and assert the generated .sql files contain zero references to
excluded schemas/tables. Also wire the cdc-filtering target into
tests/Makefile: it was listed in CI but had no target, so it never ran.
@teknogeek0
teknogeek0 merged commit fccbd0c into main Jul 25, 2026
96 checks passed
@teknogeek0
teknogeek0 deleted the fix/cdc-filter-at-transform-time branch July 25, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant