Skip to content

Document Python BigQuery CDC record format - #40021

Open
mukeshbhandarkar wants to merge 1 commit into
apache:masterfrom
mukeshbhandarkar:docs/39768-bigquery-cdc-format
Open

Document Python BigQuery CDC record format#40021
mukeshbhandarkar wants to merge 1 commit into
apache:masterfrom
mukeshbhandarkar:docs/39768-bigquery-cdc-format

Conversation

@mukeshbhandarkar

Copy link
Copy Markdown

What does this PR do?

Documents the Python BigQuery Change Data Capture (CDC) input format for WriteToBigQuery.

The new documentation explains:

  • the required STORAGE_WRITE_API, use_at_least_once, and use_cdc_writes settings
  • primary key requirements
  • the expected row_mutation_info / record PCollection structure
  • UPSERT and DELETE mutation types
  • change_sequence_number
  • how Beam maps mutation metadata to BigQuery _CHANGE_TYPE and _CHANGE_SEQUENCE_NUMBER
  • dictionary input requirements

The use_cdc_writes Python API documentation is also clarified.

Fixes #39768

Testing

  • git diff --check
  • Python 3.12 compilation
  • Verified Hugo shortcode delimiters
  • Confirmed only the intended documentation/API docstring files changed

I did not run ./gradlew :website:testWebsite locally because it requires building the website environment and related dependencies that are not available in my local setup.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@stankiewicz
stankiewicz self-requested a review September 9, 2026 07:26
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.
R: @Abacn for label website.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: document BigQuery CDC record format

2 participants