Add Lakebase ETL and Reverse ETL recipes (CLI-focused)#6
Closed
sav-maya wants to merge 1 commit intodatabricks:mainfrom
Closed
Add Lakebase ETL and Reverse ETL recipes (CLI-focused)#6sav-maya wants to merge 1 commit intodatabricks:mainfrom
sav-maya wants to merge 1 commit intodatabricks:mainfrom
Conversation
Two new recipes for Lakebase Autoscaling data movement: - ETL (Lakehouse Sync): Replicate Lakebase Postgres tables into Unity Catalog as SCD Type 2 Delta history tables via CDC - Reverse ETL (Synced Tables): Sync Unity Catalog tables into Lakebase Postgres for sub-10ms app reads via CLI Both recipes are CLI-first with explicit callouts where steps still require the Databricks UI. Tested end-to-end against a live workspace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
etl-lakehouse-sync-autoscaling.md): Replicate Lakebase Postgres tables into Unity Catalog as SCD Type 2 Delta history tables using Lakehouse Sync (CDC-based, native Lakebase feature)reverse-etl-synced-tables-autoscaling.md): Sync Unity Catalog tables into Lakebase Postgres for sub-10ms application reads usingdatabricks database create-synced-database-tableBoth recipes are CLI-first with explicit "not yet available via CLI" callouts (with condensed UI instructions) where no CLI alternative exists yet.
What was validated
databricks database --helpanddatabricks postgres --help(CLI v0.295.0)create-synced-database-table→get-synced-database-table→ queried synced data in Postgres via psql (all 5 rows returned)/postgres/API caveat (project created via postgres API correctly returns "Database instance is not found" for synced table creation)Test plan