Skip to content

Add Lakebase ETL and Reverse ETL recipes for Autoscaling#7

Merged
andrelandgraf merged 6 commits intodatabricks:mainfrom
sav-maya:savannah/streamline-lakebase-recipes
Mar 20, 2026
Merged

Add Lakebase ETL and Reverse ETL recipes for Autoscaling#7
andrelandgraf merged 6 commits intodatabricks:mainfrom
sav-maya:savannah/streamline-lakebase-recipes

Conversation

@sav-maya
Copy link
Collaborator

Summary

  • Adds two new recipes for Lakebase Autoscaling data sync workflows:
    • ETL: Lakehouse Sync — replicate Lakebase Postgres tables into Unity Catalog as SCD Type 2 Delta history tables
    • Reverse ETL: Synced Tables — serve Unity Catalog tables through Lakebase Postgres with sub-10ms latency
  • Both recipes follow the standard recipe heading format (h2 title, numbered h3 steps, h4 References)

Test plan

  • Verify both recipe files render correctly in local dev (npm run dev)
  • Confirm heading levels and numbered steps match the convention in existing recipes
  • Verify all external links resolve

Two new recipes for Lakebase Autoscaling data movement:

- ETL (Lakehouse Sync): Replicate Lakebase Postgres tables into Unity
  Catalog as SCD Type 2 Delta history tables via CDC
- Reverse ETL (Synced Tables): Sync Unity Catalog tables into Lakebase
  Postgres for sub-10ms app reads via CLI

Both recipes are CLI-first with explicit callouts where steps still
require the Databricks UI. Tested end-to-end against a live workspace.
Demote heading levels, number action steps, and rename "Learn more"
to "References" to match the convention used by all other recipes.
Lockfile was out of sync, causing `bun install --frozen-lockfile` to
fail in CI.
Format etl-lakehouse-sync-autoscaling.md and copy-markdown.spec.ts.
Slug parity check requires every content/recipes/*.md file to have a
matching entry in the recipes registry (plugins/content-entries.ts).
@andrelandgraf andrelandgraf merged commit 081fc4f into databricks:main Mar 20, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants