Allow configuring cascade deletion in DABs#5846
Conversation
Add a `cascade` field to the pipeline bundle resource that controls whether destroying a pipeline also deletes its datasets (materialized views, streaming tables, and views). When unset, the server default applies (cascade); set `cascade: false` to retain the datasets on destroy. The field is delete-time only: it is not part of the pipeline spec, so it is never sent on create/update. The direct engine persists it in state (via a PipelineState wrapper around CreatePipeline, mirroring the sql_warehouse lifecycle pattern) and reads it at delete time, force-sending cascade so an explicit false survives query-string omitempty (honored for query params as of databricks-sdk-go v0.152.0). A cascade-only change is a state-only update with no pipeline API call. cascade is classified input_only so it does not show remote drift. The terraform engine drops the field for now: the pinned provider has no such attribute yet (pending terraform-provider-databricks#5860), so terraform-engine support is a follow-up. Co-authored-by: Isaac
Approval status: pending
|
| PipelineId: id, | ||
| Cascade: *state.Cascade, | ||
| Force: false, | ||
| ForceSendFields: []string{"Cascade"}, |
There was a problem hiding this comment.
Could you add a comment about why this is used for maintainers?
| pipelines: | ||
| my_pipeline: | ||
| name: test-pipeline-cascade | ||
| cascade: false |
There was a problem hiding this comment.
We should review the name of this propetty and whether we should fold it into any more general 'lifecycle' concept.
Note that, overall, I think this approach is sound: we have a pipeline-level property that defines the deletion behavior. As opposed to a parameter for 'databricks bundle destroy'.
Let me add @denik here in case he has considerations and I'll bring it up in the DABs Core meeting tomorrow.
There was a problem hiding this comment.
We discussed this:
| cascade: false | |
| cascade_on_destroy: false |
would be the right name.
We have a purge_on_destroy that works the same way for Lakebase.
| >>> print_requests.py //api/2.0/pipelines | ||
|
|
||
| === Destroy: delete reads the updated cascade=true from persisted state | ||
| >>> [CLI] pipelines destroy --auto-approve |
There was a problem hiding this comment.
This should actually also work without --auto-approve: we should not show a warning when we know the remote pipeline datasets won't be deleted.
Related, when we do show a warning, the warning might need to mention this property.
| pipelines: | ||
| my_pipeline: | ||
| name: test-pipeline-cascade | ||
| cascade: false |
There was a problem hiding this comment.
Could you make sure the default template also sets purge_on_destroy: false? With a very short comment to the side explaining it? Could be a separate PR.
| # --cascade only affects the direct engine's delete request; the terraform engine deletes | ||
| # pipelines through the provider and ignores it. Restrict to direct so the recorded request |
There was a problem hiding this comment.
Could you add a check to make sure there' a hard error when the new property is used with the Terraform engine?
… message fix - Rename the pipeline config field cascade -> cascade_on_destroy, matching the Lakebase purge_on_delete precedent (per review discussion). - Add ValidateCascadeOnDestroy mutator: hard error when the field is used with the terraform engine, which does not support it yet. - Fix the destroy approval message so a pipeline with cascade_on_destroy: false is not described as deleting its STs/MVs, and mention the property otherwise. - Comment why ForceSendFields is needed in DoDelete. - Regenerate schema, refschema, and affected acceptance outputs. Co-authored-by: Isaac
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
2 participants
Changes
Add a
cascadefield to the pipeline bundle resource that controls whether destroying a pipeline also cascades to its datasets. When unset, we maintain the existing default of true.This change applies to the direct engine. We have a separate change on the Terraform side: databricks/terraform-provider-databricks#5860
Why
Previously, pipeline deletion would automatically cascade to its tables. After user feedback, we decided to make this configurable in the DeletePipeline API. This extends support to DABs as well, which was another common user ask.
Tests
Unit tests