Fix oss-source pipeline errors when using the Databricks CLI

[shreyas-goenka]

Changes

Removes validation that incorrectly rejected bundles using the definitions field in databricks.yml.

Why

PR #3797 inadvertently applied validation to reject OSS Spark Declarative Pipelines to all Databricks CLI bundles. This error was only meant for the pipelines CLI. Customers commonly use the definitions field for YAML anchors (e.g., to reuse cluster configurations), and this validation blocked their deployments with the error:

Error: databricks.yml seems to be formatted for open-source Spark Declarative Pipelines.
Pipelines CLI currently only supports Lakeflow Declarative Pipelines development.

The validation has been removed entirely. The Pipelines CLI is not yet in active use, and the pipelines team can add a better context-aware check later that only applies when running as the Pipelines CLI.

Tests

• New acceptance test: acceptance/bundle/validate/definitions_yaml_anchors/ verifies bundles with definitions field validate successfully
• Existing test: acceptance/pipelines/deploy/oss-spark-error/ ensures the error continues to work for the pipelines CLI.

[eng-dev-ecosystem-bot]

Run: 19139802824

	Env	❌​FAIL	🟨​KNOWN	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip
💚​	aws linux			1	1	356	595
💚​	aws windows			1	1	357	594
💚​	aws-ucws linux			1	1	479	490
💚​	aws-ucws windows			1	1	480	489
💚​	azure linux			1	1	356	594
💚​	azure windows			1	1	357	593
💚​	azure-ucws linux			1	1	475	489
💚​	azure-ucws windows			1	1	476	488
❌​	gcp linux	23	1		1	332	596
❌​	gcp windows	21	1		1	335	595

38 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R	🟨​K	🟨​K
TestAccept/bundle/deployment/bind/job/generate-and-bind	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/destroy/jobs-and-pipeline	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/local_state_staleness	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/local_state_staleness/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/dashboards/change-embed-credentials	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/dashboards/change-embed-credentials/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/resources/jobs/fail-on-active-runs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/jobs/fail-on-active-runs/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/models/basic	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/pipelines/allow-duplicate-names	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/resources/pipelines/allow-duplicate-names/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/resources/pipelines/allow-duplicate-names/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/resources/pipelines/auto-approve	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/run/app-with-job	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
TestAccept/bundle/templates/default-python/combinations/classic	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=yes	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=no	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/templates/default-python/integration_classic	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F
TestAccept/selftest/record_cloud/pipeline-crud	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/selftest/record_cloud/workspace-file-io	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestAccept/selftest/record_cloud/workspace-file-io/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F
TestFsCpDir	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDir/dbfs_to_local	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F	✅​p
TestSyncNestedFolderDoesntFailOnNonEmptyDirectory	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	❌​F

[denik]

This scopes it down to deploy, but it seems this is something that would be useful in plan & validate as well, no?

[shreyas-goenka]

This was the previous implementation, the error only triggered on deploy. I'll leave it updo the pipelines team if they want this during dry run.

Note pipeliens CLI does not have plan or validate, only a dryrun command.

[denik]

It's a bit odd to have this here, this package orchestrates existing phases rather than having actual validation logic.

What is we keep it in initialize.go() and then detect pipelines exe via argv[0]?

[shreyas-goenka]

Detecting via argv[0] feels hacky. I don't mind recording IsPipelineCLI in the bundle tree maybe and then record this in initialize.

IMO the current implementation is fine but I'm open to any alternate suggestions.

[shreyas-goenka]

One advantage is it's obvious which code is for pipelines only. You don't have to dive into the guts (the initialize phase here) to see whether some code is for pipelines CLI only or not.