experimental/bundletest: offline variable resolution for the local backend - #6632
Draft
Sankalp-Mittal wants to merge 4 commits into
Draft
experimental/bundletest: offline variable resolution for the local backend#6632Sankalp-Mittal wants to merge 4 commits into
Sankalp-Mittal wants to merge 4 commits into
Conversation
…es loudly
The local backend read databricks.yml with raw yaml.safe_load, so ${var.x},
targets and presets were left literal — local could diverge from deploy on any
interpolated field. Resolve ${var.name} offline (BUNDLE_VAR_* env override, else
the declared default), including a variable whose value references another
variable. No auth, no network — the local tier stays offline.
References only the workspace can resolve — ${workspace.*}, ${resources.*}, a
lookup variable, or an unset variable — are NOT hand-resolved (that would be the
reimplementation trap); they're left literal and rejected loudly via
LocalUnsupported at the use site (get_resource / a job's sql path), never
silently passed through as "${...}". Plain ${var.name} substitution is a small,
stable spec, so resolving it locally is safe; the auth-requiring parts are
exactly the ones that aren't locally meaningful anyway.
The example bundle's warehouse_id now has a placeholder default so it resolves
offline. tests/test_variables.py covers default/env-override/nested resolution
and the loud skip for workspace/lookup/unset references.
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/dabs-var-resolution
branch
from
September 11, 2026 12:59
04d2b7d to
d96098b
Compare
Sankalp-Mittal
changed the base branch from
sankalp-mittal/dabs-testing-framework
to
sankalp-mittal/dabs-cloud-backend
September 11, 2026 13:01
Sankalp-Mittal
added this pull request to stack #6604
September 11, 2026 13:01
Collaborator
Integration test reportCommit: c1fb887
Top 6 slowest tests (at least 2 minutes):
|
Replace the hand-rolled Python ${var} resolver in the local (DuckDB) backend
with cmd/offline-resolve, a tiny Go helper that loads and resolves the bundle
offline by reusing the CLI's own mutators — no auth, no network, no
reimplementation. deploy() now subprocesses it and parses the resolved config
as JSON, so includes, target overrides, presets, and ${var.*}/${bundle.*} all
resolve exactly as `bundle validate` would (the old resolver read only the
single databricks.yml and handled only ${var.*}).
The helper applies only the mutators that are safe offline, stopping before the
first auth call (PopulateCurrentUser) and skipping ResolveLookupVariables. It
passes offline-only prefixes {bundle, variables} to the resolver so ${workspace.*}
stays literal. A variable the workspace alone can resolve (a lookup or unset
variable) would otherwise abort resolution (SetVariables errors on an unset
required variable; dynvar errors on an unresolvable lookup reference), so the
helper seeds such variables a sentinel default and drops their lookup, letting
resolution complete with the sentinel flowing into the output.
The backend's loud-skip guard now matches both a residual ${...} and the
sentinel, so a resource or sql path that needs a workspace is LocalUnsupported
at the use site with a specific reason. Adds tests proving include and
target-override resolution, which the old resolver could not do.
Co-authored-by: Isaac <no-reply@databricks.com>
…d' into sankalp-mittal/dabs-var-resolution
The local backend now resolves bundle config by running the in-repo Go helper cmd/offline-resolve, so deploy()-based tests shell out to `go run`. The bundletest CI job was uv-only, so add actions/setup-go (pinned to the repo's go.mod version) before the pytest step, otherwise those tests go red on a runner without a matching Go. Also note the Go + CLI-repo requirement in the README. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The local (DuckDB) backend read
databricks.ymlwith rawyaml.safe_load, so${var.x}was left literal — local config could diverge from whatdeployrenders on any interpolated field. This resolves${var.name}offline and rejects workspace-dependent references loudly.Stacked on the base (
sankalp-mittal/dabs-testing-framework); independent of the resource-handles and cloud PRs (touches onlyduckdb.py+ the example).How
${var.name}←BUNDLE_VAR_<name>env override, else the declareddefault(incl. shorthandvariables: {name: value}). A variable whose value references another variable resolves too. No auth, no network — the local tier stays offline.${workspace.*},${resources.*}, alookupvariable, or an unset variable can't be resolved without the workspace. These are not hand-resolved (that would be a reimplementation trap) — they're left literal and raiseLocalUnsupportedat the use site (get_resource, or a job'ssql_taskpath), so they skip with a reason, never silently pass through as${...}.Why this split is safe: plain
${var.name}substitution is a small, stable spec, so resolving it locally won't diverge from DABs; the auth-requiring parts (lookup/workspace) are exactly the ones that aren't locally meaningful anyway (alookupliterally queries the live workspace).Changes
backends/duckdb.py:_resolve_variablesindeploy;_online_referenceguard inget_resourceand on a job'ssql_taskpath inrun_job.examples/orders_bundle/databricks.yml:warehouse_idgets a placeholderdefaultso it resolves offline.tests/test_variables.py: default / env-override / nested resolution, and the loud skip for workspace / lookup / unset references.Verification
This pull request and its description were written by Isaac.