[VPEX][4/8] Add local-env formatting-preserving pyproject.toml merge by rugpanov · Pull Request #5827 · databricks/cli

rugpanov · 2026-07-03T12:27:42Z

Why

local-env must apply the resolved Python version and constraints to the user's pyproject.toml without disturbing their own content — comments, ordering, formatting, and unrelated config must survive untouched.
Re-running must be safe and idempotent, and a greenfield project needs a sensible file created from scratch.
This is the most intricate logic in the feature, so it lands in its own PR for focused review.

What

merge.go — a formatting-preserving merge that rewrites only the env-owned regions (requires-python, the databricks-connect entry in [dependency-groups].dev, and a marker-bracketed managed [tool.uv] block) and preserves every other byte incl. CRLF; idempotent. RenderFreshPyproject builds a complete managed file for a greenfield project.
Scoping/robustness: managed constraint-dependencies nests header-less inside an existing user [tool.uv] (never a duplicate header); single- vs multi-line array detection tracks real bracket depth outside strings/comments; the databricks-connect rewrite is confined to dev and leaves trailing comments alone; requires-python's inline comment is preserved; table-header parsing tolerates inline comments and recognizes [[array.of.tables]].

Testing strategy

Unit tests that parse the merged output as TOML (not just substring checks), covering idempotency, CRLF preservation, user-key preservation, the duplicate-[tool.uv] case, bracket-in-element arrays, sibling-group/comment non-clobbering, and [[tool.uv.index]] children (merge_test.go).
Gates: go build, go test, golangci-lint, deadcode, gofmt — all green.
Reviewed with codex to a clean pass (multiple TOML-corruption edge cases were caught and fixed).

About this stack

This is one of a series of small, stacked PRs that together add the databricks local-env python sync command — it provisions a local Python environment (Python version, databricks-connect pin, and dependency constraints) matched to a selected Databricks compute target. The work was split from one large branch into single-concern layers so each is independently reviewable; the command is kept hidden until the final PR so nothing is user-visible mid-stack.

Review bottom-up. Each PR targets the previous one as its base branch, so its diff shows only that layer.

#	PR	What
1	#5823	foundation: result types + env-key mapping
2	#5824	compute-target resolution
3	#5826	constraint fetch + offline cache
4	#5827 ← you are here	formatting-preserving pyproject.toml merge
5	#5828	six-phase pipeline + detection + package-manager interface
6	#5832	uv backend + CLI command (registered hidden)
7	#5833	acceptance tests
8	#5835	unveil (unhide + help + changelog)

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-07-03T12:54:20Z

Integration test report

Commit: 49b1825

Run: 28869122864

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux		4	4	230	1064	5:31
💚	aws windows		4	4	232	1062	5:46
💚	aws-ucws linux		4	4	314	982	6:49
💚	aws-ucws windows		4	4	316	980	6:25
💚	azure linux		4	4	230	1063	6:13
💚	azure windows		4	4	232	1061	6:17
💚	azure-ucws linux		4	4	316	979	6:56
🔄	azure-ucws windows	2	2	4	318	977	8:06
💚	gcp linux		4	4	229	1065	5:08
💚	gcp windows		4	4	231	1063	5:49

8 interesting tests: 4 SKIP, 2 RECOVERED, 2 flaky

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🔄	TestFetchRepositoryInfoAPI_FromRepo/root	💚R	💚R	💚R	💚R	💚R	💚R	💚R	🔄f	💚R	💚R
🔄	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚R	💚R	💚R	💚R	💚R	💚R	💚R	🔄f	💚R	💚R

Top 11 slowest tests (at least 2 minutes):

duration	env	testname
5:07	azure windows	TestAccept
5:03	aws-ucws windows	TestAccept
5:03	azure-ucws windows	TestAccept
5:01	gcp windows	TestAccept
4:57	aws windows	TestAccept
3:05	aws-ucws windows	TestFilerWorkspaceFilesExtensionsReadDir
3:03	aws-ucws linux	TestAccept
2:56	azure linux	TestAccept
2:53	gcp linux	TestAccept
2:49	aws linux	TestAccept
2:46	azure-ucws linux	TestAccept

First of a stacked series adding `databricks local-env python sync`, which provisions a local Python environment matched to a Databricks compute target. The feature lands across small, single-concern PRs; each layer is independently reviewable and adds no user-facing surface until the final PR wires the command in. This PR is the foundation the rest of the stack builds on: - result.go: the result types and the --json / E_* error contract that every phase reports through (Result, PipelineError, ErrorCode, PhaseName, PhaseStatus, Mode, TargetInfo, ResolvedInfo, Plan, Warning), plus the command-path constants (local-env / python / sync) defined once. - envkey.go: mapping a compute target to an environment key and parsing the Python minor from a requires-python specifier. Nothing imports this package yet, so the CLI is unchanged. The unexported filesystem/artifact constants and the canonical phase-order slice live with the pipeline that consumes them (a later PR in the stack). Co-authored-by: Isaac

…pecifiers Review of the foundation layer flagged that PythonMinorFromRequires took the first MAJOR.MINOR in the string via first-match regex. For a multi-clause requires-python where the exclusive upper bound comes first — e.g. "<3.13,>=3.10" — it returned 3.13, the version the "<3.13" clause forbids, because PEP 440 clause order is arbitrary. The result feeds PM.EnsurePython, so the tool could target a Python the constraint excludes. Prefer a lower-bound / pinning clause (>=, >, ==, ~=, ===) and only fall back to the first version when none is present. Adds multi-clause test coverage; the prior tests exercised only single-bound specifiers. Co-authored-by: Isaac

…dden version Round-2 review of the foundation layer noted that when a requires-python has no lower-bound/pin clause at all (only upper-bound or exclusion, e.g. "<3.13,!=3.12"), PythonMinorFromRequires fell back to the first number and returned 3.13 — a version the specifier forbids. Such a spec has no floor to install from, so it now errors rather than guessing. A bare "3.12" (no operator) is still accepted as a valid floor. Co-authored-by: Isaac

Round-4 review found isDatabricksConnectDep matched only case, missing PEP 503 name equivalence: "databricks_connect" and "databricks.connect" (and other whitespace) were not recognized. The check now extracts the leading package name up to the first PEP 508 delimiter and compares it under PEP 503 normalization (lowercase, collapse runs of -, _, . to a single -), so all spellings match while a distinct package like databricks-connectors does not. Co-authored-by: Isaac

Hardening the fetch of untrusted remote constraint artifacts, per review: - Use a dedicated http.Client with a 30s timeout instead of http.DefaultClient (which has none), so the request is bounded even if the caller's context carries no deadline. - Cap the response body with io.LimitReader (1 MiB, far above any real pyproject.toml) so a misbehaving or hostile host can't read an unbounded body into memory; an over-cap body is rejected. Adds a test for the oversized-body rejection. Note: the separate (blocking) review point about the production artifact host being a personal GitHub repo is tracked on the PR and gated on the unveil PR; it is a hosting/ownership decision, not a code change here. Co-authored-by: Isaac

rugpanov · 2026-07-06T11:49:23Z

@anton-107 both correctness gaps addressed in a711aa44a: the merge now refuses (E_MERGE) on a TOML multi-line string and on a missing [project] table, rather than risking silent corruption / a skipped pin. Tests added for both. Ready for another look.

anton-107

Both correctness gaps are addressed:

Multi-line strings — MergeManaged now detects a """ / ''' delimiter (comment-aware via commentStart) and bails with errMultilineString → E_MERGE rather than scanning a string body as structure. The detection is intentionally conservative (even a single-line """x""" is refused), which is safe, and the test plants a fake header + fake constraint-dependencies inside the string. This preserves the no-corruption guarantee.
No [project] table — now fails loudly with errNoProjectTable → E_MERGE instead of silently skipping requires-python; greenfield still routes through RenderFreshPyproject.

LGTM.

…al-repo default Per review, the constraint artifacts must not be sourced from a hardcoded personal GitHub repo. This parameterizes the host: RepoConstraintBaseURL reads the hosting repo ("owner/name") from the DATABRICKS_LOCALENV_CONSTRAINT_REPO environment variable and builds a raw.githubusercontent.com main-branch URL. The built-in default is intentionally empty and resolution errors when no repo is configured, so no untrusted default controls what the CLI installs. This is temporary: once the Databricks-owned databricks/environments repo can publish the artifacts (its GitHub Actions are currently disabled), defaultConstraintRepo becomes that constant and the env var is no longer required. Tracked as a follow-up; the command stays hidden until the unveil PR regardless. Co-authored-by: Isaac

Fourth in the stacked local-env series. merge.go rewrites only the env-owned sections of a pyproject.toml and preserves every other byte (comments, ordering, whitespace, CRLF). It updates requires-python and the databricks-connect pin in place, and maintains a marker-bracketed managed [tool.uv] constraint block. The operation is idempotent: feeding its own output back in is byte-identical. RenderFreshPyproject produces a complete managed file for a greenfield project. Two correctness properties this file has to get right, both covered by tests that parse the result as TOML rather than asserting on strings: - When the user's pyproject.toml already has a [tool.uv] table with a non-constraint key, the managed constraint-dependencies nests header-less inside that table instead of emitting a second [tool.uv] header (two headers for one table is invalid TOML that uv rejects). - Single- vs multi-line constraint-dependencies detection tracks real bracket depth outside strings and comments, so an opening line that contains a "]" inside an element (e.g. "requests[security]~=2.0") or a trailing comment is not misread as single-line and mis-stripped. Depends on the constraints PR for the Constraints type. Still dormant. Co-authored-by: Isaac

Review of the merge layer found the databricks-connect rewrite was not scoped to [dependency-groups].dev: - It walked every line of [dependency-groups] and rewrote the first databricks-connect element found, so a pin in a sibling group (docs/test) was clobbered instead of the dev entry. It now locates the dev assignment and edits only within that array's line span. - The single-line branch replaced the databricks-connect token anywhere on the dev line, including inside a trailing comment (user content). Replacement is now confined to the array portion (through its closing "]"); the trailing comment is preserved byte-for-byte. - mergeRequiresPython replaced the whole line, dropping an inline comment such as `requires-python = ">=3.10" # maintained by platform team`. It now reattaches the trailing comment, honoring the byte-preservation contract for everything outside the managed value. Adds tests for each: sibling-group untouched, comment not clobbered, inline comment preserved. Co-authored-by: Isaac

Round-2 review found the merge did not tolerate a trailing comment on a table header line (e.g. "[project] # note"). Two consequences: mergeRequiresPython could not find a commented [project] header (managed value silently not updated), and worse, the [dependency-groups] end bound could run past a commented sibling header, so a dev key in a following table was mistaken for [dependency-groups].dev and rewritten. tableHeaderRe now allows a trailing comment and a new headerName helper matches a header by its bracketed name ignoring the comment; both the table lookup and the [tool.uv] attachment check use it. Also documents that line endings are a whole-file property (a CRLF-anywhere file is emitted entirely as CRLF), which is faithful for real single-ending pyproject.toml files. Co-authored-by: Isaac

…ldren Round-3 review found tableHeaderRe did not match TOML array-of-tables headers like "[[tool.uv.index]]". A [tool.uv] table's end bound therefore ran through its [[tool.uv.index]] children, and the header-less managed constraint block could be inserted inside the last index item instead of under [tool.uv], producing wrong or invalid uv config. tableHeaderRe now matches both "[...]" and "[[...]]"; headerName returns the full "[[...]]" token so an array-of-tables header is never treated as the same table as its "[...]" parent. Adds a merge test with a [[tool.uv.index]] child. Co-authored-by: Isaac

…oject files Two correctness gaps in the line-based merge, from review: - The scanner does not track TOML multi-line string state ("""...""" / '''...''') across lines, so a line inside such a string that looks like a table header, key, or bracket could mis-scope the managed-region edits and silently corrupt the file. MergeManaged now detects a multi-line string delimiter and returns an error (surfaced as E_MERGE) rather than risking corruption — the guarantee the merge exists to uphold. Multi-line strings are rare in a pyproject.toml. - A partial existing file with no [project] table would be "merged" with requires-python silently skipped. MergeManaged now errors when [project] is absent (greenfield goes through RenderFreshPyproject, which always writes it). Adds tests for both bail-outs. Co-authored-by: Isaac

eng-dev-ecosystem-bot · 2026-07-07T16:31:00Z

Integration test report

Commit: 4a55027

Run: 28872585952

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🔄	aws linux			3	3	4	527	993	42:58
🔄	aws windows			6	4	4	480	1006	49:10
❌	aws-ucws linux	26	3		3	2	962	808	137:01
❌	aws-ucws windows	20	1		5	2	910	826	131:24
🔄	azure linux			6	4	4	517	994	45:29
🔄	azure windows			12	4	4	468	1007	59:03
❌	azure-ucws linux	1	2	7	4	2	886	839	109:23
🔄	azure-ucws windows			13	6	2	823	857	114:36
🔄	gcp linux			3	3	4	512	999	41:49
💚	gcp windows				4	4	471	1012	48:18

62 interesting tests: 32 FAIL, 22 flaky, 3 KNOWN, 3 RECOVERED, 2 SKIP

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🔄f	💚R	🟨K	🟨K	💚R	💚R	🟨K	💚R	🔄f	💚R
❌	TestAccept/bundle/deploy/spark-jar-task	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/generate/pipeline_and_deploy	✅p	🔄f	❌F	❌F	✅p	🔄f	🔄f	🔄f	✅p	✅p
❌	TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct	✅p	🔄f	❌F	✅p	✅p	✅p	🔄f	🔄f	✅p	✅p
❌	TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	❌F	✅p	🔄f	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/generate/python_job_and_deploy	✅p	🔄f	❌F	❌F	✅p	🔄f	🔄f	🔄f	✅p	✅p
❌	TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	✅p	✅p	🔄f	🔄f	🔄f	✅p	✅p
❌	TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	🔄f	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/invariant/continue_293	🙈s	🙈s	✅p	✅p	🙈s	🙈s	🔄f	🔄f	🙈s	🙈s
🔄	TestAccept/bundle/invariant/continue_293/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl			✅p	✅p			🔄f	🔄f
🟨	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🟨K	💚R	🙈S	🙈S	🟨K	💚R	🙈S	🙈S
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=			❌F	✅p			✅p	✅p
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1			❌F	✅p			✅p	✅p
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=			❌F	✅p			✅p	✅p
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1			❌F	✅p			✅p	✅p
🔄	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=database_catalog.yml.tmpl/READPLAN=			✅p	✅p			✅p	🔄f
🔄	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl/READPLAN=			✅p	✅p			🔄f	🔄f
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=synced_database_table.yml.tmpl/READPLAN=1			✅p	✅p			❌F	🔄f
🔄	TestAccept/bundle/resources/alerts/basic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p
🔄	TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p
🔄	TestAccept/bundle/resources/apps/inline_config	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p
🔄	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/data_security_mode	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/simple	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/update-after-create	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/lifecycle-started	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/lifecycle-started-toggle	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	✅p	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/resize-terminated-fallback	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	❌F	❌F	✅p	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/dashboards/change-name	✅p	✅p	❌F	❌F	✅p	🔄f	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/dashboards/change-name/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	❌F	❌F	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/resources/pipelines/allow-duplicate-names	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
🔄	TestAccept/bundle/resources/pipelines/allow-duplicate-names/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	🔄f	✅p
🔄	TestAccept/bundle/resources/pipelines/allow-duplicate-names/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/resources/pipelines/auto-approve	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
❌	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈s	🙈s	✅p	❌F	🙈s	🙈s	🙈s	🙈s	🙈s	🙈s
❌	TestAccept/bundle/resources/postgres_projects/update_display_name/DATABRICKS_BUNDLE_ENGINE=terraform			✅p	❌F
🟨	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🟨K	💚R	🙈S	🙈S	💚R	💚R	🙈S	🙈S
❌	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name/DATABRICKS_BUNDLE_ENGINE=direct			❌F	✅p			✅p	✅p
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestAccept/bundle/templates/default-python/combinations/classic	🔄f	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes/READPLAN=	🔄f	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.9	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13	✅p	✅p	✅p	✅p	🔄f	🔄f	✅p	✅p	✅p	✅p
🔄	TestAccept/selftest/record_cloud/pipeline-crud	✅p	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=direct	✅p	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
💚	TestFetchRepositoryInfoAPI_FromRepo/root	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
💚	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R

Top 50 slowest tests (at least 2 minutes):

duration	env	testname
14:04	aws-ucws windows	TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
12:33	aws-ucws linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
12:01	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
10:23	aws-ucws windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
9:18	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:54	azure-ucws windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:52	azure-ucws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
8:24	aws-ucws linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:09	azure-ucws linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:06	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:52	aws linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:37	azure-ucws windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:36	aws-ucws windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:26	gcp windows	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
7:19	aws windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:16	aws-ucws windows	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:16	azure windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:14	aws-ucws windows	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
7:08	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:05	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:04	aws-ucws windows	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:58	azure linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:54	aws-ucws windows	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:53	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:44	azure-ucws windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
6:37	aws linux	TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
6:29	azure-ucws linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:28	gcp linux	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
6:07	aws-ucws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform
5:11	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:08	aws windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
5:02	azure linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
4:59	azure windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
4:59	gcp windows	TestAccept
4:57	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:53	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:43	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:39	aws-ucws windows	TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
4:39	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
4:30	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:29	azure-ucws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct
4:26	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:26	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:23	azure-ucws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:20	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct
4:19	azure-ucws windows	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
4:16	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:05	azure windows	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
3:57	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:50	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct

databricks#5823) ## Why - The `local-env` feature needs a shared vocabulary before any behavior can be built: the result shape, the error taxonomy, and how a compute target maps to an environment key. - Landing these contract types first lets every later layer (resolve / fetch / merge / pipeline / command) depend on stable, reviewed definitions. - Kept deliberately minimal and dependency-free so it reviews on its own and stays `unused`/`deadcode`-clean with no consumers yet. ## What - **`result.go`** — the `--json` / `E_*` output contract: `Result`, `PipelineError`, `ErrorCode`, `PhaseName`, `PhaseStatus`, `Mode`, `TargetInfo`, `ResolvedInfo`, `Plan`, `Warning`; plus the command-path constants (`local-env` / `python` / `sync`) defined in one place. - **`envkey.go`** — `EnvKeyForServerless` / `EnvKeyForSparkVersion` / `NormalizeServerless`, and `PythonMinorFromRequires` (clause-aware: returns the effective highest lower bound of a `requires-python`). - No wiring into `cmd/`, so the CLI is unchanged. Filesystem/artifact constants and the phase-order slice deliberately live with their consumer (PR 5). ## Testing strategy - Unit tests for the error/type contract (`result_test.go`) and env-key mapping incl. multi-clause / strict-`>` / no-floor `requires-python` cases (`envkey_test.go`). - Gates: `go build`, `go test`, `golangci-lint`, `deadcode`, `gofmt` — all green. - Reviewed with codex across several rounds to convergence (all findings fixed or explicitly rejected as speculative). --- ## About this stack This is one of a series of small, stacked PRs that together add the `databricks local-env python sync` command — it provisions a local Python environment (Python version, `databricks-connect` pin, and dependency constraints) matched to a selected Databricks compute target. The work was split from one large branch into single-concern layers so each is independently reviewable; the command is kept hidden until the final PR so nothing is user-visible mid-stack. **Review bottom-up.** Each PR targets the previous one as its base branch, so its diff shows only that layer. | # | PR | What | |---|----|------| | 1 | **databricks#5823 ← you are here** | foundation: result types + env-key mapping | | 2 | databricks#5824 | compute-target resolution | | 3 | databricks#5826 | constraint fetch + offline cache | | 4 | databricks#5827 | formatting-preserving pyproject.toml merge | | 5 | databricks#5828 | six-phase pipeline + detection + package-manager interface | | 6 | databricks#5832 | uv backend + CLI command (registered hidden) | | 7 | databricks#5833 | acceptance tests | | 8 | databricks#5835 | unveil (unhide + help + changelog) | This pull request and its description were written by Isaac.

## Why - `local-env` must turn the user's compute selection into a single environment key before it can fetch anything, and the selection can come from several places with a defined precedence. - Isolating resolution behind a narrow seam keeps it testable without a live workspace and keeps SDK details out of the engine. ## What - **`target.go`** — `ResolveTarget` with ordered precedence `--cluster` → `--serverless` → `--job` → bundle target, producing a `TargetInfo` + env key. - Compute lookups go through the narrow `ComputeClient` interface (stubbable in tests). - `ValidateTargetFlags` rejects more than one target flag; `ResolveTarget` runs it up front so a non-Cobra caller can't silently resolve the wrong target. - Classic-compute jobs read the Spark version from the documented first return of `GetJobSparkVersion` (not the recorded-version third return). ## Testing strategy - Unit tests against a stub `ComputeClient` covering each precedence branch, the mutually-exclusive-flags error, and the job classic-compute contract (`target_test.go`). - Gates: `go build`, `go test`, `golangci-lint`, `deadcode`, `gofmt` — all green. - Reviewed with codex to a clean pass. --- ## About this stack This is one of a series of small, stacked PRs that together add the `databricks local-env python sync` command — it provisions a local Python environment (Python version, `databricks-connect` pin, and dependency constraints) matched to a selected Databricks compute target. The work was split from one large branch into single-concern layers so each is independently reviewable; the command is kept hidden until the final PR so nothing is user-visible mid-stack. **Review bottom-up.** Each PR targets the previous one as its base branch, so its diff shows only that layer. | # | PR | What | |---|----|------| | 1 | databricks#5823 | foundation: result types + env-key mapping | | 2 | **databricks#5824 ← you are here** | compute-target resolution | | 3 | databricks#5826 | constraint fetch + offline cache | | 4 | databricks#5827 | formatting-preserving pyproject.toml merge | | 5 | databricks#5828 | six-phase pipeline + detection + package-manager interface | | 6 | databricks#5832 | uv backend + CLI command (registered hidden) | | 7 | databricks#5833 | acceptance tests | | 8 | databricks#5835 | unveil (unhide + help + changelog) | This pull request and its description were written by Isaac.

…icks#5826) ## Why - Once a target resolves to an env key, `local-env` needs the pinned Python version, `databricks-connect` version, and dependency constraints published for that key. - The fetch must degrade gracefully offline and distinguish “this environment isn't published” from “the network is down,” because those call for different user action. - The artifact host must be a Databricks-owned, access-controlled location and must never default to a personal repo — whoever controls the host controls what the CLI installs. ## What - **`constraints.go`** — fetches the per-environment `pyproject.toml`, parses `requires-python`, the `databricks-connect` pin, and `[tool.uv]` `constraint-dependencies`, and caches it on disk. - **Host parameterization** — no host is hardcoded: `RepoConstraintBaseURL` reads the repo (`owner/name`) from the temporary `DATABRICKS_LOCALENV_CONSTRAINT_REPO` env var and builds a `raw.githubusercontent.com/<repo>/main` URL; the built-in default is empty. When unset it returns `""` and `FetchConstraints` reports the missing source as a fetch-phase `E_FETCH` error (so there is no untrusted default, and the failure flows through the normal phase/JSON reporting). Once `databricks/environments` can publish, that becomes the hardcoded default and the env var is no longer required. - Failure classification: **404** → `E_ENV_UNSUPPORTED` (no cache fallback — a distinct non-transient condition); **transport / non-404** → `E_FETCH` with fallback to the last-good cached copy. - Robustness: validate the body (parse + require `requires-python`) **before** caching so a bad 2xx can't poison the cache; atomic cache write (mkdir + temp-file + rename); dedicated `http.Client` with a 30s timeout; body read bounded by `io.LimitReader` at 1 MiB; `databricks-connect` matched by leading package name under PEP 503 normalization (so `Databricks_Connect` matches, `databricks-connectors` does not); cache filename = readable slug + sha256 suffix to prevent collisions. ## Testing strategy - Unit tests with an `httptest` server: 200-parse, 404 → `E_ENV_UNSUPPORTED`, transport failure + cache fallback, missing-`requires-python` rejection, PEP 503 name matching, cache-dir creation, collision-free filenames, oversized-body rejection (`constraints_test.go`). - Host resolution: `TestRepoConstraintBaseURL` (env var → URL, unset → `""`, whitespace treated as unset) and `TestFetchConstraintsNoSourceConfigured` (empty host → `E_FETCH` naming the env var). - Gates: `go build`, `go test`, `golangci-lint`, `deadcode`, `gofmt` — all green. - Reviewed with codex to a clean pass (several fetch/cache edge-case fixes landed from review). --- ## About this stack This is one of a series of small, stacked PRs that together add the `databricks local-env python sync` command — it provisions a local Python environment (Python version, `databricks-connect` pin, and dependency constraints) matched to a selected Databricks compute target. The work was split from one large branch into single-concern layers so each is independently reviewable; the command is kept hidden until the final PR so nothing is user-visible mid-stack. **Review bottom-up.** Each PR targets the previous one as its base branch, so its diff shows only that layer. | # | PR | What | |---|----|------| | 1 | databricks#5823 | foundation: result types + env-key mapping | | 2 | databricks#5824 | compute-target resolution | | 3 | **databricks#5826 ← you are here** | constraint fetch + offline cache | | 4 | databricks#5827 | formatting-preserving pyproject.toml merge | | 5 | databricks#5828 | six-phase pipeline + detection + package-manager interface | | 6 | databricks#5832 | uv backend + CLI command (registered hidden) | | 7 | databricks#5833 | acceptance tests | | 8 | databricks#5835 | unveil (unhide + help + changelog) | This pull request and its description were written by Isaac.

rugpanov added the ai-assisted label Jul 3, 2026

rugpanov temporarily deployed to test-trigger-is July 3, 2026 12:28 — with GitHub Actions Inactive

rugpanov mentioned this pull request Jul 3, 2026

[VPEX][5/8] Add local-env pipeline, detection, and package-manager interface #5828

Open

rugpanov force-pushed the dbconnect/03-constraints branch from d867d1f to 64d8996 Compare July 3, 2026 13:20

rugpanov force-pushed the dbconnect/04-merge branch from ccc19f1 to 92f1dda Compare July 3, 2026 13:20

rugpanov temporarily deployed to test-trigger-is July 3, 2026 13:21 — with GitHub Actions Inactive

rugpanov changed the title ~~Add dbconnect formatting-preserving pyproject.toml merge~~ [VPEX][4/8] Add local-env formatting-preserving pyproject.toml merge Jul 3, 2026

rugpanov force-pushed the dbconnect/03-constraints branch from 64d8996 to 3fd0bfe Compare July 3, 2026 15:26

rugpanov force-pushed the dbconnect/04-merge branch from 92f1dda to b96bf8b Compare July 3, 2026 15:26

rugpanov temporarily deployed to test-trigger-is July 3, 2026 15:27 — with GitHub Actions Inactive

rugpanov force-pushed the dbconnect/03-constraints branch from 3fd0bfe to ce01647 Compare July 3, 2026 15:31

rugpanov force-pushed the dbconnect/04-merge branch from b96bf8b to 6477a4c Compare July 3, 2026 15:31

rugpanov temporarily deployed to test-trigger-is July 3, 2026 15:31 — with GitHub Actions Inactive

rugpanov force-pushed the dbconnect/04-merge branch from 6477a4c to f4cc1c7 Compare July 3, 2026 15:41

rugpanov temporarily deployed to test-trigger-is July 3, 2026 15:42 — with GitHub Actions Inactive

rugpanov temporarily deployed to test-trigger-is July 3, 2026 17:55 — with GitHub Actions Inactive

rugpanov force-pushed the dbconnect/03-constraints branch from 797c4ac to 03b4a2b Compare July 3, 2026 18:15

rugpanov force-pushed the dbconnect/04-merge branch from 56b496d to 7b23583 Compare July 3, 2026 18:15

rugpanov temporarily deployed to test-trigger-is July 3, 2026 18:15 — with GitHub Actions Inactive

rugpanov added 2 commits July 6, 2026 13:39

rugpanov force-pushed the dbconnect/03-constraints branch from 79890ca to 0ef2220 Compare July 6, 2026 11:47

rugpanov force-pushed the dbconnect/04-merge branch from 79113e4 to a711aa4 Compare July 6, 2026 11:47

rugpanov temporarily deployed to test-trigger-is July 6, 2026 11:47 — with GitHub Actions Inactive

anton-107 approved these changes Jul 6, 2026

View reviewed changes

rugpanov added 6 commits July 7, 2026 10:34

rugpanov force-pushed the dbconnect/04-merge branch from a711aa4 to 41d052a Compare July 7, 2026 08:35

rugpanov temporarily deployed to test-trigger-is July 7, 2026 08:35 — with GitHub Actions Inactive

Base automatically changed from dbconnect/03-constraints to main July 7, 2026 13:12

Merge branch 'main' into dbconnect/04-merge

49b1825

rugpanov temporarily deployed to test-trigger-is July 7, 2026 13:16 — with GitHub Actions Inactive

rugpanov enabled auto-merge July 7, 2026 13:47

rugpanov added this pull request to the merge queue Jul 7, 2026

Merged via the queue into main with commit 4a55027 Jul 7, 2026
23 checks passed

rugpanov deleted the dbconnect/04-merge branch July 7, 2026 14:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[VPEX][4/8] Add local-env formatting-preserving pyproject.toml merge#5827

[VPEX][4/8] Add local-env formatting-preserving pyproject.toml merge#5827
rugpanov merged 21 commits into
mainfrom
dbconnect/04-merge

rugpanov commented Jul 3, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

rugpanov commented Jul 6, 2026

Uh oh!

anton-107 left a comment

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rugpanov commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Testing strategy

About this stack

Uh oh!

eng-dev-ecosystem-bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

rugpanov commented Jul 6, 2026

Uh oh!

anton-107 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Jul 7, 2026

Integration test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rugpanov commented Jul 3, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Jul 3, 2026 •

edited

Loading