Skip to content

[VPEX][5b] Add local-env six-phase pipeline orchestrator#5851

Open
rugpanov wants to merge 1 commit into
dbconnect/05a-pkgmanagerfrom
dbconnect/05b-pipeline
Open

[VPEX][5b] Add local-env six-phase pipeline orchestrator#5851
rugpanov wants to merge 1 commit into
dbconnect/05a-pkgmanagerfrom
dbconnect/05b-pipeline

Conversation

@rugpanov

@rugpanov rugpanov commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Why

  • The earlier layers (resolve / fetch / merge) need an orchestrator that runs them in order, reports structured per-phase status, and provisions the environment — without mutating disk in --check dry-run mode.
  • This is the core half of the split described in [VPEX][5a] Add local-env package-manager interface, detection, and preflight #5850: the six-phase pipeline itself, stacked on the package-manager/detection layer.

What

  • pipeline.go — the six-phase orchestrator (preflight → resolve → fetch → merge → provision → validate). It records per-phase status into Result and returns typed PipelineErrors carrying FailurePhase and DiskMutated. Under --check it computes and reports the plan (with a diff) and performs no writes, skipping the writability probe and package-manager availability (neither is needed to compute the plan).
  • Backups are created only when no .bak exists and the original is a genuine os.ErrNotExist-free read; an unreadable or unstattable existing file fails loudly rather than being misread as greenfield and overwritten.

Testing strategy

  • End-to-end unit tests of the phase machine against a fake PackageManager + stub compute + httptest server: --check mutates nothing (even on a read-only dir / without a package manager), greenfield vs. existing, backup safety on unreadable/unstattable files, constraints-only omission, and phase/error attribution (pipeline_test.go).
  • Gates: go build, go test, golangci-lint (0 issues), deadcode, gofmt — all green.

About this stack

Review bottom-up. This PR targets #5850 as its base, so its diff shows only the pipeline layer.

This PR and #5850 together supersede #5828. The libs/localenv tree at the tip of this branch is byte-identical to the tip of #5828's branch, apart from the pyprojectFile const relocation and a small deadcode-guard test added in #5850.

This pull request and its description were written by Isaac.

@rugpanov rugpanov temporarily deployed to test-trigger-is July 8, 2026 09:58 — with GitHub Actions Inactive
@rugpanov rugpanov temporarily deployed to test-trigger-is July 8, 2026 09:58 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

Could not determine reviewers from git history.
Round-robin suggestion: @renaudhartert-db

Eligible reviewers: @andrewnester, @anton-107, @denik, @janniklasrose, @pietern, @renaudhartert-db, @shreyas-goenka, @simonfaltum

Suggestions based on git history. See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c2e8b69

Run: 28951704133

Env 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 230 1069 3:58
💚​ aws windows 4 4 232 1067 4:41
💚​ aws-ucws linux 4 4 314 987 6:18
💚​ aws-ucws windows 4 4 316 985 4:57
💚​ azure linux 4 4 230 1068 4:26
💚​ azure windows 4 4 232 1066 4:23
🟨​ azure-ucws linux 3 1 4 316 984 11:09
🟨​ azure-ucws windows 3 1 4 318 982 11:27
💚​ gcp linux 4 4 229 1070 4:01
💚​ gcp windows 4 4 231 1068 4:30
8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:39 gcp windows TestAccept
3:38 aws windows TestAccept
3:34 azure-ucws windows TestAccept
3:10 azure windows TestAccept
2:56 aws-ucws windows TestAccept
2:12 azure-ucws windows TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform

The orchestrator runs the layers in order (preflight → resolve → fetch →
merge → provision → validate), records per-phase status into Result, and
returns typed PipelineErrors carrying FailurePhase and DiskMutated. Under
--check it computes and reports the plan (with a diff) and performs no
writes — skipping the writability probe and package-manager availability,
neither of which is needed to compute the plan.

Backups are created only when no .bak exists and the original is a genuine
os.ErrNotExist-free read; an unreadable or unstattable existing file fails
loudly rather than being misread as greenfield and overwritten.

Stacked on the package-manager/detection layer; provisions through the
PackageManager interface against a fake in tests.

Co-authored-by: Isaac
@rugpanov rugpanov force-pushed the dbconnect/05a-pkgmanager branch from cb166cb to f24c320 Compare July 8, 2026 14:44
@rugpanov rugpanov force-pushed the dbconnect/05b-pipeline branch from af9d41b to c2e8b69 Compare July 8, 2026 14:44
@rugpanov rugpanov temporarily deployed to test-trigger-is July 8, 2026 14:44 — with GitHub Actions Inactive
@rugpanov rugpanov temporarily deployed to test-trigger-is July 8, 2026 14:44 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants