Auto-migrate to direct engine when opted in and dry-run is clean#5858
Draft
denik wants to merge 5 commits into
Draft
Auto-migrate to direct engine when opted in and dry-run is clean#5858denik wants to merge 5 commits into
denik wants to merge 5 commits into
Conversation
Collaborator
Integration test reportCommit: 6c09183
10 interesting tests: 4 SKIP, 3 KNOWN, 2 flaky, 1 RECOVERED
Top 5 slowest tests (at least 2 minutes):
|
When bundle.engine is set to "direct" (or DATABRICKS_BUNDLE_ENGINE=direct) and the just-deployed terraform state can be dry-run migrated with no errors and no warnings, commit the migration in place instead of just recording telemetry: rename the converted state to resources.json, move terraform.tfstate to .backup, and push the new state to the workspace. If the dry-run surfaces issues, emit a warning that auto-migration is stopped and leave the state as-is. Threads the resolved EngineSetting from process.go through phases.Deploy and deployCore into CheckDirectMigration so the same "config > env" priority already used by PullResourcesState decides opt-in. Renames the misleading targetEngine parameter to stateEngine (it's the state's engine, not any deploy target). Co-authored-by: Isaac
The state_present test's first stanza exercised DATABRICKS_BUNDLE_ENGINE=direct against a seeded terraform state to assert "state wins over env var mismatch." Auto-migration now treats that env var as an opt-in signal and rewrites the state at the end of the deploy, which changes what the follow-up assertions observe. The rest of the file already covers state precedence for empty and terraform env-var values, so drop the direct-env stanza to keep the test focused on the resolver contract. Auto-migration under a direct-opt-in signal is covered separately in acceptance/bundle/migrate/auto-migrate-envvar. Co-authored-by: Isaac
The previous synchronize event did not fire the build workflow for some reason; nudge it with an empty commit. Co-authored-by: Isaac
- Warning when direct is requested and state is terraform now reads "Using X engine for deployment, then will try to migrate your state to direct engine" so the user knows the state-wins outcome is only temporary. Regenerated the golden in state/engine_mismatch. - "Migrated N resource(s) to direct deployment engine." now reports the resource count. - Emit new bool telemetry keys migrated-via-config / migrated-via-env when auto-migration actually rewrites the state, so we can slice by opt-in source. - Test coverage: put the new auto-migrate tests under EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"] instead of [] so they participate in the CI engine-filter; extend auto-migrate-clean to end with a no-op deploy, a config edit, and a with/without --plan deploy of the resulting update (READPLAN matrix). Co-authored-by: Isaac
- Rename bundle/statemgmt/check_direct_migration.go to direct_migration.go
and the top-level function CheckDirectMigration → MigrateToDirect. The
old name suggested a passive check, but the function now can rewrite
state when the user opts in.
- Split metrics: DirectDryMigrate{Success,Warnings} now fire ONLY when
the conversion is truly a dry run (no opt-in). When the user opted in,
emit the new direct_migrate_error / direct_migrate_commit_error /
direct_migrate_warnings keys instead, matching the two failure paths
(state conversion vs. commit) and the warnings case.
Co-authored-by: Isaac
denik
force-pushed
the
denik/engine-migrate-optin
branch
from
f385901 to
6c09183
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bundle.engine: direct(orDATABRICKS_BUNDLE_ENGINE=direct) but the state file is still terraform, a deploy currently deploys under terraform and only dry-runs the migration for telemetry. This PR commits that dry-run in place when it's clean: rename the converted state toresources.json, back upterraform.tfstate, and push the new state to the workspace.databricks bundle deployment migratemanually.EngineSettingfromprocess.gothroughphases.Deploy→deployCore→CheckDirectMigrationso the same "config > env" priority used byPullResourcesStatedecides opt-in. Renames the misleadingtargetEngineparameter tostateEngine.Test plan
acceptance/bundle/migrate/auto-migrate-clean— coversbundle.engine: directopt-in pathacceptance/bundle/migrate/auto-migrate-envvar— coversDATABRICKS_BUNDLE_ENGINE=directopt-in pathacceptance/bundle/telemetry/deploystill passes (dry-run telemetry unchanged when not opted in)acceptance/bundle/migrateandacceptance/bundle/deploysuites passThis pull request and its description were written by Isaac.