Skip to content

acc: Select subset of tests that cover all output#4800

Merged
denik merged 8 commits intomainfrom
tests-subset
Mar 23, 2026
Merged

acc: Select subset of tests that cover all output#4800
denik merged 8 commits intomainfrom
tests-subset

Conversation

@denik
Copy link
Contributor

@denik denik commented Mar 20, 2026

Changes

If -subset (or -update) option is passed, only a few tests selected from EnvMatrix-generated test cases: by default a single test is selected from the whole matrix. The configurations with DATABRICKS_BUNDLE_ENGINE=direct setting get 10x weight, so they are preferred over terraform (but we still run a few tests on terraform).

We can do this, because all subtests are supposed to generate the same output.

The exception is tests that do > out.$DATABRICKS_BUNDLE_ENGINE.txt redirects. Those need both terraform and direct variants to run. For such tests, we select two variants, one with "terraform" and another with "direct" engine. The exact rule for triggering this: either script in current directory or _script in any parent directories contains "$DATABRICKS_BUNDLE_ENGINE" substring.

Why

  • Speed up "make test-update".
  • Potentially more correct update, since parallel runs do not write the same files at the same time.

Tests

Just like in #4795 remove all output files and check that "make test-update" regenerates them all.

This branch:

% git grep -l 'Local = true'  '**/out.test.toml' | find_out_files.py | xargs rm
% time make test-update # This branch 
…
make test-update  561.48s user 266.50s system 659% cpu 2:05.49 total

main:

% git grep -l 'Local = true'  '**/out.test.toml' | find_out_files.py | xargs rm
…
% time make test-update # main
make test-update  1063.22s user 442.75s system 709% cpu 3:32.27 total

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants