Skip to content

[Python] Deflake anomaly transform tests - #39669

Merged
claudevdm merged 2 commits into
apache:masterfrom
bvolpato:bvolpato/deflake-anomaly-detectors
Sep 10, 2026
Merged

[Python] Deflake anomaly transform tests#39669
claudevdm merged 2 commits into
apache:masterfrom
bvolpato:bvolpato/deflake-anomaly-detectors

Conversation

@bvolpato

@bvolpato bvolpato commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Related to #37779.

The ensemble tests compare each row with an online Z-score calculated in input order, but each detector branch reshuffles independently. Use explicit score fields for grouping and aggregation assertions so valid reordering does not change the expected outputs. These tests still check features, thresholds, keys, examples, model IDs, scores, prediction counts, and AnyVote aggregation.

Keep the existing single-detector Z-score tests and add keyed and unkeyed ensemble tests using real Z-score models on constant observations. The new tests assert exactly two warmup predictions per model and key, regardless of processing order, so lost model state still fails the suite.

The extra state coverage matters because #37779 shows repeated warmup predictions that reordering alone does not explain. This PR removes an invalid ordering assumption and preserves checks for lost model state; it does not establish the cause of that historical runner failure.

Validation:

  • 16 targeted tests pass in transforms_test.py and detectors/zscore_test.py.
  • Reversing the input fails all four original ensemble cases and passes their updated versions.
  • Both new state tests fail when model state is deliberately discarded.
  • YAPF, Ruff, and git diff --check pass.

No changelog entry is needed for this test-only change.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes. Not applicable because this is a test-only change.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement. Not applicable because this is a focused test-only change.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@github-actions github-actions Bot added the python label Aug 7, 2026
@bvolpato
bvolpato marked this pull request as draft August 7, 2026 04:16
@bvolpato
bvolpato force-pushed the bvolpato/deflake-anomaly-detectors branch 3 times, most recently from 5ba530e to 41bb0af Compare August 13, 2026 02:59
@bvolpato
bvolpato force-pushed the bvolpato/deflake-anomaly-detectors branch from 41bb0af to 99fe2db Compare August 29, 2026 19:16
@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.13

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Run Python_ML PreCommit 3.10

@bvolpato
bvolpato force-pushed the bvolpato/deflake-anomaly-detectors branch from 015f01d to 9e0725b Compare September 10, 2026 05:52
@bvolpato
bvolpato marked this pull request as ready for review September 10, 2026 05:53
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@claudevdm
claudevdm merged commit a7b702e into apache:master Sep 10, 2026
100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants