Add test stability mode to the test tool#9296
Draft
nikolakukrika wants to merge 1 commit into
Draft
Conversation
Re-runs an existing AL test suite under configurable presets (random seed variation, WorkDate shift, one-by-one isolation and reverse execution order), individually or combined, and captures every test outcome to surface flaky and order/data-dependent tests. - New Stability folder in the Test Runner module: context, preset parser, configuration + result tables, event subscribers, orchestrator, UI pages and page extensions on the AL Test Tool and Command Line Test Tool. - Seed override lives in the Any app (new "Any Seed Override" codeunit); Any and Library - Random honor it so SEED-<n> works even when tests hardcode a seed. Test Runner takes a dependency on Any for this. - PowerShell entry point build/scripts/StabilityTests/RunStabilityTestsInBcContainer.ps1 drives the run and writes a JSON artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
Build: scripts & configs
Contributor
|
Could not find a linked ADO work item. Please link one by using the pattern 'AB#' followed by the relevant work item number. You may use the 'Fixes' keyword to automatically resolve the work item when the pull request is merged. E.g. 'Fixes AB#1234' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a test stability mode to the test tool. It re-runs an existing AL test suite under one or more presets (individually or combined) to surface flaky, order-dependent and data-dependent tests, and captures the outcome of every test method (pass and fail) — including the preset combination, seed, WorkDate, execution order, error message and call stack — so failures are easy to reproduce and troubleshoot.
Runnable from both the UI (AL Test Tool) and PowerShell / CI (Command Line Test Tool).
Presets
Configured as
+-separated CODE tokens (one string = one combination):BASELINESEED-<n>AnyandLibrary - Random(even when a test hardcodes its seed)WORKDATEFUTURE-<n>YEAR/-<n>MONTHWorkDateinto the futureONEBYONEREVERSE-CODEUNITS/REVERSE-METHODSExample:
SEED-2+WORKDATEFUTURE-1YEAR+REVERSE-METHODS. A default, editable set is created per suite.What's included
Stabilityfolder in the Test Runner module (minimal-change, Code-Coverage-style): single-instance context, preset parser, Stability Run Configuration + Stability Run Result tables, event subscribers (OnBeforeTestMethodRun/OnAfterTestMethodRun), orchestrator, two pages, and page extensions on the AL Test Tool and Command Line Test Tool.Any Seed Overridecodeunit);AnyandLibrary - Randomconsult it, soSEED-<n>is effective even for tests that hardcode a seed. The Test Runner app takes a dependency onAny.build/scripts/StabilityTests/RunStabilityTestsInBcContainer.ps1drives a run and writes a JSON artifact (shape mirrorsunstable-tests.json).README.mddocumenting the feature, tokens and grammar.Order & WorkDate handling
Execution order is realized purely by
Line No.numbering when cloning the suite (no Test Runner change). The WorkDate offset is re-applied before every test method and coordinates withALTestRunner Reset Environment(which restores WorkDate after each codeunit).Notes
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com