Test | Add flaky test quarantine zone

[mdaigle]

At the moment, we have a number of flaky tests in our pipelines (a couple of which I'll take credit for) that cause random failures, necessitate pipeline reruns, and cheapen the value of our regression testing suite. When tests fail regularly, we pay less attention when we see a failing test. A common pattern to deal with flaky tests is to establish a "quarantine zone" where these tests can live temporarily until they're improved. The quarantine zone is separate from the main build pipelines and does not block main builds, but runs regularly so that we can keep an eye on how flaky tests are performing.

To establish a quarantine zone, I'm planning to use the "flaky" category introduced in this PR: #3488. Tests in this category will be run in a separate testing stage immediately after the corresponding Unit/Functional/Manual testing stage (e.g. Unit => UnitFlaky => Functional => FunctionalFlaky ...). The flaky test stage will not block the pipeline, and failures will be ignored using the continueOnError property: steps.task definition | Microsoft Learn

The ActiveIssue tag will remain reserved for tests that cannot pass due to a platform specific bug, pipeline issue, or other limitation that causes the test to always fail.

I started off by tagging our top offenders based on this dashboard. As we discover more flaky tests, the DRI (or anyone) can open a PR adding the flaky tag to the offending tests. Flaky tests can be addressed either via prioritization during sprint planning or when working on a related section of code.

[Copilot]

Pull request overview

This PR simplifies test filtering to support a quarantine zone for flaky tests by updating the --filter expressions in test execution targets. The changes remove platform-specific category exclusions and standardize filtering to only exclude tests marked as "failing" or "flaky" across all test types (Functional and Manual).

• Simplifies test filters to exclude only "failing" and "flaky" categories
• Removes platform-specific category filters that are no longer needed
• Applies consistent filtering logic across Functional and Manual test targets for both Windows and Unix platforms

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 12 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 13 comments.

[edwardneal]

Can we also include the diagnostic testing please? These seemed to be flaky on ARM64.

[paulmedynski]

Approved, assuming the Copilot comments will be addressed.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.68%. Comparing base (ec842b1) to head (bddb84e).
⚠️ Report is 18 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (ec842b1) and HEAD (bddb84e). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (ec842b1)	HEAD (bddb84e)
addons	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3856       +/-   ##
===========================================
- Coverage   90.82%   69.68%   -21.15%     
===========================================
  Files           6      254      +248     
  Lines         316    64067    +63751     
===========================================
+ Hits          287    44642    +44355     
- Misses         29    19425    +19396     

Flag	Coverage Δ
addons	?
netcore	69.68% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.