Skip to content

Test | Add flaky test quarantine zone #3856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mdaigle merged 16 commits into from
Jan 6, 2026
Merged

Test | Add flaky test quarantine zone #3856

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
fa22414
Skip flaky tests. Try removing platform filters which should already …
mdaigle Dec 19, 2025
dba39e7
Make filter configurable in build.proj
mdaigle Dec 19, 2025
13b684c
Add quarantined testing steps.
mdaigle Dec 19, 2025
8a4f8e0
Fix closing quote.
mdaigle Dec 19, 2025
4d14a1e
Fix closing quote.
mdaigle Dec 19, 2025
ae19c47
Fix quote wrapping for filter argument.
mdaigle Dec 19, 2025
29ecebc
Fix continueOnError indent.
mdaigle Dec 19, 2025
3413a96
Tag flaky tests.
mdaigle Dec 20, 2025
4c4cf1e
Tag xevent test as flaky.
mdaigle Dec 20, 2025
2fe2941
Fix indentation.
mdaigle Jan 5, 2026
c56e1b0
Mark DiagnosticTest as flaky.
mdaigle Jan 5, 2026
bde6d56
Merge branch 'main' of github.com:dotnet/SqlClient into dev/mdaigle/f…
mdaigle Jan 5, 2026
c846ec8
Remove stray preprocessor directive.
mdaigle Jan 5, 2026
a4a1864
Don't collect code coverage for flaky tests.
mdaigle Jan 5, 2026
13661d3
Add preliminary cleanup in case previous runs left files behind.
mdaigle Jan 6, 2026
bddb84e
Use parameter to control code cov upload.
mdaigle Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 22 additions & 13 deletions build.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@
<NugetPackProperties Condition="'$(TF_BUILD)' == 'true'">$(NugetPackProperties);ContinuousIntegrationBuild=true;</NugetPackProperties>
</PropertyGroup>

<!-- Test configuration properties -->
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<!--Collect code coverage unless explicitly disabled-->
<CollectCodeCoverage Condition="'$(CollectCodeCoverage)' == ''">true</CollectCodeCoverage>
<CollectStatement Condition="'$(CollectCodeCoverage)' == 'true'">--collect "Code coverage"</CollectStatement>
</PropertyGroup>

<!-- Release Build properties must be turned on for Release purposes, and turned off for Code Coverage calculations -->
<PropertyGroup>
<BuildForRelease Condition="$(BuildForRelease) == ''">true</BuildForRelease>
Expand Down Expand Up @@ -230,15 +239,15 @@
-p:Configuration=$(Configuration)
-p:Target$(TFGroup)Version=$(TF)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running unit tests for Windows via command: $(TestCommand)"/>
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)"/>

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net8_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net462_AnyCPU_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22 net9_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net9_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net8_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_x86 net462_AnyCPU_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         C:\x86\dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_NativeSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql19 net9_0_AnyCPU_ManagedSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql19 net462_AnyCPU_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_ManagedSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net462_AnyCPU_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net462_AnyCPU_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.
</Target>

<!-- Run all unit tests applicable to Unix. -->
Expand All @@ -251,11 +260,11 @@
-p:Configuration=$(Configuration)
-p:TargetNetCoreVersion=$(TF)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Unixnetcoreapp-$(TestSet)"
</TestCommand>
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running unit tests for Unix via command: $(TestCommand)"/>
Expand All @@ -277,9 +286,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Functional-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -300,9 +309,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Functional-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -326,15 +335,15 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Manual-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running Manual test for Windows via command: $(TestCommand)" />
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)" />

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_2)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:ReferenceType=Project         -p:TestSet=2         -p:TestTargetOS=Windowsnetfx         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetfx-2"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_ARM64_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net462_AnyCPU_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetfx-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_ARM64_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win22_Sql22 net8_0_AnyCPU_ManagedSNI_2)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=2         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-2"       " exited with code 1.
</Target>

<!-- Run all Manual tests applicable to Unix. -->
Expand All @@ -349,9 +358,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Manual-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand Down
20 changes: 20 additions & 0 deletions eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,20 @@ jobs:
dotnet tool install --global dotnet-reportgenerator-globaltool
displayName: Install dotnet tools

- pwsh: |
Write-Host "Removing leftover coverage files from previous runs..."
Remove-Item $(netFxDir) -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item $(netCoreDir) -Recurse -Force -ErrorAction SilentlyContinue

Write-Host "Removing leftover merged XML files from previous runs..."
Remove-Item coverageNetFxXml -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coverageNetCoreXml -Recurse -Force -ErrorAction SilentlyContinue

Write-Host "Removing leftover reports from previous runs..."
Remove-Item coveragereportNetFx -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportNetCore -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportAddOns -Recurse -Force -ErrorAction SilentlyContinue

- ${{ each targetFramework in parameters.targetFrameworks }}:
- task: DownloadPipelineArtifact@2
displayName: 'Download Coverage Reports [${{ targetFramework }}]'
Expand Down Expand Up @@ -196,3 +210,9 @@ jobs:
./codecov --verbose upload-process --fail-on-error -t $(CODECOV_TOKEN) -f "coveragereportNetCore\Cobertura.xml" -F netcore
./codecov --verbose upload-process --fail-on-error -t $(CODECOV_TOKEN) -f "coveragereportAddOns\Cobertura.xml" -F addons
displayName: Upload to CodeCov

- pwsh: |
Write-Host "Removing reports..."
Remove-Item coveragereportNetFx -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportNetCore -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportAddOns -Recurse -Force -ErrorAction SilentlyContinue
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky"'

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
Loading
Loading