-
Notifications
You must be signed in to change notification settings - Fork 746
[CI] Fix Outerloop and Quarantine workflows #13218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13218Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13218" |
d32ff25 to
137d1de
Compare
.. workflows.
These workflows have been running out of space on github runners while
just building the solution, even though it is just the test projects.
What we really need is a few test projects with the relevant attribute
like `[OuterloopTest("..")]`, and generate the runsheet for them. So we
add a new shell script which can find such projects, and then build
only those.
This makes it much faster too.
Fixes dotnet#13210 .
137d1de to
11ecfb3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR optimizes the Outerloop and Quarantine workflows to avoid disk space issues by building only test projects containing the relevant test attributes instead of building the entire solution. The optimization adds a new shell script to identify relevant test projects and passes them to the build system via MSBuild properties.
Key Changes:
- Introduces a new script to identify test projects containing specific attributes (QuarantinedTest/OuterloopTest)
- Modifies build system to accept custom project lists via BeforeBuildPropsPath
- Updates workflows to conditionally skip package builds when not needed
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/scripts/generate-specialized-test-projects-list.sh | New script that scans for test files with specific attributes and generates MSBuild props file |
| eng/Build.props | Adds support for importing custom props and overriding ProjectToBuild items |
| .github/workflows/tests-quarantine.yml | Adds attributeName input parameter to pass "QuarantinedTest" to the reusable workflow |
| .github/workflows/tests-outerloop.yml | Adds attributeName input parameter to pass "OuterloopTest" to the reusable workflow |
| .github/workflows/specialized-test-runner.yml | Implements new build optimization logic with conditional package building |
| mkdir -p `dirname $OUTPUT_FILE` | ||
|
|
||
| # Find all test files with the attribute and extract unique top-level test directories | ||
| PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \ |
Copilot
AI
Nov 26, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The regex pattern only matches attributes with required string arguments like [QuarantinedTest(\"...\")], but according to the attribute definitions, both QuarantinedTest and OuterloopTest accept optional parameters and can be used without arguments (e.g., [QuarantinedTest] or [OuterloopTest]). The pattern should also match attributes without parentheses: \"^ *\\[${ATTRIBUTE_NAME}\\]\"
| PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \ | |
| PROJECTS=$(grep -Erl "^ *\[${ATTRIBUTE_NAME}( *\(.*\))?\]" "$REPO_ROOT/tests" 2>/dev/null \ |
| <ItemGroup Condition="'@(OverrideProjectToBuild)' != ''"> | ||
| <ProjectToBuild Remove="@(ProjectToBuild)" /> | ||
|
|
||
| <ProjectToBuild Include="@(OverrideProjectToBuild)" /> | ||
| </ItemGroup> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do it this way? Instead of just setting <ProjectToBuild> to the right thing from the start?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would mean adding the check to all the conditions. I felt overriding at the end was better. But either is fine by me.
| mkdir -p `dirname $OUTPUT_FILE` | ||
|
|
||
| # Find all test files with the attribute and extract unique top-level test directories | ||
| PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is kind of gross. Why can't we use normal Test traits / categories / etc? Test frameworks have this stuff built in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to build all the test projects to do that. We do that on main right now. But that keeps running out of disk space. I had fixed it few weeks back to by limiting the build to only the test projects. We could maybe do some other stuff to reduce usage too, but it seemed like a losing game.
An alternative, albeit slower, would be to build one test project a time, use the test framework to get the runsheet generated, delete the project's artifacts, and move on to the next.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are essentially trying to limit the build to a small subset of test projects, so we need to find out which projects have any projects for outerloop or quarantined run.
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
[CI] Build only the relevant projects for Quarantined/Outerloop
.. workflows.
These workflows have been running out of space on github runners while
just building the solution, even though it is just the test projects.
What we really need is a few test projects with the relevant attribute
like
[OuterloopTest("..")], and generate the runsheet for them. So weadd a new shell script which can find such projects, and then build
only those.
This makes it much faster too.
Fixes #13210 .