Skip to content

Conversation

@radical
Copy link
Member

@radical radical commented Nov 26, 2025

[CI] Build only the relevant projects for Quarantined/Outerloop

.. workflows.

These workflows have been running out of space on github runners while
just building the solution, even though it is just the test projects.

What we really need is a few test projects with the relevant attribute
like [OuterloopTest("..")], and generate the runsheet for them. So we
add a new shell script which can find such projects, and then build
only those.

This makes it much faster too.

Fixes #13210 .

@github-actions
Copy link
Contributor

github-actions bot commented Nov 26, 2025

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13218

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13218"

.. workflows.

These workflows have been running out of space on github runners while
just building the solution, even though it is just the test projects.

What we really need is a few test projects with the relevant attribute
like `[OuterloopTest("..")]`, and generate the runsheet for them. So we
add a new shell script which can find such projects, and then build
only those.

This makes it much faster too.

Fixes dotnet#13210 .
@radical radical added the area-engineering-systems infrastructure helix infra engineering repo stuff label Nov 26, 2025
@radical radical marked this pull request as ready for review November 26, 2025 23:15
Copilot finished reviewing on behalf of radical November 26, 2025 23:17
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes the Outerloop and Quarantine workflows to avoid disk space issues by building only test projects containing the relevant test attributes instead of building the entire solution. The optimization adds a new shell script to identify relevant test projects and passes them to the build system via MSBuild properties.

Key Changes:

  • Introduces a new script to identify test projects containing specific attributes (QuarantinedTest/OuterloopTest)
  • Modifies build system to accept custom project lists via BeforeBuildPropsPath
  • Updates workflows to conditionally skip package builds when not needed

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
eng/scripts/generate-specialized-test-projects-list.sh New script that scans for test files with specific attributes and generates MSBuild props file
eng/Build.props Adds support for importing custom props and overriding ProjectToBuild items
.github/workflows/tests-quarantine.yml Adds attributeName input parameter to pass "QuarantinedTest" to the reusable workflow
.github/workflows/tests-outerloop.yml Adds attributeName input parameter to pass "OuterloopTest" to the reusable workflow
.github/workflows/specialized-test-runner.yml Implements new build optimization logic with conditional package building

mkdir -p `dirname $OUTPUT_FILE`

# Find all test files with the attribute and extract unique top-level test directories
PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex pattern only matches attributes with required string arguments like [QuarantinedTest(\"...\")], but according to the attribute definitions, both QuarantinedTest and OuterloopTest accept optional parameters and can be used without arguments (e.g., [QuarantinedTest] or [OuterloopTest]). The pattern should also match attributes without parentheses: \"^ *\\[${ATTRIBUTE_NAME}\\]\"

Suggested change
PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \
PROJECTS=$(grep -Erl "^ *\[${ATTRIBUTE_NAME}( *\(.*\))?\]" "$REPO_ROOT/tests" 2>/dev/null \

Copilot uses AI. Check for mistakes.
Comment on lines +69 to +73
<ItemGroup Condition="'@(OverrideProjectToBuild)' != ''">
<ProjectToBuild Remove="@(ProjectToBuild)" />

<ProjectToBuild Include="@(OverrideProjectToBuild)" />
</ItemGroup>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do it this way? Instead of just setting <ProjectToBuild> to the right thing from the start?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would mean adding the check to all the conditions. I felt overriding at the end was better. But either is fine by me.

mkdir -p `dirname $OUTPUT_FILE`

# Find all test files with the attribute and extract unique top-level test directories
PROJECTS=$(grep -rl "^ *\[${ATTRIBUTE_NAME}(\"[^\"]*\")\]" "$REPO_ROOT/tests" 2>/dev/null \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kind of gross. Why can't we use normal Test traits / categories / etc? Test frameworks have this stuff built in.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to build all the test projects to do that. We do that on main right now. But that keeps running out of disk space. I had fixed it few weeks back to by limiting the build to only the test projects. We could maybe do some other stuff to reduce usage too, but it seemed like a losing game.

An alternative, albeit slower, would be to build one test project a time, use the test framework to get the runsheet generated, delete the project's artifacts, and move on to the next.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are essentially trying to limit the build to a small subset of test projects, so we need to find out which projects have any projects for outerloop or quarantined run.

@radical radical closed this Dec 2, 2025
@radical radical reopened this Dec 2, 2025
@dotnet-policy-service dotnet-policy-service bot added this to the 13.1 milestone Dec 2, 2025
@radical radical enabled auto-merge (squash) December 2, 2025 08:38
@radical radical merged commit 5f27538 into dotnet:main Dec 2, 2025
573 of 576 checks passed
@radical radical deleted the fix-workflows branch December 2, 2025 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-engineering-systems infrastructure helix infra engineering repo stuff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Outerloop and Quarantined workflows are broken - out of space

3 participants