Skip to content

Commit ed324e0

Browse files
mthalmanmthalman
authored
Isolate package flow between repos (#18557)
As a prerequisite to enabling parallel builds of the repos in the VMR, it's necessary to isolate the package flow between repos. This ensures that repo dependencies are defined correctly. Prior to these changes, the packages outputted by repos went to the same directory (Shipping or NonShipping) and those two directories were used as the package feed inputs to dependent repos. This would be dangerous in the context of running builds in parallel across repos because any ill-defined dependencies could lead to race conditions. For example, consider the msbuild repo's dependency on System.Text.Json from the runtime repo. If runtime was not defined as a dependency of msbuild, then there's no guarantee that System.Text.Json will exist when it restores it. Based on timing, it may exist in one build and then not exist in another build. To solve this problem, the packages output from a repos build are placed in a repo-specific package location as a sub-directory of the package location (Shipping or NonShipping). Previously, all repos would output their packages to `artifacts/packages/Release/[NonShipping|Shipping]`. With these changes, they output to `artifacts/packages/Release/[NonShipping|Shipping]/<repo-name>`. This isolates all packages on a per-repo basis. The next step is to provide access to these packages based on dependencies. Going back to msbuild's dependency on runtime, this is accomplished by modifying msbuild's nuget.config file to include feeds specific to runtime: ```xml <add key="source-built-runtime" value="/vmr/artifacts/packages/Release/Shipping/runtime/" /> <add key="source-built-transport-runtime" value="/vmr/artifacts/packages/Release/NonShipping/runtime/" /> ```
1 parent 0a73f81 commit ed324e0

28 files changed

+304
-187
lines changed

‎src/SourceBuild/content/eng/tools/generate-graphviz/generate-graphviz.proj

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎src/SourceBuild/content/repo-projects/Directory.Build.props

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,11 @@
3333
<NuGetConfigFile Condition="'$(OriginalNuGetConfigFile)' != ''">$(BaseIntermediateOutputPath)$([System.IO.Path]::GetFileName('$(OriginalNuGetConfigFile)'))</NuGetConfigFile>
3434

3535
<SourceBuiltSdksDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsDir)', 'source-built-sdks'))</SourceBuiltSdksDir>
36+
<RepoArtifactsShippingPackagesDir Condition="'$(RepositoryName)' != 'source-build-reference-packages'">$([MSBuild]::NormalizeDirectory('$(ArtifactsShippingPackagesDir)', '$(RepositoryName)'))</RepoArtifactsShippingPackagesDir>
37+
<RepoArtifactsShippingPackagesDir Condition="'$(RepositoryName)' == 'source-build-reference-packages'">$(ReferencePackagesDir)</RepoArtifactsShippingPackagesDir>
38+
<RepoArtifactsNonShippingPackagesDir Condition="'$(RepositoryName)' != 'source-build-reference-packages'">$([MSBuild]::NormalizeDirectory('$(ArtifactsNonShippingPackagesDir)', '$(RepositoryName)'))</RepoArtifactsNonShippingPackagesDir>
39+
<RepoArtifactsNonShippingPackagesDir Condition="'$(RepositoryName)' == 'source-build-reference-packages'">$(ReferencePackagesDir)</RepoArtifactsNonShippingPackagesDir>
40+
3641

3742
<!-- Set the bootstrap version to the VMR's version if empty. (no bootstrap set). -->
3843
<ArcadeBootstrapVersion>$([MSBuild]::ValueOrDefault('$(ARCADE_BOOTSTRAP_VERSION)', '$(ArcadeSdkVersion)'))</ArcadeBootstrapVersion>

‎src/SourceBuild/content/repo-projects/Directory.Build.targets

Lines changed: 130 additions & 38 deletions
Large diffs are not rendered by default.

‎src/SourceBuild/content/repo-projects/aspnetcore.proj

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,16 @@
2222

2323
<ItemGroup>
2424
<RepositoryReference Include="arcade" />
25-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
26-
<RepositoryReference Include="runtime" />
27-
<RepositoryReference Include="msbuild" />
2825
<RepositoryReference Include="roslyn" />
29-
<RepositoryReference Include="roslyn-analyzers" />
26+
<RepositoryReference Include="runtime" />
27+
<RepositoryReference Include="xdt" />
28+
</ItemGroup>
29+
30+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
31+
<RepositoryReference Include="nuget-client" />
32+
<RepositoryReference Include="source-build-externals" />
33+
<RepositoryReference Include="source-build-reference-packages" />
34+
<RepositoryReference Include="symreader" />
3035
</ItemGroup>
3136

3237
<ItemGroup>

‎src/SourceBuild/content/repo-projects/cecil.proj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,8 @@
44
<RepositoryReference Include="arcade" />
55
</ItemGroup>
66

7+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
8+
<RepositoryReference Include="source-build-reference-packages" />
9+
</ItemGroup>
10+
711
</Project>

‎src/SourceBuild/content/repo-projects/command-line-api.proj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,8 @@
1010
<RepositoryReference Include="arcade" />
1111
</ItemGroup>
1212

13+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
14+
<RepositoryReference Include="source-build-reference-packages" />
15+
</ItemGroup>
16+
1317
</Project>

‎src/SourceBuild/content/repo-projects/deployment-tools.proj

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,11 @@
22

33
<ItemGroup>
44
<RepositoryReference Include="arcade" />
5-
<RepositoryReference Include="runtime" />
5+
</ItemGroup>
6+
7+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
8+
<RepositoryReference Include="source-build-externals" />
9+
<RepositoryReference Include="source-build-reference-packages" />
610
</ItemGroup>
711

812
<ItemGroup>

‎src/SourceBuild/content/repo-projects/diagnostics.proj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,8 @@
44
<RepositoryReference Include="arcade" />
55
</ItemGroup>
66

7+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
8+
<RepositoryReference Include="source-build-reference-packages" />
9+
</ItemGroup>
10+
711
</Project>

‎src/SourceBuild/content/repo-projects/dotnet.proj

Lines changed: 4 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -11,42 +11,12 @@
1111
- If we have a repo that is not in sdk's dependency tree, we can still build it by including it here. -->
1212

1313
<ItemGroup>
14-
<!-- Toolsets -->
15-
<RepositoryReference Include="source-build-reference-packages" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
16-
<RepositoryReference Include="arcade" />
17-
18-
<!-- Product Repos -->
19-
<RepositoryReference Include="command-line-api" />
20-
<RepositoryReference Include="sourcelink" />
21-
<RepositoryReference Include="diagnostics" />
22-
<RepositoryReference Include="emsdk" />
23-
<RepositoryReference Include="cecil" />
24-
<RepositoryReference Include="symreader" />
25-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
26-
<RepositoryReference Include="runtime" />
27-
<RepositoryReference Include="roslyn" />
28-
<RepositoryReference Include="windowsdesktop" Condition="'$(TargetOS)' == 'windows'" />
29-
<RepositoryReference Include="xdt" />
30-
<RepositoryReference Include="msbuild" />
31-
<RepositoryReference Include="roslyn-analyzers" />
32-
<RepositoryReference Include="aspnetcore" />
33-
<RepositoryReference Include="razor" />
34-
<RepositoryReference Include="deployment-tools" />
35-
<RepositoryReference Include="format" />
36-
<RepositoryReference Include="nuget-client" />
37-
<RepositoryReference Include="templating" />
38-
<RepositoryReference Include="test-templates" />
39-
<RepositoryReference Include="fsharp" />
40-
<RepositoryReference Include="vstest" />
41-
<RepositoryReference Include="sdk" />
42-
<RepositoryReference Include="aspire" />
4314
<RepositoryReference Include="installer" />
44-
45-
<!-- Package source-build artifacts -->
46-
<RepositoryReference Include="package-source-build" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
47-
48-
<!-- Testing. -->
4915
<RepositoryReference Include="scenario-tests" />
5016
</ItemGroup>
5117

18+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
19+
<RepositoryReference Include="package-source-build" />
20+
</ItemGroup>
21+
5222
</Project>

‎src/SourceBuild/content/repo-projects/emsdk.proj

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,8 @@
1414
<RepositoryReference Include="arcade" />
1515
</ItemGroup>
1616

17+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
18+
<RepositoryReference Include="source-build-reference-packages" />
19+
</ItemGroup>
20+
1721
</Project>

‎src/SourceBuild/content/repo-projects/format.proj

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,17 @@
22

33
<ItemGroup>
44
<RepositoryReference Include="arcade" />
5+
<RepositoryReference Include="command-line-api" />
56
<RepositoryReference Include="roslyn" />
7+
<RepositoryReference Include="runtime" />
8+
</ItemGroup>
9+
10+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
11+
<RepositoryReference Include="msbuild" />
612
<RepositoryReference Include="roslyn-analyzers" />
7-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
13+
<RepositoryReference Include="source-build-externals" />
14+
<RepositoryReference Include="source-build-reference-packages" />
15+
<RepositoryReference Include="symreader" />
816
</ItemGroup>
917

1018
</Project>

‎src/SourceBuild/content/repo-projects/fsharp.proj

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,12 @@
2020

2121
<ItemGroup>
2222
<RepositoryReference Include="arcade" />
23-
<RepositoryReference Include="msbuild" />
24-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
23+
</ItemGroup>
24+
25+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
26+
<RepositoryReference Include="msbuild" />
27+
<RepositoryReference Include="runtime" />
28+
<RepositoryReference Include="source-build-reference-packages" />
2529
</ItemGroup>
2630

2731
</Project>

‎src/SourceBuild/content/repo-projects/installer.proj

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,17 +37,26 @@
3737

3838
<ItemGroup>
3939
<RepositoryReference Include="arcade" />
40+
<RepositoryReference Include="aspire" />
4041
<RepositoryReference Include="aspnetcore" />
42+
<RepositoryReference Include="command-line-api" />
43+
<RepositoryReference Include="deployment-tools" />
4144
<RepositoryReference Include="emsdk" />
4245
<RepositoryReference Include="fsharp" />
4346
<RepositoryReference Include="msbuild" />
44-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
4547
<RepositoryReference Include="nuget-client" />
4648
<RepositoryReference Include="roslyn" />
4749
<RepositoryReference Include="runtime" />
4850
<RepositoryReference Include="sdk" />
51+
<RepositoryReference Include="symreader" />
4952
<RepositoryReference Include="test-templates" />
5053
<RepositoryReference Include="vstest" />
54+
<RepositoryReference Include="windowsdesktop" Condition="'$(TargetOS)' == 'windows'" />
55+
</ItemGroup>
56+
57+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
58+
<RepositoryReference Include="source-build-externals" />
59+
<RepositoryReference Include="source-build-reference-packages" />
5160
</ItemGroup>
5261

5362
<!--

‎src/SourceBuild/content/repo-projects/msbuild.proj

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,12 @@
99

1010
<ItemGroup>
1111
<RepositoryReference Include="arcade" />
12-
<RepositoryReference Include="runtime" />
12+
</ItemGroup>
13+
14+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
1315
<RepositoryReference Include="roslyn" />
16+
<RepositoryReference Include="runtime" />
17+
<RepositoryReference Include="source-build-reference-packages" />
1418
</ItemGroup>
1519

1620
</Project>

‎src/SourceBuild/content/repo-projects/nuget-client.proj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,9 @@
88
<BuildScript>$([MSBuild]::NormalizePath('$(ProjectDirectory)', 'eng', 'source-build', 'build$(ShellExtension)'))</BuildScript>
99
</PropertyGroup>
1010

11-
<ItemGroup>
12-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
11+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
1312
<RepositoryReference Include="msbuild" />
13+
<RepositoryReference Include="source-build-externals" />
1414
<RepositoryReference Include="xdt" />
1515
</ItemGroup>
1616

‎src/SourceBuild/content/repo-projects/package-source-build.proj

Lines changed: 30 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -14,56 +14,51 @@
1414
<Target Name="CustomRepoBuild"
1515
AfterTargets="RepoBuild"
1616
DependsOnTargets="CreateBuildInputProps;DetermineSourceBuiltSdkVersion">
17-
<PropertyGroup>
18-
<SourceBuildTarballStagingDir>$([MSBuild]::NormalizeDirectory('$(BaseIntermediateOutputPath)', '$(SourceBuiltArtifactsTarballName)'))</SourceBuildTarballStagingDir>
19-
<SourceBuildReferencePackagesDestination>$([MSBuild]::NormalizeDirectory('$(SourceBuildTarballStagingDir)', 'SourceBuildReferencePackages'))</SourceBuildReferencePackagesDestination>
20-
</PropertyGroup>
21-
22-
<!-- Copy PVP to staging dir in order to package them together. -->
23-
<Copy SourceFiles="$(CurrentSourceBuiltPackageVersionPropsPath)"
24-
DestinationFiles="$(SourceBuildTarballStagingDir)PackageVersions.props" />
25-
26-
<ItemGroup>
27-
<SourceBuildReferencePackagesNupkgFiles Include="$(ReferencePackagesDir)**/*.nupkg" />
28-
</ItemGroup>
29-
30-
<!-- Copy reference packages from ReferencePackagesDir to staging dir. -->
31-
<Copy
32-
SourceFiles="@(SourceBuildReferencePackagesNupkgFiles)"
33-
DestinationFiles="@(SourceBuildReferencePackagesNupkgFiles -> '$(SourceBuildReferencePackagesDestination)%(Filename)%(Extension)')"
34-
Condition="'@(SourceBuildReferencePackagesNupkgFiles)' != ''" />
35-
36-
<ItemGroup>
37-
<ArtifactsPackage Include="$(ArtifactsPackagesDir)**\*.nupkg" />
38-
</ItemGroup>
39-
40-
<!-- Copy packages to staging dir. -->
41-
<Copy SourceFiles="@(ArtifactsPackage)"
42-
DestinationFolder="$(SourceBuildTarballStagingDir)"
43-
Condition="'@(ArtifactsPackage)' != ''" />
44-
4517
<PropertyGroup>
4618
<SourceBuiltTarballName>$(ArtifactsAssetsDir)$(SourceBuiltArtifactsTarballName).$(SourceBuiltSdkVersion).$(TargetRid)$(ArchiveExtension)</SourceBuiltTarballName>
4719
<SourceBuiltVersionFileName>.version</SourceBuiltVersionFileName>
4820
</PropertyGroup>
4921

50-
<!-- Content of the .version file to include in the tarball. Write to staging dir. -->
22+
<!-- Content of the .version file to include in the tarball -->
5123
<ItemGroup>
5224
<VersionFileContent Include="$(RepositoryCommit);$(SourceBuiltSdkVersion)" />
5325
</ItemGroup>
5426

5527
<WriteLinesToFile
56-
File="$(SourceBuildTarballStagingDir)$(SourceBuiltVersionFileName)"
28+
File="$(BaseIntermediateOutputPath)$(SourceBuiltVersionFileName)"
5729
Lines="@(VersionFileContent)"
5830
Overwrite="true" />
5931

60-
<MakeDir Directories="$([System.IO.Path]::GetDirectoryName('$(SourceBuiltTarballName)'))" />
61-
<Exec Command="tar --numeric-owner -czf $(SourceBuiltTarballName) $(SourceBuiltVersionFileName) *.nupkg *.props SourceBuildReferencePackages/"
62-
WorkingDirectory="$(SourceBuildTarballStagingDir)" />
32+
<ItemGroup>
33+
<_AllRepoFiles Include="$(ArtifactsShippingPackagesDir)/**" />
34+
<_AllRepoFiles Include="$(ArtifactsNonShippingPackagesDir)/**" />
35+
<_ReferencePackageFiles Include="$(ReferencePackagesDir)**" />
36+
</ItemGroup>
6337

64-
<Message Importance="High" Text="Packaged source-built artifacts to $(SourceBuiltTarballName)" />
38+
<!-- Create a layout directory for the files that are to be included in the artifacts tarball. Since there are a large number of files, this will use symlinks
39+
instead of copying files to make this execute quickly. -->
40+
<PropertyGroup>
41+
<_SourceBuiltLayoutDir>$([MSBuild]::NormalizeDirectory('$(BaseIntermediateOutputPath)', 'artifacts-layout'))</_SourceBuiltLayoutDir>
42+
<SourceBuildReferencePackagesDestinationDirName>SourceBuildReferencePackages</SourceBuildReferencePackagesDestinationDirName>
43+
</PropertyGroup>
44+
45+
<Copy SourceFiles="@(_AllRepoFiles)"
46+
DestinationFolder="$(_SourceBuiltLayoutDir)"
47+
UseSymbolicLinksIfPossible="true" />
48+
<Copy SourceFiles="$(BaseIntermediateOutputPath)$(SourceBuiltVersionFileName)"
49+
DestinationFolder="$(_SourceBuiltLayoutDir)"
50+
UseSymbolicLinksIfPossible="true" />
51+
<Copy SourceFiles="$(CurrentSourceBuiltPackageVersionPropsPath)"
52+
DestinationFiles="$(_SourceBuiltLayoutDir)PackageVersions.props"
53+
UseSymbolicLinksIfPossible="true" />
54+
<Copy SourceFiles="@(_ReferencePackageFiles)"
55+
DestinationFolder="$(_SourceBuiltLayoutDir)$(SourceBuildReferencePackagesDestinationDirName)"
56+
UseSymbolicLinksIfPossible="true" />
6557

66-
<RemoveDir Directories="$(SourceBuildTarballStagingDir)" />
58+
<Exec Command="tar --numeric-owner -czhf $(SourceBuiltTarballName) $(SourceBuiltVersionFileName) *"
59+
WorkingDirectory="$(_SourceBuiltLayoutDir)" />
60+
61+
<Message Importance="High" Text="Packaged source-built artifacts to $(SourceBuiltTarballName)" />
6762
</Target>
6863

6964
</Project>

‎src/SourceBuild/content/repo-projects/razor.proj

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,11 @@
22

33
<ItemGroup>
44
<RepositoryReference Include="arcade" />
5-
<RepositoryReference Include="runtime" />
5+
</ItemGroup>
6+
7+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
68
<RepositoryReference Include="aspnetcore" />
9+
<RepositoryReference Include="source-build-reference-packages" />
710
</ItemGroup>
811

912
</Project>

‎src/SourceBuild/content/repo-projects/roslyn-analyzers.proj

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,12 @@
88

99
<ItemGroup>
1010
<RepositoryReference Include="arcade" />
11+
</ItemGroup>
12+
13+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
1114
<RepositoryReference Include="runtime" />
12-
<RepositoryReference Include="roslyn" />
15+
<RepositoryReference Include="source-build-externals" />
16+
<RepositoryReference Include="source-build-reference-packages" />
1317
</ItemGroup>
1418

1519
</Project>

‎src/SourceBuild/content/repo-projects/roslyn.proj

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,12 @@
2828

2929
<ItemGroup>
3030
<RepositoryReference Include="arcade" />
31-
<RepositoryReference Include="command-line-api" />
32-
<RepositoryReference Include="source-build-externals" Condition="'$(DotNetBuildSourceOnly)' == 'true'" />
31+
</ItemGroup>
32+
33+
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true'">
3334
<RepositoryReference Include="runtime" />
35+
<RepositoryReference Include="source-build-externals" />
36+
<RepositoryReference Include="source-build-reference-packages" />
3437
</ItemGroup>
3538

3639
<ItemGroup>

0 commit comments

Comments
 (0)