Design and implement proper version flow for Unified Build

[mmitche]

We need a design as to how to properly determine dependency versions in Unified Build. This design must hold these 3 scenarios simultaneously:

• What happens when the repo builds and ships in isolation (if this is needed, like with msbuild but not runtime)
• What happens when the repo builds in source-only UB mode (if this is needed, like with msbuild, but not winforms)
• What happens when the repo builds in MSFT UB mode (if needed, like with msbuild, but not source-build-externals)

To illustrate the problem, consider a dependency Foo that msbuild takes. Today, msbuild has control over what version of Foo is used when building in isolation. It may be required to choose a specific version of Foo, say 1.2.3, because msbuild's outputs that depend on Foo are used in a context where it must depend on Foo at 1.2.3.

So far so good. Then source-only builds come along. Let's say that Foo is produced by runtime and redistributed with msbuild in the .NET SDK. Now we have a new context that msbuild's outputs that depend on Foo are used in, and we have a new invariant. When building these source-only builds, msbuild must depend on Foo produced by runtime within the same source-only build, let's say that's 2.0.1. This is not ideal, because msbuild ships two binaries that depend on different versions of Foo, built from the same source. One of those is tested less than the other. However, this is not a problem that is resolvable in any reasonable fashion.

We have built up infra to deal with this dual way of doing things over time:

• Maestro powers the MSFT way of doing things. Active Maestro subscriptions will update versions to latest based on a channel if desired, dependencies can be pinned if they shouldn't be changed, or subscriptions can simply not be added for certain dependencies. How and whether versions flow is based on decisions made by repo owners.
• The source-only build infrastructure is designed around defining the dependency graph based on what dependencies must flow to comply with the need to build against the latest for binaries that will be redisted in packages. So if Foo is not just a ref assembly (if it's implementation), we actively flow its version from runtime to msbuild. This is done by defining a repo dependency graph in the source-only infra, harvesting versions of packages produced by repositories built before msbuild, and then applying those to msbuild based on what is in its Version.Details.xml file.

Unified Build shares the source-only build infra. Now we have a 3rd scenario. The Microsoft built product out of the VMR should look the same in terms of its dependency flow as a repo built in isolation. This is particularly important for those repos still maintaining a separate official build, like VS components.

The challenge here is that the dependency graph in the traditional build infra is held in data in Maestro and in people's heads (X should take latest of Y and so I need an active weekly maestro subscription, or X should match VS's version of Y and so I'll update manually as need be). The source-only build also holds a dependency graph, but it focuses on source-only flow. We need to figure out how to augment this to work for all scenarios.

Goals would be:

• Define a set of rules for how dependencies should flow.
• Define a set of infra for these rules.
• Rules should not involve sprinkling vast numbers of "If DotNetBuildSourceOnly" all over the place.

Some initial thoughts:

• We can alter the graph in source-only vs. non-source only. This mimics what Maestro does. - This would require additional infra that @mthalman is currently working on for parallelism. You need to read dependency versions based on the combo of the repo graph as defined in the repo projects + the VD.xml file. So in source-only mode you'd say "msbuild depends on the outputs of runtime". And in normal modes, you wouldn't.
• We could add metadata in the VD.xml file indicating that versions are "live". And that info would be context aware based on the build mode.

It's possible we would use some combo of the two. In addition, it would be great to have a solution we could use for the short term. This problem should not be underestimated and we don't want to get blocked.

Work Items

• Implement temporary 'cheap' solution to match product version flow in non-source only #4077
• Do not float TFMs when building in non-source-only modes for tooling repos #4075
• Create final version flow design. #4076

[mthalman]

This somewhat reminds me of #3928 in potentially there needing to be conditional aspects within Version.Details.xml. If we go that route, it would be good to have a design which can satisfy both scenarios.

[mmitche]

I think this is complete at this point. @mthalman?

[mthalman]

You're referring to my changes in dotnet/installer#18557? What about the other two open issues linked under Work Items above?

[mmitche]

I think that covers the first solution and represents a good final design. It's simple. It matches the product graph, it can change over time as need be, etc.