Unable to publish ASP.NET builds with Microsoft.Build and VS2022

[Androvald]

Issue Description

I am maintaining a CI application that uses Microsoft.Build and it now fails to run publish builds for ASP.NET projects. It now fails with the following exception:

C:\Program Files\dotnet\sdk\9.0.301\Sdks\Microsoft.NET.Sdk.StaticWebAssets\targets\Microsoft.NET.Sdk.StaticWebAssets.targets:679
(<Redacted>.csproj) C:\Program Files\dotnet\sdk\9.0.301\Sdks\Microsoft.NET.Sdk.StaticWebAssets\targets\Microsoft.NET.Sdk.StaticWebAssets.targets(679,5): Error MSB4018: The "DefineStaticWebAssets" task failed unexpectedly. System.TypeInitializationException: The type initializer for 'DefineStaticWebAssetsSerializerContext' threw an exception. ---> System.MissingMethodException: Method not found: 'System.Text.Json.JsonEncodedText System.Text.Json.JsonEncodedText.Encode(System.String, System.Text.Encodings.Web.JavaScriptEncoder)'. at Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets.DefineStaticWebAssetsSerializerContext..cctor() --- End of inner exception stack trace --- at Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets.DefineStaticWebAssetsCache.ReadOrCreateCache(TaskLoggingHelper log, String manifestPath) at Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets.GetOrCreateAssetsCache() at Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets.Execute() at Microsoft.Build.BackEnd.TaskExecutionHost.Execute() at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext() (Help Keyword: MSBuild.FatalTaskError)

Steps to Reproduce

I've simplified the reproduction to this project:
ASP.NET Publish Broken.zip
It has a sample ASP.NET app with a publish profile added, as well as a simple builder.

Expected Behavior

It should deploy the ASP.NET app to the specified folder.

Actual Behavior

The build fails with the above exception.

Analysis

I am not entirely sure why this fails, as the relevant assemblies do appear to be loaded.
Dump.zip

Versions & Configurations

Visual Studio 2022 17.14.4
Bundled MSBuild version 17.14.10+8b8e13593 for .NET Framework 17.14.10.27608

Machine 1:
Windows 11 24H2 (26100.4349)
dotnet CLI 10.0.100-preview.5.25277.114
SDKs: 2.1.526 / 5.0.416 / 8.0.117 / 8.0.411 / 9.0.300 / 10.0.100-preview.5.25277.114

Machine 2:
Windows Server 2016 version 1607 (14393.7606)
dotnet CLI 9.0.301
SDKs: 2.1.526 / 9.0.301

[YuliiaKovalova]

Hi @Androvald ,

Thank you for the details. Unfortunately, I can't repro it locally.
Could you please provide binlog for the failure case? I will help me to see what versions of System.Text.Json were loaded.
https://aka.ms/binlog

[Androvald]

Certainly, here you go:
MSBuildReproLogs.zip

Though I did notice that I sometimes get a different, but maybe related exception instead today:
OtherException.zip

[Androvald]

I had a look myself at the bin logs generated, and found that the logs where cut off. I apparently needed a call to EndBuild to get them to come through correctly. As such, here are the full ones:
MSBuildReproLogsFixed.zip
OtherExceptionFixed.zip

[YuliiaKovalova]

@Androvald , Thank you, could you also upload binlog for the scenario when it passes?

[YuliiaKovalova]

in the meantime, @javiercn could you please check the logs and tell us if this issue in DefineStaticWebAssets task looks familiar to you ?

[Androvald]

For reference, I am not able to successfully run this build with the reference code. However I am able to build it with MSBuild and Visual Studio 2022. As such, here are the binlogs from an MSBuild run.

MSBuild.zip

[YuliiaKovalova]

@Androvald your fresh binlog helped me to confirm my assumption, there is a version conflict:

Broken build logs:
MSBuildReproLogsFixed.zip
Assembly loaded during TaskRun (Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets): System.Text.Encodings.Web, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (location: C:\Program Files\dotnet\sdk\10.0.100-preview.5.25277.114\Sdks\Microsoft.NET.Sdk.StaticWebAssets\tasks\net472\System.Text.Encodings.Web.dll, MVID: 34553c85-daa9-492b-9f2f-061f10e4aa05, AppDomain: [Default])

Assembly loaded during TaskRun (Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets): System.Text.Encodings.Web, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (location: C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin\System.Text.Encodings.Web.dll, MVID: 4333a908-4e30-442b-9f2f-c018267c1c32, AppDomain: [Default])

OR

OtherExceptionFixed.zip
Assembly loaded during TaskRun (Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets): System.Text.Encodings.Web, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (location: C:\Program Files\dotnet\sdk\10.0.100-preview.5.25277.114\Sdks\Microsoft.NET.Sdk.StaticWebAssets\tasks\net472\System.Text.Encodings.Web.dll, MVID: 34553c85-daa9-492b-9f2f-061f10e4aa05, AppDomain: [Default])

Assembly loaded during TaskRun (Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets): System.ValueTuple, Version=4.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (location: C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.ValueTuple\v4.0_4.0.0.0__cc7b13ffcd2ddd51\System.ValueTuple.dll, MVID: 1b57dc61-2a2b-406d-bb2c-afd7e8e5e6af, AppDomain: [Default])

Successful run:
MSBuild.zip
Assembly loaded during TaskRun (Microsoft.AspNetCore.StaticWebAssets.Tasks.DefineStaticWebAssets): System.Text.Encodings.Web, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 (location: C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin\System.Text.Encodings.Web.dll, MVID: 4333a908-4e30-442b-9f2f-c018267c1c32, AppDomain: [Default])

Let me pull other people in the conversation.

[Androvald]

Oh yeah, I can totally see that, even in my original dump file it has both the 8 and 9 version loaded. Curiously the task DLL is the exact same for both.

I had a look at the app config included with MSBuild.exe and it does specifically do a binding redirect on that DLL to bind it specifically to 9.0.0.0. But it has a ton of of binding redirects that I don't necessarily want to include in my source project directly. In my actual CI tool I load this kind of builder in its own AppDomain, so it can load different MSBuild configurations for different versions of Visual Studio to match the solution file (VS2012 / 2017 / 2022), and unload the domain after it is complete.

[YuliiaKovalova]

in the meantime, @javiercn could you please check the logs and tell us if this issue in DefineStaticWebAssets task looks familiar to you ?

@javiercn could you please take a look at the conversation above?