NuGet failing with Response status code does not indicate success: 503 (Service Unavailable)

[jkotas]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=300923&view=results

Build leg reported

Build / Installer Build and Test coreclr windows_x86 Debug / Build

Pull Request

dotnet/runtime#78801

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Failed to retrieve information about",
   "BuildRetry": true
}

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
1054128	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116081
2718054	dotnet-sdk	Build	Log
1050603	dotnet/aspire	Build	Log	dotnet/aspire#9503
1050602	dotnet/aspire	Build	Log	dotnet/aspire#9503
2715230	dotnet-sdk	Build	Log	#50353
2715169	dotnet-performance	Run performance job script	Log
2714872	dotnet-sdk	Build	Log
2714871	dotnet-sdk	Build	Log
2714870	dotnet-sdk	Build	Log
2714661	dotnet-sdk	Build	Log	#50199
1048102	dotnet/runtime	Build native test components	Log
2714591	dotnet-dotnet	Build	Log
1047421	dotnet/runtime	Build Tests	Log
2714115	dotnet-sdk	Build	Log	#50250
1047365	dotnet/runtime	Build product	Log	dotnet/runtime#115818
1047313	dotnet/runtime	Send to Helix	Log
1047280	dotnet/runtime	Copy native test components to test output folder	Log	dotnet/runtime#115788
2713588	dotnet-dotnet	Build	Log
2713577	dotnet-sdk	Build	Log	#50250
2713541	dotnet-installer	Build	Log	#50249
2713323	dotnet-sdk	Build	Log	#50199
2713322	dotnet-sdk	Build (centosStream9)	Log	#50201
2713321	dotnet-sdk	Build (centosStream9)	Log	#50200
2713108	dotnet-sdk	Build	Log
2712990	dotnet-runtime	Build product	Log
1045756	dotnet/runtime	Build product	Log	dotnet/runtime#115743
1045734	dotnet/runtime	Build Tests	Log	dotnet/runtime#115393
2712421	dotnet-sdk	Build	Log
2712372	dotnet-sdk	Build	Log
2712232	dotnet-sdk	Build	Log
2712233	dotnet-sdk	Build	Log
2712230	dotnet-sdk	Build	Log
2711868	dotnet-sdk	Build	Log
2711793	dotnet-sdk	Build	Log	#50177
2711794	dotnet-sdk	Build	Log	#50178
2711715	dotnet-sdk	Build	Log	#50175
2711333	dotnet-sdk	Build	Log	#50168
2710616	dotnet-sdk	Build	Log	#49772
2710454	dotnet-sdk	Build	Log	#50097
2710445	dotnet-sdk	Build	Log
2710446	dotnet-sdk	Build	Log
2710443	dotnet-sdk	Build (cblMariner20Fpm)	Log
1042349	dotnet/runtime	Generate CORE_ROOT	Log	dotnet/runtime#115573
2709525	dotnet-sdk	Build	Log
1041632	dotnet/runtime	Send to Helix (Windows)	Log	dotnet/runtime#115563
1041677	dotnet/aspnetcore	Run eng/scripts/CodeCheck.ps1	Log
2708095	dotnet-sdk	🟣 Build	Log
2704739	dotnet-sdk	Build	Log
2704415	dotnet-sdk	Build	Log
2704416	dotnet-sdk	Build	Log
2704145	dotnet-fsharp	Build	Log
2704143	dotnet-fsharp	Build	Log
2704120	dotnet-fsharp	Build	Log
2704113	dotnet-fsharp	Build	Log
2703984	dotnet-roslyn	Build	Log
2703983	dotnet-roslyn	Build	Log
2703846	dotnet-aspnetcore	Build shared fx	Log
2703854	dotnet-aspnetcore	Build shared fx	Log
1034364	dotnet/sdk	Build	Log	dotnet/sdk#48735
2701540	dotnet-aspnetcore	Build shared fx	Log
1033103	dotnet/runtime	Send tests to Helix (Unix)	Log
2701345	dotnet-aspnetcore	Build shared fx	Log
2701351	dotnet-aspnetcore	Build shared fx	Log
1032878	dotnet/runtime	Build product	Log	dotnet/runtime#114828
2700702	dotnet-sdk	Build	Log

Build	Definition	Test	Pull Request
1050594	dotnet/runtime	Workloads-ST-Wasm.Build.NativeRebuild.Tests.FlagsChangeRebuildTests.WorkItemExecution
2716251	dotnet-roslyn	Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateVisualBasicTemplateProjects
2705944	dotnet-roslyn	Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateVisualBasicTemplateProjects
1038011	dotnet/roslyn	Microsoft.CodeAnalysis.MSBuild.UnitTests.NewlyCreatedProjectsFromDotNetNew.ValidateCSharpTemplateProjects(templateName: "razorclasslib")	dotnet/roslyn#78537
2703565	dotnet-sdk	dotnet.Tests.dll.9.WorkItemExecution	#49771

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	6	70

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=300923
Result validation: ✅ Known issue matched with the provided build.

[andriipatsula]

Updated an ErrorMessage

[jkotas]

Updated an ErrorMessage

This error message is too specific. The package name does not matter for 503 errors. You can see it in the log file attached to this issue. The build failure was caused by multiple different packages failing to be retrieved - for example, look for Failed to retrieve information about 'Microsoft.Build.Tasks.Git'.

[andriipatsula]

Yeh, most probably Failed to retrieve information about pattern should work.

[ulisesh]

15 hits in the last 24 hours, I'm going to enable build retry

[MattGal]

Created IcM to ask for investigation here as well, https://portal.microsofticm.com/imp/v3/incidents/details/353857134/home

[ulisesh]

More hits today.

Update from the IcM: "Will check throttling limits to see if that's kicking in here"

[epananth]

I updated the ICM and received no response from them.

[dougbu]

Where are 500s from the public AzDO feeds tracked❔ I don't see efcore-ci build #20230213.2 mentioned here for example. That build matches the ErrorMessage in the description above.

[ulisesh]

Does the EFCore repo uses build analysis? Known issues tracking is part of build analysis

[dougbu]

Might need a separate Known Build Error for the dotnet-public-npm feed errors we're seeing in dotnet/aspnetcore (a lot). For example,

  error An unexpected error occurred: "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/istanbul-lib-report/-/istanbul-lib-report-3.0.0.tgz: Request failed \"500 Internal Server Error\"".
  info If you think this is a bug, please open a bug report with the information provided in "/Users/runner/work/1/s/src/SignalR/clients/ts/common/yarn-error.log".
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
  Process stalled
  Active handles:
    - Socket
    - Socket
    - Socket
    - TLSSocket
    - TLSSocket
    - TLSSocket
    - TLSSocket
    - TLSSocket
/Users/runner/work/1/s/eng/targets/Npm.Common.targets(45,5): error MSB6006: "yarn" exited with code 1. [/Users/runner/work/1/s/src/SignalR/clients/ts/common/common.npmproj]
##[error]eng/targets/Npm.Common.targets(45,5): error MSB6006: (NETCORE_ENGINEERING_TELEMETRY=Restore) "yarn" exited with code 1.

Affected aspnetcore-ci rolling builds over the last week:

• #20230211.3
• #20230212.5
• #20230213.40
• #20230214.6

[dougbu]

Does the EFCore repo uses build analysis? Known issues tracking is part of build analysis

No.

/cc @bricelam and @ajcvickers because I don't remember why they didn't opt in and don't know if those reasons remain valid.

[MattGal]

@dougbu this one's on me, the NPM failures you're talking about are #11737. While the IcM for it is unanswered, the known issues system said no repros for a week so I had closed it. Feel free to reactivate. It won't make Azure Devops any more responsive, but it will keep it on our radar.

[dougbu]

Would that issue have picked up the efcore-ci failure given it occurred in a rolling build (where build analysis isn't relevant) for a repo where build analysis isn't enabled anyhow❔

[dougbu]

In any case, the ErrorMessage above means this issue will capture a superset of those tracked in #11737. I'll reopen the other one just because 500 and 503 aren't the same number 😁

[dougbu]

Oh, ignore my last comment. You were talking about the dotnet-public-npm failures. Right @MattGal❔

[dougbu]

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

[MattGal]

Oh, ignore my last comment. You were talking about the dotnet-public-npm failures. Right @MattGal❔

Correct I was keying off you talking about "pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public-npm/npm/registry/istanbul-lib-report/-/istanbul-lib-report-3.0.0.tgz" since that's not nuget-y

[markwilkie]

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

@AlitzelMendez - any feedback here?

[epananth]

@ilyas1974 assigning this to you, cos you were looking at a similar issue

[AlitzelMendez]

Side note: The Known Build Errors infrastructure is tilted pretty far toward PR failures. It's far too manual (AFAICT) to use for rolling build failures and the https://msit.powerbi.com/groups/de8c4cb8-b06d-4af8-8609-3182bb4bdc7c/reports/f0702582-7c04-47ca-a145-6ac37fd25813/ReportSectioncb62e8e5baebca8883e1?experience=power-bi doesn't seem to automatically report matches for such issues (nor help enough in creating them).

@AlitzelMendez - any feedback here?

so after some months I saw this comment :) I think we explored this option in the past but I never found the dashboard in which we wanted to add a link to create known issue, but I think is a great and reasonable idea. reopening: #8794