The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2722903	dotnet-runtime	Run performance job script	Log
1058703	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116216
1058642	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#116066
1058624	dotnet/runtime	Send job to Helix (Unix)	Log
2722811	dotnet-runtime	Run performance job script	Log
2722691	dotnet-runtime	Run performance job script	Log
2722671	dotnet-razor	Restore, Build and Test	Log
2722621	dotnet-aspire	Build	Log	#50600
2722578	dotnet-dotnet	Build (centOSStreamContainer)	Log
1058259	dotnet/runtime	Send to Helix	Log
1058315	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116085
1058318	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116085
2722586	dotnet-performance	Send job to Helix (Unix)	Log
2722520	dotnet-dotnet	Validate Hosted Pool Information (1ES PT)	Log
1058072	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116265
1058287	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115966
1058282	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116281
1058269	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116222
1058248	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116072
2722515	dotnet-dotnet	Build (centOSStreamContainer)	Log
1058240	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116269
1058260	dotnet/runtime	Send to Helix	Log
2722490	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
1058224	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115800
1058221	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116264
1058217	dotnet/runtime	Send to Helix (Unix)	Log
1058215	dotnet/runtime	AOT compile CoreCLR tests	Log
2722419	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
1058173	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116277
1058170	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116271
1058117	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116080
1057760	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116074
1058108	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116148
1058086	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116107
1058082	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116221
1058076	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116256
1058068	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115802
1058057	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116230
1058103	dotnet/runtime	Send job to Helix (Unix)	Log
2722387	dotnet-dotnet	🔒 Publish Artifacts (On Success)	Log
2722137	dotnet-dotnet	Build (centOSStreamContainer)	Log
1057809	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116148
1057804	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116255
1057698	dotnet/runtime	Send to Helix	Log
1057662	dotnet/runtime	Set skipComponentGovernanceDetection variable	Log	dotnet/runtime#115774
1057712	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114619
1057738	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116253
1057720	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116011
1057696	dotnet/runtime	Send to Helix	Log
1057689	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115513
1057669	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116095
1057652	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116235
1057638	dotnet/runtime	Send tests to Helix (Windows)	Log
1057697	dotnet/runtime	Send job to Helix (Unix)	Log
1057544	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116245
1057637	dotnet/runtime	AOT compile CoreCLR tests	Log
1057589	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116050
1057650	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#49014
1057607	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115856
1057579	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116232
1057596	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116250
1057572	dotnet/roslyn	Run Integration Tests	Log
1057174	dotnet/runtime	Send to Helix	Log
1057313	dotnet/roslyn	Run Integration Tests	Log
1057249	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116231
1057253	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116232
1057243	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116201
2721883	dotnet-runtime	Run performance job script	Log
1057231	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116198
1057228	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116222
1057209	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116229
1057200	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116228
1057172	dotnet/runtime	Send to Helix	Log
1057102	dotnet/runtime	AOT compile CoreCLR tests	Log
2721577	dotnet-dotnet-buildtools-prereqs-docker	🟪 Build Image for Image Builder	Log
2721627	dotnet-runtime	Run performance job script	Log
2721437	dotnet-runtime	Send UTF8Fuzzer to OneFuzz	Log
2721465	dotnet-runtime	Run performance job script	Log
1056600	dotnet/runtime	AOT compile CoreCLR tests	Log
1056448	dotnet/roslyn	Run Integration Tests	Log
1056329	dotnet/runtime	AOT compile CoreCLR tests	Log
1056312	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116085
2721108	dotnet-runtime	Run performance job script	Log
1056315	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#116096
1056275	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116070
1056267	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116187
1056128	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116070
1056070	dotnet/runtime	AOT compile CoreCLR tests	Log
2720751	dotnet-dotnet	Compare signing	Log
1055893	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#112740
1055889	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#114466
1055892	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#112740
1055886	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116177
1055877	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116085
1055871	dotnet/runtime	Send tests to Helix (Windows)	Log
1055861	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116182
1055869	dotnet/runtime	AOT compile CoreCLR tests	Log
1055850	dotnet/runtime	Send job to Helix (Unix)	Log	dotnet/runtime#116096
1055668	dotnet/runtime	AOT compile CoreCLR tests	Log
2720451	dotnet-runtime	Run performance job script	Log
Displaying 100 of 1105 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
40	198	1105

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)