The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1200112	dotnet/arcade	Run Helix Tests	Log	dotnet/arcade#16283
2836252	dotnet-dotnet-monitor	Build	Log
1199107	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121503
1199975	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121524
1199933	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121448
2836146	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
1199906	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121246
1199897	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121418
1199881	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121526
1199782	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121111
1199817	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121525
1197086	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#118957
1199742	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121282
1199710	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121479
2836046	dotnet-performance	Send job to Helix (Windows)	Log
1199672	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121468
2836019	dotnet-runtime	Send job to Helix (Windows)	Log
1199656	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121435
1199481	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121515
1199639	dotnet/runtime	Send tests to Helix (Unix)	Log
1199638	dotnet/runtime	Send tests to Helix (Unix)	Log
1199626	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#119996
1199619	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121246
1199609	dotnet/runtime	Send tests to Helix (Unix)	Log
1199631	dotnet/machinelearning	Run Helix Tests	Log	dotnet/machinelearning#7550
1199594	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121494
1199614	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51619
1199615	dotnet/performance	Send job to Helix (Unix)	Log	dotnet/performance#5035
1199604	dotnet/performance	Send job to Helix (Unix)	Log	dotnet/performance#5036
1199583	dotnet/arcade	Run Helix Tests	Log	dotnet/arcade#16283
2835869	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainerscache.azurecr.io	Log
2835908	dotnet-extensions	Restore	Log	#55128
2835865	dotnet-runtime	Send job to Helix (Windows)	Log
1199478	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121365
1199543	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121508
1199208	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121217
1199457	dotnet/runtime	Send tests to Helix (Windows)	Log
2835812	dotnet-runtime	Send job to Helix (Windows)	Log
1199509	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120688
1199502	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121435
1199497	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121513
1199488	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121516
1199464	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119432
2835793	dotnet-arcade-validation	Run Tests (Windows)	Log
1199461	dotnet/roslyn	Test	Log
1199456	dotnet/roslyn	Run Integration Tests	Log
1199441	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121510
1199417	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121506
1199334	dotnet/runtime	Send to Helix	Log
1199193	dotnet/runtime	Send to Helix	Log
2835733	dotnet-performance	Send job to Helix (Windows)	Log
1199353	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121058
1199319	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121512
2835678	dotnet-runtime	Send job to Helix (Windows)	Log
2835655	dotnet-dotnet	Build	Log
1199298	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121180
1199289	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121316
1199281	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121418
1199274	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121494
1199304	dotnet/roslyn	Test	Log	dotnet/roslyn#81145
1199252	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121491
1199206	dotnet/runtime	Send to Helix	Log
1199297	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51592
1199285	dotnet/aspnetcore	Run build.cmd helix target	Log
1199150	dotnet/runtime	Send tests to Helix (Windows)	Log
1199240	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121508
1198232	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121479
1198775	dotnet/machinelearning	Run Helix Tests	Log	dotnet/machinelearning#7540
1199207	dotnet/runtime	Send to Helix	Log
1199159	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117788
1199186	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116574
2835614	dotnet-runtime	Send job to Helix (Windows)	Log
1199098	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121450
1199084	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121373
1199167	dotnet/roslyn	Run Integration Tests	Log
1199182	dotnet/runtime	Send job to Helix (Unix)	Log
1199146	dotnet/runtime	Send to Helix	Log
1199173	dotnet/roslyn	Test	Log	dotnet/roslyn#80653
1199166	dotnet/roslyn	Test	Log
1199148	dotnet/roslyn	Test	Log	dotnet/roslyn#81142
1199009	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121431
1197157	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121363
1199036	dotnet/runtime	Send to Helix	Log
1199053	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121492
1198802	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121496
1198964	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121502
1198976	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121186
2835587	dotnet-install-scripts	Run Tests	Log
1198942	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51307
1199022	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121453
1198906	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121499
1199030	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51532
1199029	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#51531
2835532	dotnet-dotnet-monitor	Build	Log
1198886	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121417
1198901	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121489
1198890	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#119996
1198865	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121251
1198874	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121498
1198849	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121497
Displaying 100 of 1840 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
72	476	1840

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez